Framing the university ranking game: actors, motivations, and actions

: Any formulation of the university ranking game involves the perspectives of the 3 key actors: (1) graduating high-school students, (2) universities, and (3) ranking publications. These university rankings are developed and maintained by for-profit publications or magazines, which must balance 2 potentially conflicting motives: (1) to provide students with information to help them decide which university to attend and (2) to increase the revenues of the publication. The actions of the students involve their decision on which universities to apply to and which university to attend among those they are admitted to. The universities seek to attract the best students and seek to improve their ranking to do so. We frame these diverse motives and the ensuing actions of these 3 sets of actors as the university ranking game and discuss the potential inefficien-cies in the game and the possibility for unethical behavior by publications and universities. reversibility in statistical but seems to for a of the in

on admissions statistics (e.g.matriculation rates and the quality of admitted and entering students) at topranked USNWR colleges and universities.Monks & Ehrenberg (1999) find that an increase (downgrade) in a university's rank (e.g. from 4th to 5th) in USNWR results in an increase in the university's admissions rate, a decrease in matriculation rates, and a decrease in the quality of the entering class (as measured by entrance examination scores).Bowman & Bastardo (2009) find that the effect on admissions statistics of an improvement in rank is highly significant when a university moves into the top 25 (i.e. the front page of the USNWR ranking).
Students read the rankings not only to gain information about university attributes (e.g.statistics about the student quality and university expenditures on academic programs), but also to learn the ranks themselves.Controlling for college and university attributes salient to student matriculation decisions, Luca & Smith (2013) find that improvements in ranks cause increases in student applications.Examining applicant-level data of a highly-selective US college, Griffith & Rask (2007) also find a positive effect of an improvement in the college's rank on admissions statistics.
While magazines and websites that conduct and publish rankings may wish to influence student decisions, most such publications are for-profit enterprises seeking to maximize profit through newsstand sales or, in some cases, advertising revenue.
There are conflicts in these motivations, at least for universities and publications, that can help explain their (sometimes questionable) practices.As universities value rankings and a ranking competition is zero sum (i.e. gain in rank for one university implies loss in rank for others) since a university can only improve its rank by displacing another, universities have incentives to act strategically to enhance those rankings.Some questionable behaviors that are aligned with that objective include (1) rejecting top candidates, who can harm a university's rank if they are admitted and choose not to come, reducing the university's yield rate and (2) encouraging low quality candidates to apply, only to reject them, thereby raising the university's exclusivity rate-the percent of applicants that receive acceptances (Golden 2001).
Publications like USNWR also have incentives to act strategically.Unless there is volatility in university rankings, there is little news value in reporting them, reducing the need for prospective students to acquire the most recent issue with the most recent rankings.Many publications change their methodology regularly (the variables or attributes they use in their model and the weights they give to the model inputs), claiming they are refining and improving their rankings.The main purpose, however, may be to increase the volatility of the rankings, leading to increased visibility and interest.
Here, we view the university ranking game (particularly as played in the USA) as the interplay between rankings publications, universities, and students.We provide a framework to investigate the operation of the resulting marketplace by viewing the motivations and action spaces (set of actions that a decision maker can potentially take) of these actors and how those motivations and actions interact.We hope this perspective will result in a better understanding of the behaviors of these 3 sets of actors.We also hope to demonstrate the need for research on (1) ways that the system might reduce the negative impact on the quality of education from strategic moves by publications to generate profit and (2) ways to remove incentives for questionable practices by universities seeking to improve rankings, as well as (3) to improve the value to students of the information provided by the rankings.

UNIVERSITY RANKINGS: THE GAME AND THE PLAYERS
University rankings first appeared in the 1870s with the objective of informing higher education scholars and professionals and government officials (Stuart 1995).Rankings gained mass appeal in the USA in 1983, almost a century after they were first introduced, when USNWR, using a survey of university presidents, published its first rankings of undergraduate academic quality, uncovering the potential interest that potential applicants had in those rankings (Stuart 1995).In 1987 USNWR adopted its current multidimensional methodology, aggregating more objective attributes along with assessments by academic leaders of their peer institutions.The 2013 USNWR ranking includes university scores in attribute categories including assessment by administrators at peer institutions, retention of students, faculty resources, student selectivity, financial resources, alumni giving, and graduation performance.When USNWR introduced its university rankings issue in 1983, the publication ranked the top 25 national universities and top 25 national colleges.In 1998, USNWR expanded its rankings of the national universities to the top 50 universities.In the 2004 ranking, USNWR expanded its rankings by creating rankings for 3 categories of US colleges and universities: national doctoral universities, regional master's universities, and national colleges.
The 2013 version expanded further to rank the world's top 400 universities, both overall and by subject, and also to rank the universities of Asia and Latin America separately.In addition to its rankings of undergraduate programs, USNWR also ranks graduate schools in 11 areas of study including, for example, business, medicine, and education, and US high schools.
The success of USNWR spawned other rankings of undergraduate programs.BusinessWeek's ranking of undergraduate business programs, introduced in 2006, is noteworthy in that it uses a methodology similar to that of USNWR, namely a publisher-determined multi-attribute ranking.As with USNWR, universities self-report data about various attributes to Business-Week.Unlike USNWR, BusinessWeek collects its own data about some attributes, surveying college recruiters, for example, for the recruiter satisfaction attribute used in its survey.Internationally, rankings such as the Times Higher Education World Ranking, first published in 2004, have also been increasing in popularity among students (Rauhvargers 2013).
University administrators in the US, while sometimes criticizing the existence of published rankings, recognize that these rankings are publicly visible performance scorecards.For example, Hobart and William Smith College fired a senior vice president in 2000 after she failed to submit fresh data to USNWR, resulting in a major drop in the College's rank (Graham & Thompson, Broken Ranks, Washington Monthly, September 2001, www.washingtonmonthly .com/features/2001/0109.graham.thompson.html).And Richard Beeman, Dean of the College of Arts and Sciences at the University of Pennsylvania, in a letter to the New York Times (September 17, 2002) commented '…I breathed a sigh of relief when my university continued to appear in the [USNews] top 10.' Hence, given the importance of university rankings to students in their college decision-making process, most universities choose to participate in these rankings by offering data about attributes that are of interest to those students, their (potential) customers.
The success and influence of USNWR and the actions taken by universities to manipulate rankings sparked an interest in the academia to construct a new ranking methodology to eliminate some of this strategic manipulation.While the USNWR use of university acceptance rates of applications and, in the past, the yield of accepted applicants to measure student preferences encourages university manipulation of admissions decisions to improve these statistics, the Avery et al. (2013) one-dimensional university ranking mechanism, based on the actual matriculation decisions made by students, does not encourage such manipulation.
Each popular US undergraduate university ranking -BusinessWeek and USNWR in particular and even the one-time ranking in Avery et al. (2013) -includes a subset of 4 classes of university attributes: the preferences or utilities of entering students; the inputs to the university (i.e. the quality of the entering freshman class); the quality of the student experience (as measured, for example, by retention and alumni giving); and the graduate school and employment placement of university graduates.To the best of our knowledge, no US university ranking includes all 4 attribute classes.Avery et al. (2013) base their ranking on student revealed preferences (i.e.their matriculation decisions).USNWR bases its ranking on the attributes of the entering class and on measures of the student experience.The BusinessWeek ranking of undergraduate business programs in cludes information about student quality (e.g.average SAT scores), student experience (e.g.student perceptions of the quality of teaching), and output information (e.g.recruiter ratings and starting salaries).
Multi-attribute rankings such as those of Business-Week and USNWR use the following type of ranking methodology.They determine the level the university merits on each attribute and then map those levels into an overall score using a publication-determined aggregation function (which might be a weighted sum of the attribute levels or a more complicated function).They then rank-order the universities based on the universities' overall scores.Most of the included attributes are themselves multi dimensional and can be measured in a number of ways.For example, in scoring the quality of a university's entering class, a publication might use the 25th and 75th percentiles of the class's SAT scores, the rank index (i.e. a measure of high school performance) of the class, or any of a number of other measures.Note that different publications use different attributes and different measures of those attributes in their rankings.
While we examine the USNWR methodology in this paper, it is important to note that internationally, rankings publications, beginning in the late 1990s, altered the USNWR methodology by introducing 'personalized college ranking' systems in which the publications score attributes, but readers choose the weights associated with each of the attributes to determine the self-determined rankings.These personalized college ranking systems include Canada's Maclean's and Globe and Mail, The Netherland's Studychoice.nl,Forbes' Do It Yourself ranking in the US, and HEEACT's College Navigator in Taiwan.
In the US, with the possible exception of the Avery et al. (2013) ranking, the goal of each publication is to garner revenue by generating traffic through sales of the magazine or access to the website, with the possible associated sale of advertising, also linked to level of access.To generate traffic, rankings must provide value to students, which they do only if students expect that the information provided by the rankings will affect their decisions about which universities to spend time investigating, applying to, and eventually attending.Rankings publications make 4 choices that affect student value: prices for access and advertising; the attributes to include in their rankings; their investments in the measurement of attributes; and the function used to aggregate attribute scores into a ranking.Here, we comment on (1) a publication's strategic choice of the aggregation function in a publication-determined multi-attribute ranking, and (2) strategic moves by universities to improve their rankings.We consider a publication's decision in the context of the amount of information that students have about universities and also their degree of decision-making sophistication.

STUDENT DECISION MAKING, UTILITY OF RANKINGS, AND STRATEGIC CHANGES IN RANKING METHODOLOGY
A student's value of a publication's ranking depends on whether the attributes included in its ranking are important to the student, on whether those attributes are properly measured, and on whether the publication's aggregation function is consistent with that student's preferences.A publication's decision on whether to include attributes that are aligned with student preferences depends partly on whether the publications can collect data on such attributes (e.g.universities are not willing to give applicant-level data to rankings publications) and on the cost of collecting information about attributes.In terms of investment in the measurement of attributes, university rankings are quite different from consumer product rankings by publications like Consumer Reports.While the popular US independent consumer-product test company, Consumer Union (which publishes Consumer Reports, consumerreports.org) in vests in the testing of consumer products, USNWR largely relies on universities to self-report data about their attributes (www.businessweek.com/bschools/ rankings).BusinessWeek relies on a com bination of universities' self-reported data and data based on BusinessWeek's surveys of students and recruiters (www.busi ness week.com/ articles/ 2013-06-24/ the-best-under grad u ate -business-schools-by-specialty-2013).
Apparently, Con sumer Union makes financial investments in rating consumer products while university-ranking publications do not.
Rankings can play 2 different roles in the student decision process; they can inform students about university attributes, or they can persuade students, by altering students' utilities.In the neoclassical economics view of decision making, students assign utilities to universities, and then use those utilities to determine which universities to investigate, to apply to, to visit, and eventually to matriculate (see Chade &Smith 2006 andDearden &Seyhan 2013 as examples of the neoclassical economics student decision process).We view a student's utility of attending a particular university as a function both of the university's attributes and of the university's rank as reported in the publication.Note that the student's utility may not be affected by differences in the levels of some attributes and that any particular ranking measures only a subset of a university's attributes that might be important to a student.
This view of rankings as being either informative or persuasive stems from the perspective of economists and marketers that advertising is either persuasive in the sense of altering consumers' tastes and creating perceived product differentiation, or informative, providing information to consumers about product attributes, price, or points of sale (Bagwell 2007).Rankings persuade or inform students in much the same way that advertising persuades or informs consumers, and we classify students by whether rankings persuade or inform them.
We characterize students by whether they are novice or sophisticated decision makers, an assumption consistent with the findings of Griffith & Rask (2007) and Luca & Smith (2013).A novice decision maker is inexperienced at calculating his or her utility of attending a university as a function of university attributes.While the novice decision maker's utility of a university relies somewhat on the attributes included in the publication, this type of decision maker's utility is highly dependent on the publication's ranking of the university.Therefore, the information about university attributes provided in a ranking publication is less important to novice decision makers than are the university ranks reported in the publication.Thus, the persuasive role of a ranking has greater impact than does the informative role of ranking to a novice decision maker.(Note that both novice and sophisticated decision makers can both be rational decision makers in the neoclassical economics sense.Alternatively, as in Luca & Smith (2013), one could view novice decision makers from a behavioral perspective as using rankings to reduce the time required to form preferences.) A sophisticated decision maker is able to calculate his or her utility of attending a university as a function of university attributes and may not seem to need to place much value on the publication's aggregation function and associated ranking of universities.However, a university's rank in a publication may also affect the sophisticated decision maker's utility of attending the university: for example, a top student may want to attend the topranked university that other top students will also want to attend, requiring information for such coordination.If so, this decision maker's utility is also a function of a university's rank as reported in a ranking publication.Therefore, a ranking publication can both inform (through providing information about university attributes) and persuade (through reporting ranks) sophisticated decision makers as well as novices.
For a rankings publication to provide an appropriate service for students, the aggregation function used should seemingly reflect the utility functions of sophisticated decision makers.If it does, then the publication is providing the best information to students at the lowest time and money access cost to students.If the publication changes its aggregation function periodically, even though the utility functions of sophisticated student decision makers are stable over time, then the publication is acting strategically.A key issue in our analysis is whether a publication strategically changes its methodology -the attributes included, the attribute scoring rule, and the aggregation function -from year to year solely to generate interest in their rankings.
USNWR advises readers that the publication regularly changes its methodology: 'Since we may change our methodology from year to year, we do not invite readers to track colleges' annual moves in the rankings' (Morse & Flanigan 2001, p. 28).Ehrenberg (2002) contends that a ranking publication has an incentive to change its ranking methodology regularly by changing the attributes included, changing how the attributes are measured, or changing the aggregation function: 'Because of year-to-year changes in their rankings on the various dimensions that USNWR considers or yearto-year changes in the weight that each dimension is given in computing the overall ranking, an institution's ranking within its group may change from year to year.It is the change in the numerical rankings of institutions near the top of each institutional category, as well as the changes in the quartile rankings of some lower-ranked institutions from year to year that sells lots of copies of magazines.After all, if the ranking of institutions did not vary over time, there would be no need for families to have the most recent year's issues.(Ehrenberg 2002, p. 147)' Consider the effect of a publication's choice of methodology, particularly its choice of aggregation function, on student utility.For a university-ranking publisher to increase its revenues by strategically changing its aggregation function, the publisher must affect the value of the ranking to students.That value will increase only if the ranking has a positive probability of affecting a student's university search, application or matriculation decision: the greater the probability that a ranking affects a student's decision, the greater is the student's expected value of obtaining the ranking and, hence, the higher the price the student is willing to pay for access.
If a ranking publication adopts a new methodology by changing its aggregation function, then the utilities of novice decision makers will clearly be affected, while those of sophisticated decision makers may not be affected.Therefore, we consider the effect of the aggregation function on 3 categories of student decision-makers: (1) novice decision makers who are incompletely informed about university attributes; (2) sophisticated decision makers who are incompletely informed about university attributes; and (3) sophisticated and well-informed decision makers (we assume that the novice but completely informed decision maker is not a sensible category).Uninformed and novice students value rankings, using them to gain information about universities and form utilities (Table 1).Uninformed and sophisticated students at a minimum seek information about university attributes.Even in a stable environment in which university attributes and the types of students vary little over time, informed and sophisticated students can value rankings if a ranking publication changes its ranking process regularly.For each of the 3 types of students, a publisher can make strategic moves that increase the student's expected value of the publisher's ranking.

A RANKING PUBLICATION'S STRATEGIC MOVES
Consider the environment where the degree of sophistication of the students, the distribution of information among students, and the utilities of the student decision makers are all stable over time, and where school attributes are also stable over time.In such an environment, the only reason for a publication to change its aggregation function is for strategic reasons.
Consider student choices of whether to access a ranking within our taxonomy of sophisticated/novice and informed/uninformed students and also whether the rankings are informative or persuasive.We begin with the informative role of the ranking.Informed, sophisticated students have little (informative) need for university rankings as they know school attributes and have well-formed utilities about how they rank universities.Published university rankings should be unimportant to these students as they would either not access a ranking publication or would spend little time on the website.For uninformed, sophisticated decision makers, the published rankings themselves are unimportant but the information about university attributes is important.Therefore, their choice of whether to access a published ranking depends on information about university attributes and is independent of the aggregation function.The value of a published ranking to uninformed, novice decision makers depends on the reputation the ranking has earned in helping these types of students.A publication earns a positive reputation if students believe that the publication's aggregation function closely matches their own utility function.If these student utilities, once they are well-formed, are stable over time, then for a ranking publication to increase student value it should use the same aggregation function from year to year.
Hence, considering only the informative role of rankings when university attributes are stable over time, to generate the greatest access to a ranking publication from each of the 3 types of students, it is in the best interest of the publication to use a stable aggregation function, and to choose one that best matches student utilities.Now consider the persuasive role of a ranking, also when university attributes are stable over time.
Here, publications can generate access by strategically manipulating rankings.The publication can strategically manipulate the ranks of the schools either by changing the weights in the aggregation function or by changing the attributes included.By manipulating the aggregation function, students see value in accessing the current version of the published ranking because they expect to update their utility functions based on the current ranking.Sophisticated decision makers may seek information on the current rankings which may play a role in their utilities, while novice decision makers see value in all aspects of the current information on attributes and rankings.
When the levels of university attributes actually change over time, a rankings publication does not need to change its aggregation function to generate access because, even with a stable aggregation function, the ranks of universities will change.Hence, in such a turbulent (university) environment, all students are uninformed about current university attributes and see value in accessing them.In this case, publications have no need to strategically manipulate their ranking methodology.However, if university attributes are stable over time, then a publication has an incentive to strategically manipulate its ranking methodology to take advantage of the persuasive role its rankings play.

UNIVERSITY STRATEGIC MOVES: DIVERTING RESOURCES AND PROVIDING INFORMATION
Grewal et al. ( 2008) examine the marginal effect of a change in a university's attributes on the change in its USNWR ranking.By characterizing these marginal effects, they identify the gains to schools of strategically manipulating their attribute scores.Ehrenberg (2002) outlines numerous types of actions and resource movements taken by universities to improve their rankings including strategic manipulation of university admissions decisions.

Admitting students with greater probabilities of matriculation
USNWR adds to the incentives for universities to accept students by early admission or early action.Early admissions rules dictate that students apply to only one university as an early admission candidate.If accepted, the student commits to attend that university.Early action, however, does not require this type of commitment.Avery & Levin (2010) show that students who apply as early action candidates matriculate with greater probability than do regular admission candidates.By accepting more students early, universities can reject more regular admission candidates, thereby lowering a university's acceptance rate, or exclusivity score, an attribute in the USNWR ranking.Roth & Xiaolin (1994) show this tendency for markets for college admissions to push decisions to earlier dates.The same property holds for demonstrated interest: Dearden et al. (2013) show that students who demonstrate interest in a university by, for example, visiting campus, are more likely to enroll.Using admissions data from a highly-selective private university, they show that the university is more likely to accept applicants who demonstrate interest.

Misreporting data
Universities manipulate information by misreporting data to rankings publications or influence responses to survey questions used in rankings.Recognizing the importance of rankings to universities, self-reports lead to fraudulent reports (Bucknell, Emory, and George Washington Universities and Claremont McKenna College are recent offenders, see Hoover 2012, Hechinger 2013), which reduces the reliability of the rankings.Even when a publica-tion invests in collecting survey information, rankings can be manipulated.For the BusinessWeek rankings, universities often coach their students that it is in their best interest to give their own universities the best possible survey responses, since a higher rank increases the reputational value of the students' university investment.

DISCUSSION
Our analysis points to 2 policy questions: whether ranking publications using publication-determined weights about multiple university attributes to rank universities improve the efficiency of the college search, admissions, and matriculation process; and whether publications set and change methodologies to increase publication revenues at the expense of the efficiency of the search, admissions, and matriculation process.The result in Luca & Smith (2013) that university ranks in USNWR, independent of university attributes, are important to students underscores the value of publication-determined rankings (as opposed to personalized college rankings).Luca & Smith (2013) suggest 2 possible reasons for the student preference for publication-determined rankings: the rankings serve as a simplifying tool for applicants, and better ranks conferred by a dominant ranking publication confer a halo effect.A third reason could be that the best students use the ranking publication to coordinate their decisions of attending the same university by attending the publication's top-ranked university.
If university attributes and student utilities are actually stable over time, then it seems that the sole purpose for a ranking publication to change its methodology is to directly affect student utility, to persuade students.Without a publication-determined ranking, it may be that collective wisdom ranks Harvard University as the number-one US University and all top students will seek to attend Harvard.With a ranking publication that changes methodology from year to year, students have an incentive to access the ranking publication to find a given year's number one school, which might not be Harvard.(Note that Harvard is tied with Princeton for the #1 position in the 2013 US-NWR national university ranking, but is #3 behind MIT and Cambridge University in the USNWR world's best universities ranking.)With or without the ranking, top students attain their goal of attending the top university.Therefore, in this stable environment, because students spend time and money to access a ranking, the ranking publication could reduce the ef-ficiency of the university search, admissions, and matriculation process.
This inefficiency hinges on the stability of university attributes and student preferences.Dichev (2001) empirically demonstrates that year-to-year movements in rankings are largely noise.
'The empirical evidence is based on an investigation of time-series predictability in changes for the two most important rankings, USNWR's Top 25 rankings of national universities and national liberal arts colleges.The main finding is that changes in the USNWR rankings have a strong tendency to revert in the next two rankings.The reversibility in rankings is strong not only in statistical terms but seems to account for a strikingly large part of the total variation in rankings changes.' (Dichev 2001, p. 239) Recognizing the persuasive aspect of rankings and the incentives of ranking publications to increase access implies that these publications have an incentive to change their methodologies more often than is appropriate.Of course, overly-stable methodologies have a cost as well and, as Emerson famously noted, 'A foolish consistency is the hobgoblin of little minds' (Self-Reliance 1841).If student utilities change over time or innovations in higher education call for changes in attributions or changes in the aggregation function, then a stable methodology would inappropriately overlook those changes.
Ranking publications also encourage inefficient, fraudulent, or otherwise unethical university actions, reducing the value of a ranking.The inefficiency arises if actions aimed at improving rankings cause schools to focus resources on manipulating rankings rather than on furthering their academic missions (see Jacob et al. 2013 for a recent take on what they call 'College as Country Club').Furthermore, the shift toward early admission and the emphasis on demonstrated interest can lead to inefficient admissions decisions.Misreporting SAT scores and other student attributes is fraudulent and rejecting top applicants because they would matriculate with low probabilities is seen by many as unethical (see Golden 2001, Hoover 2012, Hechinger 2013).Avery et al. (2013) developed their revealed-preference ranking in part as a response to these negative manipulations by universities to improve their USNWR rankings.
While publication-determined rankings potentially improve the efficiency of the university application, admissions, and matriculation process, they also encourage behavior on the parts of the publications and universities that potentially reduce the efficiency of the process.It is possible that greater support for not-for-profit publisher-determined rankings could improve the efficiency of the process.For example, the revealed-preference not-for-profit Avery et al. (2013) ranking could serve well to guide novice decision makers, and also to provide a coordination mechanism and halos for sophisticated decision makers.Furthermore, because the ranking in Avery et al. (2013) is based on only one attribute -student ma triculation decisions -it is not as subject to methodological manipulation of an aggregation function as the multi-attribute approaches.Hence, this singleattribute ranking combined with the elimination of the profit incentives of the ranking should lead to reduced incentives to manipulate the methodology to gain readers.The key would be to convince students to shift from using for-profit multi-attribute publication-determined rankings to a not-for-profit revealedpreference ranking.
In this article, we have raised more questions than we have answered.Our arguments have been qualitative and, we feel, merit mathematical formalization or further empirical assessment.In addition, we have focused largely on the institutional context of the USA, so our comments may be less germane to geographies outside the USA than to those within.However, we hope that our remarks will generate debate and encourage further research in this challenging and fascinating domain.

Table 1 .
Reasons why students read rankings, by student types (novice, sophisticated).N/A: not applicable