Google Scholar as a new source for citation analysis

: Traditionally, the most commonly used source of bibliometric data is Thomson ISI Web of Knowledge, in particular the Web of Science and the Journal Citation Reports (JCR), which provide the yearly Journal Impact Factors (JIF). This paper presents an alternative source of data (Google Scholar, GS) as well as 3 alternatives to the JIF to assess journal impact (h-index, g-index and the number of citations per paper). Because of its broader range of data sources, the use of GS generally results in more comprehensive citation coverage in the area of management and international business. The use of GS particularly benefits academics publishing in sources that are not (well) covered in ISI. Among these are books, conference papers, non-US journals, and in general journals in the field of strategy and international business. The 3 alternative GS-based metrics showed strong correlations with the traditional JIF. As such, they provide academics and universities committed to JIFs with a good alternative for journals that are not ISI-indexed. However, we argue that these metrics provide additional advantages over the JIF and that the free availability of GS allows for a democratization of citation analysis as it provides every academic access to citation data regardless of their institution’s financial means.


INTRODUCTION
Traditionally, the most commonly used source of bibliometric data is Thomson ISI Web of Knowledge, in particular the Web of Science and the Journal Citation Reports (JCR).For journals, the most commonly used metric is the Journal Impact Factor (JIF), as calculated in the JCRs, whilst for individual academics it is the number of citations as reported in the Thomson ISI Web of Science.
This paper presents an alternative source of data (Google Scholar, GS) as well as 3 alternative metrics to assess journal impact: the h-index (Hirsch 2005), g-index (Egghe 2006) and the number of citations per paper (CPP).We first present an overview of the advantages and disadvantages of using GS versus Thomson ISI Web of Science (WoS) when assessing citation impact for individual academics.We then report on 2 experiments in which GS and the JIF were compared to assess the im-pact of journals, using the alternative metrics presented above.Finally, we assess the difference between GS and the WoS when comparing the impact of books.
We have chosen to focus on the academic fields of management and international business, as these fields -and the Social Sciences in general -have been under-researched in the area of bibliometrics (Harzing 2005).All analyses were conducted in early September 2007 using the WoS, the ISI JCRs or Publish or Perish.Publish or Perish is a software programme that retrieves and analyses academic citations.It uses Google Scholar to obtain the raw citations, then analyses these and presents a wide range of citation metrics in a user-friendly format.The results are available onscreen and can also be copied to the Windows clipboard (for pasting into other applications) or saved to a variety of output formats (for future reference or further analysis).Publish or Perish was developed by Tarma Software Research (www.tarma.com)with input from A.-W. K. Harzing and is provided free of charge for personal non-profit use (available at: www.harzing.com/pop.htm).

COMPARING WOS AND GS FOR CITATION ANALYSES OF INDIVIDUAL ACADEMICS
In this section we will compare the respective advantages and disadvantages of Thomson ISI WoS and GS for citation analyses of individual academics.An important practical advantage of GS is that it is freely available to anyone with an Internet connection and is generally praised for its speed (Bosman et al. 2006).The WoS is only available to those academics whose institutions are able and willing to bear the (quite substantial) subscription costs of the WoS and other databases in Thomson ISI's Web of Knowledge.As Pauly & Stergiou (2005, p. 34)

indicate:
Free access to […] data provided by GS provides an avenue for more transparency in tenure reviews, funding and other science policy issues, as it allows citation counts, and analyses based thereon, to be performed and duplicated by anyone.
They also point to the advantage of the no-cost GS option for research and academic institutions not only in developing countries, but also for modestly endowed institutions in developed countries.

General caveat: citations are subject to many forms of error
Before we move to a comparison of the 2 sources of citation data, a general caveat is in order.Whilst we do believe, as detailed in the section 'The disadvantage of using WoS for citation analyses', that in most cases GS presents a more complete picture of an academic's impact than the WoS, all databases have their own limitations, most of which are discussed in detail in the section 'The disadvantage of using Google Scholar for citation analyses'.More generally, citations are subject to many forms of error, from typographical errors in the source paper, to errors in GS parsing of the reference, to errors due to some non-standard reference formats.Publications such as books or conference proceedings are treated inconsistently, both in the literature and in GS.Thus, citations to these works can be complete, completely missing, or anywhere in between.

Google Scholar critics assessed
Several academics have been very critical of GS.Péter Jacsó in particular has published some highly critical papers in Online Information Review (Jacsó 2005(Jacsó , 2006a,b) ,b) discussing a limited number of GS failures in great detail.Whereas no doubt some of his critique is justified, we were unable to reproduce most of the GS failures detailed in his 2006b paper, suggesting that either they resulted from faulty searches or that GS has rectified these failures.Some of these unreproduced errors are detailed in Appendix 1. Jacsó (2006b) criticizes Pauly & Stergiou (2005) for handpicking of what he claims to be 'a tiny sample' of 114 articles.His own article is based on about a dozen GS mistakes that appear to be handpicked.We wonder why his results should be valid if the results of a sample 10 to 20 times as large are not.Jacsó's (2006b) claim that GS reports higher citation counts for certain disciplines, but not for the social sciences and humanities is certainly inaccurate, as a much larger-scale study (Bosman et al. 2006) finds the opposite result.Most importantly, the bulk of Jacsó's (2006b) critique is levelled at inconsistent results for keyword searches, which are not relevant for the author and journal impact searches conducted in this paper.In addition, the summary metrics used in the present paper are fairly robust and insensitive to occasional errors as they will not generally change the h-index or g-index and will only have a minor impact on the number of citations per paper.

Rule of thumb for citation analyses
When using GS for citation analyses, we suggest the following general rule of thumb.If an academic shows good citation metrics, i.e. if his or her work is well-cited, it is very likely that he or she has made a significant impact on the field.If an academic shows weak citation metrics, this may be caused by a lack of impact on the field.However, it may also be caused by working in a small field, publishing in a language other than English (LOTE), or publishing mainly (in) books.Although GS performs better than the WoS in this respect, it is still not very good at capturing LOTE articles and citations, or citations in books or book chapters.Google books might be a better alternative for the latter.As a result, citation metrics in the social sciences and even more so in the humanities will always be underestimated, as in these disciplines publications in LOTE and books/book chapters are more likely than in the sciences.

The disadvantage of using WoS for citation analyses
The major disadvantage of the WoS is that it may provide a substantial underestimation of an individual academic's actual citation impact.This is true equally for the 'general search' function and for the WoS 'cited reference' function, the 2 functions most generally used to perform citation analyses.However, the WoS 'general search' function performs more poorly in this respect than the 'cited reference' function.For example, the current (September 2007) number of citations to Harzing's work is around 120 with the 'general search' function, around 310 with the 'cited reference' function and 828 with GS.Harzing's h-index is 7 with the 'general search' function, 12 with the 'cited reference' function and 15 with GS.
Differences might not be as dramatic for all scholars 1 , but many academics show a substantially higher number of citations in GS than in WoS.For instance, Nisonger (2004) found that (excluding self-citations) WoS captured only 28.8% of his total citations, 42.2% of his print citations, 20.3% of his citations from outside the USA, and a mere 2.3% of his non-English citations.He suggests that librarians and academics should not rely solely on WoS author citation counts, especially when demonstration of international impact is important.Nisonger (2004) also summarises several other studies that found WoS citation data to be incomplete.

Large-scale comparison between WoS, Scopus and GS
Meho & Yang (2007) conducted a large-scale comparison between WoS, Scopus (Elsevier's alternative to Thomson ISI's WoS) and GS covering citations of over 1000 scholarly works of all 15 faculty members of the School of Library and Information Science at Indiana University, Bloomington, between 1996 and 2005.They found the overlap in citations between the 3 databases to be rather small, which given the extreme dependence on WoS is quite an important conclusion.The overlap between WoS and Scopus was 58.2%.The overlap between GS and the union of WoS and Scopus was only 30.8%.This small overlap is largely caused by the fact that GS produced more than twice as many citations as WoS and nearly twice as many citations as Scopus.Many of those additional citations came from conference papers, doctoral dissertations, master's theses and books and book chapters.
At the same time, both sources (WoS and GS) rank specific groups of scholars in a relatively similar way.Saad (2006) found that for his subset of 55 scientists in consumer research, the correlation between the 2 hindices was 0.82.Please note that this does not invalidate the earlier argument, as it simply means that most academics' h-indices are underestimated by a similar magnitude by WoS.Meho & Yang (2007) also found that when GS results were added to those of WoS and Scopus separately its results did not significantly change the ranking of the 15 academics in their survey.The correlation between GS and WoS was 0.874; between GS and the union of WoS and Scopus it was 0.976.Meho & Yang (2007) conclude that GS can help identify a significant number of unique citations.These unique citations might not significantly alter one's citation ranking in comparison to other academics in the same field and might not all be of the same quality as those found in the WoS or Scopus.However, they can be very useful in showing evidence of broader intellectual and international impact than is possible with WoS and Scopus.Hence, they further conclude that GS could be particularly helpful for academics seeking, for example, promotion, tenure, faculty positions, research grants.

WHY DOES WOS UNDERESTIMATE TRUE CITATION IMPACT?
WoS 'general search' is limited to ISI-listed journals In the general search function, WoS only includes citations to journal articles published in ISI indexed journals (Roediger 2006).Citations to books, book chapters, dissertations, theses, working papers, reports, conference papers, and journal articles published in non-ISI journals are not included.Whilst in the natural sciences this may give a fairly comprehensive picture of an academic's total output, in the social sciences and humanities only a small proportion of journals are ISI listed.Also, in both the social sciences and the humanities, books and book chapters are very important publication outlets.GS includes citations to all academic publications regardless of whether they appear in ISIlisted journals (Belew 2005, Meho & Yang 2007).Even in the natural sciences however, some sub-disciplines or geographical areas such as the Mediterranean and the tropics only have limited ISI coverage.In a review of the literature on Mediterranean marine and freshwater fishes, Stergiou & Tsikliras (2006) showed that less than 40% of the publications were published in ISI journals.Stergiou & Tsikliras (2006, p. 16) conclude that: …searching only ISI's database produces an underrepresentation of the scientific output of professionals studying marine ecosystems.

1
Factors that might influence this difference are related to GS's more comprehensive data sources as discussed below.Academics with a higher number of citations in, e.g.non-ISI-indexed journals, books and book chapters, conference proceedings, LOTE articles, will show a larger difference than academics who are mainly cited in ISI-indexed journals.As discussed these factors are highly influenced by the discipline in question

WoS 'cited reference' is limited to citations from ISI-listed journals
In the cited reference function WoS does include citations to non-ISI publications.However, it only includes citations from journals that are ISI-listed (Meho & Yang 2007).As already indicated, in the social sciences and humanities only a limited number of journals are ISI-listed.Butler (2006) analysed the distribution of publication output by field for Australian universities between 1999 and 2001.She found that whereas for the chemical, biological, physical and medical/health sciences between 69.3% and 84.6% of the publications are in ISI listed journals, for management, history, education and arts only 4.4 to 18.7% of the publications are published in ISI listed journals.ISI estimates that of the 2000 new journals reviewed annually only 10 to 12% are selected to be included in the WoS (Testa 2004).Archambault & Gagné (2004) found that US-and UK-based journals are both significantly over-represented in the WoS in comparison to Ulrich's journal database, which is recognized as one of the most exhaustive directories of journals available.This overrepresentation was stronger for the social sciences (40%) and humanities than for the natural sciences (23%).
Further, in many areas of engineering, conference proceedings are very important publication outlets.For example, our search for one of the most cited computer scientists (Hector Garcia-Molina) results in more than 20 000 citations in GS, with most of his papers being published and cited in conference proceedings.In WoS he has a mere 240 citations to his name!In contrast to WoS, GS includes citations from all academic publications regardless of where they appeared.As a result, GS provides a more comprehensive picture of recent impact, especially for the social sciences and humanities where more than 5 yr can elapse between research appearing as a working or conference paper and research being published in a journal (Harzing 2005).This also means that GS usually gives a more accurate picture of impact for junior academics.However, it must be acknowledged that although GS captures more citations in books and book chapters than WoS (which captures none) it is by no means comprehensive in this respect.Google Book Search may provide a better alternative for book searches.

WoS 'cited reference' counts citations to non-ISI journals only with respect to the first author
Whilst the cited reference function of WoS does include citations to non-ISI journals, it only includes these publications for the first author.Hence, any publications in non-ISI journals where the academic in question is the second or further author are not included.GS includes these publications for all listed authors.For instance, the publication Feely & Harzing (2003) in Cross Cultural Management shows no citations in the WoS for Harzing whilst it shows 18 citations in GS.

WoS has poor aggregation of minor variations of the same title
In the general search function, WoS does not include citations to the same work that have small mistakes in their referencing (which especially for books and book chapters occurs very frequently).In the cited reference function WoS does include these citations, but they are not aggregated with the other citations.GS appears to have a better aggregation mechanism than WoS.Even though duplicate publications that are referenced in a (slightly) different way still occur, GS has a grouping function that resolves the worst ambiguities.For instance, Harzing & Hofstede's (1996) publication in the research annual Research in the Sociology of Organizations draws 15 WoS citations, but these are spread over 7 different appearances.GS shows 29 citations and has only 1 appearance for the publication.Belew (2005) confirms that GS has lower citation noise than WoS.In the WoS, only 60% of the articles were listed as unique entries (i.e.no citation variations), while for GS this was 85%.None of the articles in his sample had more than 5 separate listings within GS, while 13% had 5 or more entries in the WoS.

WoS has very limited coverage of non-English sources
The WoS includes only a very limited number of journals in languages other than English (LOTE) and hence citations in non-English journals are generally not included in any WoS citation analysis.Whilst GS's LOTE coverage is far from comprehensive, it does include a larger number of publications in other languages and indexes documents in French, German, Spanish, Italian and Portuguese (Noruzi 2005).Meho & Yang (2007) found that 6.94% of GS citations were from LOTE, while this was true for only 1.14% for WoS and 0.70% for Scopus.Archambault & Gagné (2004) found that Thomson's ISI's journal selection favours English, a situation attributable to ISI's inability to analyse the content of journals in LOTE.

GS includes some non-scholarly citations
GS sometimes includes non-scholarly citations, such as student handbooks, library guides or editorial notes.However, incidental problems in this regard are unlikely to distort citation metrics, especially robust ones such as the h-index.An inspection of Harzing's papers shows that in general more than 75% of the citations are in academic journals, with the bulk of the remainder appearing in books, conference papers, working papers and student theses.Few non-scholarly citations were found.Moreover, we would argue that even a citation in student handbooks, library guides, textbooks or editorial notes shows that the academic has an impact on the field.

Not all scholarly journals are indexed in GS
Not all scholarly journals are indexed in GS.Unfortunately, GS is not very open about its coverage and hence it is unclear what its sources are.It is generally believed that Elsevier journals are not included (Meho & Yang 2007) because Elsevier has a competing commercial product in Scopus.However, we were able to find many Elsevier journals in the social sciences.On the other hand, Meho & Yang (2007) did find that GS missed 40.4% of the citations found by the union of WoS and Scopus, suggesting that GS does miss some important refereed citations.It must also be said though that the union of WoS and Scopus misses 61.04% of the citations in GS.Further, Meho & Yang (2007) found that most of the citations uniquely found by GS are from refereed sources.

GS coverage might be very uneven across different fields of study
Although for reasons discussed above GS generally provides a higher citation count than ISI, this might not be true for all fields of study.The social sciences, arts and humanities, and engineering in particular seem to benefit from GS's better coverage of (citations in) books, conference proceedings and a wider range of journals.The natural and health sciences are generally well covered in ISI and hence GS might not provide higher citation counts.In addition, user feedback received for Publish or Perish seems to indicate that for some disciplines in the natural and health sciences GS's journal coverage is very patchy.This leads to citation counts in these areas that might actually be much lower than those in ISI.In a systematic comparison of 64 articles in different disciplines, Bosman et al. (2006) found overall coverage of GS to be comparable with both WoS and Scopus and slightly better for articles published in 2000 than in 1995.
However, huge variations were apparent between disciplines, with chemistry and physics in particular showing very low GS coverage and science and medicine also showing lower coverage than in WoS (Bosman et al. 2006).More detailed and recent comparisons by academics working in the respective areas would be necessary before we can draw general conclusions.However, as a general rule of thumb, we would suggest that using GS might be most beneficial for 3 of the GS categories: (1)

GS does not perform as well for older publications
GS does not perform as well for older publications, as these publications and the sources that cite them have not (yet) been posted on the web.Pauly & Stergiou (2005) found that GS had less than half of the citations of the WoS for a specific set of papers published in a variety of disciplines (mostly in the sciences) between 1925 and 1989.However, for papers published in the 1990 to 2004 period both sources gave similar citation counts.The authors expect GS's performance to improve for old articles as journals' back issues are posted on the web.Meho & Yang (2007) found the majority of the citations from journals and conference papers in GS to be from after 1993.Belew (2005) found GS to be competitive in terms of coverage for references published in the last 20 yr, but the WoS superior before then.This means that GS might underestimate the impact of scholars who have mainly published before 1990.

GS's automatic processing creates occasional nonsensical results
GS's processing is done automatically without human intervention and hence sometimes provides nonsensical results.For instance one of the citations to Harzing's (1999) Managing the Multinationals book lists as its title 'K., 1999'.The author of the citing paper listed Harzing's initials with a comma after the first 2 initials and hence GS interpreted the third initial and year as the title.Automatic processing can also result in double counting citations when 2 or 3 versions of the same paper are found online.However, incidental mistakes like this are unlikely to have a major impact on citation metrics, especially those as robust as the h-index.Moreover, GS indicates on its website that it is committed to fixing mistakes.

GS is not updated as often as WoS
GS is not updated as often as Thomson ISI WoS.Whilst GS does not provide information about its update frequency, our experience suggests it is updated at least every 2 to 3 months, whilst more recently minor updates seem to have occurred more frequently.The lack of daily updating might be problematic for accessing the latest research information in fields that change quickly.However, for the purpose of citation analysis this should be not be a huge problem as many academic journals only have 4 to 6 issues a year and hence GS would generally not be more than 1 issue behind.

Names with diacritics or apostrophes are problematic
Both GS and Thomson ISI WoS have problems with academics that have names including either diacritics (e.g.Özbilgin or Olivas-Luján) or apostrophes (e.g.O'Rourke).In Thomson ISI WoS, a search with diacritics provides an error message and no results.In GS a search for the name with diacritics will usually not provide any results either.For both databases doing a search without the diacritic will generally provide the best result.A search for 'O'Rourke K*' in WoS results in only one citation to the work of the economic historian Kevin H O'Rourke, whereas a search for 'ORourke K*' results in more than 350 citations.GS performs much better.Originally, a search for 'K O'Rourke' in GS provided very few results as GS treated both K and O as initials and hence searched for KO Rourke.Adding an additional blank space before O'Rourke solved this problem.However, GS seems to have fixed this problem, as searches for 'K O'Rourke' (without the additional blank space) now (September 2007) result in more than 1850 citations.

Names with ligatures are problematic
If an academic's name includes a sequence of characters that is ligatured in traditional typesetting ('fi', 'ff', 'fl', and others in other languages) and he/she prepares papers with LaTeX (as do most academics in mathematics and computer science), then GS does not find the publications.For example to find most of the publications of J* Bradfield, one must search for J* Bradeld (omitting the 'fi' ligature created by LaTeX).In Google Scholar 'J* Bradfield' only results in some 190 cites for computer scientist Julian Bradfield, whereas 'J* Bradeld' results in nearly 400 cites for the same person.It should be mentioned that WoS does not find the publications showing up under Bradeld either as they usually concern books or conference proceedings.'Bradfield J*' results in only about 50 cites for Julian Bradfield in WoS, 'Bradeld J*' results in none.

COMPARING GS AND WOS FOR SPECIFIC SEARCHES
In this section, we will report the results of 3 specific experiments to assess the coverage of GS versus JCR/WoS for the field of management and international business: first, we compared the impact of the top-20 journals in management, as measured by the 2006 ISI journal impact factor (JIF): second, we compared the impact of all mainstream international business (IB) journals: third, we compared the total number of citations in WoS and GS to books that won the Terry Book award between 1991 and 2001.
To attenuate idiosyncrasies for individual years, we also calculated the average impact factor for 2002 to 2006 for the first 2 experiments, by averaging the 5 individual impact factors listed by ISI for each of the years.The resulting average impact factor hence refers to citations to articles published in 2000 to 2005.GS metrics were calculated over the same dataset, i.e. papers published in the journal in question between 2000 and 2005.Note that we can only measure data from GS at one point in time (the time of search), rather than calculate the impact metrics for each year independently.All 3 GS metrics were therefore computed over a longer time-frame than the ISI JIF, i.e. citations in September 2007 to articles published between 2000 and 2005, rather than citations in each of the years 2002 to 2006 to articles published in the preceding 2 yr as is the case with the JIF.As a result, the data will not be completely identical to ISI data.However, we are measuring citations to the same set of papers, i.e. those published between 2000 and 2005.
For our GS journal searches we use the h-index, g-index and citations per paper as new metrics to compare journal impact with the traditional ISI JIF.For our GS book searches, we used the total number of citations.

The h-index
The h-index was introduced by Hirsch (2005) and is defined as follows: 'A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np -h) papers have no more than h citations each' (Hirsch 2005, p. 16 569).It aims to measure the cumulative impact of a researcher's output by looking at the number of citations his/her work has received.Hirsch argues that the h-index is preferable to other single-number criteria, such as the total number of papers, the total number of citations and citations per paper.
The advantage of the h-index is that it combines an assessment of both quantity (number of papers) and quality (impact, or citations to these papers) (Glänzel 2006).An academic cannot have a high h-index without publishing a substantial number of papers.However, this is not enough.These papers need to be cited by other academics in order to count for the hindex.As such the h-index is said to be preferable over the total number of citations as it corrects for 'one hit wonders', i.e. academics who might have authored (or co-authored) 1 highly-cited paper or a limited number of highly-cited papers, but have not shown a sustained and durable academic performance.It is also preferable over the number of papers as it corrects for papers that are not cited.Hence, the h-index favours academics that publish a continuous stream of papers with lasting and above-average impact (Bornmann & Daniel 2007).
The h-index has been found to have considerable face validity.Hirsch calculated the h-index of Nobel Prize winners and found 84% of them to have an h-index of at least 30.Newly elected members in the National Academy of Sciences in physics and astronomy in 2005 had a median h-index of 46.Bornmann & Daniel (2005) found that on average the h-index for successful applications for postdoctoral research fellowships was consistently higher than for nonsuccessful applicants.Cronin & Meho (2006) found that faculty rankings in information sciences based on raw citation counts and on the h-index showed a strong positive correlation, but claim that the h-index provides additional discriminatory power.Van Raan (2006) calculated the h-index for 147 chemistry research groups in the Netherlands and found a correlation of 0.89 between the h-index and the total number of citations.Both the h-index and more traditional bibliometric indices also related in a quite comparable way with peer judgements.Finally, maybe the strongest indication that the h-index is becoming a generally accepted measure of academic achievement is that Thompson ISI has now included it as part of its new 'citation report' feature in the Web of Science.
Examples of the application of the h-index to journals are still scarce (but see Braun et al. 2005, Saad 2006).However, the arguments above would also apply to journals.We are interested in whether journals publish a continuous stream of papers with lasting and above-average impact.

The g-index
The h-index ignores the number of citations to each individual article beyond what is needed to achieve a certain h-index.Hence an academic or journal with an h-index of 5 could theoretically have a total of 25 citations (5 for each paper), but could also have more than a 4000 citations (4 papers with 1000 citations each and 1 paper with 5 citations).In reality these extremes will be unlikely.However, once a paper belongs to the top h papers, its subsequent citations no longer 'count'.Such a paper can double or triple its citations without influencing the h-index (Egghe 2006).Hence, in order to give more weight to highlycited articles Leo Egghe (2006) proposed the g-index.The g-index is defined as follows: Given a set of articles ranked in decreasing order of the number of citations that they received, the g-index is the (unique) largest number such that the top g articles received (together) at least g 2 citations.Although the g-index has not yet attracted much attention or empirical verification, it would seem to be a very useful complement to the h-index.

Citations per paper
The number of citations per paper was calculated simply by dividing the total number of citations for the 2000 to 2005 timeframe by the number of articles.In this calculation, we only included regular journal articles, excluding items such as book reviews, editorial comments, call for papers and other non-article material.We also excluded the occasional 'submitted/forthcoming/under revision' papers for the journal in question reported by GS.All of these 'non regular article' publications have no or few citations, and including them would drastically lower the average number of citations per paper for journals with for instance a large number of book reviews, such as Administrative Science Quarterly.
For our book search, the total number of citations in both WoS and GS was calculated simply by adding up the citations to different variations of the book title and author name (i.e.number of initials included).

Comparison of top 20 management journals
Table 1 lists the 20 top journals in management according to the 2006 journal impact factor.They are ranked by the average impact factor for 2002 to 2006.We further report the journal impact factor for 2006, the h-index, the g-index and the average number of citations per paper.For each additional metric we also report how the journal ranks on this measure.

Strong agreement between the various metrics
Table 2 shows strong and significant (all p < 0.01) correlations between the various impact metrics for the top 20 management journals.This is certainly true for the different metrics from the same source (i.e.ISI JCR or GS).However, even correlations between the metrics derived from the ISI JCR and GS are fairly high.The GS derived citations-per-paper metric in particular shows a very strong correlation with the ISI JCR journal impact measures.This is not surprising given the fact that the citations-per-paper metric is conceptually most strongly related to the ISI JIF; both are variants of a citation-per-paper measure.When the Human Resource Management anomaly (see below) is excluded, the correlation between the GS citations- per-paper metric and the WoS JIFs is even higher than its correlation with the GS h-index.Overall, it would seem that the different impact metrics can be used as alternative measures of journal impact.However, although overall correlations are high, Table 1 shows that there are fairly substantial differences in journal rankings for the different metrics.We contacted Thomson ISI for an explanation of this difference and the support officer claimed that the JCR used a slightly different calculation than the cited reference search and that the difference was mainly made up by references to misspelled versions of this journal.However, the examples of misspellings given referred to Human Resource Management Review and Human Resource Management Journal, which are distinct journals, and to books with Human Resource Management in their titles.Moreover, when we double-checked the JIF of 5 other journals (Organization Studies, Leadership Quarterly, Information and Management, Information Systems Research and Journal of Operations Management), we found the cited reference search to provide 50 to 100% more citations than the JCR.Hence, we consider it likely that the GS ranking for Human Resource Management is more accurate than the ISI JCR ranking.

Journals ranking higher in GS
Six journals show a significant improvement in their ranking when using GS: Strategic Management Journal, Management Science, Journal of International Business Studies, Journal of Management Information Systems, Journal of Management Studies and Journal of Product Innovation Management.We argue that this change in ranking might be caused by 3 separate effects.
The first is a lack of ISI journal coverage in specific areas of management.Very few journals in the field of strategy and international business are covered in ISI (a separate analysis for international business follows in the next section).Hence, any citations to the 2 top journals in these fields (Strategic Management Journal and Journal of International Business Studies) in journals that are not covered in ISI will not be accounted for in ISI, whilst they will be included in the rankings based on GS.Hence these journals will show a higher ranking when using GS.
Second, the fact that engineers in general and computer scientists in particular tend to publish a lot of their work in conference proceedings (which are included in GS, but not in ISI) might explain the fact that journals that may be referred to by engineers and computer scientists, such as Management Science, Journal of Product Innovation Management and Journal of Management Information Systems, show higher rankings in GS.
Finally, ISI coverage tends to be heavily concentrated on North American journals.This means that journals that are traditionally more focused towards a non-North American audience might gather more citations in GS, which includes a larger proportion of non-North American journals.To test this assumption, we analysed the proportion of ISI citations by North-American authors for the different journals in our sample.As is shown in Table 1 for many top North American journals this proportion is between 2 ⁄ 3 and 3 ⁄ 4 of the total number of citations.However, for the European-based journals (Information and Management, Organization Studies, Journal of Management Studies) this proportion lies between 1 ⁄ 3 and 1 ⁄ 2 of the total number of citations.One of these journals -Journal of Management Studies -does indeed show a significantly higher ranking in GS than in ISI.However, the 2 other journals do not show an improved ranking in GS and hence the validity of this argument should be subject to further validation in a larger group of journals.

Conclusion: GS presents a more comprehensive picture
Overall, we would argue that for the field of management the various GS-based citation metrics provides a more comprehensive picture of journal impact than the ISI JIF.However, the very high correlations between the ISI JIF and GS citations-per-paper measures do indicate that this GS based measure could be an excellent alternative to the ISI JIF for academics and universities who insist on measures closely related to the ISI JIF.In fact, if we exclude the 3 largest ranking differences from the top 20 journal set (Human Resource Management, Leadership Quarterly, and Journal of Management Information Systems) the correlation between the ISI JIF 2002 to 2006 and the GS citations-per-paper metric increases to 0.89, whilst correlations with the h-index (0.67) and gindex (0.74) are also very high.This then brings us to our next experiment which compares journals in the field of international business that are ISI listed with journals that are not listed.

Comparison of international business journals
As indicated above, there are some areas of management where coverage of journals in ISI is very limited.Here we discuss one particular area -the main research area of the first author -in a bit more detail.Dubois & Reeb (2000) provided a comprehensive ranking of journals publishing papers in the area of international business.Of this list, we only included the socalled mainstream international business journals.Journals in the area of international marketing (e.g.International Marketing Review) or international economics (e.g.International Trade Journal) for instance were excluded.
We followed the same search strategies as for the top 20 management journals detailed above.However, because of the smaller number of journals and papers involved, we also manually excluded duplicate titles (i.e.titles that also appeared with incorrect spelling and hence few citations).These errors did not impact on the h-index and g-index, but did result in a marginal rise in the citations-per-paper metric.
As Table 3 shows only 2 of the 7 mainstream IB journals are ISI ranked.These 2 journals do indeed appear to be the highest impact journals in terms of the h-index and citations-per-paper metric.However, the main distinction appears to be between the Journal of International Business Studies and the other IB journals.In fact, International Business Review has an h-index and citations-per-paper mean that is not substantially different from Journal of World Business, and its g-index is even slightly higher.It therefore seems entirely justified that this journal was included in ISI in 2005; its first JIF will be listed in 2007.
The citations-per-paper impact metrics for Management International Review and Journal of International Management are fairly high as well.In fact, they are not much lower than the bottom 2 journals in the list of top 20 management journals, let alone those of ISI-listed journals lower down the ranks such as Services Industries Journal (mean citations per paper 3.30) and Total Quality Management and Business Excellence (mean citations per paper 3.03).Given these results, it would seem rather artificial to make a distinction between ISI-listed journals and non-ISI listed journals as many journals that are not ISI listed might actually have a higher impact on the field than journals that are ISI listed.

Terry book awards
Our final experiment involves a comparison between WoS and GS for the impact of books.We have chosen to look at books winning the Terry book award between 1991-2001 for 2 reasons.Firstly, the Terry Book award is a prestigious award given yearly by the Academy of Management, the most important professional organization in the field.Secondly, Pfeffer & Fong (2002) strongly criticised business school research for its lack of impact on practice.One of their arguments was based on an analysis of the impact of books winning the Terry Book award between 1991 and 2001.Pfeffer & Fong (2002) claim that even these supposedly highly influential books only had an average of 6.80 citations per year.Walsh (in Walsh et al. 2007) concludes that this shows that our best books are not particularly well read even by our scholarly peers.We wondered whether the same conclusion would be drawn if we used a more inclusive source of citation impact, i.e.GS instead of the WoS.

Inaccurate citation analysis creates myths
In order to assess this, we first repeated the analysis conducted by Pfeffer & Fong (2002).Our results show that the Terry Book Award winners on average received 346 cites in WoS for an average of approximately 28 citations a year, not exactly a performance which we would consider to show low impact (see Table 4).Hence, our conclusion strongly contradicts that of Pfeffer & Fong (2002).It is unclear why our analysis resulted in so many more citations.One reason might be the fact that our analysis was conducted 5 yr later and hence the books had had more time to gather citations.This shows that the impact of books might take some time to effectuate and hence JIF-like indices that use 2 yr time-spans would not be very useful for books or for research fields that take a long time for work to penetrate.Another reason might be that Pfeffer & Fong (2002) did not systematically include misspellings or appearances with different author initials.A final reason might be that in the case of the Handbook of Organization Studies they did not include citations to individual chapters.Whatever the reason, it shows that one should be careful before drawing rather far-reaching conclusions.Whilst we would not necessarily negate Pfeffer & Fong's (2002) general conclusion that business schools do not have much impact on practice, their conclusion that even our best books do not have an academic impact -or Walsh's even stronger conclusion that 'our best books are not particularly read by business people or by their scholarly peers' (in Walsh et al. 2007, p. 129) -does not seem to be supported by the data.This note of caution is all the more important since myths are easily created by subsequent citations that seem to endorse the message, making it unassailable (see also Harzing 2002).The Pfeffer & Fong (2002) article had already gathered 57 ISI and 169 GS citations by September 2007.

GS reports 2.5 times as many citations as WoS
Even more remarkable than the difference between our WoS search and that of Pfeffer & Fong (2002), is the difference in impact when using GS as a base for citations.On average, GS reports nearly 2.5 times as many citations as the ISI WoS, for an average of 833 citations per book and 68 citations per book per year.Both measures show that these books have (had) a very considerable impact on the field.The differences are particularly large for 2 books in the area of strategic management (Haspeslagh & Jemison 1991, Mintzberg 1994), reflecting our earlier observation that strategy journals are not particularly well covered in ISI.In the case of Haspeslagh, the fact that this author and academic is not working at a North American university might also have led to a modest impact in ISI listed journals, given the focus of ISI on North American journals.

CONCLUSIONS
The use of GS generally results in more comprehensive coverage in the area of management and international business, which benefits academics publishing in sources that are not (well) covered in ISI.Among these are books, conference papers, non US journals and generally journals in the field of strategy and international business.
The 3 alternative GS-based metrics showed strong correlations with the traditional JIF.As such, they provide academics and universities committed to JIFs with a good alternative for journals that do not currently have a JIF.However, we argue that these metrics also provide some additional advantages over the JIF.
First, all 3 metrics were computed over a longer timeframe, i.e. citations in September 2007 to articles published between 2000 and 2005, rather than citations in each of the years 2002 to 2006 to articles published in the preceding 2 yr (as is the case with the JIF).This accommodates the strong concern that ISI's 2 yr time period is too short for 'slow response' disciplines.
Second, the h-index, and to a lesser extent the gindex, attenuates the impact of one highly-cited article because -in contrast to the citations-per-paper measures -these indices are not based on mean scores.In a citations-per-paper metric (either our GSbased CPP or the ISI JIF) one highly cited article can cause a very strong increase in the mean number of citations per paper for the journal in question, leading to highly idiosyncratic results.When evaluating journal quality through citation impact measures we are interested in the overall citation impact of the journal, not in the citation impact of 1 or 2 highly cited individual papers.Hence, just as the h-index for authors provides a measure of their sustained and durable research performance, the h-index for journals provides a robust measure of sustained and durable performance of journals.
We would therefore strongly encourage both individual academics and university administrators to take GS-based impact measures into account when evaluating the impact of both journals and individual academics in the areas of management and international business, as they lead to a more comprehensive and possibly more accurate measure of true impact.Moreover, the free availability of GS allows for a democratization of citation analysis as it provides every academic with access to citation data regardless of their institution's financial means.

® ®
Comparing the JIF 2002 to 2006 with the JIF 2006 shows that Organization Science, Journal of International Business Studies, Journal of Organizational Behaviour and Journal of Management Studies have recently improved their JIF ranking, whilst Administrative Science Quarterly, Human Resource Management, Leadership Quarterly, Management Science and Organization Studies have experienced a drop in ranking.It should be noted though that for the last 3 journals the actual JIF has improved, but because of the overall improvement in JIF over the years, their ranking relative to other journals has dropped.Journals ranking lower in GS Comparing the various GS metrics with the ISI JIF 2002 to 2006 shows that there are 3 journals that on all 3 GS metrics rank 3 or more places lower than on the ISI JIF 2002 to 2006: Human Resource Management, Journal of Operations Management and Leadership Quarterly.For the Journal of Operations Management the drop in ranking is relatively modest, but for Leadership Quarterly and in particular for Human Resource Management it is very important.The large difference in ranking for Human Resource Management prompted us to double check this journal's JIF.The JCRs for 2006 report 115 citations to this journal.However, a cited reference search with Hum Resource Manage (the official ISI abbreviation for Human Resource Management) as the cited work resulted in only 67 citations, 42% less than for the JCR.
business, administration, finance & economics; (2) engineering, computer science & mathematics; (3) social sciences, arts & humanities.Although broad comparative searches can be done for other disciplines, we would not encourage heavy reliance on GS for individual academics working in other areas without verifying results with either Scopus or WoS.

Table 1 .
Top 20 management journals according to different impact metrics.Rankings 3 or more places lower (higher) than JIF 2002 to 2006 are shown as italic (bold) text for both the JIF 2006 and the 3 Google Scholar metrics (h-index, g-index and CPP).Journal titles are italicized (bold) if all 3 GS rankings provide a lower (higher) ranking than the JIF 2002-2006.JIF: journal impact factor; CPP: citations per paper; % NA cites: percentage of total citations by North American authors

Table 3 .
Mainstream international business journals ranked by citations per paper.JIF: ISI journal impact factor; CPP: citations per paper.-: journals not ISI-indexed

Table 4 .
Citations to Terry Book Award winners.ISI: Institute of Scientific Information; GS: Google Scholar