Google Scholar: a review of literature examining its effectiveness as a search tool

By Helena Korjonen, PhD
Postdoctoral Researcher, Sustainable Food Practices, University of Luxembourg

Google Scholar (GS) is the top search engine used by those who are looking for scholarly content¹. There are many reasons for this, not least that it feels familiar ², ³. However, a review of the literature reveals that there are significant limitations to its effectiveness.

Lack of curation

The internet is full of ‘scholarly’ content, information, and data in various formats and of various quality. As John Naisbitt said, “We are drowning in information, but starved for knowledge”

When Google Scholar trawls the internet to seek out scholarly content, it does not assess it. It does not try to make sense of it for researchers in a logical indexed structure. And, unlike many trusted curated tools⁴, Google Scholar does not have transparent indexing guidelines that define what they collect, and why they collect it and make it available.

Recent research shows that Google Scholar is “unsuitable as a primary review for resources”⁵. Other research argues that Google Scholar has flaws and lacks technological features to enable systematic searching. As a result, it fails to adhere to several FAIR principles (Findable, Accessible, Interoperable and Reusable) and cannot be classed as a professional searching tool⁶.

Google Scholar is not a curated controlled database. Content is harvested automatically.

Whilst there are some advantages to this, for example in the retrieval of grey literature⁷, there are also significant disadvantages.

Search results rankings

Research has shown that Google Scholar results can be dated. This presents an issue when most people don’t go beyond the first page of results when searching for papers.

The citation count of an article is a major factor in the Google Scholar results ranking⁸,⁹. This benefits publications from high Impact Factor sources, which are cited or linked to extensively around the Internet.

Unfortunately, new papers have what’s called a citation lag, and older papers are cited more often¹⁰. Newly published papers may be retrieved by GS, but they will be further down the results list and you may not see them.

Papers can also be manipulated into a higher rank than it ‘deserves’ by placing words in titles or abstracts to increase recall or manipulate citing data³.

Should you need to include multi-lingual items in your search, there is a definite bias in Google Scholar towards English-language content. Documents published in languages other than English are relegated to positions that make them virtually invisible⁸.

screenshot-of-google-search-on-phone

Replicating your search

It is nearly impossible to replicate a search in GS, to retrieve the same references again. For this reason, it’s recommended to carefully document a search done in Google Scholar for full transparency⁶. That can be extremely time consuming.

References are often dropped unknowingly from GS⁵, ¹¹, ¹² and you won’t even know the ones that might be missing¹.

Using your results

In Google Scholar, once you have the results, it does not allow you to see beyond 1,000 records or to download more than 20 references to your bibliographic tool at a time. Most curated databases allow you to run your search, save it, download the results to your tool and revisit it.

How to search more effectively

IFIS has published a guide called Best Practice for Literature Searching. This provides guidance on where to search; how to formulate research questions and evaluate results. Click here to access the guide

(Image Credit: Kaitlyn Baker and Solen Feyissa at Unsplash)

References

Gardner T, Inger S. How readers discover content: Trends in reader behaviour from 2005 to 2021. Renew Consultants. 2021; 17. Available from: https://renewconsultants.com/wp-content/uploads/2021/07/How-Readers-Discover-Content-2021.pdf
Hemminger BM, Lu D, Vaughan KTL, Adams SJ. Information seeking behavior of academic scientists. Journal of the American Society for Information Science and Technology. 2007; 58(14): 2205-2225. Available from: https://doi.org/10.1002/asi.20686
Georgas H. Google vs. the library: Student preferences and perceptions when doing research using Google and a federated search tool. portal: Libraries and the Academy. 2013; 13(2): 165-185.Available from: https://doi.org/10.1353/pla.2013.0011
Halevi G, Moed H, Bar-Ilan J. "Suitability of Google Scholar as a source of scientific information and as a source of data for scientific evaluation—Review of the literature. ." Journal of Informetrics. 2017; 11(3): 823-834. Available from: https://doi.org/10.1016/j.joi.2017.06.005
Gusenbauer M, Haddaway NR. "Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources." Research Synthesis Methods 2020; 11(2): 181-217. Available from: https://doi.org/10.1002/jrsm.1378
Boeker M, Vach W, Motschall E. Google Scholar as replacement for systematic literature searches: Good relative recall and precision are not enough. BMC Medical Research Methodology. 2013; 13(1):1-2. Available from: https://doi.org/10.1186/1471-2288-13-131
Haddaway NR, Collins AM, Coughlin D, Kirk S. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PloS One. 2015; 17;10(9): e0138237. Available from: https://doi.org/10.1371/journal.pone.0138237
Beel J, Gipp B. Google Scholar's ranking algorithm: The impact of citation counts (an empirical study). Proceedings of the 2009 3rd International Conference on Research Challenges in Information Science, RCIS 2009; 439–446. Available from: http://doi.org/10.1109/RCIS.2009.5089308
Rovira C, Codina L, Guerrero-Solé F, Lopezosa C. Ranking by relevance and citation counts, a comparative study: Google Scholar, Microsoft Academic, WoS and Scopus. Future Internet. 2019 Sep;11(9):202. Available from: https://doi.org/10.3390/fi11090202
Khare R, Leaman R, Lu Z. Accessing biomedical literature in the current information landscape. Biomedical Literature Mining. 2014; 11-31. Available from: https://doi.org/10.1007/978-1-4939-0709-0_2
Nicholas D, Boukacem‐Zeghmouri C, Rodríguez‐Bravo B, Xu J, Watkinson A, Abrizah A, Herman E, Świgoń M. Where and how early career researchers find scholarly information. Learned Publishing. 2017; 30(1): 19-29. Available from: https://doi.org/10.1002/leap.1087.
Jamali HR, Asadi S. Google and the scholar: The role of Google in scientists' information‐seeking behaviour. Online Information Review. 2010 Apr 20.. Available from: https://doi.org/10.1108/14684521011036990.

Research Skills Blog