Research Skills

Google Scholar: a review of literature examining its effectiveness as a search tool

hands-typing-on-a-laptop

By Helena Korjonen, PhD
Postdoctoral Researcher, Sustainable Food Practices, University of Luxembourg

Google Scholar (GS) is the top search engine used by those who are looking for scholarly content1. There are many reasons for this, not least that it feels familiar 2, 3. However, a review of the literature reveals that there are significant limitations to its effectiveness.  

Lack of curation 

The internet is full of ‘scholarly’ content, information, and data in various formats and of various quality. As John Naisbitt said, “We are drowning in information, but starved for knowledge”  

When Google Scholar trawls the internet to seek out scholarly content, it does not assess it. It does not try to make sense of it for researchers in a logical indexed structure. And, unlike many trusted curated tools4, Google Scholar does not have transparent indexing guidelines that define what they collect, and why they collect it and make it available.  

Recent research shows that Google Scholar is “unsuitable as a primary review for resources”5. Other research argues that Google Scholar has flaws and lacks technological features to enable systematic searching. As a result, it fails to adhere to several FAIR principles (Findable, Accessible, Interoperable and Reusable) and cannot be classed as a professional searching tool6.   

Google Scholar is not a curated controlled database. Content is harvested automatically.  

Whilst there are some advantages to this, for example in the retrieval of grey literature7, there are also significant disadvantages. 

Search results rankings 

Research has shown that Google Scholar results can be dated. This presents an issue when most people don’t go beyond the first page of results when searching for papers.   

The citation count of an article is a major factor in the Google Scholar results ranking8, 9. This benefits publications from high Impact Factor sources, which are cited or linked to extensively around the Internet.   

Unfortunately, new papers have what’s called a citation lag, and older papers are cited more often10. Newly published papers may be retrieved by GS, but they will be further down the results list and you may not see them.   

Papers can also be manipulated into a higher rank than it ‘deserves’ by placing words in titles or abstracts to increase recall or manipulate citing data3.   

Should you need to include multi-lingual items in your search, there is a definite bias in Google Scholar towards English-language content. Documents published in languages other than English are relegated to positions that make them virtually invisible8. 

screenshot-of-google-search-on-phone

Replicating your search 

It is nearly impossible to replicate a search in GS, to retrieve the same references again. For this reason, it’s recommended to carefully document a search done in Google Scholar for full transparency6.  That can be extremely time consuming.  

References are often dropped unknowingly from GS5, 11, 12 and you won’t even know the ones that might be missing1 

Using your results 

In Google Scholar, once you have the results, it does not allow you to see beyond 1,000 records or to download more than 20 references to your bibliographic tool at a time. Most curated databases allow you to run your search, save it, download the results to your tool and revisit it.  

How to search more effectively 

IFIS has published a guide called Best Practice for Literature Searching. This provides guidance on where to search; how to formulate research questions and evaluate results. Click here to access the guide

(Image Credit: Kaitlyn Baker and Solen Feyissa at Unsplash)

Do you know which search tool to use and when? It is a common mistake to use search tools designed for information access for the task of information discovery, but doing this means you are doing your searching backwards! Find out more in our Guide for Effective Literature Searching


References

  1. Gardner T, Inger S. How readers discover content: Trends in reader behaviour from 2005 to 2021. Renew Consultants. 2021; 17. Available from: https://renewconsultants.com/wp-content/uploads/2021/07/How-Readers-Discover-Content-2021.pdf
  2. Hemminger BM, Lu D, Vaughan KTL, Adams SJ. Information seeking behavior of academic scientists. Journal of the American Society for Information Science and Technology. 2007; 58(14): 2205-2225. Available from: https://doi.org/10.1002/asi.20686
  3. Georgas H. Google vs. the library: Student preferences and perceptions when doing research using Google and a federated search tool. portal: Libraries and the Academy. 2013; 13(2): 165-185.Available from: https://doi.org/10.1353/pla.2013.0011
  4. Halevi G, Moed H, Bar-Ilan J. "Suitability of Google Scholar as a source of scientific information and as a source of data for scientific evaluation—Review of the literature. ." Journal of Informetrics. 2017; 11(3): 823-834. Available from: https://doi.org/10.1016/j.joi.2017.06.005
  5. Gusenbauer M, Haddaway NR. "Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources." Research Synthesis Methods 2020; 11(2): 181-217. Available from: https://doi.org/10.1002/jrsm.1378 
  6. Boeker M, Vach W, Motschall E. Google Scholar as replacement for systematic literature searches: Good relative recall and precision are not enough. BMC Medical Research Methodology. 2013; 13(1):1-2. Available from: https://doi.org/10.1186/1471-2288-13-131 
  7. Haddaway NR, Collins AM, Coughlin D, Kirk S. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PloS One. 2015; 17;10(9): e0138237. Available from: https://doi.org/10.1371/journal.pone.0138237 
  8. Beel J, Gipp B. Google Scholar's ranking algorithm: The impact of citation counts (an empirical study). Proceedings of the 2009 3rd International Conference on Research Challenges in Information Science, RCIS 2009; 439–446. Available from: http://doi.org/10.1109/RCIS.2009.5089308 
  9. Rovira C, Codina L, Guerrero-Solé F, Lopezosa C. Ranking by relevance and citation counts, a comparative study: Google Scholar, Microsoft Academic, WoS and Scopus. Future Internet. 2019 Sep;11(9):202. Available from: https://doi.org/10.3390/fi11090202
  10. Khare R, Leaman R, Lu Z. Accessing biomedical literature in the current information landscape. Biomedical Literature Mining. 2014; 11-31. Available from: https://doi.org/10.1007/978-1-4939-0709-0_2
  11. Nicholas D, Boukacem‐Zeghmouri C, Rodríguez‐Bravo B, Xu J, Watkinson A, Abrizah A, Herman E, Świgoń M. Where and how early career researchers find scholarly information. Learned Publishing. 2017; 30(1): 19-29. Available from: https://doi.org/10.1002/leap.1087.
  12. Jamali HR, Asadi S. Google and the scholar: The role of Google in scientists' information‐seeking behaviour. Online Information Review. 2010 Apr 20.. Available from: https://doi.org/10.1108/14684521011036990.

 


Subscribe to our Research Skills blog