While the systematic review methodology is known to be a powerful tool in evidence synthesis, many studies which use the method have been found to be poorly conducted, in recent scientific literature. Food science experts have raised their concerns about the increasing number of poor-quality systematic reviews and their impact on future food research. This post looks at some of the most reported issues seen in systematic reviews related to food topics, by analysing a sample of relevant systematic review articles. We've put together this information to increase awareness of current quality issues among the food research community, and to help researchers ensure they avoid such problems.
Current quality issues related to poor methods of conduct
Systematic review methodology is increasingly used for reassessment of the existing body of knowledge in different areas of food research. A search for systematic reviews in the Web of Science Core Collection for food science related articles showed the number has increased from only one in 2000 to over 850 articles by January 2020. A sample of more than 200 articles, under the category for food science and technology were selected and articles were scanned and analysed for issues related to conduct and how frequently they occurred or reported. This experiment looked for issues related to use of resources, data collection and reporting.
This graph shows the increasing number of systematic review studies in food science and technology over the last two decades. The numbers have increased in a faster pace for the last five years, with the figures for 2020 standing three times higher than the numbers in 2016.
Systematic review method dominantly used within the context of human health
The application of the method tends to vary considerably in different areas of food research, with some areas using the method more consistently than others. Reviewed articles were grouped into eight subject areas: dietary interventions, animals in agriculture, food microbiology, food safety, food security, consumer science, veterinary medicine, and environmental health. The most frequent research questions were assessing effectiveness of interventions and food safety risks.
Limited number of databases used for literature searching
Number and types of databases used for literature searching varied significantly across different subject areas. However, health related research questions followed a more consistent pattern in the use of databases. Surprisingly, about 15% of the 200 reviewed articles only used one or two databases for searching, with most individuals only using multidisciplinary platforms like Web of Science or free-to-access databases like PubMed and Google Scholar. Subject-specific databases of food-related topics were only reported in certain subject areas, including agriculture and nutrition.
Following guidelines, tools and frameworks was only reported in certain topics
PRISMA was the most frequently used framework, and its use was limited to studies assessing effectiveness of interventions. Systematic review studies in other subject areas including food technology and agriculture related topics did not refer to or state a standard framework but judging from their methodology, the majority followed the core steps of the process as their workflow strategy. A small number used a conceptual framework or reported using a quality assessment tool or risk of bias tool.
A small number of systematic reviews within certain topics reported they assessed risk of bias
Most studies did not report that they have assessed literature for risk of bias and almost all of those that did also reported various levels of bias, from weak to significant degrees. About 15% of the studies reported bias, including publication bias or that they have used a tool for assessing risk of bias. Lack of data and inconsistent or poor reporting appeared across all subject areas. It is worth noting that some of the recorded issues were specific to individual studies which might be only answered by the expertise of reviewers in developing well-defined study plans and protocols.
So how can we restore the power of the systematic review methodology?
In conclusion, it is evident that the quality of systematic reviews is impacted by poor methods of conduct and reviewers should be made aware of the cost of such practices for their time and efforts. More importantly, to support reviewers of different topic areas, relevant guidelines are needed to fill the gaps in knowledge and current practice. Ultimately, the effectiveness of the systematic review methodology relies on reviewers' adherence to the principles of comprehensiveness, transparency and reproducibility in gathering, documenting and assessing evidence.
IFIS are currently writing a guide on the use of systematic review methodology for food related topics. The purpose of this resource will be to provide researchers conducting systematic reviews of food related topics with guidance on good review practices in different topic areas which are likely to improve their methods of conduct and help to produce high quality systematic reviews.
To receive updates about this guide, along with other upcoming resources from IFIS, subscribe to our monthly newsletter here.
References (available in FSTA)
- Lichtenstein, A. H., Yetley, E. A., & Lau, J. (2008). Application of Systematic Review Methodology to the Field of Nutrition. The Journal of Nutrition, 138(12), 2297–2306.
- Kelley, G. A., & Kelley, K. S. (2019). Systematic reviews and meta-analysis in nutrition research. British Journal of Nutrition, 122(11), 1279–1294.
- Kroeger, C. M., Garza, C., Lynch, C. J., Myers, E., Rowe, S., Schneeman, B. O., Sharma, A. M., & Allison, D. B. (2018). Scientific rigor and credibility in the nutrition research landscape. American Journal of Clinical Nutrition, 107(3), 484–494.
- Perez-Cueto, F. J. A. (2019). An umbrella review of systematic reviews on food choice and nutrition published between 2017 and-2019. Nutrients, 11(10).
Ioannidis JPA (2016) The mass production of redundant, misleading, and conflicted systematic reviews and meta- analyses. Milbank Q 94, 485–514.