October 19th, 2020, 8 common problems with literature reviews and how to fix them.
3 comments | 318 shares
Estimated reading time: 5 minutes
Literature reviews are an integral part of the process and communication of scientific research. Whilst systematic reviews have become regarded as the highest standard of evidence synthesis, many literature reviews fall short of these standards and may end up presenting biased or incorrect conclusions. In this post, Neal Haddaway highlights 8 common problems with literature review methods, provides examples for each and provides practical solutions for ways to mitigate them.
Enjoying this blogpost? 📨 Sign up to our mailing list and receive all the latest LSE Impact Blog news direct to your inbox.
Researchers regularly review the literature – it’s an integral part of day-to-day research: finding relevant research, reading and digesting the main findings, summarising across papers, and making conclusions about the evidence base as a whole. However, there is a fundamental difference between brief, narrative approaches to summarising a selection of studies and attempting to reliably and comprehensively summarise an evidence base to support decision-making in policy and practice.
So-called ‘evidence-informed decision-making’ (EIDM) relies on rigorous systematic approaches to synthesising the evidence. Systematic review has become the highest standard of evidence synthesis and is well established in the pipeline from research to practice in the field of health . Systematic reviews must include a suite of specifically designed methods for the conduct and reporting of all synthesis activities (planning, searching, screening, appraising, extracting data, qualitative/quantitative/mixed methods synthesis, writing; e.g. see the Cochrane Handbook ). The method has been widely adapted into other fields, including environment (the Collaboration for Environmental Evidence ) and social policy (the Campbell Collaboration ).
Despite the growing interest in systematic reviews, traditional approaches to reviewing the literature continue to persist in contemporary publications across disciplines. These reviews, some of which are incorrectly referred to as ‘systematic’ reviews, may be susceptible to bias and as a result, may end up providing incorrect conclusions. This is of particular concern when reviews address key policy- and practice- relevant questions, such as the ongoing COVID-19 pandemic or climate change.
These limitations with traditional literature review approaches could be improved relatively easily with a few key procedures; some of them not prohibitively costly in terms of skill, time or resources.
In our recent paper in Nature Ecology and Evolution , we highlight 8 common problems with traditional literature review methods, provide examples for each from the field of environmental management and ecology, and provide practical solutions for ways to mitigate them.
Problem | Solution |
---|---|
Lack of relevance – limited stakeholder engagement can produce a review that is of limited practical use to decision-makers | Stakeholders can be identified, mapped and contacted for feedback and inclusion without the need for extensive budgets – check out best-practice guidance |
Mission creep – reviews that don’t publish their methods in an a priori protocol can suffer from shifting goals and inclusion criteria | Carefully design and publish an a priori protocol that outlines planned methods for searching, screening, data extraction, critical appraisal and synthesis in detail. Make use of existing organisations to support you (e.g. the Collaboration for Environmental Evidence). |
A lack of transparency/replicability in the review methods may mean that the review cannot be replicated – a central tenet of the scientific method! | Be explicit, and make use of high-quality guidance and standards for review conduct (e.g. CEE Guidance) and reporting (PRISMA or ROSES) |
Selection bias (where included studies are not representative of the evidence base) and a lack of comprehensiveness (an inappropriate search method) can mean that reviews end up with the wrong evidence for the question at hand | Carefully design a search strategy with an info specialist; trial the search strategy (against a benchmark list); use multiple bibliographic databases/languages/sources of grey literature; publish search methods in an a priori protocol for peer-review |
The exclusion of grey literature and failure to test for evidence of publication bias can result in incorrect or misleading conclusions | Include attempts to find grey literature, including both ‘file-drawer’ (unpublished academic) research and organisational reports. Test for possible evidence of publication bias. |
Traditional reviews often lack appropriate critical appraisal of included study validity, treating all evidence as equally valid – we know some research is more valid and we need to account for this in the synthesis. | Carefully plan and trial a critical appraisal tool before starting the process in full, learning from existing robust critical appraisal tools. |
Inappropriate synthesis (e.g. using vote-counting and inappropriate statistics) can negate all of the preceding systematic effort. Vote-counting (tallying studies based on their statistical significance) ignores study validity and magnitude of effect sizes. | Select the synthesis method carefully based on the data analysed. Vote-counting should never be used instead of meta-analysis. Formal methods for narrative synthesis should be used to summarise and describe the evidence base. |
There is a lack of awareness and appreciation of the methods needed to ensure systematic reviews are as free from bias and as reliable as possible: demonstrated by recent, flawed, high-profile reviews. We call on review authors to conduct more rigorous reviews, on editors and peer-reviewers to gate-keep more strictly, and the community of methodologists to better support the broader research community. Only by working together can we build and maintain a strong system of rigorous, evidence-informed decision-making in conservation and environmental management.
Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below
Image credit: Jaeyoung Geoffrey Kang via unsplash
Neal Haddaway is a Senior Research Fellow at the Stockholm Environment Institute, a Humboldt Research Fellow at the Mercator Research Institute on Global Commons and Climate Change, and a Research Associate at the Africa Centre for Evidence. He researches evidence synthesis methodology and conducts systematic reviews and maps in the field of sustainability and environmental science. His main research interests focus on improving the transparency, efficiency and reliability of evidence synthesis as a methodology and supporting evidence synthesis in resource constrained contexts. He co-founded and coordinates the Evidence Synthesis Hackathon (www.eshackathon.org) and is the leader of the Collaboration for Environmental Evidence centre at SEI. @nealhaddaway
Why is mission creep a problem and not a legitimate response to an unexpected finding in the literature? Surely the crucial points are that the review’s scope is stated clearly and implemented rigorously, not when the scope was finalised.
#9. Most of them are terribly boring. Which is why I teach students how to make them engaging…and useful.
Your email address will not be published. Required fields are marked *
Notify me of follow-up comments by email.
November 16th, 2020.
January 18th, 2017.
March 29th, 2021.
February 8th, 2024.
Visit our sister blog LSE Review of Books
Why are literature reviews necessary.
Your research is seen as a contribution to knowledge in the field and it needs to indicate, therefore, that there is an awareness of what that knowledge comprises. The reasons for literature reviews are:
Literature reviews depend upon extensive literature searching. This is:
Literature reviews do not usually appear as a separate chapter as such but form a considerable part of the theoretical perspectives chapter (often your second chapter) because they indicate previous work in the field, the context into which your own work fits and the different theories from which your own work springs, which inform it. You will be reading in areas of the field, of related fields, of critical and theoretical questioning and approaches to properly inform and drive your own work.
While the literature review you do is largely written up in the introduction, you continue to refer to key themes, texts, writers and experts as and when their work informs and relates to yours throughout the thesis or dissertation.
If you are an international student you will most probably find that you need either to seek a translation of the work of international theorists, critics or experts you hope to use in your own arguments and research, or to provide your own translation of the quotations you use if they do not publish in English.
The literature review is an essential part of planning your research and helps you to develop your own line of thought. As an ongoing process it also helps you to keep abreast of developments in your subject and field and possibly enables you to get in touch with others working in the same field (you can contact other researchers and discuss work with them). Your examiners will be looking for how far your thesis contributes to knowledge in the field, to which the literature review element of your work is central.
There are several activities associated with handling a literature review.
Every researcher needs to become familiar with the use of the libraries available to them - not always your local library but often a specialist library perhaps at a distance - and to make good use of the information available on the Internet also. Using email and the web to keep in touch with other researchers and your supervisor is important.
Trawling for information is fascinating when you know how - and many of us do this every day. However, there is often too much information, and on the Internet it is not likely to be organised in the way you need it, so be careful with copying it. Do manage it, organise and sift it. There is a real concern with students at all levels merely downloading topical material from the net. This is plagiarism as serious as merely copying from a book. The other problem with material on the net is that it is put on the net without any quality control checks and some of it is incorrect and poorly written. You have been warned!
For further information see Chapter 5 of The Postgraduate Research Handbook by Gina Wisker.
For further information see Chapter 12 of The Postgraduate Research Handbook by Gina Wisker
Bryan Greetham
This engaging guide by bestselling author Bryan Greetham takes students step-by-step through the process of writing a literature review, and...
Find out more
Martin Davies
The definitive guide to successful study as an international postgraduate student. Chapters cover all the core academic skills, including time...
COMMENTS
Reason #3: Setting a Theoretical Framework. It can help to think of the literature review as the foundations for your study, since the rest of your work will build upon the ideas and existing research you discuss therein. A crucial part of this is formulating a theoretical framework, which comprises the concepts and theories that your work is ...
"A substantive, thorough, sophisticated literature review is a precondition for doing substantive, thorough, sophisticated research". Boote and Baile 2005 . Authors of manuscripts treat writing a literature review as a routine work or a mere formality. But a seasoned one knows the purpose and importance of a well-written literature review.
Besides the obvious reason for students -- because it is assigned! -- a literature review helps you explore the research that has come before you, to see how your research question has (or has not) already been addressed. You identify: core research in the field. experts in the subject area. methodology you may want to use (or avoid)
Example: Predictors and Outcomes of U.S. Quality Maternity Leave: A Review and Conceptual Framework: 10.1177/08948453211037398 ; Systematic review: "The authors of a systematic review use a specific procedure to search the research literature, select the studies to include in their review, and critically evaluate the studies they find." (p. 139).
Examples of literature reviews. Step 1 - Search for relevant literature. Step 2 - Evaluate and select sources. Step 3 - Identify themes, debates, and gaps. Step 4 - Outline your literature review's structure. Step 5 - Write your literature review.
A literature review is an integrated analysis-- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question. That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question ...
A formal literature review is an evidence-based, in-depth analysis of a subject. There are many reasons for writing one and these will influence the length and style of your review, but in essence a literature review is a critical appraisal of the current collective knowledge on a subject. Rather than just being an exhaustive list of all that ...
Literature reviews establish the foundation of academic inquires. However, in the planning field, we lack rigorous systematic reviews. In this article, through a systematic search on the methodology of literature review, we categorize a typology of literature reviews, discuss steps in conducting a systematic literature review, and provide suggestions on how to enhance rigor in literature ...
The purpose of a literature review is to: Provide a foundation of knowledge on a topic; Identify areas of prior scholarship to prevent duplication and give credit to other researchers; Identify inconstancies: gaps in research, conflicts in previous studies, open questions left from other research;
There are several reasons to conduct a literature review at the beginning of a research project: To familiarize yourself with the current state of knowledge on your topic. To ensure that you're not just repeating what others have already done. To identify gaps in knowledge and unresolved problems that your research can address.
Important aspects of a systematic literature review (SLR) include a structured method for conducting the study and significant transparency of the approaches used for summarizing the literature (Hiebl, 2023).The inspection of existing scientific literature is a valuable tool for (a) developing best practices and (b) resolving issues or controversies over a single study (Gupta et al., 2018).
While there might be many reasons for conducting a literature review, following are four key outcomes of doing the review. Assessment of the current state of research on a topic. This is probably the most obvious value of the literature review. Once a researcher has determined an area to work with for a research project, a search of relevant ...
INTRODUCTION. Writing the literature review (LR) is often viewed as a difficult task that can be a point of writer's block and procrastination in postgraduate life.Disagreements on the definitions or classifications of LRs may confuse students about their purpose and scope, as well as how to perform an LR.Interestingly, at many universities, the LR is still an important element in any ...
Literature reviews are in great demand in most scientific fields. Their need stems from the ever-increasing output of scientific publications .For example, compared to 1991, in 2008 three, eight, and forty times more papers were indexed in Web of Science on malaria, obesity, and biodiversity, respectively .Given such mountains of papers, scientists cannot be expected to examine in detail every ...
A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories.A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that ...
9.3. Types of Review Articles and Brief Illustrations. EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic.
The first step of any research project is to review the field. So let's think about surveying, synthesising, critically analysing and presenting in more detail. A literature review does the following. Identifies gaps in current knowledge. Avoids reinventing the wheel - i.e. it saves you wasting time researching something that's already ...
Systematic Approaches to a Successful Literature Review by Andrew Booth; Anthea Sutton; Diana Papaioannou Showing you how to take a structured and organized approach to a wide range of literature review types, this book helps you to choose which approach is right for your research. Packed with constructive tools, examples, case studies and hands-on exercises, the book covers the full range of ...
A literature review is "the systematic examination of the (…) literature about one's topic. It critically analyses, evaluates, and synthesizes research findings, theories, and practices by scholars and researchers that are related to an area of focus" (Efron & Ravid, 2019, p. 2). Knowledge in your field can appear as different types of ...
Systematic reviews provide a synthesis of evidence for a specific topic of interest, summarising the results of multiple studies to aid in clinical decisions and resource allocation. They remain among the best forms of evidence, and reduce the bias inherent in other methods. A solid understanding of the systematic review process can be of ...
Importance of a Literature Review. There are various reasons for carrying out literature review. Majorly, literature review helps in: Assessing the current state and level of research on a given topic; Identifying experts related to particular research; Identifying questions that need further research and exploration
A literature review helps you create a sense of rapport with your audience or readers so they can trust that you have done your homework. As a result, they can give you credit for your due diligence: you have done your fact-finding and fact-checking mission, one of the initial steps of any research writing.
Some Issues in Liter ature R eview. 1. A continuous and time consuming process runs. through out r esearch work (more whil e selecting. a resear ch problem and writing 'r eview of. liter ature ...
In our recent paper in Nature Ecology and Evolution, we highlight 8 common problems with traditional literature review methods, provide examples for each from the field of environmental management and ecology, and provide practical solutions for ways to mitigate them. Problem. Solution. Lack of relevance - limited stakeholder engagement can ...
Carrying out the literature review There are several activities associated with handling a literature review. You need to scour the library and associated libraries, probably using a computer to help in your search, but not substituting it completely for looking around the shelves in the area where you find a useful book.