critical appraisal of literature reviews

Conducting a Literature Review

• Literature Review
• Developing a Topic
• Planning Your Literature Review
• Developing a Search Strategy
• Managing Citations
• Critical Appraisal Tools
• Writing a Literature Review

Appraise Your Research Articles

The structure of a literature review should include the following :

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

• Provenance  -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology  -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity  -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness  -- which of the author's theses are most convincing or least convincing?
• Value  -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

Reviewing the Literature

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what the articles are saying, but how are they saying it.

Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?
• When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Tools for Critical Appraisal

Now, that you have found articles based on your research question you can appraise the quality of those articles. These are resources you can use to appraise different study designs.

Centre for Evidence Based Medicine (Oxford)

University of Glasgow

"AFP uses the Strength-of-Recommendation Taxonomy (SORT), to label key recommendations in clinical review articles."

• SORT: Rating the Strength of Evidence    American Family Physician and other family medicine journals use the Strength of Recommendation Taxonomy (SORT) system for rating bodies of evidence for key clinical recommendations.

• The Interprofessional Health Sciences Library
• 123 Metro Boulevard
• Nutley, NJ 07110
• [email protected]
• Visiting Campus
• News and Events
• Parents and Families
• Web Accessibility
• Career Center
• Public Safety
• Accountability
• Privacy Statements
• Report a Problem
• Login to LibApps

• Mayo Clinic Libraries
• Systematic Reviews
• Critical Appraisal by Study Design

Systematic Reviews: Critical Appraisal by Study Design

• Knowledge Synthesis Comparison
• Knowledge Synthesis Decision Tree
• Standards & Reporting Results
• Materials in the Mayo Clinic Libraries
• Training Resources
• Review Teams
• Develop & Refine Your Research Question
• Develop a Timeline
• Project Management
• Communication
• PRISMA-P Checklist
• Eligibility Criteria
• Register your Protocol
• Other Resources
• Other Screening Tools
• Grey Literature Searching
• Citation Searching
• Data Extraction Tools
• Minimize Bias
• Synthesis & Meta-Analysis
• Publishing your Systematic Review

Tools for Critical Appraisal of Studies

“The purpose of critical appraisal is to determine the scientific merit of a research report and its applicability to clinical decision making.” 1 Conducting a critical appraisal of a study is imperative to any well executed evidence review, but the process can be time consuming and difficult. 2 The critical appraisal process requires “a methodological approach coupled with the right tools and skills to match these methods is essential for finding meaningful results.” 3 In short, it is a method of differentiating good research from bad research.

Critical Appraisal by Study Design (featured tools)

• Non-RCTs or Observational Studies
• Diagnostic Accuracy
• Animal Studies
• Qualitative Research
• Tool Repository
• AMSTAR 2 The original AMSTAR was developed to assess the risk of bias in systematic reviews that included only randomized controlled trials. AMSTAR 2 was published in 2017 and allows researchers to “identify high quality systematic reviews, including those based on non-randomised studies of healthcare interventions.” 4 more... less... AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews)
• ROBIS ROBIS is a tool designed specifically to assess the risk of bias in systematic reviews. “The tool is completed in three phases: (1) assess relevance(optional), (2) identify concerns with the review process, and (3) judge risk of bias in the review. Signaling questions are included to help assess specific concerns about potential biases with the review.” 5 more... less... ROBIS (Risk of Bias in Systematic Reviews)
• BMJ Framework for Assessing Systematic Reviews This framework provides a checklist that is used to evaluate the quality of a systematic review.
• CASP Checklist for Systematic Reviews This CASP checklist is not a scoring system, but rather a method of appraising systematic reviews by considering: 1. Are the results of the study valid? 2. What are the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Systematic Reviews Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• JBI Critical Appraisal Tools, Checklist for Systematic Reviews JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• NHLBI Study Quality Assessment of Systematic Reviews and Meta-Analyses The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• RoB 2 RoB 2 “provides a framework for assessing the risk of bias in a single estimate of an intervention effect reported from a randomized trial,” rather than the entire trial. 6 more... less... RoB 2 (revised tool to assess Risk of Bias in randomized trials)
• CASP Randomised Controlled Trials Checklist This CASP checklist considers various aspects of an RCT that require critical appraisal: 1. Is the basic study design valid for a randomized controlled trial? 2. Was the study methodologically sound? 3. What are the results? 4. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CONSORT Statement The CONSORT checklist includes 25 items to determine the quality of randomized controlled trials. “Critical appraisal of the quality of clinical trials is possible only if the design, conduct, and analysis of RCTs are thoroughly and accurately described in the report.” 7 more... less... CONSORT (Consolidated Standards of Reporting Trials)
• NHLBI Study Quality Assessment of Controlled Intervention Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• JBI Critical Appraisal Tools Checklist for Randomized Controlled Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• ROBINS-I ROBINS-I is a “tool for evaluating risk of bias in estimates of the comparative effectiveness… of interventions from studies that did not use randomization to allocate units… to comparison groups.” 8 more... less... ROBINS-I (Risk Of Bias in Non-randomized Studies – of Interventions)
• NOS This tool is used primarily to evaluate and appraise case-control or cohort studies. more... less... NOS (Newcastle-Ottawa Scale)
• AXIS Cross-sectional studies are frequently used as an evidence base for diagnostic testing, risk factors for disease, and prevalence studies. “The AXIS tool focuses mainly on the presented [study] methods and results.” 9 more... less... AXIS (Appraisal tool for Cross-Sectional Studies)
• NHLBI Study Quality Assessment Tools for Non-Randomized Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. • Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies • Quality Assessment of Case-Control Studies • Quality Assessment Tool for Before-After (Pre-Post) Studies With No Control Group • Quality Assessment Tool for Case Series Studies more... less... NHLBI (National Heart, Lung, and Blood Institute)
• Case Series Studies Quality Appraisal Checklist Developed by the Institute of Health Economics (Canada), the checklist is comprised of 20 questions to assess “the robustness of the evidence of uncontrolled, [case series] studies.” 10
• Methodological Quality and Synthesis of Case Series and Case Reports In this paper, Dr. Murad and colleagues “present a framework for appraisal, synthesis and application of evidence derived from case reports and case series.” 11
• MINORS The MINORS instrument contains 12 items and was developed for evaluating the quality of observational or non-randomized studies. 12 This tool may be of particular interest to researchers who would like to critically appraise surgical studies. more... less... MINORS (Methodological Index for Non-Randomized Studies)
• JBI Critical Appraisal Tools for Non-Randomized Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis. • Checklist for Analytical Cross Sectional Studies • Checklist for Case Control Studies • Checklist for Case Reports • Checklist for Case Series • Checklist for Cohort Studies
• QUADAS-2 The QUADAS-2 tool “is designed to assess the quality of primary diagnostic accuracy studies… [it] consists of 4 key domains that discuss patient selection, index test, reference standard, and flow of patients through the study and timing of the index tests and reference standard.” 13 more... less... QUADAS-2 (a revised tool for the Quality Assessment of Diagnostic Accuracy Studies)
• JBI Critical Appraisal Tools Checklist for Diagnostic Test Accuracy Studies JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• STARD 2015 The authors of the standards note that “[e]ssential elements of [diagnostic accuracy] study methods are often poorly described and sometimes completely omitted, making both critical appraisal and replication difficult, if not impossible.”10 The Standards for the Reporting of Diagnostic Accuracy Studies was developed “to help… improve completeness and transparency in reporting of diagnostic accuracy studies.” 14 more... less... STARD 2015 (Standards for the Reporting of Diagnostic Accuracy Studies)
• CASP Diagnostic Study Checklist This CASP checklist considers various aspects of diagnostic test studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Diagnostic Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• SYRCLE’s RoB “[I]mplementation of [SYRCLE’s RoB tool] will facilitate and improve critical appraisal of evidence from animal studies. This may… enhance the efficiency of translating animal research into clinical practice and increase awareness of the necessity of improving the methodological quality of animal studies.” 15 more... less... SYRCLE’s RoB (SYstematic Review Center for Laboratory animal Experimentation’s Risk of Bias)
• ARRIVE 2.0 “The [ARRIVE 2.0] guidelines are a checklist of information to include in a manuscript to ensure that publications [on in vivo animal studies] contain enough information to add to the knowledge base.” 16 more... less... ARRIVE 2.0 (Animal Research: Reporting of In Vivo Experiments)
• Critical Appraisal of Studies Using Laboratory Animal Models This article provides “an approach to critically appraising papers based on the results of laboratory animal experiments,” and discusses various “bias domains” in the literature that critical appraisal can identify. 17
• CEBM Critical Appraisal of Qualitative Studies Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• CASP Qualitative Studies Checklist This CASP checklist considers various aspects of qualitative research studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• Quality Assessment and Risk of Bias Tool Repository Created by librarians at Duke University, this extensive listing contains over 100 commonly used risk of bias tools that may be sorted by study type.
• Latitudes Network A library of risk of bias tools for use in evidence syntheses that provides selection help and training videos.

References & Recommended Reading

1.     Kolaski, K., Logan, L. R., & Ioannidis, J. P. (2024). Guidance to best tools and practices for systematic reviews .  British Journal of Pharmacology ,  181 (1), 180-210

2.    Portney LG.  Foundations of clinical research : applications to evidence-based practice.  Fourth edition. ed. Philadelphia: F A Davis; 2020.

3.     Fowkes FG, Fulton PM.  Critical appraisal of published research: introductory guidelines.   BMJ (Clinical research ed).  1991;302(6785):1136-1140.

4.     Singh S.  Critical appraisal skills programme.   Journal of Pharmacology and Pharmacotherapeutics.  2013;4(1):76-77.

5.     Shea BJ, Reeves BC, Wells G, et al.  AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.   BMJ (Clinical research ed).  2017;358:j4008.

6.     Whiting P, Savovic J, Higgins JPT, et al.  ROBIS: A new tool to assess risk of bias in systematic reviews was developed.   Journal of clinical epidemiology.  2016;69:225-234.

7.     Sterne JAC, Savovic J, Page MJ, et al.  RoB 2: a revised tool for assessing risk of bias in randomised trials.  BMJ (Clinical research ed).  2019;366:l4898.

8.     Moher D, Hopewell S, Schulz KF, et al.  CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials.  Journal of clinical epidemiology.  2010;63(8):e1-37.

9.     Sterne JA, Hernan MA, Reeves BC, et al.  ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.  BMJ (Clinical research ed).  2016;355:i4919.

10.     Downes MJ, Brennan ML, Williams HC, Dean RS.  Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS).   BMJ open.  2016;6(12):e011458.

11.   Guo B, Moga C, Harstall C, Schopflocher D.  A principal component analysis is conducted for a case series quality appraisal checklist.   Journal of clinical epidemiology.  2016;69:199-207.e192.

12.   Murad MH, Sultan S, Haffar S, Bazerbachi F.  Methodological quality and synthesis of case series and case reports.  BMJ evidence-based medicine.  2018;23(2):60-63.

13.   Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y, Chipponi J.  Methodological index for non-randomized studies (MINORS): development and validation of a new instrument.   ANZ journal of surgery.  2003;73(9):712-716.

14.   Whiting PF, Rutjes AWS, Westwood ME, et al.  QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.   Annals of internal medicine.  2011;155(8):529-536.

15.   Bossuyt PM, Reitsma JB, Bruns DE, et al.  STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies.   BMJ (Clinical research ed).  2015;351:h5527.

16.   Hooijmans CR, Rovers MM, de Vries RBM, Leenaars M, Ritskes-Hoitinga M, Langendam MW.  SYRCLE's risk of bias tool for animal studies.   BMC medical research methodology.  2014;14:43.

17.   Percie du Sert N, Ahluwalia A, Alam S, et al.  Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0.  PLoS biology.  2020;18(7):e3000411.

18.   O'Connor AM, Sargeant JM.  Critical appraisal of studies using laboratory animal models.   ILAR journal.  2014;55(3):405-417.

• << Previous: Minimize Bias
• Next: GRADE >>
• Last Updated: Jun 14, 2024 12:57 PM
• URL: https://libraryguides.mayo.edu/systematicreviewprocess

Kelvin Smith Library

Systematic Reviews

• Introduction
• Formulating a Research Question
• Inclusion and Exclusion Criteria
• Search Strategy
• Grey Literature
• Exporting Your Results and Screening Articles
• Critical Appraisal
• Synthesis and Analysis
• More Research Guides

Critically Appraising the Literature

Much like the question formulation, search, and selection processes, the process of evaluating the quality of the literature in a systematic review should be structured. In the critical appraisal stage, the researchers analyze the design of each included study, the validity of the results in light of the design, potential for bias, and the relevance of each study to the larger literature and the research question. There are tools that researchers can use to help them approach this process systematically and objectively. 

• Critical Appraisals Skills Programme (CASP) CASP is a UK-based health research organization that has made a set of tools for critical appraisal freely avialable
• Center for Evidence-Based Medicine (CEBM) Oxford's CEBM has produced a number of worksheets and guidelines for critical appraisal.
• Center for Evidence-Based Management (CEBMa)a The CEBa provides a number of checklists for critical appraisal, as well as a mobile app.
• Strengthening the Reporting of Observational Studies in Epidemiology (Strobe) STROBE, an international organization of epidemiology researchers, has produced checklists for systematic reviews as well.
• Newcastle-Ottawa Scale These guidelines for appraising non-randomized studies was developed by researchers in the University of Ottawa's Epidemiology and Community Medicine Department.
• PRISMA PRISMA, mentioned earlier in this guide, has helpful guidelines for critical appraisal as well.
• QUADAS-2 QUADAS-2 is a critical appraisal tool developed at the University of Bristol.
• Grading of Recommendations Assessment, Development and Evaluation (GRADE) GRADE, a working group of health care researchers, has developed a guide to critical appraisal that they believe is more comprehensive and holistic than some of the checklists available.
• Students 4 Best Evidence Students 4 Best Evidence, a student-run organization, has produced guidelines for systematic reviews as well.
• Understanding Health Research This step-by-step tutorial on critical appraisal was developed by the Social and Public Health Sciences Unit at the University of Glasgow.
• Revised Cochrane Risk of Bias Tool (ROB 2.0) This tool, developed by Cochrane Reviews, is useful for assessing bias in randomly controlled studies. The 2.0 version was released October 2018.
• Joanna Briggs Institute Critical Appraisal Tools The Joanna Briggs Institute offers a suite of critical appraisal tools.

More Resources

• How To Read A Paper, by Trisha Greenbalgh This book explores the theory of critical appraisal in great detail. CWRU holds it in its collection, both in print and digital form.
• How to Critically Appraise an Article This article by Jane M. Young and Michael J. Solomon discusses critical appraisal theory and offers suggestions for researchers.

These are key issues to consider when appraising various types of medical studies. 

Key Issues in Appraising Therapy Studies:

• Randomization and concealed allocation
• Follow-up of all patients (ideally 80% or better)
• Blinding (concealment) of patients, clinicians and study personnel to the treatment being provided
• Intention to Treat analysis
• Baseline similarities between groups (established at the start of the trial)

Key Issues in Appraising Diagnostic Studies:

• Independent blind comparison with a gold standard
• Appropriate spectrum of patients
• All patients receive both tests
• Key Issues in Appraising Prognosis Studies:
• Well-defined sample of patients
• Similar prognostic factors
• Objective outcome criteria

Key Issues for Etiology/Harm Studies:

• Similarity of comparison groups
• Outcomes and exposure measured the same for both groups
• Follow-up of sufficient length
• << Previous: Exporting Your Results and Screening Articles
• Next: Synthesis and Analysis >>
• Last Updated: Mar 1, 2024 3:57 PM
• URL: https://researchguides.case.edu/systematicreviews

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Critical Appraisal of a Systematic Review: A Concise Review

Affiliations.

• 1 Division of Pulmonary and Critical Care Medicine, Department of Medicine, Medical College of Wisconsin, Milwaukee, WI.
• 2 Department of Anesthesiology, University Hospital RWTH Aachen University, Aachen, Germany.
• 3 Department of Intensive Care Medicine, University Hospital RWTH Aachen University, Aachen, Germany.
• 4 Department of Anesthesiology, Faculty of Medicine, University of Malaya, Kuala Lumpur, Malaysia.
• 5 Clinical Evaluation Research Unit, Department of Critical Care Medicine, Queen's University, KGH Research Institute, Kingston Health Sciences Centre, Kingston, ON, Canada.
• 6 Department of Anesthesiology and Intensive Care Medicine, University Hospital Wuerzburg, Wuerzburg, Germany.
• PMID: 35853198
• DOI: 10.1097/CCM.0000000000005602

Objectives: Concise definitive review of how to read and critically appraise a systematic review.

Data sources: None.

Study selection: Current literature describing the conduct, reporting, and appraisal of systematic reviews and meta-analyses.

Data extraction: Best practices for conducting, reporting, and appraising systematic review were summarized.

Data synthesis: A systematic review is a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant original research, and to collect and analyze data from the studies that are included in the review. Critical appraisal methods address both the credibility (quality of conduct) and rate the confidence in the quality of summarized evidence from a systematic review. The A Measurement Tool to Assess Systematic Reviews-2 tool is a widely used practical tool to appraise the conduct of a systematic review. Confidence in estimates of effect is determined by assessing for risk of bias, inconsistency of results, imprecision, indirectness of evidence, and publication bias.

Conclusions: Systematic reviews are transparent and reproducible summaries of research and conclusions drawn from them are only as credible and reliable as their development process and the studies which form the systematic review. Applying evidence from a systematic review to patient care considers whether the results can be directly applied, whether all important outcomes have been considered, and if the benefits are worth potential harms and costs.

Copyright © 2022 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.

PubMed Disclaimer

Conflict of interest statement

Dr. Hill’s institution received funding from Fresenius Kabi and the Medical Faculty Rheinisch-Westfälische Technische Hochschule Aachen; she received funding from Fresenius Kabi. The remaining authors have disclosed that they do not have any potential conflicts of interest.

Similar articles

• How to Critically Appraise and Interpret Systematic Reviews and Meta-Analyses of Diagnostic Accuracy: A User Guide. Frank RA, Salameh JP, Islam N, Yang B, Murad MH, Mustafa R, Leeflang M, Bossuyt PM, Takwoingi Y, Whiting P, Dawit H, Kang SK, Ebrahimzadeh S, Levis B, Hutton B, McInnes MDF. Frank RA, et al. Radiology. 2023 May;307(3):e221437. doi: 10.1148/radiol.221437. Epub 2023 Mar 14. Radiology. 2023. PMID: 36916896 Free PMC article. Review.
• Folic acid supplementation and malaria susceptibility and severity among people taking antifolate antimalarial drugs in endemic areas. Crider K, Williams J, Qi YP, Gutman J, Yeung L, Mai C, Finkelstain J, Mehta S, Pons-Duran C, Menéndez C, Moraleda C, Rogers L, Daniels K, Green P. Crider K, et al. Cochrane Database Syst Rev. 2022 Feb 1;2(2022):CD014217. doi: 10.1002/14651858.CD014217. Cochrane Database Syst Rev. 2022. PMID: 36321557 Free PMC article.
• Characteristics, quality and volume of the first 5 months of the COVID-19 evidence synthesis infodemic: a meta-research study. Abbott R, Bethel A, Rogers M, Whear R, Orr N, Shaw L, Stein K, Thompson Coon J. Abbott R, et al. BMJ Evid Based Med. 2022 Jun;27(3):169-177. doi: 10.1136/bmjebm-2021-111710. Epub 2021 Jun 3. BMJ Evid Based Med. 2022. PMID: 34083212 Free PMC article. Review.
• Behavioural interventions for smoking cessation: an overview and network meta-analysis. Hartmann-Boyce J, Livingstone-Banks J, Ordóñez-Mena JM, Fanshawe TR, Lindson N, Freeman SC, Sutton AJ, Theodoulou A, Aveyard P. Hartmann-Boyce J, et al. Cochrane Database Syst Rev. 2021 Jan 4;1:CD013229. doi: 10.1002/14651858.CD013229.pub2. Cochrane Database Syst Rev. 2021. PMID: 33411338
• The future of Cochrane Neonatal. Soll RF, Ovelman C, McGuire W. Soll RF, et al. Early Hum Dev. 2020 Nov;150:105191. doi: 10.1016/j.earlhumdev.2020.105191. Epub 2020 Sep 12. Early Hum Dev. 2020. PMID: 33036834
• Advantages of unilateral percutaneous kyphoplasty for osteoporotic vertebral compression fractures-a systematic review and meta-analysis. Cao DH, Gu WB, Zhao HY, Hu JL, Yuan HF. Cao DH, et al. Arch Osteoporos. 2024 May 15;19(1):38. doi: 10.1007/s11657-024-01400-8. Arch Osteoporos. 2024. PMID: 38750277 Review.
• Evaluating COVID-19 vaccine acceptance among parents in Saudi Arabia: a systematic review examining attitudes, hesitancy, and intentions. Sayed AA. Sayed AA. Front Public Health. 2024 Mar 22;12:1327944. doi: 10.3389/fpubh.2024.1327944. eCollection 2024. Front Public Health. 2024. PMID: 38584927 Free PMC article.
• Utilization of Mirror Visual Feedback for Upper Limb Function in Poststroke Patients: A Systematic Review and Meta-Analysis. Kim H, Lee E, Jung J, Lee S. Kim H, et al. Vision (Basel). 2023 Nov 15;7(4):75. doi: 10.3390/vision7040075. Vision (Basel). 2023. PMID: 37987295 Free PMC article. Review.
• Comparison of clinical outcomes between aggressive and non-aggressive intravenous hydration for acute pancreatitis: a systematic review and meta-analysis. Li XW, Wang CH, Dai JW, Tsao SH, Wang PH, Tai CC, Chien RN, Shao SC, Lai EC. Li XW, et al. Crit Care. 2023 Mar 22;27(1):122. doi: 10.1186/s13054-023-04401-0. Crit Care. 2023. PMID: 36949459 Free PMC article.
• Evidence-Based Medicine Working Group: Evidence-based medicine. A new approach to teaching the practice of medicine. JAMA 1992; 268:2420–2425
• Guyatt G: Users’ Guides to the Medical Literature. Third Edition. New York, NY, McGraw Hill Education, 2015
• Djulbegovic B, Guyatt GH: Progress in evidence-based medicine: A quarter century on. Lancet 2017; 390:415–423
• Doig GS, Roberts I, Bellomo R: The tens of thousands of lives saved by randomized clinical trials in critical care. Intensive Care Med 2015; 41:701–704
• Davidoff F, Haynes B, Sackett D, et al.: Evidence based medicine. BMJ 1995; 310:1085–1086

Publication types

• Search in MeSH

LinkOut - more resources

Full text sources.

• Ingenta plc
• Ovid Technologies, Inc.
• Wolters Kluwer

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Literature Review: Evaluating sources and critical appraisal of literature

• Traditional or narrative literature reviews
• Scoping Reviews
• Systematic literature reviews
• Annotated bibliography
• Keeping up to date with literature
• Finding a thesis
• Evaluating sources and critical appraisal of literature
• Managing and analysing your literature
• Further reading and resources

Evaluating literature

While you are searching for relevant information about your topic you will need to think about the accuracy of the information, whether the information is from a reputable source, whether it is objective and current. The basic criteria for assessing information and questions you might want to think about are listed below. 

Accuracy        

• Is the information reliable?
• Is the information error-free?
• Is the information based on proven facts?
• Can the information be verified against other reliable sources?

Authority       

• Who is the author?
• Does he or she have the qualifications to speak/write on that topic?
• Is the author affiliated with a reputable university or organization in this subject field?
• Is the source peer reviewed or refereed?

Objectivity      

• What is the intended purpose of the information?
• Is the information fact or opinion?
• Is the information biased?

Currency        

• When was the information published?
• Is the information current or out-dated?
• Does currency matter in this topic?

Coverage        

• Does the information covered meet your information needs?
• Does it provide basic or in depth coverage

Further Reading:

Critical appraisal or critiquing the literature

Reading critically

The Sage Research Methods Online database (SRMO) is a good source of full text electronic Books, chapters, and articles on a range of research methodologies.  It includes a wide range of items in relation to literature review processes, and importantly how to read critically.

Goodwyn, A. & Stables, A. W. (2004). Learning to Read Critically: Learning to read critically in language and literacy : SAGE Publications Ltd doi: 10.4135/9781849209410

Quinton, S. & Smallbone, T. (2006). How to read critically . In Sage Study Skills: Postgraduate research in business (pp. 81-96). : SAGE Publications Ltd doi: 10.4135/9781849209564.n6

• << Previous: Finding a thesis
• Next: Managing and analysing your literature >>
• Last Updated: Jun 3, 2024 9:25 AM
• URL: https://libguides.csu.edu.au/review

Charles Sturt University is an Australian University, TEQSA Provider Identification: PRV12018. CRICOS Provider: 00005F.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Published: 08 April 2022

How to appraise the literature: basic principles for the busy clinician - part 1: randomised controlled trials

• Aslam Alkadhimi 1 ,
• Samuel Reeves 2 &
• Andrew T. DiBiase 3  

British Dental Journal volume  232 ,  pages 475–481 ( 2022 ) Cite this article

852 Accesses

1 Citations

2 Altmetric

Metrics details

Critical appraisal is the process of carefully, judiciously and systematically examining research to adjudicate its trustworthiness and its value and relevance in clinical practice. The first part of this two-part series will discuss the principles of critically appraising randomised controlled trials. The second part will discuss the principles of critically appraising systematic reviews and meta-analyses.

Evidence-based dentistry (EBD) is the integration of the dentist's clinical expertise, the patient's needs and preferences and the most current, clinically relevant evidence. Critical appraisal of the literature is an invaluable and indispensable skill that dentists should possess to help them deliver EBD.

This article seeks to act as a refresher and guide for generalists, specialists and the wider readership, so that they can efficiently and confidently appraise research - specifically, randomised controlled trials - that may be pertinent to their daily clinical practice.

Evidence-based dentistry is discussed.

Efficient techniques for critically appraising randomised controlled trials are described.

Important methodological and statistical considerations are explicated.

This is a preview of subscription content, access via your institution

Access options

Subscribe to this journal

Receive 24 print issues and online access

251,40 € per year

only 10,48 € per issue

Buy this article

• Purchase on Springer Link
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

How to appraise the literature: basic principles for the busy clinician - part 2: systematic reviews and meta-analyses

Making sense of the literature: an introduction to critical appraisal for the primary care practitioner

Critical appraisal of systematic reviews of intervention in dentistry published between 2019-2020 using the AMSTAR 2 tool

Burls A. What is critical appraisal? 2014. Available at http://www.whatisseries.co.uk/whatiscritical-appraisal/ (accessed April 2021).

Hong B, Plugge E. Critical appraisal skills teaching in UK dental schools. Br Dent J 2017; 222: 209-213.

Isham A, Bettiol S, Hoang H, Crocombe L. A Systematic Literature Review of the Information-Seeking Behaviour of Dentists in Developed Countries. J Dent Educ 2016; 80: 569-577.

Critical Appraisal Skills Programme. CASP Checklist. Available at https://casp-uk.net/wp-content/uploads/2018/01/CASP-Randomised-Controlled-Trial-Checklist-2018.pdf (accessed April 2021).

Schulz K F, Altman D G, Moher D, CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Int Med 2010; 152 : 726-732.

Sterne J A C, Savović J, Page M J et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 2019; DOI: 10.1136/bmj.l4898.

Petrou S, Grey A. Economic evaluation alongside randomised controlled trials: design, conduct, analysis, and reporting. BMJ 2011; DOI: 10.1136/bmj.d1548.

Black W C. The CE plane: a graphic representation of cost-effectiveness. Med Decis Making 1990; 10: 212-214.

Download references

Author information

Authors and affiliations.

Senior Registrar in Orthodontics, The Royal London Hospital Barts Health NHS Trust and East Kent Hospitals University NHS Foundation Trust, London, UK

Aslam Alkadhimi

Dental Core Trainee, East Kent Hospitals University NHS Foundation Trust, UK

Samuel Reeves

Consultant Orthodontist, East Kent Hospitals University NHS Foundation Trust, UK

Andrew T. DiBiase

You can also search for this author in PubMed   Google Scholar

Contributions

Aslam Alkadhimi contributed to conceptualisation, literature search, original draft preparation and drafting and critically revising the manuscript; Samuel Reeves contributed to original draft preparation and editing; and Andrew DiBiase contributed to supervision, draft editing and critically revising the manuscript.

Corresponding author

Correspondence to Aslam Alkadhimi .

Ethics declarations

The authors declare no competing interests.

Ethical approval and consent to participate did not apply to this study.

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Alkadhimi, A., Reeves, S. & DiBiase, A. How to appraise the literature: basic principles for the busy clinician - part 1: randomised controlled trials. Br Dent J 232 , 475–481 (2022). https://doi.org/10.1038/s41415-022-4096-y

Download citation

Received : 31 January 2021

Accepted : 25 April 2021

Published : 08 April 2022

Issue Date : 08 April 2022

DOI : https://doi.org/10.1038/s41415-022-4096-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Please enter both an email address and a password.

Account login

• Show/Hide Password Show password Hide password
• Reset Password

Need to reset your password?  Enter the email address which you used to register on this site (or your membership/contact number) and we'll email you a link to reset it. You must complete the process within 2hrs of receiving the link.

We've sent you an email.

An email has been sent to Simply follow the link provided in the email to reset your password. If you can't find the email please check your junk or spam folder and add [email protected] to your address book.

• About RCS England

• Dissecting the literature: the importance of critical appraisal

08 Dec 2017

Kirsty Morrison

This post was updated  in 2023.

Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context.

Amanda Burls, What is Critical Appraisal?

Why is critical appraisal needed?

Literature searches using databases like Medline or EMBASE often result in an overwhelming volume of results which can vary in quality. Similarly, those who browse medical literature for the purposes of CPD or in response to a clinical query will know that there are vast amounts of content available. Critical appraisal helps to reduce the burden and allow you to focus on articles that are relevant to the research question, and that can reliably support or refute its claims with high-quality evidence, or identify high-level research relevant to your practice.

Critical appraisal allows us to:

• reduce information overload by eliminating irrelevant or weak studies
• identify the most relevant papers
• distinguish evidence from opinion, assumptions, misreporting, and belief
• assess the validity of the study
• assess the usefulness and clinical applicability of the study
• recognise any potential for bias.

Critical appraisal helps to separate what is significant from what is not. One way we use critical appraisal in the Library is to prioritise the most clinically relevant content for our Current Awareness Updates .

How to critically appraise a paper

There are some general rules to help you, including a range of checklists highlighted at the end of this blog. Some key questions to consider when critically appraising a paper:

• Is the study question relevant to my field?
• Does the study add anything new to the evidence in my field?
• What type of research question is being asked? A well-developed research question usually identifies three components: the group or population of patients, the studied parameter (e.g. a therapy or clinical intervention) and outcomes of interest.
• Was the study design appropriate for the research question? You can learn more about different study types and the hierarchy of evidence here .
• Did the methodology address important potential sources of bias? Bias can be attributed to chance (e.g. random error) or to the study methods (systematic bias).
• Was the study performed according to the original protocol? Deviations from the planned protocol can affect the validity or relevance of a study, e.g. a decrease in the studied population over the course of a randomised controlled trial .
• Does the study test a stated hypothesis? Is there a clear statement of what the investigators expect the study to find which can be tested, and confirmed or refuted.
• Were the statistical analyses performed correctly? The approach to dealing with missing data, and the statistical techniques that have been applied should be specified. Original data should be presented clearly so that readers can check the statistical accuracy of the paper.
• Do the data justify the conclusions? Watch out for definite conclusions based on statistically insignificant results, generalised findings from a small sample size, and statistically significant associations being misinterpreted to imply a cause and effect.
• Are there any conflicts of interest? Who has funded the study and can we trust their objectivity? Do the authors have any potential conflicts of interest, and have these been declared?

And an important consideration for surgeons:

• Will the results help me manage my patients?

At the end of the appraisal process you should have a better appreciation of how strong the evidence is, and ultimately whether or not you should apply it to your patients.

Further resources:

• How to Read a Paper by Trisha Greenhalgh
• The Doctor’s Guide to Critical Appraisal by Narinder Kaur Gosall
• CASP checklists
• CEBM Critical Appraisal Tools
• Critical Appraisal: a checklist
• Critical Appraisal of a Journal Article (PDF)
• Introduction to...Critical appraisal of literature
• Reporting guidelines for the main study types

Kirsty Morrison, Information Specialist

Share this page:

• Library Blog

• Student Supports
• About U of M
• Current students
• Faculty and staff
• News and events
• Elizabeth Dafoe, Sciences & Technology, and Neil John Maclean Health Sciences Libraries are open for students, faculty, and staff. Archives is closed.
• Physical materials can be requested for contactless pickup at the pickup lockers at Elizabeth Dafoe Library (Fort Garry Campus) and Neil John Maclean Health Sciences Library (only accessible for Bannatyne campus students, staff, and faculty who have swipe card access).
• Our online information resources are available 24x7. Begin your search here .
• Need help? Use our Ask Us chat service and stay tuned to our update page for the most current libraries' services information.

How to Write a Literature Review: Step #3: Critical Appraisal

• Step #1: Build a Search Question This link opens in a new window
• Boolean Operators
• Subject Headings and Keywords
• Where to Search
• Step #3: Reading & Analyzing
• Step #4: Writing
• Step #5: Citation Management This link opens in a new window
• Last Updated: May 31, 2024 11:43 AM
• URL: https://libguides.lib.umanitoba.ca/literaturereview

Best Practice for Literature Searching

• Literature Search Best Practice
• What is literature searching?
• What are literature reviews?
• Hierarchies of evidence
• 1. Managing references
• 2. Defining your research question
• 3. Where to search
• 4. Search strategy
• 5. Screening results
• 6. Paper acquisition
• 7. Critical appraisal
• Further resources
• Training opportunities and videos
• Join FSTA student advisory board This link opens in a new window
• Chinese This link opens in a new window
• Italian This link opens in a new window
• Persian This link opens in a new window
• Portuguese This link opens in a new window
• Spanish This link opens in a new window

Evidence hierarchies

Levels of hierarchies can be useful for assessing the quality of evidence. In health sciences, these are portrayed as a pyramid with levels for the different types of study design.

Understanding study designs will help you judge the limitations of what can be concluded from a particular study. 

While in health-related research this hierarchy of study types can be used as a guideline, you cannot rely on the hierarchy to substitute for critical appraisal .  A strong cohort study would be more useful than a flawed systematic review.  There are times when evidence is better sorted by its usefulness for your own research question than by type of study design.

In food research, there is no consensus around the hierarchy of evidence.*  Best practice is to decide, when you plan your search, on the type of research you are looking for and what study designs would be appropriate for it. 

For example, if you are interested in qualitative studies of people’s behaviour towards nutrition, or in animal studies, you are unlikely to find large RCTs as it is not ethical or possible to run randomised studies in certain areas, so they may not exist. You are more likely to find qualitative studies that may include survey and interview data, write-ups from focus groups, or other types of studies that exist involving crops or animals (often observational or non-randomised studies).  Similarly, you might decide that challenge studies are the most appropriate way to research packaging and shelf life.

A systematic review is a study of studies, where researchers follow a predetermined and published protocol to find all the primary research studies done on a question, weigh the reliability of each one, and, if possible, extract the data from the studies in order to draw a conclusion from the combined evidence.  

Randomised control trials (RCTs)   randomly allocate participants to either an intervention or a control group, so that conclusions can be drawn about the efficacy of an intervention.  

Cohort studies are observational, longitudinal studies that look at a group of people with a shared experience or characteristic to see how they fare over time in regards to a particular factor. 

Case-controlled studies are observational studies in which a group of cases (i.e. people with a condition or disease) is compared to an analogous  group (similar to the case group except that they don't have the condition) to see if a causal attribute can be found for the case group.   

Cross-sectional studies are observational studies that describe a population at a certain point in time. These studies can locate correlations but not causal relationships.   

*Reported in Research Synthesis Methods and EFSA Journal.

• << Previous: What is critical appraisal?
• Next: Practical guide to literature searching >>
• Last Updated: May 17, 2024 5:48 PM
• URL: https://ifis.libguides.com/literature_search_best_practice

Searching the literature

• Getting Started
• Defining the question
• Formulate your search
• Search Tips
• Where to search
• Cited reference searching. Snowballing
• Hand-searching
• Record your search and manage your references

Evaluating your results

Critical appraisal, evaluating grey literature.

• Keep up to date
• Too many or too few results?

Is the material you have found suitable for your purpose?  The Open University Evaluating information  guide recommends using the PROMPT  criteria to evaluate sources of information. The OU Safari Evaluating information workbook  (PDF) (2014) may also be useful. 

Presentation

Is the information presented in a clear and readable way? Are there relevant diagrams and photographs?  Is it written objectively or is it emotive?

Is it relevant and appropriate for your needs? Does it cover the countries or regions which interest you? Does it cover all aspects of your topic?

Objectivity

Is it balanced or is there some bias? Can you easily establish who the authors are and what their authority might be? Are there vested interests behind the website? Is it trying to sell you something?

How was the information gathered together? Are the methods clearly stated?  Ask yourself basic questions about sample size, use of control groups. questionnaire design etc.

Provenance or Authority

Who or what originated the information and are they reliable sources? Are the authors acknowledged experts in this area? What else have they published on this topic?  Do you they belong to well-known institutions? If you're looking at a journal article, is it from a peer-reviewed journal? If it is a website, did you find the link on a trusted site, such as NHS Evidence or a professional body or a university?

Is the information up-to-date and can you tell if it has been superseded? Is it clear when the website was produced?

Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context.  

Here are different online courses to choose from. Each includes explanations and checklists to help you appraise the material you have retrieved.

• Centre for Evidence-Based Medicine: Critical Appraisal CEBM based at Oxford University, provides worksheets, checklists and other tools to use for critical appraisal. The CEBM also provides other tools for EBM, such as likelihood ratios and the CATmaker, a computer-assisted critical appraisal tool.
• Critical Appraisal Skills Programme (CASP) Oxford-based CASP runs workshops and provides a wealth of checklists and training materials on critical appraisal. Useful, downloadable checklists for different study types, such as systematic review or randomised clinical trials.
• Health Knowledge: Finding and Appraising the Evidence Online course from HealthKnowledge. Aimed at public health practitioners who are new to evidence based practice and critical appraisal. There are six modules including an overall introduction to critical appraisal, systematic reviews, economic evaluations and making sense of results.
• AACODS Checklist

Grey literature has not been through any sort of peer review process.  Therefore it is particularly important that you evaluate material very carefully to decide whether to use it.

The AACODS checklist is designed to enable evaluation and critical appraisal of grey literature. The checklist was designed and made available by Jess Tyndall at Flinders University. 

Who is responsible for the intellectual content?

Does it state aims, methods, peer-review, supporting work?

Are the limits or scope of the material clearly stated?

Can you identify bias, balance or opinion in the material?

Is the date relevant and does it meet your needs?

Significance

Is the item meaningful? What does it add?

• << Previous: Record your search and manage your references
• Next: Keep up to date >>
• Last Updated: Apr 24, 2024 9:39 AM
• URL: https://lancaster.libguides.com/Litsearch

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Can Fam Physician
• v.39; 1993 May

Critical appraisal of review articles.

Review articles are an important source of clinical information for family physicians. However, the volume of available reviews is vast and their scientific quality varies enormously. Family physicians must be able to identify trustworthy reviews quickly. This article outlines practical and flexible guidelines for critical appraisal and discusses the respective roles of review articles and original research reports in guiding clinical practice.

Full text is available as a scanned copy of the original print version. Get a printable copy (PDF file) of the complete article (1.1M), or click on a page image below to browse page by page. Links to PubMed are also available for Selected References .

Images in this article

p1099-a on p.1099

Click on the image to see a larger version.

Selected References

These references are in PubMed. This may not be the complete list of references from this article.

• Horwitz RI. Complexity and contradiction in clinical trial research. Am J Med. 1987 Mar; 82 (3):498–510. [ PubMed ] [ Google Scholar ]
• Yusuf S, Peto R, Lewis J, Collins R, Sleight P. Beta blockade during and after myocardial infarction: an overview of the randomized trials. Prog Cardiovasc Dis. 1985 Mar-Apr; 27 (5):335–371. [ PubMed ] [ Google Scholar ]
• Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC. Meta-analyses of randomized controlled trials. N Engl J Med. 1987 Feb 19; 316 (8):450–455. [ PubMed ] [ Google Scholar ]
• Mulrow CD. The medical review article: state of the science. Ann Intern Med. 1987 Mar; 106 (3):485–488. [ PubMed ] [ Google Scholar ]
• Oxman AD, Guyatt GH. Guidelines for reading literature reviews. CMAJ. 1988 Apr 15; 138 (8):697–703. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Mulrow CD, Thacker SB, Pugh JA. A proposal for more informative abstracts of review articles. Ann Intern Med. 1988 Apr; 108 (4):613–615. [ PubMed ] [ Google Scholar ]
• Haynes RB, Mulrow CD, Huth EJ, Altman DG, Gardner MJ. More informative abstracts revisited. Ann Intern Med. 1990 Jul 1; 113 (1):69–76. [ PubMed ] [ Google Scholar ]