Dissecting the literature: the importance of critical appraisal

This is an updated version of a post from 2017.

Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context.

Amanda Burls, What is Critical Appraisal?

Why is critical appraisal needed?

Literature searches using databases like MEDLINE or Embase often result in an overwhelming volume of results which can vary in quality. Similarly, those who browse medical literature for the purposes of CPD or in response to a clinical query will know that there are vast amounts of content available. Critical appraisal helps to reduce the burden and allow you to focus on articles that are relevant to the research question, and that can reliably support or refute its claims with high-quality evidence, or identify high-level research (PDF) relevant to your practice (see also the "Hierarchy of Scientific Evidence diagram" below for research levels).

Image: "Hierarchy of Scientific Evidence", CC BY-SA 4.0, via Wikimedia Commons.

While most of us know not to believe everything we may read in a newspaper (or on Twitter), it’s also true that we cannot rely 100% on papers written in even the most prestigious academic journals. Different types of studies reported in the literature also have different strengths and weaknesses. Even if the contents of a research paper are reliable, it is sometimes difficult to find the specific information you are looking for and interpret it accurately.

Critical appraisal allows us to:

• reduce information overload by eliminating irrelevant or weak studies
• identify the most relevant papers
• distinguish evidence from opinion, assumptions, misreporting, and belief
• assess the validity of the study
• assess the usefulness and clinical applicability of the study
• recognise any potential for bias.

Critical appraisal helps to separate what is significant from what is not. One way we use critical appraisal in the Library is to prioritise the most clinically relevant content for our Specialty Updates.

How to critically appraise a paper

There are some general rules to help you, including a range of checklists highlighted at the end of this blog. Some key questions to consider when critically appraising a paper:

• Is the study question relevant to my field?
• Does the study add anything new to the evidence in my field?
• What type of research question is being asked?
  A well-developed research question usually identifies three components: the group or population of patients, the studied parameter (e.g. a therapy or clinical intervention) and outcomes of interest.
• Was the study design appropriate for the research question?
  You can learn more about different study types and the hierarchy of evidence here.
• Did the methodology address important potential sources of bias?
  Bias can be attributed to chance (e.g. random error) or to the study methods (systematic bias).
• Does the study population represent the patient population, e.g. with regards to ethnicity?
• Was the study performed according to the original protocol?
  Deviations from the planned protocol can affect the validity or relevance of a study, e.g. a decrease in the studied population over the course of a randomised controlled trial.
• Does the study test a stated hypothesis?
  Is there a clear statement of what the investigators expect the study to find which can be tested, and confirmed or refuted?
• Were the statistical analyses performed correctly?
  The approach to dealing with missing data, and the statistical techniques that have been applied should be specified. Original data should be presented clearly so that readers can check the statistical accuracy of the paper.
• Do the data justify the conclusions?
  Watch out for definite conclusions based on statistically insignificant results, generalised findings from a small sample size, and statistically significant associations being misinterpreted to imply a cause and effect.
• Are there any conflicts of interest?
  Who has funded the study and can we trust their objectivity? Do the authors have any potential conflicts of interest, and have these been declared?

And an important consideration for surgeons:

• Will the results help me manage my patients?

At the end of the appraisal process you should have a better appreciation of how strong the evidence is, and ultimately whether or not you should apply it to your patients.

Further resources

Background on critical appraisal

• How to read a paper: the basics of evidence based medicine by Trisha Greenhalgh
• The doctor’s guide to critical appraisal by Narinder Kaur Gosall
• How to read a systematic review and meta-analysis and apply the results to patient care: users' guides to the medical literature

Critical appraisal for antiracism

Traditional appraisal tools do not prompt appraisers to consider issues such as minoritised ethnic groups’ under-representation, and the effect of racial bias on medical research and its application. To address these issues, Ramona Naicker, Librarian and Information Specialist, has developed a supplementary tool to support appraisers. A short checklist of five questions is provided to be used alongside exisiting tools: Critically Appraising for Antiracism.

Appraisal tools

• CASP Checklists
• CEBM Critical Appraisal Tools
• SIGN Checklists
• JBI Critical Appraisal Tools
• AMSTAR Checklist (for systematic reviews)
• Equator Network: Reporting Guidelines for the Main Study Types
• BMJ Best Practice: Practical EBM Tools

Original author: Kirsty Morrison, Information Specialist.
Updated in 2023: Janet Clapton and Judit Sami, Information Specialists.