Skip to main content

Pharmacy Research Guide: More than just systematic reviews

An instructional pathfinder for HPD College of Pharmacy students and faculty

Systematic Review:
Does it mean what you think it means?

Systematic reviews are an increasingly prevalent form of health science research and publication.  
A search of PubMed using their publication date and systematic review filters (systematic[sb]) and shows a 213% increase in published systematic reviews from 2006 to 2016: 

Systematic reviews are an exhaustive review and synthesis of all the relevant studies on a given topic.  They are extremely important in evidence based health care. 

“The most reliable way to identify benefits and harms associated with various treatment options is a systematic review of comparative effectiveness research.“
(Institute of Medicine. 2011. Finding What Works in Health Care:  Standards for Systematic Reviews. Washington, DC: The National Academies Press)

 The typical systematic review takes 12 months to complete:

Box 2.3.b: Timeline for a Cochrane Review
Source: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from

What are systematic reviews and meta-analyses?

Individual studies are often not big and powerful enough to provide reliable answers on their own. Or several studies on the effects of a treatment might come to different conclusions. In order to find reliable answers to research questions, you therefore have to look at all of the studies and analyze their results together.

Systematic reviews summarize the results of all the studies on a medical treatment and assess the quality of the studies. The analysis is done following a specific, methodologically sound process. In a way, it’s a “study of studies.” Good systematic reviews can provide a reliable overview of the current knowledge in a certain area.

They are normally done by teams of authors working together. The authors are usually specialists with backgrounds in medicine, epidemiology, medical statistics and research.
© IQWiG (Institute for Quality and Efficiency in Health Care)

Other Types of Reviews
Not every review is a systematic review, and despite their utility and growing prevalence,  there are many other types of reviews that may be more appropriate for your research topic, purpose, scope, or available time and manpower.  

Narrative Reviews (also known as a Literature Review) - According to the Medical Subject Headings (MeSH) scope note, a literature review describes ‘Published materials which provide an examination of recent or current literature.Review articles can cover a wide range of subject matter at various levels of completeness and comprehensiveness based on analyses of literature
that may include research findings.
Strengths: Seeks to summarize previous works and accomplishments
Weaknesses: Lack explicit intent to maximize the scope of the study, or to analyze data.  Have a higher risk of bias, intentional or inadvertent, through omission of areas of research or studies, or questioning validity of those studies.
Hall, A. & Walton, G. Information overload within the health care system: a literature review. Health Information and Libraries Journal 2004,21(2), 102–8. (FindIt)

Critical Review - Aims to demonstrate writer has extensively researched literature and critically evaluated its quality. Goes beyond mere description to include degree of analysis and conceptual innovation. Typically results in hypothesis or model. 
Strengths: Emphasizes critical evaluation of previous bodies of work
Weaknesses: Typically lack a structured or systematic approach to searching, synthesis, and analysis.
Example. Kulviwat, S., Guo, C. & Engchanil, N. Determinants of online information search: a critical review and assessment. Internet Research: Electronic Networking Applications and Policy 2004, 14 (3), 245–53. (FindIt)
Fetta, J., Starkweather, A., & Gill, J. M. (2017). Computer-Based Cognitive Rehabilitation Interventions for Traumatic Brain Injury: A Critical Review of the Literature. Journal of Neuroscience Nursing, 49(4), 235-240. (FindIt)

Mapping Review -  This type of review was developed by the Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Centre) at the Institute of Education in London. The purpose is to map and categorize existing information on a topic and identify gaps in knowledge, with the purpose of initiating further research.  The outcomes may involve further review or primary research.
Strengths: Provide context by utilizing a systematic search strategy and indentify gaps in the knowledge base.
Weaknesses: Tend to be time constrained; may be overly broad and might mask heterogeneity among studies; often do not address quality of studies included in the review. 
Gough, D., Kiwan, D., Sutcliffe, K., Simpson, D. & Houghton, N. A Systematic Map and Synthesis Review of the Effectiveness of Personal Development Planning for Improving Student Learning. London: EPPI-Centre, Social Science Research Unit, 2003. (FindIt)

Scoping Review - Provides a preliminary assessment of the size and scope of the literature that is available on a topic. Looks at the nature and extent of the available evidence.
StrengthsCan help policymakers or others determine if a fulll systematic review is needed.  The search aims to be systematic, transparent, and replicable - similar to a systematic review.
Weaknesses: Can't be considered the final product on their own.  They are usually done relatively quickly and often lack rigor.  This leaves them vulnerable to biased conclusions.  For example, they do not usually assess the quality of included studies.  They should not be used to recommend policy or practice.  
Weeks, L. C. & Strudsholm, T. A scoping review of research on complementary and alternative medicine (CAM) and the mass media: looking back, moving forward. BMC Complementary and Alternative Medicine 2008, 19(8), 43. (FindIt)

Rapid Review -  Provide an assessment of current policy or practice, using systematic review methods for searching and appraising, done in when time constraints are short.
Strengths: Aim to be rigorous and use an explicit method, making them systematic, but may lack depth or breadth.  Often use a more focused question than a systematic review, broader and less focused search strategies, and may limit the use of grey literature.  
Weaknesses: The shorter timeframe increases the risk of bias, in particular publication bias, since the search may not find less visible research.  The research question must be carefully chosen to avoid providing a focused answer on the wrong question. Documenting methodology and acknowledging limitations is important in guarding against possible bias.  
Lacey Bryant, S. & Gray, A. Demonstrating the positive impact of information
support on patient care in primary care: a rapid literature review. Health Information and Libraries Journal 2006, 23(2), 118–25.  (FindIt)

Systematized Review - Done in a systematic manner, but without the extensiveness of a true systematic review.  They are frequently conducted as postgraduate student assignments, while recognizing that time and manpower constraints are limiting factors. 
Strengths: The search might be comprehensive or the included studies might be thoroughly analyzed, although frequently only one or the other is done.  This allows the author to demonstrate knowledge of the review process.
Weaknesses: Searches may miss important studies or analysis may be cursory.  The risk of bias is significant.  Often, completion as an academic assignment is more important than solid methodology. 
Cornet, R. & de Keizer, N. Forty years of SNOMED: a literature review. BMC Medical Informatics and Decision Making 2008, 8(Suppl. 1), S2. (FindIt)

Umbrella Review:  Essentially a review of systematic reviews. Umbrella reviews focus on broad conditions or problems when there are more than one potential interventions.  
Strengths:  Provide readers with quick overviews that cover multiple interventions without excessive loss of detail and specificity.
Weaknesses: Can only be done for topics where there are sufficient numbers of existing systematic reviews.   
Seida, J. K., Ospina, M. B., Karkhaneh, M., Hartling, L., Smith, V. & Clark, B. Systematic reviews of psychosocial interventions for autism: an umbrella review. Developmental Medicine and Child Neurology 2009, 51(2), 95–104. (FindIt)

Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), 91-108. (FindIt)