Clinical vs Statistical Heterogeneity
A meta-analysis may attempt to address a compelling clinical dilemma. But one of the key questions to ask when appraising meta-analyses is whether the pooling of the included studies is appropriate. Clinical heterogeneity reflects clinical differences between study populations, the intervention, co-interventions and/or outcomes when pooling studies in meta-analysis. This is distinct from statistical heterogeneity which can be determined by visually assessing the forest plot, measuring the I2 statistic or the Cochran’s Q. Always ask yourself if the meta-analysis is combining apples with apples.
Cochrane Collaboration Tool for Assessing Risk of Bias in Intervention Trials
Dr. Ian Stiell September 2014
The use of scales for assessing quality or risk of bias in intervention trials is explicitly discouraged in Cochrane reviews, including the commonly-used scale was developed by Jadad and colleagues for randomized trials in pain research (Jadad 1996). For assessing bias, the Cochrane Collaboration recommends a two-part tool that addresses seven specific domains (sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective outcome reporting and ‘other issues’). Each domain in the tool includes one or more specific entries in a ‘Risk of bias’ table.
Goals of Systematic Literature Reviews
Dr. Lisa Calder April 2014
We often consider systematic literature reviews (SLR) and meta-analyses as tools to help us better understand the effectiveness of a given therapy when there are multiple conflicting studies in the literature. Another reasonable goal, however, is to advance the science in a given domain. On occasion, there lack rigorous study definitions to allow for effective research to answer a given clinical question or perhaps studies have been published looking repetitively at the same issue without any advancement on the clinically relevant question. Researchers can use SLRs to highlight gaps in the current literature and the need for a specific study design. This can focus researchers towards answering the question rather than multiple haphazard approaches.
Heterogeneity of Studies in Overviews
Network Meta-analysis
PICOS Format
Dr. Lisa Calder March 2012
When reviewing systematic literature reviews, pay attention to how the research question is phrased. High quality studies will follow the PRISMA guidelines’ suggestion to use the PICOS (participants, interventions, comparisons, outcomes, study design) format. A precise research question will guide study selection and should also dictate how table 1 summarizes the included studies. Having these details will also assist the reader in evaluating the included studies for generalizability and clinical heterogeneity.
Pooled Analysis vs. Meta-Analysis
In a meta-analysis, researchers assess heterogeneity across studies, examine subgroups of studies to determine if selected subsets of the research data provide similar or different results, and calculate summary relative risk estimates. A pooled analysis is similar to a traditional meta-analysis, except that data are combined (or pooled) from multiple studies and are analyzed as a single dataset. If the data and methods are consider homogeneous across studies and the data are available, then a pooled analysis is a very legitimate approach.
PRISMA Statement for Reporting Systematic Reviews:
The PRISMA statement (revised 2009 to replace QUORUM http://www.prisma-statement.org) was developed by an international group to establish preferred reporting guidelines for systematic reviews and meta-analyses. David Moher of the Ottawa Hospital Research Institute is the lead author on the paper that includes the PRISMA 27-item checklist and four-phase flow diagram.
Publication Bias in Systematic Reviews
Dr. Ian Stiell March 2012
Even when individual studies included in best evidence summaries have a low risk of bias, publication bias can result in substantial overestimates of effect. Authors should suspect publication bias when available evidence comes from a number of small studies, most of which have been commercially funded. A number of approaches based on examination of the pattern of data are available to help assess publication bias. The most popular of these is the funnel plot; all, however, have substantial limitations. The likelihood of publication bias is less for studies that are not evaluating new drugs or devices.
QUADAS-2 Tool for Evaluation of Systematic Reviews of Diagnostic Accuracy Studies
QUOROM statement
Dr. Lisa Calder December 2012
When critically appraising systematic literature reviews and meta-analyses, the reader will find the QUOROM statement to be a helpful tool. This guide helps you assess adequacy of search strategy, article selection and quality assessment. Furthermore, it provides the key elements which should be reported to allow you to assess the overall validity of the results. This reporting template also assists authors of SLR and MA to both design and write-up their studies in a rigorous way.
Reporting Standards for Systematic Reviews
The PRISMA statement (http://www.prisma-statement.org) was developed to establish preferred reporting guidelines for systematic reviews and meta-analyses and includes a 27-item checklist and four-phase flow diagram. Observational studies are considered a lower level of evidence for interventions and have had two sets of guidelines developed: MOOSE (Meta-analysis Of Observational Studies in Epidemiology) and STROBE (STrengthening the Reporting of OBservational studies in Epidemiology).
Review of Grey Literature in Systematic Reviews
Should Observational Studies be Included in a Systematic Review?
Generally only RCTs should be included because the results of non-randomized, observational studies on interventions are subject to a number of biases and often over-estimate the effects. The problem we frequently see in emergency medicine is that there may be very few RCTs in particular content area, like acute pericarditis. Hence, while reviews that include non-randomized studies may be informative, readers must take the results with several grains of salt.
Statistical Heterogeneity in Systematic Reviews
Subgroup analyses in Meta-analysis
Systematic review versus meta-analysis
Dr. Ian Stiell November 2012
To avoid the biases of an unsystematic review (i.e. review article), a systematic review incorporates explicit inclusion and exclusion criteria, a comprehensive search for the evidence, and a summary of the results according to explicit rules. When a systematic review pools data across studies to provide a quantitative estimate of the treatment effect, this is called a meta-analysis. When the data cannot be pooled, the systematic review will provide a narrative synthesis of the evidence.
To Pool or Not to Pool
A meta-analysis may attempt to address a compelling clinical question. But one of the key questions to ask when appraising meta-analyses is whether the pooling of the included studies is appropriate. Clinical heterogeneity reflects clinical differences between study populations, the intervention, co-interventions and/or outcomes when pooling studies in meta-analysis. This is distinct from statistical heterogeneity. Always ask yourself if the meta-analysis is combining apples with apples.
Trackbacks/Pingbacks