As part of our Journal Club summaries our JC Chairs (Drs. Lisa Calder and Ian Stiell @EMO_Daddy) have been tasked with explaining Epidemiological concepts so that everyone in our department can analyze the literature and appraise articles on their own. For this Blog post we have all the “Epi Lesson” as they relate to “Overview Articles”. More to follow in the coming weeks.

Clinical vs Statistical Heterogeneity

A meta-analysis may attempt to address a compelling clinical dilemma. But one of the key questions to ask when appraising meta-analyses is whether the pooling of the included studies is appropriate. Clinical heterogeneity reflects clinical differences between study populations, the intervention, co-interventions and/or outcomes when pooling studies in meta-analysis. This is distinct from statistical heterogeneity which can be determined by visually assessing the forest plot, measuring the I2 statistic or the Cochran’s Q. Always ask yourself if the meta-analysis is combining apples with apples. 

Cochrane Collaboration Tool for Assessing Risk of Bias in Intervention Trials

Dr. Ian Stiell  September 2014
The use of scales for assessing quality or risk of bias in intervention trials is explicitly discouraged in Cochrane reviews, including the commonly-used scale was developed by Jadad and colleagues for randomized trials in pain research (Jadad 1996). For assessing bias, the Cochrane Collaboration recommends a two-part tool that addresses seven specific domains (sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective outcome reporting and ‘other issues’).  Each domain in the tool includes one or more specific entries in a ‘Risk of bias’ table.

Goals of Systematic Literature Reviews 

Dr. Lisa Calder  April 2014
We often consider systematic literature reviews (SLR) and meta-analyses as tools to help us better understand the effectiveness of a given therapy when there are multiple conflicting studies in the literature. Another reasonable goal, however, is to advance the science in a given domain. On occasion, there lack rigorous study definitions to allow for effective research to answer a given clinical question or perhaps studies have been published looking repetitively at the same issue without any advancement on the clinically relevant question. Researchers can use SLRs to highlight gaps in the current literature and the need for a specific study design. This can focus researchers towards answering the question rather than multiple haphazard approaches.

Heterogeneity of Studies in Overviews 

Dr Ian Stiell  September 2012  
Heterogeneity refers to differences between patients or differences in the results of different studies and must be carefully evaluated in a meta-analysis to determine if results can be statistically pooled. The classical measure of heterogeneity is Cochran’s Q, which is calculated as the weighted sum of squared differences between individual study effects and the pooled effect across studies, with the weights being those used in the pooling method. The I² statistic describes the percentage of variation across studies that is due to heterogeneity rather than chance.  Always ask yourself if the meta-analysis is combining apples with apples.

Network Meta-analysis

Dr. Lisa Calder  December 2014 
Network meta-analyses should not be viewed as the same as other meta-analyses. This specialized methodology is designed to answer complex management questions when there are multiple options possible. This is particularly helpful when a head-to-head RCT is not practical to conduct. The key requirement is that the studies being compared have at least one common or equivalent therapy among 

PICOS Format

Dr. Lisa Calder  March 2012
When reviewing systematic literature reviews, pay attention to how the research question is phrased. High quality studies will follow the PRISMA guidelines’ suggestion to use the PICOS (participants, interventions, comparisons, outcomes, study design) format. A precise research question will guide study selection and should also dictate how table 1 summarizes the included studies.  Having these details will also assist the reader in evaluating the included studies for generalizability and clinical heterogeneity.

Pooled Analysis vs. Meta-Analysis 

Dr. Ian Stiell   May 2012
In a meta-analysis, researchers assess heterogeneity across studies, examine subgroups of studies to determine if selected subsets of the research data provide similar or different results, and calculate summary relative risk estimates. A pooled analysis is similar to a traditional meta-analysis, except that data are combined (or pooled) from multiple studies and are analyzed as a single dataset. If the data and methods are consider homogeneous across studies and the data are available, then a pooled analysis is a very legitimate approach. 

PRISMA Statement for Reporting Systematic Reviews:   

Dr. Ian Stiell   January 2012  
The PRISMA statement (revised 2009 to replace QUORUM was developed by an international group to establish preferred reporting guidelines for systematic reviews and meta-analyses. David Moher of the Ottawa Hospital Research Institute is the lead author on the paper that includes the PRISMA 27-item checklist and four-phase flow diagram. 

Publication Bias in Systematic Reviews  

Dr. Ian Stiell   March 2012 
Even when individual studies included in best evidence summaries have a low risk of bias, publication bias can result in substantial overestimates of effect. Authors should suspect publication bias when available evidence comes from a number of small studies, most of which have been commercially funded. A number of approaches based on examination of the pattern of data are available to help assess publication bias. The most popular of these is the funnel plot; all, however, have substantial limitations. The likelihood of publication bias is less for studies that are not evaluating new drugs or devices.  

QUADAS-2 Tool for Evaluation of Systematic Reviews of Diagnostic Accuracy Studies

Dr. Ian Stiell   September 2014
The QUADAS-2 Tool is recommended for evaluating the quality of systematic reviews of primary diagnostic accuracy studies [as opposed to intervention studies]. QUADAS-2 consists of four key domains: patient selection, index test, reference standard, as well as flow and timing. Each is assessed in terms of risk of bias and the first three in terms of concerns regarding applicability. Signalling questions are included to assist in judgements about risk of bias. QUADAS-2 is applied in four phases.

QUOROM statement 

Dr. Lisa Calder  December 2012
When critically appraising systematic literature reviews and meta-analyses, the reader will find the QUOROM statement to be a helpful tool. This guide helps you assess adequacy of search strategy, article selection and quality assessment. Furthermore, it provides the key elements which should be reported to allow you to assess the overall validity of the results. This reporting template also assists authors of SLR and MA to both design and write-up their studies in a rigorous way. 

Reporting Standards for Systematic Reviews  

Dr. Ian Stiell     February 2013
The PRISMA statement ( was developed to establish preferred reporting guidelines for systematic reviews and meta-analyses and includes a 27-item checklist and four-phase flow diagram. Observational studies are considered a lower level of evidence for interventions and have had two sets of guidelines developed: MOOSE (Meta-analysis Of Observational Studies in Epidemiology) and STROBE (STrengthening the Reporting of OBservational studies in Epidemiology).

Review of Grey Literature in Systematic Reviews 

Dr. Ian Stiell   September 2015
There are many definitions of grey literature, but it is usually understood to mean literature that is not formally published in sources such as books or journal articles. Conference abstracts and other grey literature have been shown to be sources of approximately 10% of the studies referenced in Cochrane reviews. A recently updated Cochrane methodology review showed that published trials showed an overall greater treatment effect than grey literature trials. Conference abstracts are a particularly important source of grey literature although such work has not undergone the extensive peer-reviewed required for most scientific manuscripts. There are numerous on-line grey literature databases.

Should Observational Studies be Included in a Systematic Review?    

Dr. Ian Stiell    May 2012
Generally only RCTs should be included because the results of non-randomized, observational studies on interventions are subject to a number of biases and often over-estimate the effects. The problem we frequently see in emergency medicine is that there may be very few RCTs in particular content area, like acute pericarditis. Hence, while reviews that include non-randomized studies may be informative, readers must take the results with several grains of salt.

Statistical Heterogeneity in Systematic Reviews 

Dr. Christian Vaillancourt
There is no statistical substitution to evaluating “clinical” heterogeneity. Statistical heterogeneity should only be evaluated after a clinical decision is made that study populations, interventions, and outcomes were indeed comparable, and that it is appropriate to combine study results. Cochran’s Q test is a classical measure of heterogeneity. It can be underpowered to detect heterogeneity when the number of included studies is small, and overpowered when it is large. This is not the case when using the I2 instead. A fixed effect can be used to combine studies with minimal statistical heterogeneity, whereas a random effect model should be used otherwise. 

Subgroup analyses in Meta-analysis

Dr. Lisa Calder   May 2013
 Subgroup analyses can be an important source of hypothesis generation. The key element to determine is whether the subgroup analyses were planned a priori. These analyses have greater validity than post hoc analyses which result from searching for statistically significant trends in the data which could be present due to chance and which some have labeled “data dredging”.

Systematic review versus meta-analysis  

Dr. Ian Stiell    November 2012
To avoid the biases of an unsystematic review (i.e. review article), a systematic review incorporates explicit inclusion and exclusion criteria, a comprehensive search for the evidence, and a summary of the results according to explicit rules. When a systematic review pools data across studies to provide a quantitative estimate of the treatment effect, this is called a meta-analysis. When the data cannot be pooled, the systematic review will provide a narrative synthesis of the evidence. 

To Pool or Not to Pool 

Dr. Lisa Calder    October 2014
A meta-analysis may attempt to address a compelling clinical question. But one of the key questions to ask when appraising meta-analyses is whether the pooling of the included studies is appropriate. Clinical heterogeneity reflects clinical differences between study populations, the intervention, co-interventions and/or outcomes when pooling studies in meta-analysis. This is distinct from statistical heterogeneity. Always ask yourself if the meta-analysis is combining apples with apples. 

Verification Bias

Dr. Lisa Calder  October 2014
When critically appraising a study evaluating a diagnostic test, it is important to examine the study population closely. In emergency medicine, we see undifferentiated patients the majority of the time. Studies of specialist referral populations have less generalizability to the ED because inherently the diagnostic test will perform better in a population with a high pre-test probability of disease in the first place. Verification bias, or work-up bias, applies here where the likelihood of performing the gold standard is influenced by the results of the diagnostic test under study. In this case, performing MRI or U/S for rotator cuff disease is influenced by the results of the physical examination tests being studied.