What makes a ‘good’ systematic evidence synthesis?
There are lots of “bad” evidence syntheses published but what makes a “good” one and how can we spot them?
1 What makes a good systematic evidence synthesis?
There has been an proliferation of studies claiming to be systematic evidence syntheses. Many of these studies are of poor quality and have a high risk of bias. Systematic evidence synthesis strives to reduce bias and poor quality syntheses could be misleading and waste valuable resources. As pointed out by Haddaway et al. (2020) these poor quality syntheses could be eroding trust in “evidence synthesis as an academic endeavour”. I would go as far as to suggest that they are eroding trust in evidence synthesis.
1.1 Identifying a good evidence synthesis
Here are some simple criteria to help you identify a good synthesis:
1.1.1 A Clear Research Question:
A good synthesis has a well-defined research question that guides the entire review process. Often a good synthesis will use a question formulation framework/tool to help them identify the key elements. For example, a common approach is to identify the PICO elements of a question; Population, Intervention, Control, Outcome.
1.1.2 Transparent Methods:
Look for transparency in the methodology section. A good synthesis should clearly outline its search strategy, inclusion and exclusion criteria, data extraction methods, and quality assessment criteria. A predefined (before the work started on the synthesis) protocol outlining the steps and decisions to be made adds to trust in the evidence synthesis. This avoids “mission creep” where the question changes to fit the evidence found and not to answer the original question.
1.1.3 Comprehensive Search Strategy:
A comprehensive search strategy is essential to ensure that all relevant studies are identified. Look for details on the databases searched, search terms used, and any other sources consulted (e.g., grey literature, conference proceedings). Multiple databases should be searched and unpublished (i.e. grey literature) sources should be consulted.
1.1.4 Inclusion Criteria:
Ensure that the synthesis clearly states the criteria used to include or exclude studies. This helps in understanding the scope of the synthesis and the generalisability of its findings.
1.1.5 Quality Assessment:
Look for information on how the quality of included studies was assessed. This could involve evaluating study design, risk of bias, and other relevant factors. This is a vital element that is often missed.
1.1.6 Data Synthesis:
A good synthesis should provide a clear synthesis of the evidence, including any statistical analysis or qualitative synthesis methods used. Open data and open code, where appropriate, should be available to readers.
1.1.7 Risk of Bias Assessment:
The synthesis should address the risk of bias in the included studies. Look for methods used to minimise bias and consider how potential biases might impact the review findings. Risk of bias are things in the conduct of a study that could lead to misleading results. For example a small sample size might mean a study does not have enough statistical power to determine the size or direction of an effect.
1.1.8 Discussion of Limitations:
A good synthesis should discuss its limitations openly. This could include limitations related to the search strategy, study quality, or other factors that may impact the validity of the findings.
1.1.9 Conclusion Supported by Evidence:
Ensure that the conclusions drawn in the synthesis are supported by the evidence presented. Look for consistency between the findings and the data synthesised from the included studies. Do not just rely on the abstract or the lay summary.
1.2 Conclusion
By carefully assessing these criteria, you can better distinguish between good and bad syntheses and make more informed decisions about their reliability and validity.
Donnelly et al. (2018) provided a much simpler schema which condenses the criteria above in to four principles - Inclusive, Rigorous, Transparent and Accessible.
It is not easy identifying good evidence synthesis. Sometimes the criteria/principles outlined above are hard to identify. It can be tough to know if a synthesis only achieves some of the criteria how much it can be trusted.
There are many organisations whose purpose it is to help produce and share robust, high quality evidence syntheses. In the environmental field there is The Collaboration for Environmental Evidence. If you need help please reach out.