Allopathic Medicine: NSU MD Library Guide: To Systematic Review or not

A systematic review is guided filtering and synthesis of all available evidence addressing a specific, focused research question, generally about a specific intervention or exposure. The use of standardized, systematic methods and pre-selected eligibility criteria reduce the risk of bias in identifying, selecting and analyzing relevant studies. A well-designed systematic review includes clear objectives, pre-selected criteria for identifying eligible studies, an explicit methodology, a thorough and reproducible search of the literature, an assessment of the validity or risk of bias of each included study, and a systematic synthesis, analysis and presentation of the findings of the included studies. A systematic review may include a meta-analysis.

A systematic review is best deployed to test a specific hypothesis about a healthcare or public health intervention or exposure. By focusing on a single intervention or a few specific interventions for a particular condition, the investigator can ensure a manageable results set. Moreover, examining a single or small set of related interventions, exposures, or outcomes, will simplify the assessment of studies and the synthesis of their findings.

Systematic reviews are poor tools for hypothesis generation: for instance, to determine what interventions have been used to increase the awareness and acceptability of a vaccine or to investigate the ways that predictive analytics have been used in health care management. In the first case, we don't know what interventions to search for and so have to screen all the articles about awareness and acceptability. In the second, there is no agreed on set of methods that make up predictive analytics, and health care management is far too broad. The search will necessarily be incomplete, vague and very large all at the same time. In most cases, reviews without clearly and exactly specified populations, interventions, exposures, and outcomes will produce results sets that quickly outstrip the resources of a small team and offer no consistent way to assess and synthesize findings from the studies that are identified.

You might consider performing a rapid scoping review or evidence assessment . These frameworks allow iterative searching over a reduced number of data sources and no requirement to assess individual studies for risk of bias. They include built-in mechanisms to adjust the analytic framework as the work progresses and more is learned about the topic. These frameworks won't help you limit the number of records you'll need to screen (broad questions lead to large results sets) but may give you means of dealing with a large set of results.

This tool can help you decide what kind of review is right for your question.

When your Professor/Faculty Advisor/Mentor Tells You to "Just Do a Systematic Review"

Systematic reviews have become a popular research approach, but many people misunderstand them. Sometimes well-meaning faculty will advise students or residents to complete a systematic review as an “easy” research project without fully understanding the methods and resources required. These misunderstandings often lead to the development of poorly done “systematic reviews” that lack the required systematic methods defined in an appropriate protocol, lack well-developed and well-documented search strategies, and fail to follow other methodological and reporting standards (PRISMA 2020 Checklist Items 1,4,6,7) that peer reviewers should look for when considering the manuscript for publication.

A systematic review is not a suitable project for a single student/resident with no prior experience in this methodology. It also is generally not an appropriate project for a single student and a single faculty advisor who is not an expert on systematic review methodology. Most health science graduate students have not had the appropriate experience or training to independently develop robust protocols and search strategies for systematic reviews. A lack of familiarity with appropriate methods for SR projects is very likely to lead to wasted time and unpublishable work. A systematic review is not an appropriate short-term project; systematic reviews can take an average of a year or more to complete.

We recommend that students instructed to do a systematic review without appropriate support share the above statement with their faculty. We are available for consultation on when and for whom a systematic review is an appropriate project. We can also recommend training courses for faculty who are interested in further participating in and engaging students in future systematic review research.

Further Recommendations:

• Identify one or more students or trainees to work with you

• Take the coursera or other training: https://www.coursera.org/learn/systematic-review

Probably not. Systematic reviews are a lot of work. Including creating the protocol, building and running a quality search, collecting all the papers, evaluating the studies that meet the inclusion critera and extracting and analyzing the summary data, a well done review can require dozens to hundreds of hours of work that can span several months. Moreover, a systematic review requires subject expertise, statistical support and a librarian to help design and run the search only where the faculty members are the PI not students. Be aware that librarians sometimes have queues for their search time. It may take several weeks to complete and run a search. Moreover, all guidelines for carrying out systematic reviews recommend that at least two subject experts screen the studies identified in the search. The first round of screening can consume 1 hour per screener for every 100-200 records. A systematic review is a labor-intensive team effort.

Before starting out on a systematic review, check to see if someone has done it already. In PubMed you can use the systematic review subset to limit to a broad group of papers that is enriched for systematic reviews. You can invoke the subset by selecting if from the Article Types filters to the left of your PubMed results, or you can append AND systematic[sb] to your search. For example:

"neoadjuvant chemotherapy" AND systematic[sb]

The systematic review subset is very noisy, however. To quickly focus on systematic reviews (knowing that you may be missing some), simply search for the word systematic in the title:

"neoadjuvant chemotherapy" AND systematic[ti]

Any PRISMA-compliant systematic review will be identifed by this method since including the words "systematic review" in the title is a requirement of the PRISMA checklist.

You can also search for protocols that will indicate that another group has set out on a similar project. Many investigators will register their protocols in PROSPERO, a registry of review protocols. Other published protocols as well as Cochrane Review protocols appear in the Cochrane Methodology Register, a part of the Cochrane Library.                                                                                                     

                                                                                                                                                  Harvard Countway Library, Systematic Review and Meta-Analysis Guide