Evidence searches for systematic reviews.
On this page
There is no suggested maximum time to conduct evidence searches for systemartic reviews. Systematic reviews are large scale projects, and it is advisable to put-aside adequate time for searching, meetings and correspondence with the requester and their team.
Planning
You should ask the requester their timescales and when they would like the results by. If the suggested time scale is too short advise them of an appropriate date.
It is reasonable to explain to the requester that as systematic review searches are complex, carried out across multiple databases, registries and websites, and require extensive testing, longer delivery time scales are usual.
As a rough guide delivery time scales are commonly at least 10 days to complete the initial searches with revisions possible as the review progresses.
Things to consider
Systematic review or literature review
Carrying out a systematic review is a highly structured activity governed by internationally recognised guidance. Requesters can mistakenly use the blanket terminology “systematic review” when they really wish to carry out a systematic search of the evidence.
Information:
If required, refer your requester to the following for definitions and descriptions on types of review.
Sutton and others (2019) 'Meeting the review family: exploring review types and associated information retrieval requirements' identifies 48 distinct review types, rapid reviews, systematic reviews, realist reviews, and highlights the expansion of methodological approaches.
The Women and Newborn Health Service Library’s guide 'What is a Systematic Review and how does it differ from a Literature Review?'.
For a systematic search of the evidence/literature review please follow the section in this guide on evidence searching to support research/writing for publication.
For a ‘true’ systematic review, this requires the gathering, collation and analysis of all the available evidence for a given research topic and is governed by the afore-mentioned internationally recognised guidance which we will come to shortly. The topic normally will be looking at some type of intervention and its effectiveness.
Due to the scale of the work a systematic review often takes a number of months, and your help and advice may be sought at different stages of the review over this period.
Review protocols
A review protocol is written prior to beginning work on the review proper. Protocols are registered on the International Prospective Register of Systematic Reviews (PROSPERO) database, unless the requester is writing a review for Cochrane in which case it is submitted to Cochrane and from there automatically launched onto PROSPERO. You should advise the requester to submit their protocol to Prospero, if they are unaware.
Internationally recognised guidance
You shoud be familiar with the following lead sources of internationally recognised guidance governing systematic reviews. A systematic review must follow this guidance.
The Cochrane Handbook and PRISMA essentially give the same instructions but complement and overlap each other. PRISMA provides a concise checklist and is a good place to start for those new to the process.
There is also PRISMA-S which governs how the ‘searches supporting the review’ are recorded. This is of key relevance to the information professional when recording search activity.
Search strategy
Develop a search strategy as part of your planning and create a draft strategy on Medline initially.
Scoping searches and formulating research questions
Has your requester performed a scoping search and is the research question fully formulated?
If not, they may want you to undertake some scoping searches to help finalise the question, see what research has already been carried out and if a recently published review already exists on this topic.
This will give them a sense of how much research is ‘out there’, how many results they can expect to screen and will help to set timescales to manage their project.
If they have carried out some preliminary searches it is useful to see the strategies to help you understand the search topic and its scope. If they are aware of relevant published studies on the topic, ask them to send these to you to inform your search.
Is guidance needed to finalise the review question? If yes, you should refer requester to Chapter 2 of the Cochrane Handbook.
Writing the protocol
If they have finalised the review question and a scoping search has been undertaken, have they written their protocol?
If yes, you need to see this as it details the inclusion and exclusion criteria for the search in addition to giving you a clearly defined research question.
If no, check they are aware of the following 4 areas of guidance which will help them write their protocol:
Is this the requester’s first systematic review?
If yes, regarding the methodology and the reporting of review findings, the PRISMA guidance is very easy to follow and is endorsed by Cochrane.
Refer new systematic reviewers to the PRISMA checklist if a quick overview of the whole process would be helpful.
Databases, resources and filters
The databases, resources and filters that you use will depend on the question and subject area. A question that is clinical in nature will require biomedical databases such as Medline, Embase and CINAHL.
A topic relating to mental health will need to include databases with a mental health focus such as PsycINFO and PsycArticles.
See the Cochrane Handbook Chapter 4, section 4.3.1.1 for further guidance.
The Cochrane Handbook also states that Medline, Embase, and the Cochrane Library’s CENTRAL, are the recommended minimum for database coverage.
Publication bias
To avoid publication bias it is also important to search clinical trial registries. This will help to identify trials that may have been stopped, withdrawn or the results were not published.
Search clinical trials registers for in-progress trials, including (but not limited-to):
Citaion searches
It is usual to also perform a citation search to identify additional relevant studies. This involves searching for those studies that have been cited by relevant articles.
The relevant articles can be those which have been selected from the search results already obtained or could be from the articles identified as relevant by the researcher at the very beginning of the review.
Citation searches can be carried out using citation indexes such as Scopus, if available. Google Scholar and PubMed also permit limited citation searching through their “cited by” or “citations” links. However, these may refer to online publication only.
You should also search preprint servers including Europe PubMed Central (PMC).
For further resources, the following sites may also be useful.
Search filters
You should also consider using appropriate search filters if your search is looking for a particular study design or population.
The InterTASC Information Specialists' Sub-Group (ISSG) Search Filters Resource is a vast resource of published search filters designed to retrieve research by study design or focus.
If you include a search filter, you must report your decision to do so in your search methods.
Once complete you should send this to the reviewer along with a sample of the results, such as the first 50 references.
- Are they as expected?
- Are further search terms to be added or removed from the strategy?
You should include with your draft the definitions used in the scope notes of indexing terms for information, this can help the requester.
Remember to stress that for the search to be truly comprehensive they will see a significant number of references that are not on target. These will need to be identified and discarded as part of their sift.
Execution
When executing your search strategy, it is important to remember the following.
Searching databases
The database searches must be broad in scope.
It is important that search strategies are very sensitive/broad to not miss any relevant research. They should contain a combination of:
- subject headings
- free text search terms
Indexing terms are often exploded.
Look in the MeSH hierarchy to see which terms fall below your chosen term as this will determine whether it is appropriate explode your chosen term to include these.
For free text search terms, take into consideration synonyms and plurals by truncating terms and include spelling variations such as American and British English spellings.
Relevant floating subheadings and adjacency operators can also extend your coverage.
See the list of Medline and Embase subheadings.
Selecting search terms
Select a test database to test your search terms.
This would normally be Medline as it has broad coverage.
If your topic is very specialised and there is a key database for this, you may also wish to run a test search on this as well.
Run searches for specific terms related to your topic and make a note of relevant search terms that arise.
Check the indexing term’s relevancy by clicking on the term to see the Scope note.
You may also find the “Used for” list within the scope note handy for identifying other text words to include in your strategy.
Useful text mining software can also help to identify relevant search terms. These include:
- PubMed PubReMiner
- Vos Viewer
- Mesh on Demand
- Yale MeSH Analyzer
- Medline Transpose
- Litsearchr
- Swift-Review
- Polyglot Search Translator
- nowGlobe
- Citationchaser
- RCT Tagger
- Connected Papers
Comprehensiveness
The aim is not to miss any potentially relevant studies that are appropriate to the review question as this could weaken or invalidate any findings.
The search should maximize the retrieval or “recall” of articles over the “relevancy” of articles retrieved.
It is usual therefore when scanning through the results of the search that you will see quite a high number of references that are not so on target.
From this broad set of data, the researcher will make their selection of studies to be included in the review, discarding during this process the studies that are not relevant.
This approach helps to reduce the chance of important articles being missed.
Read the guidance on comprehensiveness in the Cochrane Handbook.
Language and publication bias
Language and publication bias can creep in when limiting the parameters of the search.
Language bias
To avoid language bias results should not be limited to English.
Publication bias
To avoid publication bias conference abstracts should not be excluded from the database results.
Conference abstracts give information on the latest work which is yet to be formally published and helps to identify authors or research establishments worth contacting. They extend the search beyond what is available electronically.
Grey literature
In addition to following-up on conference abstracts, grey literature should also be searched.
The following helpful guidance is available:
Population/Problem, Intervention, Comparison, Outcomes (PICO)
Search strategies should ideally follow a structured framework such as the PICO (Population/Problem, Intervention, Comparison, Outcomes) structure.
However, if PICO is not a natural fit for the topic, other structures such as SPIDER and ECLIPS(E) are available.
For more on these, see Andrew Booth's Alternative Question Structures for Different types of systematic review.
Read Chapter 4 of the Cochrane Handbook for Systematic Reviews of Interventions covering designing the search strategy or the MECIR manual.
The PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement can be useful as it provides a list of what elements the search strategy should have to be robust.
Documentation and recording the searches
The key guidance for reporting the searches is PRISMA-S.
To be compliant you will need to do the following.
- Document all strategies. The requester needs to provide details of these both in the review write up and in the protocol submission.
- Document the details of the number of records retrieved for each database search prior to the removal of duplicates.
- Save all strategies on the database servers. The requester will need re-runs of these prior to completing the review and writing it up as several months will have normally passed during this process.
- Keep a record of the search statements used for grey literature searches, if it's not possible to save online.
- Keep a back-up file of all the search strategies used for easy reference.
- Write up the search methods for publication as the paper will need to include a paragraph on the methods uses to conduct the search strategy.
Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic? covers issues relating to searching and recording search activity on resources outside academic bibliographical databases.
It is vital that accurate records are kept to inform this paragraph. The paragraph should include:
- where you searched
- how you derived the search terms (did you identify relevant terms from relevant studies)
- whether you used text mining software such as PubMed PubReMiner or Vos Viewer
- whether you got a colleague to peer review your search for you
It is also worth including a sentence that the search was developed and carried out by a librarian/information professional. Co-authorship may be appropriate where you have made a substantial contribution to the research paper.
Results
Reference management
You should save the results in the .ris file format, which is compatible with most reference management software.
If you have organisational or site-wide access to reference management software you may have ‘organisation specific’ arrangements for supplying the results such as:
- loading the results directly into the software
- removing duplicates
- giving the requester the file details to access the results directly
NHS England provides access to Refworks for NHS Knowledge and Library Services (KLS) staff, and can grant access to non-KLS research-active staff like your requester.
Find out more information about reference management software.
If the requester is based off-site, or no site package nor organisational access is available to them, they will still be able to remove the duplicates in whichever reference management software they have chosen.
Many reference management software applications allow you to upload .ris files into online libraries that you can share with users.
Systematic review software
You may wish to use systematic review software packages to aid the process of screening and data extraction.
Two popular packages are the following.
- Rayyan which is freely available to all.
- Covidence which is freely available to those writing a Cochrane review.
Find out further information and read a comparison of Rayyan and Covidence.
PRISMA
For the reporting of results in systematic reviews the requester will need to follow the PRISMA flow diagram.
You will need to provide the requester with the total number of records retrieved prior to the removal of duplicates as this is required information for the PRISMA flow diagram.
Examples
Example 1
Search for Systematic Review on “Identifying the risk of seeding to the abdominal wall when placing PEGs in head and neck cancer patients search strategy”.
Librarian’s narrative description/explanation of search strategy.
Search methods for identification of studies
Searches were developed and carried out by an Evidence and Knowledge Specialist on:
- MEDLINE
- Embase and Emcare via Ovid,
- Cumulative Index to Nursing and Allied Health (CINAHL) via EBSCO
- Cochrane Database of Systematic Reviews, Cochrane Clinical Answers
- Cochrane Central Register of Controlled Trials in The Cochrane Library
- Turning Research into Practice (Trip)
- ClinicalTrials.gov
- World Health Organization (WHO) Internationa l Clinical Trials Registry Platform
Studies were restricted to studies published in English language and no publication status restrictions were applied.
The search was developed using a combination of database-specific indexing and free text terms for head and neck cancer and percutaneous Endoscopic Gastrostomy (PEG) and seeding.
The search terms were derived from internal discussions, examining the free text and indexing terms of relevant studies and PubMed Reminer.
The results were de-duplicated and managed using RefWorks reference management software. See Supplementary material/ appendix for the complete search strategy.
The protocol for this review was registered on PROSPERO (reference).”
Search strategies
The search steps included in the strategy for each database are included. For example:
Ovid MEDLINE(R) ALL 1946 to June 30, 2022:
- "Head and Neck Neoplasms"/ 61645 [I didn’t explode this term as it would have included thyroid and parathyroid which the reviewer said is not considered ‘head and neck cancer’ in the UK]
- "Squamous cell carcinoma of head and neck"/ 9431
- exp mouth neoplasms/ 4575
- exp otorhinolaryngologic neoplasms/ 93745
- exp Odontogenic Tumors/ 8492 ...
Total pre-duplication: 1538
Total post-duplication: 1029
Example 2
Request confirmation
Request confirmed via email, after a face-to-face meeting with requester.
Subject: RE: review
Good to see you just now…, just to confirm, as per your protocol, here’s what we’re looking for:
Refractory angina + all types of treatment (including: stems cells, any interventional treatment - especially novel interventions, spinal cord stimulation, laser revascularisation, nerve stimulation, extracorporeal shockwaves, PCI options)
All types of paper (not just trials/meta-analyses/reviews)
No date limits
Adults only (no paeds)
No language/geographic limitations
I’ll use the meta-analyses you brought in as guides and will aim to have an Endnote library compiled for you by the end of next week.
I’ll keep track of numbers for PRISMA diagram and send you a narrative explanation of the search strategy also. The search will inclusive/broad, so there will be a big yield, and likely a high proportion of irrelevant articles that I will leave for you to screen from the final set of results.
Thanks and all the best…”
Covering email from librarian
Dear …
With apologies for the delay caused by technical difficulties, and thanks for your patience, please find attached bibex and RIS files of your complete deduplicated search results. I’ve also shared my Endnote web groups below with you.
I’ve tested the search and it retrieved all the papers that you had identified in your scoping search previously but if you find it has missed any then let me know and I’ll revise the strategy. I’ll get some narrative of the search to you hopefully this afternoon or if not then first-thing next week.
Here are the numbers:
- www.clinicaltrials.gov results: 30
- Cochrane Library results: 246
- Embase results: 1611
- Medline results: 1825
- Total results: 3712
- Total unique results after deduplication: 2659
After exporting the combined results to Mendeley, which automatically merges exact duplicate records that it identifies, there were 2815, but the set still contained duplicates. After using Mendeley’s “find duplicate” tool to get rid of any remaining duplicates there were 2664. After manually screening again for duplicates there are 2659.
Re: the strategies used:
In clinicaltrials.gov I searched using the terms “refractory angina”, “intractable angina”, “intractable angina”, “drug resistant angina” and “symptomatic coronary artery disease” (screen grabs attached). Search conducted on XX/XX/20XX.
In the Cochrane library I used a strategy identical to the Medline strategy below (screen grab attached). Search conducted on XX/XX/20XX.
In Embase and Medline I used the following strategy:
- EMBASE (refractory angina).ti,ab 1186
- EMBASE (symptomatic coronary arter* disease*).ti,ab 651
- EMBASE (intract?ble angina).ti,ab 221
- EMBASE (drug resist* angina).ti,ab 6
- EMBASE exp "ANGINA PECTORIS"/ 92235 ...
Searches conducted on XX/XX/20XX using NICE’s HDAS interface.
Resources
Systematic reviews
These resources are suggested for searches requiring information for systematic reviews.
Useful further information and a comparison of Rayyan and Covidence, systematic review software packages to aid the process of screening and data extraction, can be found at the following product review.
Use te below resources if you need further help structuring a systematic review search strategy.
Text mining software
The below resources are for systematic reviews which require text mining.
Page last reviewed: 29 August 2023
Next review due: 29 August 2024