Systematic review support NURS41000&70120
Support for your systematic review
Introduction to the support
For most of you, this will be your first time conducting a systematic review, though you’ll likely be familiar with many of its elements through your own academic work and reading academic literature.
This package of support, created in partnership with the Library, is intended as an independent learning resource; something that you can come to as you recognise the need for it. The content will cover the many different aspects of conducting a systematic review and will follow an order in which many people will carry out the research. You’ll probably find that conducting a systematic review is rarely a linear process and these resources are here for you to return to, however you conduct your research.
Further support will be available from the Library throughout. You will be directed to the relevant content in the first instance before a follow-up with a librarian if necessary.
Although it isn’t easy, conducting a systematic review is a great way to gain a deeper understanding of the research landscape of a particular topic. We hope that you find this content useful and wish you success in carrying out this research! We appreciate your feedback, so please do comment on what you find useful (or less useful).
What is a systematic review?
Introduction
A systematic review is a kind of tightly structured literature review that focuses on a topic with strict research parameters. The methodology used to collect research has to be consistent in order to reduce misinterpretation and misrepresentation of the data.
Systematic reviews are measured against a set of specific criteria outlined at the start to help you achieve this: a well-defined topic; a clear statement of what is in and out of scope; and how you will go about collecting the evidence you will use.
It can be helpful to think about systematic reviews as you would any other research method. You need to be able to report on what you have done in a way that would allow your research to be repeated. Keeping this in mind can help you to make sense of many of the processes involved.
What is the aim of a systematic review?
The aim of a systematic review is to identify, analyse and appraise the published research on your topic before arriving at a considered judgement or set of conclusions based on that research, all while adhering to the review’s predetermined criteria.
Why do we conduct systematic reviews?
The purpose of following this very strict protocol is to gather evidence-based research that supports a balanced and unbiased conclusion.
Recording exactly how you found the research you use in your review, and where you found it, helps to show how you got to your conclusions. Each piece of research material is examined and compared to other similar studies, resources and summarised accordingly. Some people keep records in a form or table summarising each reviewed article which they refer to when collating the evidence.
Summary
Transcript: A systematic allows you to take a broader and deeper look at a topic than you could achieve by reading one or two studies. Treat it as you would any other research method by reporting on what you have done with the aim of it being repeatable by other researchers. Your final review is only as good as the research you include in it, so take care at each stage.
Refining your topic and planning your review
Introduction
In order to conduct a robust systematic review, you need to be sure how your information resources work. To do this, you could ask yourself:
- How familiar are you with the range of search resources at your disposal?
- How familiar are you with the functionality within your chosen resources?
- Are you sure you have the right tools for the job?
- Are you sure you know the rules for Boolean to ensure they work correctly?
Have a look at the Nursing Subject Guide if you are unfamiliar with the range of resources available for your research area. This gathers the most used resources for each subject on one page. You may also want to look at other subject pages if your research area spans across multiple disciplines. For example, if you’re systematic review was looking at a topic within ‘nursing education’ you might choose an education database as part of your mix. We’ll look at this in more detail in ‘Choosing your databases’.
How I get started?
Scoping searches are a good place to start your systematic review. Initial early scoping searches may influence the future directions of your research. Scoping searches help you discover what’s out there and which search terms are likely to yield results that are useful to you and which you may have to modify. After your initial searches you may wish to revisit and tweak your parameters.
- Write down all the points you need to consider:
- Look at the scope of your study. What’s your main focus/plan/goal?
- What could you compromise on?
- What’s been done before in your research area?
- What about gaps in research? How would you address these?
Refining your topic
Video: WIN_20220921_17_53_06_Pro.mp4
Transcript: Having viewed the Question Development slides and adhering to the principles outlined there, you may have begun formulating your question. You you may however need to refine it. You can do this by conducting some initial scoping searches to find out how much research is published on your topic.
If you find too many papers it will be difficult to pick out the high quality research to include in your systematic review. If you find too few papers you won’t be able to draw out any themes within the research and will have little to write about.
Refining your topic means being more specific, or general, in order to obtain a manageable amount of research to screen and analyse.
Let me give you an example. Searching for the impact Covid-19 has had on hospital staff may give me over 10000 results — too many to manage in the time I have. I may experiment, changing my population to look at only nurses, or only UK hospital staff. I would document what I have found and discuss it with my supervisor before making any decision.
Talk to your supervisor
You may find that your scoping review returns many thousands of results (too many) or hardly any results at all (too few). If either is the case you should speak to your supervisor.
Summary
Becoming familiar with the information resources available to you and conducting quick scoping searches on your topic can save you a lot of time in the long run. As with the rest of your review, make sure you note how many results you get in your scoping review as this will help you and your supervisor discuss any refinements that need to be made to your research question.
Turning your topic into a ‘search strategy’
Introduction
Transcript: Your search strategy will determine the research you find in your searches and that you include in your review, so it’s important that you get it right! Coming up with an effective search strategy isn’t a linear process. You should spend time thinking about how other researchers might describe the ideas you are interested in; conduct some scoping searches where you experiment with different terms; read some of the relevant papers you find and use the terminology in them to improve your strategy. You’ll finally come up with a strategy that can be applied effectively across your chosen databases.
Once you have agreed your research question with your supervisor the next step is to turn it into something searchable — a ‘search strategy’. This is where you take each key idea from your research question and consider how else that term might be described.
Why do you need to create a search strategy?
We are all used to searching with Google and its clever algorithms. Google can guess what we are searching for based on the millions of occasions users have previously searched using similar terms. If I search for ‘trainers’ while looking for new shoes, it will return results for ‘sneakers’ (and importantly, not ‘teachers’ or ‘coaches’).
Most scholarly subject databases work by the opposite principles. Rather than ‘guessing’ what you are searching for it gives you complete control — it will only search for the things we tell it to. This means that if there’s more than one way to describe an idea, we must tell the database to search for each term! This process of thinking about all the ways an idea can be described, combine with our use of the special functionality of these databases, creates our ‘search strategy’.
Example
In this section, we will take you through the different steps of creating a search strategy. The research question we will use for this is: “What are the COVID-19 related risks for hospital workers?”
Search Terms
The first step to begin your review is to think about what you are searching for. At this stage, you should consider the important ideas in your research question and the terminology that may have been used in the research you are looking for. Your search terms are important because they determine the range of results you get from each resource you search.
In our research question our important ideas are ‘COVID-19’ and ‘hospital workers’. We would include both of these as search terms.
At this point it is also useful to think of synonyms for these search terms. For example, an alternative for COVID-19 would be ‘Coronavirus’ and ‘SARS-CoV-2’. for hospital workers we could also use ‘doctors’, ‘nurses’ or ‘hospital porters’. I’m sure you can think of more terms we could use for both of these important ideas!
Scoping Strategy
After you have considered the keywords and search terms that will be useful. The next step is to conduct a scoping search. This is an initial search where you test out some of your search terms and see what sort of results you get. Scoping searches give you an idea of what research is out there (or not). They are particularly useful if you are researching a new area or phenomenon (as we are in this example!)
Searching Systematically
After you have tested the viability of your research question with a scoping search, you will now need to think about how you will search systematically to ensure you are finding all the research on this topic. Alongside the search terms we have already thought about, we have two very useful functions for conducting a systematic search:
- Medical Subject Headings (MeSH)
- Boolean operators
MeSH terms
MeSH, sometimes referred to as ‘subject headings’ are similar to a thesaurus and help us to capture papers that are about a particular even if we use different terminology. Let’s use a non-medical example: if I wrote a paper mentioning ‘rising temperatures across the world’ and you wrote a paper mentioning ‘global warming’ then both would be labelled with the subject heading ‘climate change’. Selecting the subject heading ‘climate change’ would help us to capture papers that use a range of terminology relating to that topic, which is incredibly useful.
Databases on the Ovid platform, such as Medline, Embase, and Pyscinfo will try to find a matching subject heading when you search for your keyword. It can be helpful to start with one of these databases. You can find out more about databases functionality in the ‘Choosing your databases’ post.
Searching for the keywords and synonyms that you thought of in your scoping search is a good place to start and can be combined with subject headings, where they are available. You’ll find subject headings for most of your ‘health sciences’ terms and many general terms. You may find that some terms don’t match to subject heading — in these instances you would just search as a keyword. Subject heading options are not available on all database platforms and you’ll also find that subject headings differ between databases. You can find out more in the ‘Replicating your search across database platforms’ post.
Using our example and typing ‘Covid-19’ into the search box we are given a subject heading option:
Click the ‘Scope’ button to see what is covered by the subject heading — particularly useful if you have a number of potential options.
You can find out more about MeSH in the Library’s Making use of MeSH and Suggested Subject Terms blog post.
Boolean operators
Boolean operators are the ‘OR’, ‘AND’, and occasionally ‘NOT’ you see on some database platforms and are the way that we can combine our search terms.
They can help broaden or narrow your search depending upon what you need and give you more control over the literature you will find:
- OR broadens a search term
- AND narrows a search term
- NOT excludes a search term
Returning to our example we might combine our related terms — Covid-19, Coronavirus, SARS-CoV-2 with OR:
Select the terms by checking the boxes on the left and click OR below:
You will now see a new line with those terms combined:
You will now see a new line with those terms combined.
We used AND to combine separate concepts. Here we combine all of the terms to do with ‘Covid-19’ (line 4 of the search) with everything to do with ‘hospital workers’ (line 9):
You will now see a new line with those searches combined:
A word of warning if using the NOT operator — it will remove all results containing that term. Returning to our example, we may only be interested in the Covid-19 risks for nurses. If we searched for ‘NOT doctors’ we would remove any results to do with doctors — including those that are to do with both ‘doctors’ and ‘nurses’! Think carefully before using this operator, if at all.
You can find out more about Boolean operators in the Library’s ‘Using operators in your search’ resource.
Summary
Your search strategy is hugely important and worth spending time on — thinking; scoping; and revising. Once you are happy with your search strategy you can start to think about the using MeSH to get the most out of your chosen databases, and combining your terms using Boolean operators.
Defining your inclusion and exclusion criteria
Introduction
Your inclusion and exclusion criteria work alongside your search strategy to define which papers are included in your systematic review. They should be decided upon after agreeing upon and refining your research question but before you conduct your systematic searches. You define your criteria at this stage to avoid the biases inherent in choosing which papers are included and which are excluded after you have begun analysing them.
Creating your criteria
Your inclusion and exclusion criteria are derived from your research question and not picked arbitrarily. You may choose to only include quantitative research in order to provide a particular kind of analysis — not so that you avoid the task of analysing messy qualitative accounts! When choosing your criteria think about how you would justify those decisions in relation to your research question and the nature of your research. It might be reasonable to exclude all papers published before a certain date if something changed at this time (treatment; legislation; etc.) but not because it would give you fewer papers to review. You might choose to limit to include only papers published in English as it would not be possible to translate papers during the short period in which you complete your research.
Common types of inclusion and exclusion criteria
Population
Who are you interested in? Who are the study’s participants?
Intervention
What should the participants have experienced or been exposed to? What treatment have should have they received or not received?
Date
When was the paper published?
Type of study
Did the study collect quantitative, qualitative, or mixed data? How was the data collected? See the ‘Critical appraisal’ post for more on study design.
Type of publication
Was the paper published and peer reviewed? Will you include review papers or only original research? Will you include conference papers or letters?
Language
Will you include only papers in specified languages?
Summary
Your inclusion and exclusion criteria are the boundaries you place around your review — so define them carefully! Make sure that you can justify your choice of criteria and include them in your review methodology.
Choosing your databases
Introduction
Conducting a systematic review search is very different to conducting a search for an essay, because you are required to document clearly and in detail how you have searched the literature. For more information on the difference between systematic reviews and other types of assignment see our earlier post. The additional requirements of a systematic review mean that you need to think carefully about which tools you use to carry out your review. You need your searches to:
- be repeatable
- capture the range of perspectives on your topic
- aim to provide a comprehensive overview of your topic
What not to use: Library Search and Google Scholar
For previous assignments you may have used Library Search or Google Scholar to carry out your searches.
Library Search allows you to search everything in the Library Catalogue, knowing you will be able to access everything you find if you are logged into your University account. Google Scholar is great for ease of use and when you want to quickly explore what has been written about a particular subject.
Both are unsuitable for use in a systematic review for several reasons.
Number of results and manageability
Library Search searches across all the library’s subscriptions and collections; Google Scholar offers few options to refine your search and is difficult to search in a systematic way. Both mean you are likely to end up with a vast number of results which are un-manageable when you come to review them.
Repeatability
Searches in Library Search and Google Scholar are not reliably repeatable because the results you see are influenced by your individual search history or personalised settings. This means that neither is an appropriate tool for carrying out a systematic review as the results will not necessarily be the same even if you are repeating the same search. For these reasons we recommend using an appropriate subject database for systematic reviews.
Subject databases
As your systematic review will require you to screen a large number of papers within a clearly defined topic area, you will need to choose powerful and specific search tools which allow you to search in a systematic way using database functionality like subject headings, Boolean operators and filters.
A matter of perspectives
You should consider the research disciplines that may have a perspective on your topic beyond nursing — is there a medical sciences, pharmacological, psychological sciences or other healthcare element to your topic? Or even relevant perspectives from outside of the health sciences? Identifying these perspectives can help you to choose the most appropriate databases for your topic.
Where to find and access subject databases
You can find relevant subject databases using the Library’s Subject Guides and specifically, the Nursing Subject Guide. Each subject guide includes a curated list of subject specific databases along with a summary of their coverage — you can use this information to help you choose the best ones for your research. Where Google Scholar draws results from everywhere, subject specific databases tailor their coverage to a particular professional audience. For example, the CINAHL database is aimed at nurses and Medline is designed for all medical professionals.
Platforms and databases
A platform is the tool you use to search databases. For example, Ovid is the platform that you would use to search the Medline database.
Some platforms allow you to search multiple databases and some databases are available on more than one platform, so it is important in your work that you report both the database and the platform you used to access it.
Ovid Medline or PubMed?
Medline is an example of a database that can be accessed on several platforms, including Ovid and PubMed. You need to decide which to use — each has its advantages.
Ovid Medline: better for systematic searching and so recommended by the Library for systematic review research. You will also likely use Ovid to access Embase and/or PsycINFO — using Ovid Medline reduces the need to learn how to use another platform.
PubMed: a platform you are likely to use throughout your practice (and likely a platform that your supervisor is familiar with).
Key databases and why you might use them
Medline is one of the most important biomedical databases and a great place to start. It is produced by the National Library of Medicine (USA) and covers the contents of over 3,000 medical, dental and nursing journals. Its MeSH (subject heading) descriptions are informative and great to help you to create your search strategy.
CINAHL Plus provides indexing for 3,802 journals from the fields of nursing and allied health. Using CINAHL can help you to provide a broad healthcare perspective on a topic.
Similar coverage to Medline but with additional strength in pharmacological sciences (so great if there is a ‘drugs’ element to your topic). You’ll frequently see both Medline and Embase featured in the methodology of published systematic reviews.
Produced by the American Psychological Association and drawing on over 1300 psychological sciences journals, PsycINFO is a good addition if your topic has a psychological element, such as ‘behaviours’ or ‘attitudes’.
Both are huge multidisciplinary databases that allow you to conduct a systematic search (note, neither use subject headings). They are also great for reference and citation searching making it easy to access a given paper’s references and all of the papers that in turn reference it. These databases are sometimes used in systematic reviews to capture relevant papers published outside of the health sciences. You would only need to use one and should think carefully about your use of them — they often give a lot of irrelevant results in your search to find one or two relevant papers.
Summary
A systematic review’s strength rests on the quality of the articles included and choosing the right databases is an important step in writing an excellent review. Your choice should take account of the different perspectives on your topic and aim to comprehensively search for the literature relevant to your topic.
You can find out more about using these databases and platforms in the ‘How to use databases platforms’ post.
How to use databases platforms
Introduction
Before you use these database platforms you should have spent some time thinking about you search strategy and the databases that are best suited to your topic.
These guides are produced by the Library and look at the platform rather than the database. So, the ‘Introduction to the Ovid platform using PsycINFO’ post will help you with Ovid Medline and Embase.
You will notice that different platforms work in slightly different ways The ‘Replicating your search across database platforms’ post will help you to understand what to do when conducting your search in different databases.
Introduction to the Ovid platform using PsycINFO
Replicating your search across database platforms
Introduction
It’s a good idea to apply your search strategy to a single database before moving onto others. Once you are confident that you have used appropriate terms and subject headings (MeSH), and combined them correctly using Boolean operators, you can move onto replicating that search strategy in different databases. I recommend starting in Ovid Medline which is great for conducting a systematic search.
Your aim when replicating your searches is to be true to your search strategy while working within the limitations of different databases. Replicating your search exactly between different databases is rarely possible!
Databases that are on the same platform
I like to start by first replicating my search strategy on a database that is on the same platform. If I first conducted my search in Ovid Medline, I might next replicate it in Embase, which is also accessed through the Ovid platform. The way that you enter and combine terms will be the same. The main difference is with the subject headings. The subject heading for the same idea may be different. For example, you would use ‘COVID-19/’ in Ovid Medline and ‘coronavirus disease 2019/’ in Embase. Sometimes an idea might have a subject heading in one database but not another. For example, if we were looking at a surgical procedure like a tracheostomy, in Ovid Medline we would use ‘Tracheostomy/’. There is no equivalent subject heading in Psycinfo — we would search as a keyword instead. You can find out more in the ‘Turning your topic into a ‘search strategy’’ post.
Databases that are on a different platform
It can be a little trickier replicating your search strategy on a database that is on a different platform. With the above example I might next move onto CINAHL, on the EBSCO platform. My advice here is to use the ‘search history’ function to make these other database platforms look like the Ovid platform. From there you can make combinations of search lines in a similar way.
Non-health sciences databases
Some of you might choose to use a database from another scholarly area, or a large multi-disciplinary database like Web of Science or Scopus. You will not find subject headings in these databases. Think about searching as a keyword, just as you would where a subject heading did not exist in a database.
Summary
Replicating searches in different databases means applying your search strategy as closely as possible while using the features available in each database. Have a look at the ‘How to use databases platforms’ to get the best out of the databases you use.
Saving your searches
Introduction
Applying your search strategy to a database and replicating that search across other databases is rarely a linear process. Saving your searches allows you to come back to your search to make amendments or retrieve results. Each platform does this differently. The Library’s guides to saving your searches take you through the process, from creating an account to saving your searches in each platform. You can find out more about databases and platforms in the ‘Choosing your databases’ post.
How to save your search in OVID
Reporting on your search and screening results with the PRISMA Flow Diagram
Introduction
Reporting on your search and screening results is a fundamental part of any systematic review. It helps to evidence the validity of your review and enable it to be repeated by another researcher — the same qualities expected of any research method. The PRISMA Flow Diagram is a straightforward way to ensure you report on the findings at each stage of your searching and screening process. The ‘PRISMA 2020 flow diagram for new systematic reviews which included searches of databases and registers only’ is the most appropriate version for most systematic reviews.
What to report and when
Search results
TK text boxes used originally to look like the PRISMA reporting — can each of the below bits in quotes be put into boxes in Blackboard? Is that possible?
After conducting your search on your chosen databases, you need to report on the number of results from each database:
Records identified from:
Ovid Medline: 356
Embase: 762
PsycINFO: 261
CINAHL: 83
Web of Science: 793
Total with duplicates: 2255
Removal of duplicates
As databases overlap in their coverage, you will find that your results contain papers that appeared in two or more of your chosen databases — duplicates. You do not want to have to screen the same papers twice and you need to report on the number of duplicate papers. Reference management tools like EndNote can help with this.
Records removed before screening:
Duplicate records: 164
Total (with duplicates removed: 2091
Screening
You are now ready to apply your inclusion and exclusion criteria to your results to ascertain which papers will be included in your review:
Records screened: 2091
Records sought for retrieval: 28
Records excluded: 2063
You might also report on the reasons for not including papers in your review (ie, the reason for excluding them):
Review papers: 291
Sample size: 578
Population: 847
Other: 347
Retrieving papers
In our example above we are left with 28 papers that have met our criteria and are taken forward to be included in the review. You can expect the Library to have access to the majority of these results and for most other you can request access using the Order an Article service. On occasion, you may find some results unobtainable, or it may not be practical to request access in the timescales of your project. In these cases, report on the number of records that were not retrieved:
Records not retrieved: 2
Records included in review: 26
These papers will form the basis of your systematic review write-up.
Summary
Approaching a systematic review in the same way you would approach any research methodology can help you to understand when to report, and why. As the papers included in your review form the foundation of the points you make in your review, it is important to show how you got those results, at every stage.
Reference management tools for systematic reviews
Introduction
A systematic review requires you to manage the bibliographic data for hundreds, if not thousands of scholarly papers. Reference management tools are ideally suited to this task, with the bonus of helping you generate citations and bibliographies when you come to write up your review. In this post we’ll explain how reference management tools can help you throughout your review and share some tips on how to get the most out of the functionality. We’ll be using EndNote desktop for our examples but you’ll find much of the functionality in other tools. We would strongly recommend using EndNote 20 which can be downloaded via IT Services and is available on all campus PCs.
If you are already using EndNote 20 or another tool do note that you need to have a new or empty library for your review results.
Organising your results
Organising the results you find from each database helps you to report your findings. The easiest way to do this is to create Groups for each of the databases you have used and to import your results from the database to the appropriate Group.
Creating group sets and groups in EndNote desktop
EndNote organises your records at two levels — Groups and Group Sets. First, create a group set and name it Systematic Review
To create a group set:
Groups > Create Group Set
Next, with your Systematic Review group set selected, create groups for each of the databases you used:
Groups > Create group
The new group will now appear in your group set — give it the name of the database, e.g., Medline.
When you import your results from each database you will notice that the group shows the number of records it contains — making reporting easy. Make sure that you report on the number of results from each database before removing duplicates.
You should end up with something that looks like this:
Removing duplicates
Different databases often have overlapping coverage — making duplicate results a likelihood. You don’t want to include these in your final numbers when reporting, and you certainly don’t want to have to screen the same paper twice!
Removing duplicates in EndNote desktop
With the All references group selected:
Library > Find Duplicates
A dialogue box will pop up allowing you to choose which record to keep and which to discard. If you have lots of duplicates it is easier to click Cancel in the dialogue box then press Delete on your keyboard to remove all duplicates. Make sure you have reported the number of unique results before screening.
Screening your results
Specialist tools like Covidence and Rayyan are great when you are screening results as part of a research team. Reference management tools are perfectly adequate when screening results on your own.
Screening your results in EndNote
Following the same process as above create a group set and name it Screening, and within the group set create three groups — Accept, Maybe, Reject
Select the All References folder and move the record into the appropriate folder based on your inclusion/exclusion criteria. You may find having a Maybe folder useful as you find yourself getting into a flow: a mindset where you are clear on your criteria and can see from the title and/or abstract whether a paper will be accepted or rejected. Anything you are unsure about gets put in the Maybe folder which you come back to later.
You might want to organise your rejected papers by the reasons for their rejection, based on your inclusion/exclusion criteria. You could create folders for this in a similar way — e.g., Reject: qualitative; Reject: review paper, etc. You would only do this if you intend or are required to report on the reasons for rejecting papers. Do note that the screening will take longer. You can now report on the final number of papers included in your systematic review.
Your groups should look something like this:
Summary
EndNote is an excellent tool for keeping track of the literature you will use in your systematic review. It makes it easy to report on your results, remove duplicates and even screen. You can find out more about using EndNote in the Library’s ‘Getting started with EndNote 20 desktop’ post.
Critical appraisal
Introduction
Like many aspects of a systematic review, critical appraisal asks you to take something that you already do and apply it in a more systematic way. In this case you will evaluate the information sources you find with a systematic approach and often using an appraisal tool or checklist.
What is critical appraisal?
You need to be able to critically appraise research literature to be sure that your clinical practice is based upon the best available research evidence.
Critical appraisal has been defined as:
“The process of assessing and interpreting evidence by systematically considering its validity, results and relevance to an individual’s own clinical work.” (Last, 1988)
Most papers published in medical, healthcare and dental journals follow the IMRAD format (Greenhalgh,1997):
- Introduction
- Methods
- Results
- Discussion
Most papers also have an abstract at the beginning, which gives an overview of the key elements of each section. An important first step when critically appraising a paper is to consider your expectations of each section and its purpose.
Abstract
The abstract provides a summary of a paper. It can be useful when deciding whether a study is relevant. However, it should not be used to make judgements concerning the validity of the research or results. It is estimated that 18–68% of medical journal abstracts contain omissions or inaccuracies (Pitkin, 1999).
Introduction
The introduction should identify gaps in current knowledge in the area that the paper investigates. It should also outline the aim of the study. As with the abstract, an introduction is useful to establish the relevance of a study, but it will not tell you anything about the validity of the study or what the results are.
Methods
The methods section outlines how the study was conducted. If you have established the topic of the paper is relevant, the methods section will enable you to identify the validity of the study. This can indicate whether the paper is worth reading.
Results
The results section will document the findings of the study. This section will be of interest if you have previously identified from the abstract and methods sections that the study is relevant and valid.
Discussion/Conclusion
The discussion or conclusion section does not always represent the actual findings of the study. For your own research, it is important that you focus on the methods and results sections and think about what the findings mean to you and your practice.
Types of study and appropriate critical appraisal tools
Before you start to appraise a research paper, it is helpful to know what type of study it is. Different study types have their own strengths and weaknesses. Being able to identify the type of study can give you an overall indication of the quality of the research.
Subject databases such as Medline may include the publication type in the Complete Reference entry for the article. However, sometimes this is not listed, and you will need to consult the research paper to find this.
There are several critical appraisal checklists available online to help you with appraising distinct types of study. Examples of these include CASP, CEBM and AMSTAR and you may be recommended others by your supervisor. Within these you will notice different checklists for different study types. Sometimes a research paper may use more than one method of research and it is important to pick the correct tool for the research type. You might discuss your choice of critical appraisal tools with your supervisor.
Hierarchy of evidence
The hierarchy of evidence can be used as a general indication of the quality of a piece of research. There is no universally accepted hierarchy, and there are many variations of the hierarchical pyramid. The diagram below represents an accepted illustration of the relative strengths of a few of the key types of study. Note that you may already have specified study types as part of your inclusion criteria, making this less important.
Summary
Your first attempt at critical appraisal may feel slow, but with practice it becomes much quicker and almost automatic. It is important that researchers conducting a systematic review and indeed anyone wanting to engage in evidence-based practice can quickly identify high-quality, clinically relevant research articles to inform their work. By developing critical appraisal skills, you will become better at managing information overload and able to find valid and relevant literature more easily.
Note-making
As you plan and start to carry out your systematic review, making effective notes will help you to document the process you have followed and identify any points for discussion. You will need to demonstrate your inclusion and exclusion criteria, how you arrived at these and any changes you made during your review. Recording these decisions as you make them will help you identify any themes that arise in your work and will help you when you come to write up your systematic review. For more information, please see our resources on strategies for effective note making.
Referencing refresher
Referencing is a vital part of academic writing and understanding the principles of good referencing practice will help you to manage the large amounts of bibliographic data used in a systematic review. The Library provides online resources, workshops and drop-in sessions covering all aspects of referencing, details of which can be found in the Library’s referencing guide.
If you need a refresher on how and why we reference you’ll find the Library’s ‘Getting started with referencing’ resource useful.
Reference management tools can help you with many aspects of conducting a systematic review, which we’ll look at in more detail in the ‘EndNote 20 for systematic reviews’ post. If you are new to EndNote you might want to check out ‘Getting started with EndNote 20 desktop’.
Help and support from the Library
The Library can provide support in all of the aspects covered in these pages. There are several ways to get further support:
Contact us:
Email: uml.teachingandlearning@manchester.ac.uk
Library Chat via the Library homepage:
https://www.library.manchester.ac.uk/
‘Ask a University of Manchester Librarian for help’ on the Ovid platform:
and EBSCO platform:
Attend a drop in:
Library drop in (everything from searching to academic writing)
Referencing drop in (from basic referencing to using reference management software)
Skills training sessions
We’ll be running sessions on ‘Searching Systematically’ and ‘Writing Together’ in semester 1 with dates to be announced. You can also sign up to any of our open sessions on searching, referencing and other academic skills.
Feedback
We love to hear your feedback. You can speak to your supervisor or email us directly at uml.teachingandlearning@manchester.ac.uk
