.
Tove Faber Frandsen1, Christina Louise Lindhardt2, Mette Brandt Eriksen3
doi: http://dx.doi.org/10.5195/jmla.2021.1150
Volume 109, Number 3: 388-394
Received 10 2020: Accepted 03 2021
ABSTRACT
Objective:A growing volume of studies address methods for performing systematic reviews of qualitative studies. One such methodological aspect is the conceptual framework used to structure the review question and plan the search strategy for locating relevant studies. The purpose of this case study was to evaluate the retrieval potential of each element of conceptual frameworks in qualitative systematic reviews in the health sciences.
Methods:The presence of elements from conceptual frameworks in publication titles, abstracts, and controlled vocabulary in CINAHL and PubMed was analyzed using a set of qualitative reviews and their included studies as a gold standard. Using a sample of 101 publications, we determined whether particular publications could be retrieved if a specific element from the conceptual framework was used in the search strategy.
Results:We found that the relative recall of conceptual framework elements varied considerably, with higher recall for patient/population (99%) and research type (97%) and lower recall for intervention/phenomenon of interest (74%), outcome (79%), and context (61%).
Conclusion:The use of patient/population and research type elements had high relative recall for qualitative studies. However, other elements should be used with great care due to lower relative recall.
Keywords: systematic searching; qualitative reviews; PubMed; CINAHL; conceptualization models.
A qualitative systematic review or qualitative evidence synthesis provides answers or gains a deeper understanding of the what, how, or why of a phenomenon [1]. Qualitative systematic reviews focus on summarizing, analyzing, and/or interpreting. The number of qualitative systematic reviews and evidence syntheses has increased dramatically during the last two decades, with many different approaches in existence [2, 3].
The increase in the number of qualitative studies within health care leads to a growing volume of studies addressing methods of systematically reviewing, integrating, and synthesizing findings from original qualitative studies. The Cochrane Qualitative and Implementation Methods Group is continuously involved in developing guidance on the synthesis of qualitative and mixed-method implementation evidence [4–8]. Recent studies addressing methods for systematically reviewing and synthesizing findings include those on the choice of database [9], choice of search strategy [10], and use of search filters to limit to qualitative literature [11, 12].
How to structure review questions has received a great deal of attention. Within clinical research, the population-intervention-comparison-outcome (PICO) model is often used to structure the review question, although other conceptual frameworks are available that are better suited for specific review types [13]. For qualitative systematic reviews, PICO may not be adequate [14], and thus models have been specifically developed for this review type [15–17]. A review from 2016 lists as many as twelve different models for structuring the qualitative review question and recommends that “in the absence of empirical data on effectiveness of structured approaches, the question structure should be selected to match the purpose and focus on the review” [3].
Two organizations are dedicated to strengthening conceptual frameworks and guidelines for conducting qualitative systematic reviews: Cochrane Collaboration and Joanna Briggs Institute (JBI), both of which are recognized as producers of systematic reviews that undergo rigorous scrutiny [18, 19]. Cochrane argues for the strength of simple question models but also recommends the use of a formulation that includes context [20] as stressed in earlier studies [21, 22]. Thus, Cochrane recommends the use of an extended model named PerSPEcTiF [20], which consists of the following elements: perspective, setting, phenomenon of interest/problem, environment, comparison (optional), time/timing, and findings [23]. JBI recommends population, phenomena of interest, and context (PICO) and argues that there is no need for an outcome statement in qualitative synthesis, as the expression of the phenomenon of interest represents the outcome [24].
A conceptual framework is also recommended for developing search strategies [25], although not all elements provide high recall [26]. The effect of the choice of model on search quality remains inconclusive [27]. One should keep in mind that when using a conceptual framework consisting of many elements (e.g., SPIDER or PerSPEcTiF), using all elements in the search strategy will, in practice, reduce recall [16]. Consequently, a search strategy should include only some elements, but little evidence is available to inform that process. Cochrane generally recommends omitting outcome from the search strategy, which is supported by a recent case study [26]. However, in systematic qualitative reviews, outcome is not necessarily included in the conceptual framework and therefore not necessarily relevant to consider for the search strategy. Furthermore, little is known about the retrieval potential of elements in conceptual frameworks available for systematic qualitative reviews.
Essentially, qualitative reviewers need to know how to locate relevant studies, which implies knowing which databases to search and how to search them. The present study contributes to this knowledge base by examining the impact of conceptual frameworks developed for qualitative reviews on retrieval. The purpose of this case study is to evaluate the retrieval potential of each element of conceptual frameworks in qualitative systematic reviews in health sciences.
To investigate the presence of elements from conceptual frameworks in publication titles, abstracts, and controlled vocabulary, we used a set of qualitative systematic reviews identified in a study of database coverage of qualitative health literature [9] as our starting point and analyzed whether the studies included in the reviews would be retrieved if a reviewer used the elements specified in the review to perform a thorough search for relevant studies. We defined a “thorough search for relevant studies” as one that uses synonyms as well as narrower terms as both text words and subject headings. Search terms were defined in this study using the thesauri available in PubMed and CINAHL and were found using entry terms in PubMed as well as narrower terms in both thesauri. We did not actually perform the searches but instead examined whether the studies would be retrieved if using the elements specified in the review as search terms.
We followed the methods used in a recent study that evaluated the retrieval potential of each element of the PICO model used in the fields of upper gastrointestinal/pancreatic diseases and pregnancy/childbirth [26]. This previous study used a sample of Cochrane reviews to determine whether carefully constructed searches for population, intervention, comparison, or outcome elements could retrieve the EMBASE or PubMed records of studies meeting the inclusion criteria of the specific review. The relative recall of each PICO element was determined using the articles judged relevant for inclusion in the reviews [26]. The previous study found high retrieval rates for all PICO elements except the outcome element, supporting the existing recommendation not to search for outcome-related terms. In the present study, we adapted these methods to investigate conceptual frameworks used for qualitative systematic reviews. Specifically, we used 71 qualitative reviews (65 from JBI and 6 from Cochrane) published from 2013 through 2017 that included a total of 927 qualitative studies [9]. From these 71 reviews, we randomly selected reviews to achieve a minimum of a 10% sample of included studies using the true random number generator at Random.org. An overview of the 12 selected reviews is available in Appendix 1. To enable comparisons between databases, only studies included in these reviews that were indexed in both PubMed and CINAHL were included in the present study. Consequently, from a total of 210 studies included in the 12 reviews, we included 101 studies that were indexed in both PubMed and CINAHL.
We focused on PubMed and CINAHL, as they are commonly recommended among health sciences researchers searching for qualitative research. Furthermore, in a study of the coverage of qualitative health research in bibliographic databases, a combination of PubMed and CINAHL retrieved 82% of publications [9]. Although the previous study also found that a combination of Scopus and CINAHL or Scopus and Proquest Dissertations and Theses had higher retrieval rates (about 89%), Scopus does not have a controlled vocabulary of its own, whereas the aim of the present study was to investigate whether elements of conceptual frameworks are present in publication titles, abstracts, or controlled vocabulary.
For each of the 101 included studies, data were recorded using a template containing the following information:
We determined relative recall based on whether an included study could be retrieved if a particular element from the conceptual framework was used in the search strategy. A particular study was considered retrievable if at least one of the search terms for an element in the conceptual framework was identified in the bibliographic record, which we noted in the data as “1.” If we were not able to identify any of the search terms for an element, we noted this as “0” in the data.
To describe the data collection process in further detail, a review in this study that presents a synthesis of qualitative evidence of parents' and informal caregivers' views and experiences of communication about routine childhood vaccination can serve as an example [28]. The following elements are described in the review: patient/population, intervention/phenomenon of interest, second intervention/phenomenon of interest, and research type. One of the studies included in this review is on parents' choices and rationales for alternative vaccination schedules [29]. In the PubMed and CINAHL records for this study, the interventions are not described in the abstract or subject headings, meaning that a thorough search using the intervention element would not retrieve this study, whereas it would be retrievable using patient/population and research type elements.
Data were extracted by an experienced information specialist with a background in information science (Tove Faber Frandsen), and extracted data were validated by the two other authors: an experienced information specialist with a background in biomedicine (Mette Brandt Eriksen) and a nurse researcher with extensive qualitative research experience (Christina Louise Lindhardt). Any conflicts were resolved by consensus. Data were extracted from February to May 2020, and validation took place from May to September 2020. The data extraction form is available in Appendix 2.
The reviews included in this study used varying conceptual frameworks. All reviews included patient/population, intervention/phenomenon of interest, and research type. A few reviews included two different elements describing intervention/phenomenon of interest. Most reviews included either outcome or context but not both elements.
Table 1 provides an overview of the relative recall for each element in PubMed and CINAHL using either text words or subject headings. Relative recall was high for some elements. Relative recall for patient/population was high due to the frequent presence of text words describing this element; it was 97.0% for text words in both databases but was lower for subject headings: 88.1% and 84.2% in PubMed and CINAHL, respectively. Relative recall for research type was 86.1% and 85.1% for text words in PubMed and CINAHL, respectively, but differed dramatically between databases for subject headings: 51.5% in PubMed and 94.1% in CINAHL. Relative recall for the remaining elements were all below 70%. Some reviews included a second element describing intervention/phenomenon of interest, and relative recall in these cases was very low (14% for text words and 7% for subject headings in both databases).
Table 1Relative recall of conceptual framework elements using text words or subject headings in PubMed and CINAHL
| Conceptual framework element | PubMed Text words (%) | CINAHL Text words (%) | PubMed MeSH terms (%) | CINAHL Subject headings (%) |
|---|---|---|---|---|
| Patient/population | 97.0 | 97.0 | 88.1 | 84.2 |
| Intervention/phenomenon of interest | 65.3 | 64.4 | 40.6 | 38.6 |
| Second intervention/phenomenon of interest | 14.3 | 14.3 | 7.1 | 7.1 |
| Research type | 86.1 | 85.1 | 51.5 | 94.1 |
| Outcome | 60.7 | 58.9 | 35.7 | 39.3 |
| Context | 54.8 | 51.6 | 48.4 | 45.2 |
Table 1 shows differences between databases as well as between the use of text words or subject headings. However, current recommendations for systematic searches stress the importance of using several databases to retrieve relevant literature as well as using both text words and subject headings [30, 31]. Thus, Table 2 provides an overview of the relative recall for each element in PubMed and CINAHL using text words and subject headings together as well as for both databases combined.
Table 2Relative recall of conceptual framework elements using both text words and subject headings in PubMed, CINAHL, and both databases combined
| Conceptual framework element | PubMed Text words + MeSH terms (%) | CINAHL Text words + subject headings (%) | PubMed and CINAHL combined Text words + subject headings (%) |
|---|---|---|---|
| Patient/population | 99.0 | 98.0 | 99.0 |
| Intervention/phenomenon of interest | 72.3 | 70.3 | 75.3 |
| Second intervention/phenomenon of interest | 14.3 | 14.3 | 16.7 |
| Research type | 89.1 | 97.0 | 97.0 |
| Outcome | 75.0 | 73.2 | 78.6 |
| Context | 61.3 | 54.8 | 61.3 |
Relative recall for patient/population was high in both databases (99% in PubMed and 98% in CINAHL); using both databases, 99% of studies would be retrieved. Relative recall for research type was also high: 89.1% in PubMed, 97% in CINAHL, and 97% for both databases. Using text words and subject headings together, relative recall for intervention/phenomenon of interest and outcome increased to 75.3% and 78.6% for both databases respectively. Relative recall for context and the second intervention/phenomenon of interest was lower.
Elements of conceptual frameworks are typically used in combination, and adding more elements to a search strategy can lead to lower retrieval. Table 3 provides an overview of retrieval associated with various combinations of elements. Patient/population and research type, which were the elements with the highest relative recall, provided the highest recall when combined. However, adding intervention/phenomenon of interest, context, or outcome considerably reduced recall.
Table 3Recall of combinations of conceptual framework elements using both text words and subject headings in PubMed and CINAHL combined
| Conceptual framework element combinations | PubMed and CINAHL combined Text words + subject headings (%) |
|---|---|
| Patient/population, research type | 96.0 |
| Patient/population, research type, outcome | 76.8 |
| Patient/population, research type, intervention/phenomenon of interest | 73.3 |
| Patient/population, research type, context | 58.1 |
In summary, when searching PubMed and CINAHL, relative recall was high for patient/population and research type elements. Somewhat lower relative recall, but still above 75%, was found for intervention/phenomenon of interest and outcome elements. Finally, relative recall was lowest for context and the second intervention/phenomenon of interest element. When combining elements, the highest recall was found for a combination of patient/population and research type.
The results of this study show that typical elements of conceptual frameworks appearing in the titles, abstracts, or controlled vocabulary of qualitative systematic reviews are patient/population, intervention/phenomenon of interest, and research type. Other elements used are a second intervention/phenomenon of interest, outcome, and context. The latter two were not used in combination.
The relative recall of these elements varied considerably. Assuming that existing guidelines are followed and thus both PubMed and CINAHL are searched using text words and subject headings, we found high relative recall for patient/population and research type (99% and 97%, respectively). The relative recall of intervention/phenomenon of interest was 74.3%, which was similar to that of outcome (78.6%). Finally, the relative recall for context was relatively low (61.3%).
A recent study that examined the effect of using the PICO model to develop search strategies for quantitative reviews showed that the relative recall of PICO elements was high (100% for population, 100% for intervention, and no less than 90% for comparison and outcome) [26]. Considering qualitative reviews, however, we found that the relative recall of elements in conceptualization frameworks was much lower. The findings of this study suggest that searching the literature for a qualitative review requires careful planning and maybe even the use of several strategies to compensate for the lower recall of many of the elements of conceptual frameworks. If possible, patient/population and research type should be included in the search strategy, although the latter should only be included if the search is also performed in CINAHL. The remaining elements should only be used in the search strategy with care. As the recall associated with some elements is relatively low, search strategies for qualitative literature using conceptualization tools for PubMed and CINAHL should be supplemented with other strategies for finding relevant literature.
Although it indexes a relatively small number of journals [32], CINAHL is often recommended for finding qualitative studies. CINAHL can yield a high number of relevant studies, including those not available in other databases [9, 33]. Based on the findings of the present study, CINAHL should be considered for qualitative systematic reviews due to its high-quality indexing of research types. Furthermore, the challenges of finding relevant literature for qualitative systematic reviews are well known [3] and include a lack of indexing of qualitative methods [14]. However, the present findings indicate that the relative recall of research type is 97% when searching CINAHL.
Although conceptual frameworks can be used to plan the search strategy for a systematic review, some of their elements may not provide a satisfactory level of recall. When planning qualitative reviews, reviewers should be aware of low relative recall when searching for interventions/phenomenons of interest, outcome, and context. Higher relative recall is found when searching for patients/population and research type, although no elements provide 100% recall. Consequently, additional approaches should be considered when locating relevant qualitative studies. In the process of developing search strategies, it should also be taken into consideration that the relative recall of elements depends on the indexing policies of the databases searched.
Potential limitations of this study must also be considered. First, this is a case study, which has limited generalizability. Few empirical studies of search processes for finding qualitative studies exist [14, 34]. Thus, this study aimed to substantiate the assumption that when using a conceptual framework for qualitative reviews, some of its elements will result in higher recall than other elements. More research is needed to substantiate these findings. Second, we defined elements according to a selection of qualitative reviews published by Cochrane and JBI. The studies included in these reviews may not necessarily define the elements of the conceptualization framework identically to those in the review. For example, an included study could define the population more narrowly than the review, which would have an effect on recall. In this case, we chose to take the perspective of a systematic reviewer who develops a search strategy using elements from the conceptualization framework. Consequently, we used reviews' definitions of conceptual framework elements. Furthermore, we used reviews from Cochrane and JBI because they follow a set of strictly established guidelines and undergo rigorous scrutiny [35–37], whereas other reviews do not necessarily have the same characteristics. Third, we did not address subfield differences. Qualitative synthesis approaches include a wide range of methodologies including meta-ethnographies, thematic analyses, and narrative syntheses [38], for which elements of conceptual frameworks may have different relative recall. Fourth, we did not reproduce search strategies in the qualitative reviews and therefore do not know how they located the included studies. Hand-searching, snowballing, searching additional databases, and using other strategies may compensate for low database retrieval. Finally, we made subjective assessments of synonyms for text words and subjects headings. However, as qualitative concepts are complex, the use of varying synonyms and definitions can make it difficult to definitively determine relative recall. We tried to take this into account by validating the extracted data twice, which confirmed that although the assessments were subjective, the process was reproducible.
In conclusion, the results of this case study show that the relative recall of conceptual framework elements varies considerably, suggesting a need to consider existing recommendations regarding the development of search strategies for qualitative systematic reviews. Taking the limitations of this study into account, it appears that the elements of conceptual frameworks developed for qualitative reviews have lower relative recall than the original PICO elements. This suggests that qualitative systematic reviewers must plan searches carefully, as the use of some elements can lower recall considerably. Although patient/population and research type had high recall, other elements should be used with great care or perhaps in different combinations with patient/population and research type. Furthermore, this study confirms the previously noted success of using CINAHL to search for qualitative literature using a conceptual framework.
All data associated with this article are available in the Open Science Framework: http://dx.doi.org/http://doi.org/10.5281/zenodo.4548725.
Appendix 1: Qualitative systematic reviews used as golden standard in this study
Appendix 2: Data extraction template
1. Green J, Thorogood N. Qualitative methods for health research. Sage; 2018. ISBN 1526448807.
2. Campbell F, Weeks L, Booth A, Kaunelis D, Smith A. A scoping review found increasing examples of rapid qualitative evidence syntheses
and no methodological guidance. J Clin Epidemiol. 2019 Nov;115:160–71. DOI: http://dx.doi.org/10.1016/j.jclinepi.2019.05.032.
3. Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured
methodological review. Syst Rev. 2016 May 04;5:74. DOI: http://dx.doi.org/10.1186/s13643-016-0249-x.
4. Noyes J, Booth A, Flemming K, Garside R, Harden A, Lewin S, Pantoja T, Hannes K, Cargo M, Thomas J. Cochrane Qualitative and Implementation Methods Group guidance paper 3: methods for
assessing methodological limitations, data extraction and synthesis, and confidence
in synthesized qualitative findings. J Clin Epidemiol. 2018 May;97:49–58. DOI: http://dx.doi.org/10.1016/j.jclinepi.2017.06.020.
5. Noyes J, Booth A, Cargo M, Flemming K, Garside R, Hannes K, Harden A, Harris J, Lewin S, Pantoja T, Thomas J. Cochrane Qualitative and Implementation Methods Group guidance series-paper 1: introduction. J Clin Epidemiol. 2018 May;97:35–38. DOI: http://dx.doi.org/10.1016/j.jclinepi.2017.09.025.
6. Harris JL, Booth A, Cargo M, Hannes K, Harden A, Flemming K, Garside R, Pantoja T, Thomas J, Noyes J. Cochrane Qualitative and Implementation Methods Group guidance paper 2: methods for
question formulation, searching, and protocol development for qualitative evidence
synthesis. J Clin Epidemiol. 2018 May;97:39–48. DOI: http://dx.doi.org/10.1016/j.jclinepi.2017.10.023.
7. Harden A, Thomas J, Cargo M, Harris J, Pantoja T, Flemming K, Booth A, Garside R, Hannes K, Noyes J. Cochrane Qualitative and Implementation Methods Group guidance paper 5: methods for
integrating qualitative and implementation evidence within intervention effectiveness
reviews. J Clin Epidemiol. 2018 May;97:70–78. DOI: http://dx.doi.org/10.1016/j.jclinepi.2017.11.029.
8. Flemming K, Booth A, Hannes K, Cargo M, Noyes J. Cochrane Qualitative and Implementation Methods Group guidance paper 6: reporting
guidelines for qualitative, implementation, and process evaluation evidence syntheses. J Clin Epidemiol. 2018 May;97:79–85. DOI: http://dx.doi.org/10.1016/j.jclinepi.2017.10.022.
9. Frandsen TF, Gildberg FA, Tingleff EB. Searching for qualitative health research required several databases and alternative
search strategies: a study of coverage in bibliographic databases. J Clin Epidemiol. 2019 Oct;114:118–24. DOI: http://dx.doi.org/10.1016/j.jclinepi.2019.06.013.
10. Rogers M, Bethel A, Abbott R. Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO:
A comparison of search strategies. Res Synth Methods. 2018 Dec;9(4):579–86. DOI: http://dx.doi.org/10.1002/jrsm.1280.
11. Wong SS, Wilczynski NL, Haynes RB, Hedges T. Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE. Stud Health Technol Inform. 2004;107(Pt 1):311–6.
12. Wagner M, Rosumeck S, Kuffmeier C, Doring K, Euler U. A validation study revealed differences in design and performance of MEDLINE search
filters for qualitative research. J Clin Epidemiol. 2020 Apr;120:17–24. DOI: http://dx.doi.org/10.1016/j.jclinepi.2019.12.008.
13. Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z. What kind of systematic review should I conduct? A proposed typology and guidance
for systematic reviewers in the medical and health sciences. BMC Med Res Methodol. 2018 Jan 10;18(1):5. DOI: http://dx.doi.org/10.1186/s12874-017-0468-4.
14. Saini M, Shlonsky A. Systematic synthesis of qualitative research (Pocket guide to social work research methods). New York: Oxford University Press; 2012. p. xiv, 208.
15. Cooke A, Smith D, Booth A. Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual Health Res. 2012 Oct;22(10):1435–43. DOI: http://dx.doi.org/10.1177/1049732312452938.
16. Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three
search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579. DOI: http://dx.doi.org/10.1186/s12913-014-0579-0.
17. Booth A. Clear and present questions: formulating questions for evidence based practice. Libr Hi Tech. 2006;24(3):355–68. DOI: http://dx.doi.org/10.1108/07378830610692127.
18. Pearson A. Evidence-based review in policy and practice, an issue of nursing clinics. E-book. Elsevier Health Sciences; 2014. ISBN 0323326633.
19. Dosenovic S, Jelicic Kadic A, Vucic K, Markovina N, Pieper D, Puljak L. Comparison of methodological quality rating of systematic reviews on neuropathic pain
using AMSTAR and R-AMSTAR. BMC Med Res Methodol. 2018 May 8;18(1):37. DOI: http://dx.doi.org/10.1186/s12874-018-0493-y.
20. Noyes J, Booth A, Cargo M, Flemming K, Harden A, Harris J, Garside R, Hannes K, Pantoja T, Thomas J. Qualitative evidence. In: Higgins JPT, Thomas J, Chandler J, Cumptson M, Li T, Page MJ, Welch VA, eds. Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. <www.training.cochrane.org/handbook>.
21. Booth A, Moore G, Flemming K, Garside R, Rollins N, Tuncalp O, Noyes J. Taking account of context in systematic reviews and guidelines considering a complexity
perspective. BMJ Glob Health. 2019;4(Suppl 1):e000840. DOI: http://dx.doi.org/10.1136/bmjgh-2018-000840.
22. Petticrew M, Knai C, Thomas J, Rehfuess EA, Noyes J, Gerhardus A, Grimshaw JM, Rutter H, McGill E. Implications of a complexity perspective for systematic reviews and guideline development
in health decision making. BMJ Glob Health. 2019;4(Suppl 1):e000899. DOI: http://dx.doi.org/10.1136/bmjgh-2018-000899.
23. Booth A, Noyes J, Flemming K, Moore G, Tunçalp Ö, Shakibazadeh E. Formulating questions to explore complex interventions within qualitative evidence
synthesis. BMJ Global Health. 2019;4(Suppl 1):e001107. DOI: http://dx.doi.org/10.1136/bmjgh-2018-001107.
24. Aromataris E, Munn Z, eds. Joanna Briggs Institute reviewer's manual [Internet]. The Joanna Briggs Institute; 2017. p. 299. Available from <https://reviewersmanual.joannabriggs.org/>.
25. Considine J, Shaban RZ, Fry M, Curtis K. Evidence based emergency nursing: designing a research question and searching the
literature. Int Emerg Nurs. 2017 May;32:78–82. DOI: http://dx.doi.org/10.1016/j.ienj.2017.02.001.
26. Frandsen TF, Bruun Nielsen MF, Lindhardt CL, Eriksen MB. Using the full PICO model as a search tool for systematic reviews resulted in lower
recall for some PICO elements. J Clin Epidemiol. 2020;127:69–75. DOI: http://dx.doi.org/10.1016/j.jclinepi.2020.07.005.
27. Eriksen MB, Frandsen TF. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy
tool on literature search quality: a systematic review. J Med Libr Assoc. 2018;106(4):420–31. DOI: http://dx.doi.org/10.5195/jmla.2018.345.
28. Ames HM, Glenton C, Lewin S. Parents' and informal caregivers' views and experiences of communication about routine
childhood vaccination: a synthesis of qualitative evidence. Cochrane Database Syst Rev. 2017;2(2):CD0111787. DOI: http://dx.doi.org/10.1002/14651858.cd011787.pub2.
29. Saada A, Lieu TA, Morain SR, Zikmund-Fisher BJ, Wittenberg E. Parents' choices and rationales for alternative vaccination schedules: a qualitative
study. Clin Pediatr (Phila). 2015;54(3):236–43. DOI: http://dx.doi.org/10.1177/0009922814548838.
30. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page ML, Welch VA, eds. Cochrane handbook for systematic reviews of interventions. Version 6.0 (updated July 2019). Cochrane; 2019 [rev. Jul 2019]. Available from <www.training.cochrane.org/handbook>.
31. McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016 Jul;75:40–6. DOI: http://dx.doi.org/10.1016/j.jclinepi.2016.01.021.
32. Behrend D. CINAHL complete. The Charleston Advisor. 2020;22(2):26–9.
33. Wright K, Golder S, Lewis-Light K. What value is the CINAHL database when searching for systematic reviews of qualitative
studies? Syst Rev. 2015;4:104. DOI: http://dx.doi.org/10.1186/s13643-015-0069-4.
34. Mohammed MA, Moles RJ, Chen TF. Meta-synthesis of qualitative research: the challenges and opportunities. Int J Clin Pharm. 2016;38(3):695–704. DOI: http://dx.doi.org/10.1007/s11096-016-0289-2.
35. Moseley AM, Elkins MR, Herbert RD, Maher CG, Sherrington C. Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic
reviews in physiotherapy. J Clin Epidemiol. 2009;62(10):1021–30. DOI: http://dx.doi.org/10.1016/j.jclinepi.2008.09.018.
36. Biocic M, Fidahic M, Cikes K, Puljak L. Comparison of information sources used in Cochrane and non-Cochrane systematic reviews:
A case study in the field of anesthesiology and pain. Res Synth Methods. 2019;10(4):597–605. DOI: http://dx.doi.org/10.1002/jrsm.1375.
37. Gholizadeh M, Amir-Behghadami M, Janati A. Systematic reviews: are they actually well conducted and reported in accordance with
PRISMA? Bull Emerg Trauma. 2020;8(1):51. DOI: http://dx.doi.org/10.29252/beat-080110.
38. Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J. 2009;26(2):91–108. DOI: http://dx.doi.org/10.1111/j.1471-1842.2009.00848.x.
Tove Faber Frandsen, 1 t.faber@sdu.dk, Professor WSR, University of Southern Denmark, Denmark
Christina Louise Lindhardt, 2 clindhardt@health.sdu.dk, Department of Geriatric Medicine, Odense University Hospital, Odense Denmark. Department of Clinical Institute, University of Southern, Odense, Denmark
Mette Brandt Eriksen, 3 mettee@bib.sdu.dk, University of Southern Denmark, Denmark
Copyright © 2021 Tove Faber Frandsen, Christina Louise Lindhardt, Mette Brandt Eriksen
This work is licensed under a Creative Commons Attribution 4.0 International License.
Journal of the Medical Library Association, VOLUME 109, NUMBER 3, July 2021