Healthcare Policy
The Use of Research Evidence in Two International Organizations' Recommendations about Health Systems
Abstract
Background: Little is known about the extent to which research evidence informs the development of recommendations by international organizations.
Methods: We identified specific World Health Organization (WHO) and World Bank recommendations on five topics (contracting, healthcare financing, health human resources, tuberculosis control and tobacco control), catalogued the related systematic reviews and assessed the recommendations to determine their consistency with the systematic reviews that were available at the time of their formulation.
Findings: Only two of the eight publications examined were found to cite systematic reviews, and only five of 14 WHO and two of seven World Bank recommendations were consistent with both the direction and nature of effect claims from systematic reviews. Ten of 14 WHO and five of seven World Bank recommendations were consistent with the direction of effect claims only.
Conclusion: WHO and the World Bank - working with donor agencies and national governments - can improve their use of (or at least, their reporting about their use of) research evidence. Decision-makers and clinicians should critically evaluate the quality and local applicability of recommendations from any source, including international organizations, prior to their implementation.
The importance of linking research evidence to action has been well established (WHO 2004a; Haines et al. 2004). This linkage, however, is particularly essential for health systems in low- and middle-income countries (Commission on Health Research for Development 1990). Health system limitations and fragmentation have been described as a "bottleneck" that slows the full implementation of existing interventions (Travis et al. 2004; WHO 2005a). Just one package of interventions, if fully implemented, has been estimated to have the potential to reduce child mortality by two-thirds and maternal mortality by three-quarters (Jones et al. 2003; World Bank 2004). Yet, many studies have reinforced the view that policy making about health systems is often not informed by research evidence (Aaserud et al. 2005; Lush et al. 2003; Ogden et al. 2003). The need to develop mechanisms to support policy makers' use of health policy and systems research has been widely acknowledged (WHO 2005a; Lavis et al. 2004; Lavis, Davies et al. 2006; Lavis, Lomas et al. 2006), and a number of country- and region-level initiatives have been launched to address this need (Hamid et al. 2005; East African Community 2006).
The recommendations about health systems that are formulated by international organizations like the World Health Organization (WHO) and the World Bank have the potential to serve as important mediators between the best available research evidence and policy for the many low- and middle-income countries that rely on both the recommendations for technical guidance and the financial support that often accompanies a commitment to follow the recommendations (Oxman et al. 2006). Indeed, policy makers would have a much more valuable resource on which to draw in national policy making processes if international organizations were to use (among other information sources) systematic reviews of effects - the best available synthesis of global research evidence about the likely effects of different policy options - as a starting point for their deliberations and to report whether, how and why their recommendations are consistent with the direction and nature of effect claims made in these reviews (Lavis, Lomas et al. 2006; Oxman and Guyatt 2002). Yet, despite the value of systematic reviews and the practical efficiencies associated with their use (as highlighted over the past five years by WHO's "Guidelines for WHO Guidelines" [2003], World Report on Knowledge for Better Health [2004a], Task Force on Health Systems Research [2005] and Advisory Committee on Health Research [2006]), two recently published studies have revealed that systematic reviews (among other types of research evidence) are not widely used within at least one international organization - WHO (Oxman et al. 2007; Nahar Kabir and Holmgren 2005).
This study is the first of its kind to systematically compare health systems recommendations by two prominent international organizations - WHO and the World Bank - to the research evidence that was available at the time of their formulation. The overall goal was to contribute to international efforts aiming to link research to action by supporting the development of evidence-informed recommendations by international organizations that focus, at least in part, on strengthening health systems in resource-poor settings.
This study was approved by the McMaster University Faculty of Health Sciences/Hamilton Health Sciences Research Ethics Board in Hamilton, Ontario, Canada.
Methods
We examined the use of research evidence in health systems recommendations by developing a series of inventories that facilitated the purposive sampling of two international organizations, five health topics, 10 relevant publications (two per topic) and 30 recommendations (three per publication) based on explicit selection criteria (Table 1), and comparing the chosen recommendations to the nature and direction of effect claims made in systematic reviews compiled specifically for this purpose.
We selected WHO and the World Bank for this study because they are two of the largest and most prominent international organizations that operate in the health field. In addition to their work at the country level, both organizations strive to stimulate the dissemination and use of research evidence by articulating evidence-informed policy options, offering technical support and publishing hundreds of guidelines and reports each year (WHO 2006a; World Bank 2006).
We identified health topics by reviewing all resolutions of the World Health Assembly (WHA) that were adopted between 2000 and 2003 (a period that provides sufficient time for countries to act), as they often reflect the priorities of the global health community. Resolutions were catalogued based on their applicability to different country contexts (i.e., low- and middle-income, high-income and a combination of both); one specific resolution and corresponding health topic were chosen for each of governance arrangements (WHA56.25/contracting), financial arrangements (WHA53.14/healthcare financing) and delivery arrangements (WHA54.12/health human resources). One resolution and health topic were also chosen for each of clinical program content (WHA53.1/tuberculosis control) and population and public health program content (WHA56.1/tobacco control) to enable comparisons with the three health systems topic areas. These resolutions, however, were not compared to the research evidence in isolation, as they are declarative in nature and rarely contain technical guidance that could practically be compared to the available research evidence; rather, relevant WHO and World Bank recommendations-containing documents in these five health topic areas were then identified through a comprehensive search of their respective websites for major publications as well as complementary searches in their respective library catalogue systems.
| Table 1. Selection criteria for each stage of the study | |||
| Item | Target | Actual | Selection criteria |
| International organizations | 2 | 2 |
|
| Health topics addressed by the selected organizations | 5 | 5 |
|
| Publications produced on the selected health topics (1 per health topic from each organization) | 10 | 8 |
|
| Recommendations contained in the selected publications (3 per publication) | 30 | 21 |
|
One publication was then sought from each organization for each of the five topics through purposive sampling based on the selection criteria for publications; data were collected on each publication's number of pages, citation of any type of research evidence and citation of systematic reviews. Three central recommendations with effect claims (i.e., assertions about the likely impact of the intervention under consideration) were subsequently sought from each publication for a target of 30 recommendations across organizations and topics based on the availability of systematic reviews and a desire to compare WHO and World Bank recommendations on the same issue (Mucciaroni and Quirk 2006).1
The research evidence with which to compare the recommendations was subsequently compiled for each topic using existing overviews of systematic reviews on health financing (Lagarde and Palmer 2006), health human resources (Chopra et al. 2006), maternal and child health (Kakad and Oxman 2006) and from an ongoing comprehensive overview of systematic reviews of a range of governance, financial and delivery arrangements (Lavis et al. under review),2 as well as an update of each of these searches and new searches for tuberculosis and tobacco control on MEDLINE, CINAHL and EMBASE, using optimized search strategies specific for systematic reviews (Montori et al. 2005; Wong et al. 2006; Wilczynski et al. 2007). In instances where systematic reviews were found but were published after the relevant WHO or World Bank publication, the number and proportion of studies in the systematic review that were published one year prior to the recommendations-containing publication were recorded.
The systematic reviews were then assessed and coded based on whether the authors' effect claims indicated that the intervention under study works (achieves specific positive effects), doesn't work (fails to achieve specific positive effects or achieves negative effects), works in some contexts (achieves specific positive effects in some groups, jurisdictions or time periods but not others) or lacks enough high-quality research evidence to draw conclusions. This coding scheme facilitated an objective comparison by two independent reviewers of the effect claims of WHO and World Bank recommendations to those of the systematic reviews (or, in their absence, studies) that were available at the time of the recommendations' publication. The comparison of the effect claims was separated into two different assessments: (a) consistency in the direction of effect claims (i.e., whether research evidence supports use of the intervention) and (b) nature of the effect claims (i.e., whether research evidence supports the rationale for using the intervention). Where research evidence from systematic reviews existed at the time that recommendations were written and it was not utilized, an explanation for this discrepancy was sought within the publication.
Results
The search of the respective websites of WHO and the World Bank in the identified topic areas yielded 187 official documents from both organizations that were published between the 2003 release of "Guidelines for WHO Guidelines" (WHO 2003) and 2006. While a publication from WHO was selected for each of the five topics, no World Bank publications met the selection criteria for health human resources and tuberculosis control, mainly because the published documents were specific to a single country or region (Table 2). Four of the eight publications were books (de Beyer and Waverley 2003; Gottret and Schieber 2006; Harding and Preker 2003; WHO 2004b), two were technical briefs for policy makers (WHO 2005b,c), one was a set of guidelines for national governments (WHO 2004c) and one was a WHO world health report (WHO 2006b). All publications were featured prominently on the two organizations' respective websites and were made publicly available free of charge, except for one book that required a minimal payment for full access (Gottret and Schieber 2006).3
[Table 2]
Citation practices and the use of systematic reviews in these publications varied greatly across topics and between the two organizations. While all three World Bank publications used extensive citations, only two of the five WHO publications are referenced (i.e., the WHO world health report and the book on tobacco control): one of the other WHO documents cited research evidence rarely and the remaining two did not use referencing at all. Systematic reviews were cited by only two of the eight publications (i.e., one from each organization) and constituted eight of the 1,587 citations that were recorded in the six publications that referenced research evidence (see Table 2) (Buchan and Dal Poz 2002; Buchan et al. 2000; Coomarasamy and Khan 2004; Davis et al. 1995; Ekman 2004; Gosden et al. 2001; Hanson et al. 2001; Littlewood et al. 2005). The total count of citations, however, is artificially raised by the fact that six of the eight publications had end-of-chapter references that often overlapped.
The overviews and searches for additional systematic reviews on the five health topics resulted in the collection of 255 systematic reviews (including updates of systematic reviews), with five for contracting, 12 for healthcare financing, 93 for health human resources, 71 for tuberculosis control and 74 for tobacco control. This collection of systematic reviews consisted of this study's evidence base, which was compared to the recommendations contained in the selected publications.
A total of 14 WHO and seven World Bank recommendations from the eight publications were compared to the research evidence from systematic reviews that were available at the time of their formulation (Table 3) (Lagarde and Palmer 2006; Buchan and Dal Poz 2002; Coomarasamy and Khan 2004; Littlewood et al. 2005; Bordley et al. 2000; Chang et al. 2006; Fichtenberg and Glantz 2002; Gelband 2000, 2006; Grilli et al. 2002a,b; Holland et al. 2005; Horrocks et al. 2002; Jamtvedt et al. 2003; Jamtvedt et al. 2006a,b; Kaper et al. 2005; Laurant et al. 2004; Lexchin and Grootendorst 2004; Lovato et al. 2003; McAlister et al. 2004; Moher et al. 2003, 2005; Mwandumba and Squire 2000, 2001; Serra et al. 2000; Silagy et al. 2001, 2002, 2004; Sowden and Arblaster 2000; Stewart 2006; Thomson O'Brien et al. 2000; Veloski et al. 2006; Volmink and Garner 2000a,b, 2001, 2003, 2006; Volmink et al. 2000; Wellman et al. 2006; Zwarenstein and Bryant 2000). As evaluated by two independent reviewers with almost perfect agreement (kappa=0.95 [0.86, 1.04: p-value < 0.0005]), five of the 14 WHO and two of the seven World Bank recommendations were consistent with both the direction and nature of effect claims from systematic reviews; a total of 10 WHO and five World Bank recommendations were consistent with the direction of effect claims. Overall, consistency between recommendations and research evidence varied greatly across topic but not between organizations (with user fees in healthcare financing serving as an exception). Whereas every examined recommendation on health human resources and tobacco control was consistent with the direction of effect claims from the available research evidence (of which half were also consistent with the nature of effect claims), the same was not found for any of the tuberculosis control recommendations. While WHO and the World Bank provided contradictory recommendations on social insurance as a healthcare financing mechanism, the fact that no high-quality studies were found by the available systematic review meant that neither the direction nor nature of the effect claims for either recommendation were supported by research evidence. No meaningful patterns, however, emerged across health topics or organizations for the few recommendations that were found to be consistent with the specific nature of effect claims from the available research evidence.
[Table 3]
No explanation was found within any of the WHO or World Bank publications for the discrepancies between the recommendations and the existing research evidence from systematic reviews.
Discussion
Statement of principal findings
This study is the first to confirm previous hypotheses and demonstrate with evidence from purposively sampled recommendations-containing publications that systematic reviews are rarely cited by two prominent international organizations and are not consistently used (or at least reported as having been used and then weighed explicitly against competing social, political, economic or ethical considerations) in the development of their recommendations (Oxman et al. 2006). While differences can certainly be identified among the various health topics, overall there appears to be no clear rationale for the consistency between recommendations and research evidence that occurs with some health topics but not others. Neither the recommendations' date of publication nor the differences between health systems and program content recommendations appeared to explain the discrepancies. All publications appeared after the "Guidelines for WHO Guidelines" (WHO 2003), which emphasized the importance of systematic reviews, but before the creation of the WHO Guidelines Review Committee in May 2007 (WHO 2007), the development of the WHO Rapid Advice Guidelines (Schünemann et al. 2007), the introduction of continuing education opportunities for WHO staff in guideline development (Hill and Pang 2007) and the development of WHO's strategy on research for health (WHO 2008).
Strengths and weaknesses of the study
As a first attempt at systematically comparing health systems recommendations by two prominent international organizations to the research evidence that was available at the time of their formulation, the study has several strengths: (a) explicit and replicable sampling criteria were used at every stage of the recommendation-identification process and were consistently implemented by two reviewers with high inter-rater agreement; (b) existing overviews of systematic reviews and optimized search strategies were used to identify systematic reviews to compare against the recommendations; (c) comparisons were conducted both conservatively in terms of the direction of effects and more strictly in terms of the nature of effects; and (d) a mix of health topics was chosen, including both health systems topics and more traditional program content.
Several weaknesses of this study must also be recognized: (a) only a small sample of each of the two organizations' recommendations were examined and, in the case of WHO, sometimes as little as one year after the development of "Guidelines for WHO Guidelines" (WHO 2003); (b) comparisons were focused primarily on health systems recommendations, a domain in which systematic reviews have only recently begun to take hold (Lavis et al. 2004); (c) access to research evidence was restricted by the availability of relevant systematic reviews (and the inclusion of high-quality studies in these systematic reviews); and (d) systematic reviews were coded based only on the authors' conclusions (and not on a standardized grading of the recommendations' strength or a rating of the systematic reviews' quality).
Strengths and weaknesses in relation to other studies
This study builds upon previous work as the first attempt to systematically compare health systems recommendations by two prominent international organizations to the research evidence that was available at the time of their formulation. While the use of research evidence in WHO recommendations has been previously examined (Oxman et al. 2006, 2007; Nahar Kabir and Holmgren 2005; Panisset 2005), this study begins to quantify this challenge while offering data on a second international organization, the World Bank, as a comparator. Nevertheless, this study, unlike previous work, did not examine what international organizations are currently doing to support the use of research evidence but rather looked exclusively at the outcome of this process.
Meaning of the study: Possible mechanisms and implications for clinicians and policy makers
Results from this study point to the necessity of implementing and building upon the recommendations of the subcommittee of the WHO Advisory Committee on Health Research that examines the use of research evidence. This group conducted several environmental scans and literature reviews that identified strategies to improve the use of research evidence in recommendation development. Specifically, the subcommittee looked at such issues as priority setting, composition of expert committees, gathering evidence, incorporating other considerations, implementation and evaluation (Oxman et al. 2006). This comprehensive work is certainly an excellent starting point for international organizations' efforts to improve their use of research evidence to inform their recommendations.
However, the existence of the "Guidelines for WHO Guidelines" prior to the publication of the recommendations examined in this study demonstrates the limitations of such operating policies. It is clear that international organizations must not only (a) help to strengthen the research base about health systems and (b) demand the explicit use of research evidence as a standard operating policy, but also support this stipulation by (c) building institutional capacity to acquire, assess, adapt and apply research evidence, (d) allocating the necessary financial and staff resources to use research evidence and (e) adopting appropriate quality control mechanisms for recommendations and publications. A number of practical suggestions for international organizations have been identified for each of these five priority areas that build upon and extend beyond the report from the WHO's Subcommittee on the Use of Research Evidence (Table 4) (Nahar Kabir and Holmgren 2005; CHSRF 2001; Center for Global Development 2006). Given the different mandates, operating modalities and management structures of international organizations, it is likely that each will need to address the practical suggestions presented in rather different ways.
| Table 4. Possible options to enhance international organizations' use of research evidence | |
| Priority areas | Practical suggestions |
| 1. Strengthen the research base about health systems |
|
| 2. Demand the explicit use of research evidence as a standard operating policy |
|
| 3. Build institutional capacity to acquire, assess, adapt and apply research evidence |
|
| 4. Allocate the necessary financial and staff resources to use research evidence |
|
| 5. Adopt appropriate quality control mechanisms for recommendations and publications |
|
Donor organizations and national governments can also contribute to efforts in this area by demanding international organizations' accountability to the best available research evidence as a minimum expectation, highlighting in various forums the importance of reporting whether, how and why their recommendations were consistent with the direction and nature of effect claims made in available systematic reviews, and using their influence on the governing bodies of international organizations (e.g., WHO's World Health Assembly and the World Bank's Board of Governors) to apply pressure as necessary. Additional financial resources can be specifically allocated to enhance international organizations' use of research evidence, and impact evaluations of health interventions can be systematized. Decision-makers at donor organizations and national governments, and clinicians in general, should also always make sure to critically evaluate the quality and local applicability of recommendations from international organizations prior to their implementation.
Unanswered questions and future research
A dearth of research evidence still exists for evaluating the potential strategies for enhancing the use of research evidence in the development and reporting of recommendations. While a number of practical steps have been suggested, limited high-quality research evidence exists to prioritize the allocation of resources to support their implementation. Future investigations, however, must give serious consideration to the feasibility and practicality of such measures in recognition of the significant workloads and pressures placed on staff at international organizations. Further research is necessary to test the effectiveness of the practical strategies that have been suggested in this paper and to determine the most effective and feasible ways in which they can be operationalized. Qualitative research is needed to illuminate the factors that influence the use of research evidence by international organizations, and the success of any implemented interventions must also be examined so that the goal of using research evidence as a starting point for recommendations can be achieved.
Competing Interests
Steven J. Hoffman worked as an intern for the Alliance for Health Policy and Systems Research (which is co-sponsored by and housed within WHO) while conducting this study. Sara Bennett formerly led the Alliance for Health Policy and Systems Research's secretariat and John N. Lavis serves on its Scientific and Technical Advisory Committee. John N. Lavis also serves as President of the PAHO/WHO Advisory Committee on Health Research and as a member of the WHO Advisory Committee on Health Research
Utilisation des données de recherche par deux organismes internationaux dans leurs recommandations visant les systèmes de santé
Résumé
Contexte :On ne sait pas vraiment à quel point les données de recherche renseignent la formulation des recommandations émises par les organismes internationaux.
Méthode :Nous avons identifié des recommandations précises formulées par l'Organisation mondiale de la santé (OMS) et par la Banque mondiale au sujet des cinq points suivants : la sous-traitance, le financement des services de santé, les ressources humaines dans le domaine de la santé, la lutte contre la tuberculose et la lutte contre le tabagisme. Nous avons répertorié les revues systématiques pertinentes et nous avons évalué les recommandations afin de déterminer si elles sont cohérentes avec les éléments des revues systématiques qui étaient disponibles au moment de leur formulation.
Résultats : Seulement deux des huit publications examinées citaient des revues systématiques et seulement cinq des 14 recommandations de l'OMS et deux des sept recommandations de la Banque mondiale étaient cohérentes avec la direction et la nature des effets décrits par les revues systématiques. Dix des 14 recommandations de l'OMS et cinq des sept recommandations de la Banque mondiale étaient seulement cohérentes avec la direction des effets décrits.
Conclusion : L'OMS et la Banque mondiale, qui toutes deux travaillent avec des organismes donateurs et des gouvernements nationaux, peuvent améliorer leur utilisation des données de recherche (ou, du moins, leur façon d'indiquer une telle utilisation). Quelle que soit la source d'une recommandation, y compris si elle provient d'un organisme international, les décideurs et les cliniciens devraient en faire une évaluation critique en matière de qualité et d'application à l'échelle locale, avant de la mettre en application.
About the Author(s)
Steven J. Hoffman, BHSc
Faculty of Law, Department of Political Science and Munk Centre for International Studies
University of Toronto
Toronto, ON
John N. Lavis, MD, PhD
Professor and Canada Research Chair in Knowledge Transfer and Exchange
Department of Clinical Epidemiology and Biostatistics
Department of Political Science
McMaster University
Hamilton, ON
Sara Bennett, PhD
Advisor, Alliance for Health Policy and Systems Research
World Health Organization
Geneva, Switzerland
Correspondence may be directed to: John N. Lavis, McMaster University, HSC-2D3, 1200 Main St. West, Hamilton, ON L8N 3Z5; tel.: 905-525-9140, ext. 22907; fax: 905-529-5742; e-mail: lavisj@mcmaster.ca.
Acknowledgment
This study was supported by an Ontario Tobacco Research Unit Ashley Studentship for Research in Tobacco Control and a grant from the McMaster University Bachelor of Health Sciences (Honours) Program. These funders had no role in designing the study, collecting the data, analyzing the results, deciding to publish or preparing the manuscript. Significant in-kind support was also provided by the Alliance for Health Policy and Systems Research. John Lavis receives salary support as the Canada Research Chair in Knowledge Transfer and Exchange.References
Aaserud, M., S. Lewin, S. Innvaer, E.J. Paulsen, A.T. Dahlgren, M. Trommald, L. Duley, M. Zwarenstein and A.D. Oxman. 2005. "Translating Research into Policy and Practice in Developing Countries: A Case Study of Magnesium Sulphate for Pre-Eclampsia." BMC Health Services Research 5: 68.
Bordley, C., A. Chelmeinski, P. Margolis, R. Kraus, P. Szilagyi and J. Vann. 2000. "The Effect of Audit and Feedback on Immunization Delivery: A Systematic Review." American Journal of Preventative Medicine 18(4): 343-50.
Buchan, J. and M.R. Dal Poz. 2002. "Skill Mix in the Health Care Workforce: Reviewing the Evidence." Bulletin of the World Health Organization 80: 575-80.
Buchan, J., M. Thompson and F. O'May. 2000. Health Workforce Incentive and Remuneration Strategies: A Research Review. Geneva: World Health Organization.
Canada Health Services Research Foundation (CHSRF). 2001. Is Research Working for You? A Self-Assessment Tool for Health Service and Policy Organizations. Ottawa: Author.
Center for Global Development. 2006. When Will We Ever Learn? Improving Lives through Impact Evaluation. Washington, DC: Author.
Chang, K.C., C.C. Leung, W.W. Yew, S.L. Chan and C.M. Tam. 2006. "Dosing Schedules of 6 Month Regimens and Relapse for Pulmonary Tuberculosis." American Journal of Respiratory and Critical Care Medicine 174(10): 1153-58.
Chopra, M., S. Munro and G. Vist. 2006. Evidence from Systematic Reviews of Effects to Inform Policy-Making about Optimizing the Supply, Improving the Distribution, Increasing the Efficiency and Enhancing the Performance of Health Workers: A Policy Brief Prepared for the International Dialogue on Evidence-informed Action to Achieve Health Goals in Developing Countries (IDEAHealth) in Khon Kaen, Thailand, 13-16 December 2006. J.N. Lavis, ed. Geneva: Alliance for Health Policy and Systems Research.
Commission on Health Research for Development. 1990. Health Research: Essential Link to Equity in Development. New York: Oxford University Press.
Coomarasamy, A. and K.S. Khan. 2004. "What Is the Evidence That Postgraduate Teaching in Evidence-Based Medicine Changes Anything? A Systematic Review." British Medical Journal 329: 1017-21.
Davis, D.A., M.A. Thomson, A.D. Oxman and R.B. Haynes. 1995. "Changing Physician Performance: A Systematic Review of the Effect of Continuing Medical Education Strategies." Journal of the American Medical Association 274: 700-5.
de Beyer, J. and L. Waverley, eds. 2003. Tobacco Control Policy: Strategies, Successes and Setbacks. Ottawa and Washington, DC: Research for International Tobacco Control and the World Bank.
East African Community. 2006. "Regional East African Community Health (REACH) - Policy Initiative: Prospectus." Retrieved May 28, 2009. < http://www.idrc.ca/uploads/ user-S/11551301781 REACH_Prospectus.pdf > .
Ekman, B. 2004. "Community-Based Health Insurance in Low-Income Countries: A Systematic Review of the Evidence." Health Policy and Planning 19(5): 249-70.
Fichtenberg, C.M. and S.A. Glantz. 2002. "Effect of Smoke-Free Workplaces on Smoking Behaviour: Systematic Review." British Medical Journal 325: 188.
Gelband, H. 2000. "Regimens of Less Than Six Months for Treating Tuberculosis [Cochrane Review]." In The Cochrane Library, Issue 2. Chichester, UK: John Wiley and Sons.
Gelband, H. 2006. "Regimens of Less than Six Months for Treating Tuberculosis [Cochrane Review]." In The Cochrane Library, Issue 4. Chichester, UK: John Wiley and Sons.
Gosden, T., F. Forland, I.S. Kristiansen, M. Sutton, B. Leese, A. Giuffrida, M. Sergison and I. Pedersen. 2001. "Impact of Payment Method on Behaviour of Primary Care Physicians: A Systematic Review." Journal of Health Services Research and Policy 6: 44-55.
Gottret, P. and G. Schieber. 2006. Health Financing Revisited: A Practitioner's Guide. Washington, DC: World Bank.
Grilli, R., C. Ramsay and S. Minozzi. 2002a. "Mass Media Interventions: Effects on Health Services Utilization [Cochrane Review]." In The Cochrane Library, Issue 1. Chichester, UK: John Wiley and Sons.
Grilli, R., C. Ramsay and S. Minozzi. 2002b. "Mass Media Interventions: Effects on Health Services Utilization [Cochrane Review]." In The Cochrane Library, Issue 2. Chichester, UK: John Wiley and Sons.
Haines, A., S. Kuruvilla and M. Borchert. 2004. "Bridging the Implementation Gap between Knowledge and Action for Health." Bulletin of the World Health Organization 82: 724-31.
Hamid, M., T. Bustamante-Manaog, T. Dung, K. Akkhavong, H. Fu, Y. Ma, X. Zhong, R. Salmela, U. Panisset and T. Pang. 2005. "EVIPNet: Translating the Spirit of Mexico." Lancet 366: 1758-60.
Hanson, K., M.K. Ranson, V. Oliveira-Cruz and A. Mills. 2001. "Approaches to Overcoming Health Systems Constraints at a Peripheral Level: A Review of the Evidence." CMH Working Paper Series WG5:15. Geneva: World Health Organization Commission on Macroeconomics and Health.
Harding, A. and A. Preker, eds. 2003. Private Participation in Health Services. Washington, DC: World Bank.
Hill, S. and T. Pang. 2007. "Leading by Example: A Culture Change at WHO." Lancet 369: 1842-44.
Holland, R., J. Battersby, I. Harvey, E. Lenaghan, J. Smith and L. Hay. 2005. "Systematic Review of Multidisciplinary Interventions in Heart Failure." Heart 91(7): 899-906.
Horrocks, S., E. Anderson and C. Salisbury. 2002. "Systematic Review of Whether Nurse Practitioners Working in Primary Care Can Provide Equivalent Care to Doctors." British Medical Journal 324: 819-23.
Jamtvedt, G., J.M. Young, D.T. Kristoffersen, M.A. Thomson O'Brien and A.D. Oxman. 2003. "Audit and Feedback: Effects on Professional Practice and Health Care Outcomes [Cochrane Review]." In The Cochrane Library, Issue 3. Chichester, UK: John Wiley and Sons.
Jamtvedt, G., J.M. Young, D.T. Kristoffersen, M.A. Thomson O'Brien and A.D. Oxman. 2006a. "Audit and Feedback: Effects on Professional Practice and Health Care Outcomes [Cochrane Review]." In The Cochrane Library, Issue 2. Chichester, UK: John Wiley and Sons.
Jamtvedt, G., J.M. Young, D.T. Kristoffersen, M.A. O'Brien and A.D. Oxman. 2006b. "Does Telling People What They Have Been Doing Change What They Do? A Systematic Review of the Effects of Audit and Feedback." Quality and Safety in Health Care 15(6): 433-36.
Jones, G., R.W. Stekettee, R.E. Black, Z.A. Bhutta, S.S. Morris and the Bellagio Child Survival Study Group. 2003. "How Many Child Deaths Can We Prevent this Year?" Lancet 362: 65-71.
Kakad, M. and A.D. Oxman. 2006. Evidence from Systematic Reviews to Inform Decision-Making towards Achieving the Millennium Development Goals for Reducing Maternal and Child Mortality: A Background Document Prepared for an International Dialogue on Evidence-Informed Action to Achieve Health Goals in Developing Countries (IDEAHealth) in Khon Kaen, Thailand, 13-16 December 2006. Geneva: Alliance for Health Policy and Systems Research.
Kaper, J., E.J. Wagena, J.L. Severens and C.P. Van Schayck. 2005. "Healthcare Financing Systems for Increasing the Use of Tobacco Dependence Treatment [Cochrane Review]." In The Cochrane Library, Issue 1. Chichester, UK: John Wiley and Sons.
Lagarde, M. and N. Palmer. 2006. Evidence from Systematic Reviews to Inform Decision-Making Regarding Financing Mechanisms That Improve Access to Health Services for Poor People: A Policy Brief Prepared for the International Dialogue on Evidence-Informed Action to Achieve Health Goals in Developing Countries (IDEAHealth) in Khon Kaen, Thailand, 13-16 December 2006. S. Bennett, ed. Geneva: Alliance for Health Policy and Systems Research.
Laurant, M., D. Reeves, R. Hermens, J. Braspenning, R. Grol and B. Sibbald. 2004. "Substitution of Doctors by Nurses in Primary Care [Cochrane Review]." In The Cochrane Library, Issue 2. Chichester, UK: John Wiley and Sons.
Lavis, J.N., H.T.O. Davies, R.L. Gruen, K. Walshe and C.M. Farquhar. 2006. "Working Within and Beyond the Cochrane Collaboration to Make Systematic Reviews More Useful to Healthcare Managers and Policy Makers." Healthcare Policy 1(2): 21-33.
Lavis, J.N., A.C. Hammill, M.G. Wilson, J.A. Boyko, J. Grimshaw, A.D. Oxman and S. Flottorp. "Systematic Reviews That Can Inform Health Systems Management and Policy Making." Manuscript under review.
Lavis, J.N., J. Lomas, M. Hamid and N.K. Sewankambo. 2006. "Assessing Country-Level Efforts to Link Research to Action." Bulletin of the World Health Organization 84(8): 620-28.
Lavis, J.N., F.B. Posada, A. Haines and E. Osei. 2004. "Use of Research to Inform Policymaking." Lancet 364: 1615-21.
Lexchin, J. and P. Grootendorst. 2004. "Effects of Prescription Drug User Fees on Drug and Health Services Use and on Health Status in Vulnerable Populations: A Systematic Review of the Evidence." International Journal of Health Services 34(1): 101-22.
Littlewood, S., V. Ypinazar, S.A. Margolis, A. Scherpbier, J. Spencer and T. Dornan. 2005. "Early Practical Experience and the Social Responsiveness of Clinical Education: Systematic Review." British Medical Journal 331: 387-91.
Lovato, C., G. Linn, L.F. Stead and A. Best. 2003. "Impact of Tobacco Advertising and Promotion on Increasing Adolescent Smoking Behaviours [Cochrane Review]." In The Cochrane Library, Issue 3. Chichester, UK: John Wiley and Sons.
Lush, L., G. Walt and J. Ogden. 2003. "Transferring Policies for Treating Sexually Transmitted Infections: What's Wrong with Global Guidelines?" Health Policy and Planning 18(1): 18-30.
McAlister, F.A., S. Stewart, S. Ferrua and J.J.V. McMurray. 2004. "Multidisciplinary Strategies for the Management of Heart Failure Patients at High Risk for Admission: A Systematic Review of Randomized Trials." Journal of the American College of Cardiology 44(4): 810-19.
Moher, M., K. Hey and T. Lancaster. 2003. "Workplace Interventions for Smoking Cessation [Cochrane Review]." In The Cochrane Library, Issue 2. Chichester, UK: John Wiley and Sons.
Moher, M., K. Hey and T. Lancaster. 2005. "Workplace Interventions for Smoking Cessation [Cochrane Review]." In The Cochrane Library, Issue 2. Chichester, UK: John Wiley and Sons.
Montori, V.M., N.L. Wilczynski, D. Morgan, R.B. Haynes and the Hedges Team. 2005. "Optimized Search Strategies for Retrieving Systematic Reviews from MEDLINE: Analytical Survey." British Medical Journal 330: 68.
Mucciaroni, G. and P.J. Quirk. 2006. Deliberative Choices: Debating Public Policy in Congress. Chicago: University of Chicago Press.
Mwandumba, H.C. and S.B. Squire. 2000. "Fully Intermittent Dosing with Drugs for Tuberculosis [Cochrane Review]." In The Cochrane Library, Issue 4. Chichester, UK: John Wiley and Sons.
Mwandumba, H.C. and S.B. Squire. 2001. "Fully Intermittent Dosing with Drugs for Treating Tuberculosis in Adults [Cochrane Review]." In The Cochrane Library, Issue 4. Chichester, UK: John Wiley and Sons.
Nahar Kabir, Z. and J. Holmgren. 2005. Overview of Research Activities at the World Health Organization. Stockholm: Swedish International Development Cooperation Agency.
Ogden, J., G. Walt and L. Lush. 2003. "The Politics of 'Branding' in Policy Transfer: The Case of DOTS for Tuberculosis Control." Social Science and Medicine 57: 179-88.
Oxman, A.D., A. Fretheim and H.J. Schünemann for the Subcommittee on the Use of Research Evidence of the WHO Advisory Committee on Health Research. 2006. "Improving the Use of Research Evidence in Guideline Development: Introduction." Health Research Policy and Systems 4: 12.
Oxman, A.D. and G. Guyatt. 2002. "When to Believe a Subgroup Analysis." In G. Guyatt, ed., Users' Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (pp. 553-65). Chicago: AMA Press.
Oxman, A.D., J.N. Lavis and A. Fretheim. 2007. "The Use of Evidence in WHO Recommendations." Lancet 369: 1883-89.
Panisset, U. 2005. "A Review of WHO Recommendations Published in 2005." Unpublished paper.
Schünemann, H.J., S.R. Hill, M. Kakad, G.E. Vist, R. Bellamy, L. Stockman, T. Fosen Wisløff, C.D. Mar, F. Hayden, T.M. Uyeki, J. Farrar, Y. Yazdanpanah, H. Zucker, J. Beigel, T. Chotpitayasunondh, T.T. Hien, B. Özbay, N. Sugaya and A.D. Oxman. 2007. "Transparent Development of the WHO Rapid Advice Guidelines." PLoS Medicine 4(5): e119.
Serra, C., C. Cabezas, X. Bonfill and M. Pladevall-Vila. 2000. "Interventions for Preventing Tobacco Smoking in Public Places [Cochrane Review]." In The Cochrane Library, Issue 3. Chichester, UK: John Wiley and Sons.
Silagy, C., T. Lancaster, L. Stead, D. Mant and G. Fowler. 2001. "Nicotine Replacement Therapy for Smoking Cessation [Cochrane Review]." In The Cochrane Library, Issue 3. Chichester, UK: John Wiley and Sons.
Silagy, C., T. Lancaster, L. Stead, D. Mant and G. Fowler. 2002. "Nicotine Replacement Therapy for Smoking Cessation [Cochrane Review]." In The Cochrane Library, Issue 4. Chichester, UK: John Wiley and Sons.
Silagy, C., T. Lancaster, L. Stead, D. Mant and G. Fowler. 2004. "Nicotine Replacement Therapy for Smoking Cessation [Cochrane Review]." In The Cochrane Library, Issue 3. Chichester, UK: John Wiley and Sons.
Sowden, A.J. and L. Arblaster. 2000. "Mass Media Interventions for Preventing Smoking in Young People [Cochrane Review]." In The Cochrane Library, Issue 2. Chichester, UK: John Wiley and Sons.
Stewart, S. 2006. "Review: Multidisciplinary Interventions Reduce Hospital Admission and All Cause Mortality in Heart Failure." Evidence-Based Nursing 9(1): 23.
Thomson O'Brien, M.A., A.D. Oxman, D.A. Davis, R.B. Haynes, N. Freemantle and E.L. Harvey. 2000. "Audit and Feedback: Effects on Professional Practice and Health Care Outcomes [Cochrane Review]." In The Cochrane Library, Issue 2. Chichester, UK: John Wiley and Sons.
Travis, P., S. Bennett, A. Haines, T. Pang, Z. Bhutta, A.A. Hyder, N.R. Pielemeier, A. Mills and T. Evans. 2004. "Overcoming Health-Systems Constraints to Achieve the Millennium Development Goals." Lancet 364: 900-6.
Veloski, J., J.R. Boex, M.J. Grasberger, A. Evans and D.B. Wolfson. 2006. "Systematic Review of the Literature on Assessment, Feedback and Physicians' Clinical Performance: BEME Guide No. 7." Medical Teacher 28(2): 117-28.
Volmink, J. and P. Garner. 2000a. "Interventions for Promoting Adherence to Tuberculosis Management [Cochrane Review]." In The Cochrane Library, Issue 2. Chichester, UK: John Wiley and Sons.
Volmink, J. and P. Garner. 2000b. "Interventions for Promoting Adherence to Tuberculosis Management [Cochrane Review]." In The Cochrane Library, Issue 4. Chichester, UK: John Wiley and Sons.
Volmink, J. and P. Garner. 2001. "Directly Observed Therapy for Treating Tuberculosis [Cochrane Review]." In The Cochrane Library, Issue 4. Chichester, UK: John Wiley and Sons.
Volmink, J. and P. Garner. 2003. "Directly Observed Therapy for Treating Tuberculosis [Cochrane Review]. In The Cochrane Library, Issue 1. Chichester, UK: John Wiley and Sons.
Volmink, J. and P. Garner. 2006. "Directly Observed Therapy for Treating Tuberculosis [Cochrane Review]." In The Cochrane Library, Issue 2. Chichester, UK: John Wiley and Sons.
Volmink, J., P. Matchaba and P. Garner. 2000. "Directly Observed Therapy and Treatment Adherence." Lancet 355(9212): 1345-50. [Erratum in Lancet 356: 434.]
Wellman, R.J., D.B. Sugarman, J.R. DiFranza and J.P. Winickoff. 2006. "The Extent to Which Tobacco Marketing and Tobacco Use in Films Contribute to Children's Use of Tobacco: A Meta-Analysis." Archives of Pediatrics and Adolescent Medicine 160(12): 1285-96.
Wilczynski, N.L. and R.B. Haynes for the Hedges Team. 2007. "EMBASE Search Strategies Achieved High Sensitivity and Specificity for Retrieving Methodologically Sound Systematic Reviews." Journal of Clinical Epidemiology 60: 29-33.
Wong, S.S.L., N.L. Wilczynski and R.B. Haynes. 2006. "Optimal CINHAL Search Strategies for Identifying Therapy Studies and Review Articles." Journal of Nursing Scholarship 38(2): 194-99.
World Bank. 2004. The Millennium Development Goals for Health: Rising to the Challenges. Washington: Author.
World Bank. 2006. World Bank Group: Working for a World Free of Poverty. Washington: Author.
World Health Organization. 2003. "Guidelines for WHO Guidelines." EIP/GPE/EQC/2003.1. Geneva: Author. Retrieved May 28, 2009. < http://whqlibdoc.who.int/hq/2003/ EIP_GPE_EQC_2003_1.pdf > .
World Health Organization. 2004a. World Report on Knowledge for Better Health (pp. 97-130). Geneva: Author.
World Health Organization. 2004b. Building Blocks for Tobacco Control: A Handbook. Geneva: Author.
World Health Organization. 2004c. Treatment of Tuberculosis: Guidelines for National Programmes (3rd ed.). Geneva: Author.
World Health Organization. 2005a. WHA58.34: Ministerial Summit on Health Research. Fifty-Eighth World Health Assembly, A58/22, 1-7. Geneva: Author.
World Health Organization. 2005b. Application of Contracting in Health Systems: Key Messages. Technical Briefs for Policy-Makers Series No. 4. Geneva: Author.
World Health Organization. 2005c. Achieving Universal Health Coverage: Developing the Health Financing System. Technical Briefs for Policy-Makers Series No. 1. Geneva: Author.
World Health Organization. 2006a. Engaging for Health: Eleventh General Programme of Work 2006-2011: A Global Health Agenda. Geneva: Author.
World Health Organization. 2006b. World Health Report 2006: Working Together for Health. Geneva: Author.
World Health Organization. 2007. WHO Publications: Report by the Secretariat. EB122/20. Retrieved May 28, 2009. < http://www.who.int/gb/ebwha/pdf_files/EB122/ B122_20-en.pdf > .
World Health Organization. 2008. WHO's Role and Responsibilities in Health Research: Draft WHO Strategy on Research for Health: Report by the Secretariat. EB124/12. Retrieved May 28, 2009. < http://www.who.int/gb/ebwha/pdf_files/EB124/ B124_12-en.pdf > .
World Health Organization Advisory Committee on Health Research. 2006. Research for Health: A Position Paper on WHO's Role and Responsibilities in Health Research. ACHR45/05.16/Rev.1. Geneva: WHO.
World Health Organization Task Force on Health Systems Research. 2005. The Millennium Development Goals Will Not Be Attained without New Research Addressing Health System Constraints to Delivering Effective Interventions: Report of the Task Force on Health Systems Research. Geneva: WHO.
Zwarenstein, M. and W. Bryant. 2000. "Interventions to Promote Collaboration between Nurses and Doctors [Cochrane Review]." In The Cochrane Library, Issue 2. Chichester, UK: John Wiley and Sons.
Footnotes
1. See Mucciaroni and Quirk (2006) for a study that similarly assessed the validity of effect claims made by elected members of the US Congress based on information that would have been available to them at the time of their statements.
2. This inventory of systematic reviews of governance, financial and delivery arrangements within health systems is now publicly available at < http://www.researchtopolicy.ca/search/ reviews.aspx > . (Retrieved June 1, 2009.)
3. Full access to an electronic copy of Gottret and Schieber (2006) was purchased for USD$10 on March 21, 2007. This book has since been made available free of charge at < http://siteresources.worldbank.org/INTHSD/ Resources/topics/Health-Financing/ HFRFull.pdf > . (Retrieved June 1, 2009.)
Comments
Be the first to comment on this!
Personal Subscriber? Sign In
Note: Please enter a display name. Your email address will not be publically displayed