Open Access

Systematic reviews on selected nutrition interventions: descriptive assessment of conduct and methodological challenges

BMC Nutrition20151:9

https://doi.org/10.1186/s40795-015-0002-1

Received: 19 September 2014

Accepted: 5 February 2015

Published: 13 April 2015

Abstract

Background

Rigorous and transparent systematic reviews are recognized internationally as a credible source for evidence of effectiveness. However, in the field of nutrition, despite attempts at developing consensus on actions and interventions to reduce undernutrition and micronutrient deficiencies, there is lack of coordination among various groups.

Methods

The aim of this overview of systematic review is to assess the process and conduct of systematic reviews published in the past 5 years to make recommendations on improving process and methodology of systematic reviews in the field of nutrition. We identified nine interventions from four areas of nutrition through a consultative process and conducted a comprehensive search to identify systematic reviews on the selected interventions published in the last 5 years.

Results

We identified 90 systematic reviews across these nine intervention areas. The median overall Assessment of Multiple Systematic Reviews (AMSTAR) score was 8 (range 2–11) with methodological quality of Cochrane reviews being fairly consistent with a median AMSTAR score of 10 (range 8–11), while for the non-Cochrane reviews, it ranged from 2 to 11 with a median of 7.5. From the 11-point AMSTAR criteria, 91% of the reviews followed an a priori design, 81% did duplicate screening and data extraction, 88% conducted a comprehensive search, 64% used status of publication as an inclusion criteria, 44% provided the list of included and excluded studies, 60% assessed and documented the scientific quality of the included studies and used it in interpreting the results, 61% used appropriate methods to combine the results, 40% assessed publication bias and 82% stated the conflict of interest. We found considerable variation in methodological quality, lack of standardization of outcomes, lack of standardized assessment of risk of bias of included studies, variation in study designs included and variation in how heterogeneity was handled.

Conclusions

Each of these methodological choices influences the findings of the reviews, and lack of standardization across these domains increases the complexity for users of systematic reviews in interpreting results. There is a need to develop a consensus on methodologies for nutrition reviews, criteria for assessing the evidence and possibly facilitating development and collation of the evidence in the subject area.

Keywords

Systematic reviews Nutrition Methodological assessment

Background

To reach the targets set for Millennium Development Goal (MDG) 1, which is to reduce extreme poverty and hunger by half, many countries are on-track in reducing the income poverty while less than a quarter of the developing countries are on-track for achieving the goal of halving undernutrition [1]. The global burden of undernutrition remains high with little evidence of change in many countries despite economic growth. Millions of people are faced with starvation and malnutrition; the State of Food Insecurity (SOFI) in the World 2012 estimated that about 870 million people have been undernourished (in terms of dietary energy supply) in the period 2010–2012, mostly from low-middle-income countries (LMIC) [2]. About one third of deaths in children under 5 years of age are due to underlying undernutrition, which includes stunting, severe wasting, deficiencies of vitamin A and zinc and suboptimum breastfeeding [3]. The majority of an estimated 178 million children under five suffering from stunting are in sub-Saharan Africa and South-central Asia [4]. Around 55 million children are wasted, 19 million of these are severely affected and at high risk of premature death [4]. Altogether, more than 3.5 million mothers and children under five die unnecessarily each year in LMICs due to the underlying cause of undernutrition [5]. With the existing burden of undernutrition, the problem of overweight and obesity among children is also on the rise. In 2011, globally, an estimated 43 million (7%) children younger than 5 years were overweight, an increase from an estimated 28 million in 1990. This trend is expected to continue and reach a prevalence of 9.9% (64 million) in 2025 [6]. Many of the LMICs are said to have the double burden of malnutrition—continued stunting of growth and deficiencies of essential nutrients along with the emerging issue of obesity.

Although there have been several attempts at developing consensus on actions and interventions for nutrition, there is lack of coordination between various academic groups, United Nations (UN) and development agencies, and these are working in parallel silos. The Lancet Nutrition series in 2008 [7-11] and more recently in 2013 [6,12-14] are important attempts at bringing the field together; however, progress by countries has been slow and also hampered by the global increase in food and oil prices; climate change; unprecedented draughts and increased number of countries affected by fragility, conflict and emergencies. As per the World Bank in 2011, 33 countries fall in the fragile situations category as per the World Bank definition [15]. In addition, conflict and fragility also occur at the sub-national level within some strongly performing countries.

Systematic reviews and meta-analysis have become increasingly popular in evidence-based healthcare over the past two decades [16], although these have been evaluated for reliability since quite a long time [17-20]. Rigorous and transparent systematic reviews are recognized internationally as a credible source for evidence of effects and as the basis for evidence-informed policy and decisions [16,21,22]. Within the field of nutrition, systematic reviews have been used to answer many complex questions. A description of systematic review methodology with particular regard to the steps for performing systematic reviews and areas unique to the discipline of nutrition is already available, published by the US Agency for Healthcare Research and Quality (AHRQ) [23]. Furthermore, guidance on assessing equity for users and authors of systematic reviews of interventions has also been highlighted recently to improve the relevance of systematic reviews for both clinical practice and public policy-making [24]. However, several gaps pertaining to the conduct and methodological approaches still exists particularly with reference to nutrition interventions implemented through sectors other than health, such as agriculture, social welfare and trade.

Across nutrition-related systematic reviews, there is multiplicity of reviews in the same subject area with variability in methods such as intervention, target population comparison, outcomes and consideration of contextual variables related to nutritional interventions and outcomes. Additionally, with the increasing attention to social determinants, there is a need to bring in additional qualitative aspects to such reviews. The World Health Organization (WHO) has led the field with a series of activities geared towards development of nutrition policy guidelines based on the best evidence. In 2011, a new online resource, the electronic Library of Evidence on Nutrition Actions (eLENA), was launched that provides access to available sources of evidence and to WHO guidelines, as well as to ongoing clinical trials in the field of nutrition [25].

This overview of systematic reviews from the selected areas in nutrition was produced as a background document for a WHO-convened meeting on improving the quality of systematic reviews of nutrition [26]. The aim of this paper is to assess the process and conduct of systematic reviews (including systematic reviews and systematic reviews + meta-analysis) published in the past 5 years to make recommendations on improving process and methodology of systematic reviews in the field of nutrition.

Methods

We consulted the major stakeholders including leading academic groups engaged in systematic reviews, editors of the Cochrane and Campbell collaborations; International initiative for impact evaluation (3ie); UN agencies including the WHO, United Nations International Children’s Emergency Fund (UNICEF), World Food Program (WFP) and Food and Agriculture Organization (FAO); International Food Policy Research Institute (IFPRI); and interested bilateral agencies including the Department for International Development (DFID), United States Agency for International Development (USAID), Norwegian agency for International Development (NORAD) and representatives from the Bill & Melinda Gates Foundation to identify nine focused interventions under four umbrella areas for our review. These areas were selected based on consultations and WHO interests and included preventive (interventions to prevent undernutrition and micronutrient deficiencies), therapeutic (interventions to treat undernutrition and micronutrient deficiencies), food fortification strategies and delivery platforms (Table 1).
Table 1

Topic areas of landscape study

Topic area

Interventions

Preventive nutrition interventions (interventions to prevent undernutrition and micronutrient deficiencies)

Preventive zinc supplementation

Breast feeding

Complementary feeding

Preventive multiple micronutrient (MMN) supplementation

Therapeutic nutrition interventions (interventions to treat undernutrition and micronutrient deficiencies)

Undernutrition/malnutrition [ready-to-use therapeutic food for community management of severe acute malnutrition]

Therapeutic zinc supplementation

Therapeutic MMN supplementation

Fortification strategies (deliberately increasing the content of an essential micronutrient in food irrespective of whether the nutrients were originally in the food before processing or not, so as to improve the nutritional quality of the food supply)

Single nutrient fortifications (Folic Acid)

Delivery platforms (specific modes and channels of delivering interventions)

Conditional cash transfers

We defined systematic review according to the ‘Cochrane Handbook of Systematic Reviews’ [27] as ‘A systematic review attempts to collate all empirical evidence that fits prespecified eligibility criteria in order to answer a specific research question’. Meta-analysis was defined as ‘the use of statistical methods to summarize the results of independent studies’. An overview of systematic reviews was defined as ‘Systematic and explicit methods to identify select and critically appraise relevant findings from systematic reviews and/ or meta-analysis of effectiveness of interventions’. We included systematic reviews of trials as well as observational studies. We considered all available systematic reviews and overviews of systematic reviews on the predefined interventions. A separate search strategy was developed for each intervention using preidentified Medical Subject Headings (MeSH) and key search terms. A search was conducted in the Cochrane Library, Campbell Collaboration, PubMed and Google Scholar to identify systematic reviews in each of the nine intervention areas. Additional reviews were identified by hand searching references from included reviews. A priori inclusion criteria were outlined stated as below:
  • Systematic reviews pertaining to effectiveness or efficacy of nutrition-related interventions, assessed by prospective studies (including both trials and observational studies), published between 2005 and 2011 were included.

  • Overviews of systematic reviews were also included if they were found to be relevant.

  • Reviews of all languages with English translations were considered eligible for inclusion.

  • Inclusion was not restricted to any particular reported outcome.

Reviews that met the inclusion criteria were selected and double data abstracted on a standardized abstraction sheet for source journal, funding, authors, year, title, target population, intervention, study setting (high-income countries or low-middle-income countries) and comparison, outcomes, types of included studies (randomized trials, quasi trials, before-after, etc.), gaps in knowledge base (as identified by the author), and conclusions. Any disagreements on selection and abstraction of reviews between the two primary abstractors were resolved by the third reviewer. We also assessed each systematic review according to the components from the Cochrane Public Health Review Group guidance including study design; searching non-health databases; assessing risk of bias, ethics and inequality; context; sustainability and applicability [28]. For each systematic review, we assessed the quality of the review by the Assessment of Multiple Systematic Reviews (AMSTAR) rating consisting of 11 items which has been demonstrated to have good face and content validity for measuring the methodological quality of systematic reviews [29]. We then compared the reviews within the same domain with respect to their journals, funding, research question, included study designs and meta-analysis, reporting tools, and definition and assessment of outcomes.

Results

Our search identified 10,926 titles, of which 217 full texts were reviewed and 90 systematic reviews across the nine content areas were included (Figure 1). All eligible texts were available in English, and there was no exclusions based on language. We did not include any unpublished or grey literature review. A wide range of groups were responsible for commissioning and funding the reviews including governments, non-governmental organizations (NGOs), various institutions, UN agencies and private foundations (Table 2). Sixty-eight percent (53/78) of the systematic reviews from the selected nutrition domains were produced from authors based in the high-income countries (HIC) with major contributions from USA, UK, Canada and Australia; thirty-two percent (25/78) of the reviews had authors from the LMICs with Pakistan (13/25) being the major contributing site (Figure 2). From the illustrative areas reviewed, the journal publishing the most systematic reviews in the last 5 years in these nine intervention areas was the Cochrane Database of Systematic Reviews (35/90), followed by BioMed Central (7/90), ‘Journal of Nutrition’ (4/90), ‘The Lancet’ (4/90), ‘American Journal of Clinical Nutrition’ (4/90), ‘Food and Nutrition Bulletin’ (3/90) and ‘Journal of American Medical Association’ (3/90) (Figure 3). The included reviews, and overviews of reviews are summarized using the Cochrane public health guideline grid (Table 3). The assessment of the quality of systematic reviews using the AMSTAR rating, and further description of the attributes of the systematic reviews and overviews, are provided below.
Figure 1

Search flow.

Table 2

Commissioning and funding of the reviews

 

Organizations

Government

HTA program-HCA; the National Institutes of Health for the prevention of childhood disease; Canadian Institutes of Health Research; United States Agency for International Development Office of Foreign Disaster Assistance; Research Council of Norway; Department of Health and Ageing-Australia; USAID; The Department for International Development (UK); UK Department for International Development (DFID) for developing countries; Concern Worldwide; Ministry of Science and Technological Development, Serbia; Food Standards Australia New Zealand; Norwegian Agency for Development Cooperation (NORAD)

UN agencies

WHO, World Bank, UNICEF, World Food Program, United Nations System Standing Committee on Nutrition (SCN)

Private foundations

Bill & Melinda Gates Foundation, The Richard and Rhoda, Goldman Fund

NGOs

ICDDRB (Bangladesh); The Nutricia Research Foundation; International Initiative for Impact Evaluation; German Technical Co-operation (GTZ), Germany; Aubrey Sheiham Public Health and Primary Care Scholarship-UK; International Nutrition Council

Institutions

The Alberta Heritage; Medical University of Warsaw; The Monsanto Fund; Institute of Child Health, London; Royal College of Paediatrics and Child Health-UK; The Copenhagen Trial Unit; Harvard Medical School

Figure 2

Countries of the lead authors producing reviews.

Figure 3

Journals publishing systematic reviews in last 5 years.

Table 3

Cochrane public health guideline grid

Methods

Data from studies

Public health guidance

 

Preventive and promotive interventions (n = 50)

Systematic reviews: 12

Systematic review + meta: 36

Overview: 2

Therapeutic nutrition interventions (n = 34)

Systematic reviews: 10

Systematic review + meta: 24

Overview: 0

Fortification (folic acid) (n = 2)

Systematic reviews: 0

Systematic review + meta: 2

Overview: 0

Delivery platforms (CCT) (n = 4)

Systematic reviews: 3

Systematic review + meta: 0

Overview: 1

 

Study designs (primary studies)

RCTs: 25 reviews

Mixed (RCTs, quasi and other designs): 25 reviews

RCTs: 23 reviews

Mixed (RCTs, quasi and other designs): 11 reviews

RCTs: 0 reviews

Mixed (RCTs, quasi and other designs): 2 reviews

Overview: 0 reviews

RCTs: 0 reviews

Mixed (RCTs, quasi and other designs): 4 reviews

RCTs may have limited generalizability or not be feasible or ethical. Non-randomized trials or cluster RCTs may be more appropriate for the question

Searching non-health databases, number seen out of total (range)

6 (1–5)

11 (1–2)

0

3 (3–5)

Public health may require range of databases and detailed text words because terminology is imprecise

Assessing risk of bias

Tools used:

Cochrane risk of bias: 17

GRADE: 5

Jadad: 1

CONSORT: 1

Other tools: 18

Not stated: 8

Tools used:

Cochrane risk of bias: 14

GRADE: 0

Jadad: 0

CONSORT: 0

Other tools: 19

Not stated: 1

Tools used:

Cochrane risk of bias: 0

GRADE: 1

Jadad: 0

CONSORT: 0

Other tools: 1

Tools used:

Cochrane risk of bias: 2

GRADE: 0

Jadad: 0

CONSORT: 0

Other tools: 2

Appraisal criteria depend on the type of study in the review. Authors should use Cochrane risk of bias tool for RCTs

Ethics and inequalities

HIC: 9

LMIC: 21

Both: 20

HIC: 8

LMIC: 15

Both: 11

HIC: 1

LMIC: 0

Both: 1

HIC: 0

LMIC: 2

Both: 2

Public health interventions may unintentionally increase health inequalities if the more advantaged benefit more than the worst off. Authors should consider whether populations are likely to be disadvantaged across PROGRESS+ acronym factors (place of residence, race or ethnicity, occupation, gender, religion, education, socioeconomic position (SES) and social capital

Context (explore context as an effect modifier)

29

13

1

0

Public health interventions depend on social, economic and political context in which they are developed and implemented. Context should be explored as an effect modifier (e.g. host organization, target population)

Sustainability (sustainability was assessed or described)

11

6

0

4

Consider the extent to which the intervention’s effects on outcomes are sustained over time, e.g. by assessing economic and political variables, strength of implementing institutions, integration of services into existing services, training component, community involvement

Applicability (conducted subgroup analysis to assess applicability to different settings)

29

13

1

0

Authors are well-positioned to collect data needed to make judgments about applicability to different settings, including political environment, social norms, cultural preferences, resources required, educational level of target population, organizational capacity to deliver the intervention, structural barriers to intervention, health care providers have capacity to deliver this intervention

HIC high-income countries, LMIC low-middle-income countries, RCT randomized controlled trials.

Methodological quality-AMSTAR rating

The median overall AMSTAR score was 8 (range 2–11) with methodological quality of Cochrane reviews being fairly consistent with a median AMSTAR score of 10 (range 8–11), while for the non-Cochrane reviews, it ranged from 2 to 11 with a median of 7.5 (Figure 4). From the 11-point AMSTAR criteria, 91% (82/90) of the reviews followed an a priori design, 81% (73/90) did duplicate screening and data extraction, 88% (79/90) conducted comprehensive literature search, 64% (58/90) used status of publication as an inclusion criteria, 44% (40/90) provided the list of included and excluded studies, 34% (31/90) provided the list of included studies only, 74% (67/90) provided characteristics of the included studies, 60% (54/90) assessed and documented the scientific quality of the included studies and used it in interpreting the results, 61% (55/90) used appropriate methods to combine the results, 40% (36/90) assessed publication bias and 82% (74/90) stated the conflict of interest.
Figure 4

AMSTAR ratings for the included Cochrane and non-Cochrane reviews.

Research question

The research question guides the systematic review and is probably the most important step that defines the population, the intervention, the comparison and the outcomes of interest. We compared research questions for reviews identified on the same intervention to assess for overlap of questions. Although we found several reviews for the same intervention, each review was different because of differences in aspects of the population, intervention, outcome or comparison. For preventive zinc supplementation, two of the included reviews targeted pregnant women, seven targeted children under 5 years of age, one targeted adults while one targeted general population. Besides differences in the target population, reviews evaluated various outcomes; five reviews evaluated childhood infections [30-34], of which three focused on pneumonia alone [31-33]; one on respiratory illness and diarrhoea [30] and one on diarrhoea, pneumonia and malaria altogether [34]. Similarly reviews on multiple micronutrient (MMN) supplementation targeted a variety of age groups including pregnant and women of reproductive age, children, healthy adults, and HIV-infected women and children. Table 4 reports the population, topics and outcomes reported in the included reviews.
Table 4

Population, topics and outcomes reported in the included reviews

Intervention

N

(Cochrane/non-Cochrane)

Population

Topics

Outcomes

Setting

Preventive zinc

14 (7/7)

Pregnancy (2)

Children <5 (7)

Adult (1)

General population (1)

Supplementation during pregnancy and outcome assessment: 2

Supplementation to children under five and impact on growth: 2

Supplementation to children and impact on infections: 4

•Anthropometric Indices

•APGAR

•All-cause mortality

•Age-specific mortality

•Cause-specific mortality (diarrhoea, pneumonia)

•Neonatal (birth weight, linear growth)

•Morbidity (incidence and duration of ARI, diabetes, otitis media)

•Haemoglobin level

LMIC (n = 8)

HIC (n = 1)

Both (n = 2)

Breast feeding

16 (11/05)

Pregnant + reproductive age woman (7)

Children (5)

General population (2)

Interventions to promote breast feeding: 6

•Initiation, duration and exclusive breast feeding

•Neonatal and infant morbidities

•Anthropometric indices

•Maternal satisfaction

•Cognitive development

•Neonatal mortality

•Micronutrient status

•Maternal weight loss

•Breast and ovarian cancer

•Osteoporosis

•Pain

•Maternal morbidities

LMIC (n = 1)

HIC (n = 6)

Both (n = 4)

Not clear (2)

Complementary feeding

06 (1/5)

Children (6)

Education and counselling interventions to promote complementary feeding: 5

•Anthropometric indices

•Micronutrient intake and status

•Morbidity

LMIC (n = 6)

Preventive multiple micronutrient supplementation

14 (4/10)

Pregnant + reproductive age woman (6)

Children (2)

Women and children (4)

Adult men (1)

Supplementation during and prior to pregnancy and its impact on pregnancy outcome: 9

Supplementation to HIV-infected women and children: 2

•Birth weight

•Kwashiorkor

•Prematurity, SGA, LBW

•Stillbirth, neonatal and perinatal mortality

•Premature rupture of membranes

•Preeclampsia

•Miscarriage

•Congenital anomalies

•Nutritional status

•Micronutrient level

•Maternal anaemia

•Multiple pregnancies

•Psychological effects

•HIV transmission and viral load

•Prostate cancer occurrence, metastasis, mortality

LMIC (n = 4)

HIC (n = 2)

Both (n = 4)

Not clear (3)

Therapeutic zinc

12 (4/8)

Children (6)

General population (5)

Adult (1)

Adult supplementation and its impact on cold symptoms: 2

Supplementation among children and impact on diarrhoea and pneumonia: 6

•Ulcer healing (time, rate)

•Incidence, severity and duration of cold

•Pneumonia: incidence, duration and recurrence of pneumonia. Recovery time and hospitalization

•Adverse effects

•Lipid profile

•Incidence and duration of malaria and TB

•Hospital stay, cost and mortality of critically ill patients

LMIC (n = 3)

HIC (n = 4)

Both (n = 3)

Not clear (2)

Interventions for undernutrition

11 (4/7)

Children (11)

Management of under nutrition through supplementary feeding: 9

•Kwashiorkor

•Cognitive development

•Hospitalization

•Anthropometric indices

•Recurrent illnesses

•Mortality

•HIV prevalence and mortality

•Morbidity: diarrhoea, malaria and ARI

•Maternal knowledge, anxiety

LMIC (n = 11)

Therapeutic multiple micronutrient supplementation

11 (3/8)

Children (1)

Adults (4)

Adult Men (1)

Population with HIV (4)

HIV-infected populations: 5

•Infant mortality

•Anthropometric indices

•Micronutrient levels

•Anaemia

•Mental and motor development

•Pregnancy outcomes

•HIV progression, transmission and viral load

•HIV hospitalization, morbidity and mortality

•Fasting glucose and insulin resistance

•Treatment and length of hospitalization of TB

•Age-related disorders like age-related macular degeneration and cataract

•Occurrence, metastasis and mortality due to prostate cancer

LMIC (n = 1)

HIC (n = 2)

Both (n = 4)

Not clear (4)

Fortification (folic acid)

02 (0/2)

Pregnant and women of reproductive age (1)

General population (1)

 

•Neural tube defect

•Neonatal mortality for NTD

•Twining

Both (n = 2)

Conditional cash transfers (CCT)

4 (1/3)

General population (4)

All 4 reviews assessed the impact CCT on health outcomes

•Coverage of health service

•Access to health care

•Household health expenditure

•Nutritional outcomes

LMIC (n = 2)

Both (n = 2)

HIC high-income countries, LMIC low-middle-income countries, NTD neural tube defect.

Included study designs

Good quality randomized controlled trials (RCTs) are considered as the highest quality evidence because they minimize the influence of bias in their results; however, RCTs cannot always be relied upon for conducting systematic reviews as such robust study designs account for only 10% of the healthcare literature [35,36]. Within the selected nutrition interventions, we found that 53% (48/90) of the reviews included RCTs while the others included quasi and observational studies along with RCTs. Most of the Cochrane reviews (73%) included RCTs and quasi randomized studies; however, for most of the non-Cochrane reviews and some specific interventions like breastfeeding, fortification and management of malnutrition, data from pre-post designs were also included.

Quality assessment of the included studies

Table 3 specifies the quality assessment tools used by the reviews in each domain. All of the Cochrane reviews used the Cochrane risk of bias tool to assess the methodological quality of included studies, while some non-Cochrane reviews used various quality assessment scales including the CONSORT checklist, Jadad scale, GRADE quality assessment, US Preventive Task Force Quality Rating and Criteria by Glasziou et al. For fortification reviews, methodological quality was assessed using The Food Standards Australia New Zealand (FSAN) framework for assessing evidence when substantiating nutrition, health and related claims on foods. Some reviews mentioned conducting a quality assessment; however, the methodology used was not clear, while information on methodological quality assessment was found missing or unclear in a few reviews. The overview of the reviews used an informal method of quality assessment that is the Oxman et al. 1991 criteria.

Outcomes standardization and assessment

Standardized, valid outcome measures are pivotal in interpreting findings of systematic reviews, as intervention effectiveness may vary depending on the outcome measurement tools. For example, the specificity of case definitions for acute respiratory infections (ARI) substantially influenced inferences for the effectiveness of preventive and therapeutic interventions [33,37]. Various definitions have been used for defining ARI; however, in some reviews, we found that standardization measures were taken by restricting the inclusion criteria to studies in which ARI case definition aimed to distinguish lower from upper tract respiratory disease; various definitions were also scored for standardization purposes in one of the reviews [33]. Furthermore, despite showing an overall effectiveness of preventive zinc supplementation on pneumonia prevention, the impact changed from statistically significant to non-significant when the definition of pneumonia changed from ‘pneumonia defined by specific clinical criteria’ to ‘lower specificity pneumonia case definition (i.e. age specific fast breathing with or without lower chest in-drawing)’ [31,33]. Few other reviews reported considerable variation in outcome definition across studies, for example definition of exclusive breastfeeding and mixed breast feeding [38], definitions of diarrhoea [30,34] and the definition of health care workers delivering the interventions [39]. In areas where WHO standard definitions exist like undernutrition, reviews have used standard WHO definitions [40-42], while for some reviews, reviewers have self-defined the outcome for standardization [43,44]. Forty three of the included reviews considered contextual factors and conducted sensitivity and subgroup analysis to adjust the outcomes according to these factors.

Meta-analysis and heterogeneity

Nutrition interventions are inherently complex due to multifaceted interventions, diverse settings and outcomes; hence, it is challenging to pool data for meta-analysis which leads to a lack of consensus on effectiveness and contradicting findings. Almost all the reviews reported heterogeneity in the data. Out of 90 included reviews, 62 were systematic reviews with a meta-analysis, 25 were systematic reviews without meta-analysis, while almost 11% (10/90) of the systematic reviews reported heterogeneity as a reason for not conducting meta-analysis. The heterogeneity reported was both clinical as well as statistical due to diverse interventions, target population and outcome definitions. None of the reviews conducted meta-analysis using individual patient data (IPD).

Discussion

Our overview findings pertain to the process and conduct of systematic reviews and not individual studies since we did not assess the primary studies. Our findings suggest that from the 90 nutrition-related reviews, we found considerable variation in methodological quality as assessed by the AMSTAR (ranging from 2 to 11), lack of standardization of outcomes, lack of standardized assessment of risk of bias of included studies, variation in study designs included and variation in how heterogeneity was handled. Each of these methodological choices influence the findings of the reviews, and lack of standardization across these domains increases the complexity for users of systematic reviews in interpreting results—sometimes of conflicting reviews. None of the included meta-analyses used IPD, which provides more detailed analyses and a broader methodological scope due to the presence of raw data. Our analysis shows that these nutrition reviews are comparable to a random sample of systematic reviews across all topics. However, the sample of nutrition reviews had considerably more reviews led by an LMIC author, 32% compared to the 2004 sample where <20% of all reviews had the contact author outside of USA, UK, Canada, New Zealand and Germany [45]. Systematic reviews and meta-analysis have become increasingly popular in evidence-based healthcare over the past two decades [16-20]. The selected systematic reviews on nutrition came from varying sources as the nutrition-related interventions are not solely implemented by health sector. There is an evident gap of quantitative data in some major areas concerning nutrition like undernutrition that direly needs policy guidance. We found considerable overlap between the topics and questions addressed in multiple reviews; however, the outcomes reported varied.

Overall, the methodological quality of the Cochrane reviews was found to be better, which is not surprising since they are conducted according to standardized guidelines published in the ‘Cochrane Handbook’. However, within the Cochrane Collaboration, several editorial groups have undertaken nutrition-related reviews, and while the broad methods are consistent, there is variability in nutrition-specific decisions about the methods such as eligibility criteria for population, intervention, comparison and outcomes and consideration of contextual variables. Furthermore, in some areas like conditional cash transfers (CCTs), the scope is very broad and thus fails to focus on nutrition-specific indicators. There is a need for improved justification of methodological choices in nutrition systematic reviews. We propose the need to follow standards for conducting systematic reviews such as the ‘Cochrane Handbook’ [46], as well as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for reporting systematic reviews and meta-analyses [47]. With respect to public health nutrition, there is a need for standardization in defining of the population, intervention, comparison and outcomes, as well as a need for attention to aspects of context, appropriate study designs, sustainability and applicability. For outcomes, there is a need to standardize the measurement and definition of outcomes. The Core Outcome Measures in Effectiveness Trials (COMET) initiative is a major opportunity to explore this for nutrition. The differences in methods used are consistently observed in both Cochrane and non-Cochrane reviews, suggesting a need for improved guidance for nutrition reviews.

Existing evidence related to systematic review conduct in the field of nutrition highlight the fact that systematic reviews in nutrition are unique and challenging as it encompasses various nutrition-related considerations including baseline nutrient exposure, nutrient status, bioequivalence of bioactive compounds, bioavailability, multiple and interrelated biological functions, undefined nature of some interventions and uncertainties in assessment [23]. Developing capacity-building programmes, searching the primary literature for research gaps and extending reporting tools such as the PRISMA Statement to the field of nutrition are recommended to improve the conduct of systematic reviews in nutrition [48,49].

One strength of this paper is the assessment of methods used in systematic reviews of nutrition using accepted tools including the AMSTAR and the ‘Cochrane Handbook of Public Health Guidance’. One limitation is that we used a non-validated search for nutrition topics and thus may have missed some systematic reviews. However, we think this limitation is unlikely to affect the findings of this study since we intended to identify a purposive sample, rather than an exhaustive sample of systematic reviews in nine intervention areas. Furthermore, our assessment of quality of the included systematic reviews is limited to how they were reported.

Given the growth in the field, there is a clear need to develop a consensus on methodologies for nutrition reviews, criteria for assessing the evidence and possibly facilitating development and collation of the evidence in the subject area. There also needs to be clarity on the purpose of the reviews and on the questions that they are addressing, for them to be useful in the context of programme and policy guidance. The diverse methods involved in such systematic evidence syntheses and conclusions thereof have a clear bearing on impact on policy and practice. Given the differing needs of information syntheses by practitioners and public sector policymakers, a consensus on the quality indicators for such reviews and methods is an important prerequisite. While the Cochrane and Campbell collaborations have a due process with published guidelines within the ‘Cochrane Handbook’, other journals and peer review processes may not follow a standard set of criteria. Cochrane and Campbell are well-positioned to take the lead in improving quality as our analysis also suggests that the quality ratings for the Cochrane reviews were found to be fairly consistent as compared to the non-Cochrane reviews. The Campbell International Development Nutrition group is actively working on these priority areas of methodological quality, outcome standardization and avoiding duplication. Engagement with a range of stakeholders could lead to wider uptake and endorsement of such standards and priorities. This could then expand to include guidance to other journals and agencies evaluating and publishing nutrition-related systematic reviews.

Conclusions

To conclude, there is a need to develop a consensus on methodologies for nutrition reviews, criteria for assessing the evidence and possibly facilitating development and collation of the evidence in the subject area. We therefore propose the development of a potential nutrition review editorial process and a group inclusive of major stakeholders in the field to help bring the field together.

Abbreviations

AMSTAR: 

Assessment of Multiple Systematic Reviews

COMET: 

Core Outcome Measures in Effectiveness Trials

DFID: 

Department of International Development

EPPI: 

Evidence for Policy, Practice Information and Coordinating Centre

FAO: 

Food and Agriculture Organization

MeSH: 

Medical Subject Headings

MMN: 

Multiple micronutrient

OMERACT: 

Outcome Measures in Rheumatoid Arthritis Clinical Trials

OR: 

Odds ratio

PICO: 

Participants, Intervention, Comparison and Outcome

PRISMA: 

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

RCT: 

Randomized controlled trial

RR: 

Relative risk

RUTF: 

Ready-to-use therapeutic food

SAM: 

Severe acute malnutrition

SMD: 

Standard mean difference

SOFI: 

State of Food Security

UN: 

United Nations

UNICEF: 

United Nations International Children’s Emergency Fund

USAID: 

United States Agency for International Development

WFP: 

World Food Program

WHO: 

World Health Organization

Declarations

Acknowledgements

We would like to acknowledge the academic groups, editors of the Cochrane and Campbell collaborations, 3ie, UN agencies (WHO, UNICEF, WFP, FAO), IFPRI and other bilateral agencies (notably DFID, USAID, NORAD) and representatives from the Bill & Melinda Gates Foundation for funding the work and consultation to guide this study. We would also like to acknowledge Omer Bangash, who helped us with the double data extractions.

Authors’ Affiliations

(1)
Center of Excellence in Women and Child Health, Division of Woman and Child Health, Aga Khan University
(2)
Centre for Global Health, University of Ottawa
(3)
Robert Harding Chair in Global Child Health and Policy, Centre for Global Child Health, The Hospital for Sick Children

References

  1. Global Alliance for Improved Nutrition 2014. Highlighting recent advances in understanding the bioavailability of micronutrients. Available at http://www.gainhealth.org/knowledge-centre/highlighting-recent-advances-understanding-bioavailability-micronutrients/.
  2. Shah PS, Aliwalas LL, Shah V. Breastfeeding or breast milk for procedural pain in neonates. Cochrane Database Syst Rev. 2006;12(12):CD004950.Google Scholar
  3. Bhutta ZA, Salam RA. Global nutrition epidemiology and trends. Ann Nutr Metab. 2012;61(1):19-27.View ArticlePubMedGoogle Scholar
  4. Bhutta ZA. Micronutrient needs of malnourished children. Curr Opin Clin Nutr Metab Care. 2008;11(3):309–14.View ArticlePubMedGoogle Scholar
  5. World Health O. Severe malnutrition: report of a consultation to review current literature. Geneva: World Health Organization; 2005.Google Scholar
  6. Black RE, Victora CG, Walker SP, Bhutta ZA, Christian P, de Onis M, et al. Maternal and child undernutrition and overweight in low-income and middle-income countries. Lancet. 2013;382(9890):427–51.View ArticlePubMedGoogle Scholar
  7. Black RE, Allen LH, Bhutta ZA, Caulfield LE, De Onis M, Ezzati M, et al. Maternal and child undernutrition: global and regional exposures and health consequences. Lancet. 2008;371(9608):243–60.View ArticlePubMedGoogle Scholar
  8. Victora CG, Adair L, Fall C, Hallal PC, Martorell R, Richter L, et al. Maternal and child undernutrition: consequences for adult health and human capital. Lancet. 2008;371(9609):340.View ArticlePubMedPubMed CentralGoogle Scholar
  9. Bhutta ZA, Ahmed T, Black RE, Cousens S, Dewey K, Giugliani E, et al. Maternal and child undernutrition 3: what works? Interventions for maternal and child undernutrition and survival. Child-Care Health Dev. 2008;34(3):404–404.Google Scholar
  10. Bloomberg JH. Maternal and child undernutrition 4. Maternal and child undernutrition: effective action at national level. Lancet. 2008;371:510–26.View ArticleGoogle Scholar
  11. Morris SS, Cogill B, Uauy R. Effective international action against undernutrition: why has it proven so difficult and what can be done to accelerate progress? Lancet. 2008;371(9612):608–21.View ArticlePubMedGoogle Scholar
  12. Bhutta ZA, Das JK, Rizvi A, Gaffey MF, Walker N, Horton S, et al. Evidence-based interventions for improvement of maternal and child nutrition: what can be done and at what cost? Lancet. 2013;382(9890):452–77.View ArticlePubMedGoogle Scholar
  13. Gillespie S, Haddad L, Mannar V, Menon P, Nisbett N. The politics of reducing malnutrition: building commitment and accelerating progress. Lancet. 2013;382(9891):552–69.View ArticlePubMedGoogle Scholar
  14. Ruel MT, Alderman H. Nutrition-sensitive interventions and programmes: how can they help to accelerate progress in improving maternal and child nutrition? Lancet. 2013;382(9891):536–51.View ArticlePubMedGoogle Scholar
  15. The World Bank. Fragility and Conflict. Accessed from http://wbi.worldbank.org/wbi/about/topics/fragile-states.
  16. Lavis J, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005;10 Suppl 1:35–48.View ArticlePubMedGoogle Scholar
  17. Furukawa TA, Streiner DL, Hori S. Discrepancies among megatrials. J Clin Epidemiol. 2000;53(12):1193–9.View ArticlePubMedGoogle Scholar
  18. LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F. Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med. 1997;337(8):536–42.View ArticlePubMedGoogle Scholar
  19. Trikalinos TA, Moorthy D, Chung M, Yu WW, Lee J, Lichtenstein AH, Lau J: Concordance of randomized and nonrandomized studies was unrelated to translational patterns of two nutrient-disease associations. J Clin Epidemiol. 2012;65(1):16–29.Google Scholar
  20. Villar J, Carroli G, Belizan JM. Predictive ability of meta-analyses of randomised controlled trials. Lancet. 1995;345(8952):772–6.View ArticlePubMedGoogle Scholar
  21. Cook DJ, Greengold NL, Ellrodt AG, Weingarten SR. The relation between systematic reviews and practice guidelines. Ann Intern Med. 1997;127(3):210–6.View ArticlePubMedGoogle Scholar
  22. Lavis JN, Posada FB, Haines A, Osei E. Use of research to inform public policymaking. Lancet. 2004;364(9445):1615–21.View ArticlePubMedGoogle Scholar
  23. Lichtenstein AH, Yetley EA, Lau J. Application of systematic review methodology to the field of nutrition. (Prepared by the Tufts Evidence-based Practice Center under Contract No. 290–020022). AHRQ publication No. 09–0025. Rockville, MD: Agency for Healthcare Research and Quality; 2009. Available at http://www.ncbi.nlm.nih.gov/books/NBK44076/pdf/TOC.pdf.Google Scholar
  24. Tugwell P, Petticrew M, Kristjansson E, Welch V, Ueffing E, Waters E, et al. Assessing equity in systematic reviews: realising the recommendations of the Commission on Social Determinants of Health. BMJ. 2010;341:c4739.View ArticlePubMedGoogle Scholar
  25. WHO. e-Library of Evidence for Nutrition Actions (eLENA). Access from http://www.who.int/elena/en/.
  26. WHO. Use of systematic reviews of the evidence in public health nutrition. Proceedings of an informal consultation held 18 October 2011 in Geneva, Switzerland. Geneva: World Health Organization; 2013. Available at http://apps.who.int/iris/bitstream/10665/85792/1/9789241506038_eng.pdf.
  27. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org
  28. Systematic Reviews of Health Promotion and Public Health Interventions. http://ph.cochrane.org/sites/ph.cochrane.org/files/uploads/Guidelines%20HP_PH%20reviews.pdf.
  29. Shea B, Grimshaw J, Wells G, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):10.View ArticlePubMedPubMed CentralGoogle Scholar
  30. Aggarwal R, Sentz J, Miller MA. Role of zinc administration in prevention of childhood diarrhea and respiratory illnesses: a meta-analysis. Pediatrics. 2007;119(6):1120.View ArticlePubMedGoogle Scholar
  31. Lassi ZS, Haider BA, Bhutta ZA. Zinc supplementation for the prevention of pneumonia in children aged 2 months to 59 months. Cochrane Database Syst Rev. 2010;12(12):CD005978.Google Scholar
  32. Mathew JL. Zinc supplementation for prevention or treatment of childhood pneumonia: a systematic review of randomized controlled trials. Indian Pediatr. 2010;47(1):61–6.View ArticlePubMedGoogle Scholar
  33. Roth DE, Richard SA, Black RE. Zinc supplementation for the prevention of acute lower respiratory infection in children in developing countries: meta-analysis and meta-regression of randomized trials. Int J Epidemiol. 2010;39(3):795.View ArticlePubMedGoogle Scholar
  34. Yakoob MY, Theodoratou E, Jabeen A, Imdad A, Eisele T, Ferguson J, et al. Preventive zinc supplementation in developing countries: impact on mortality and morbidity due to diarrhea, pneumonia and malaria. BMC Public Health. 2011;11 Suppl 3:S23.View ArticlePubMedPubMed CentralGoogle Scholar
  35. Funai EF, Rosenbush EJ, Lee MJ, Del Priore G. Distribution of study designs in four major US journals of obstetrics and gynecology. Gynecol Obstet Invest. 2001;51(1):8–11.View ArticlePubMedGoogle Scholar
  36. Scales Jr CD, Norris RD, Peterson BL, Preminger GM, Dahm P. Clinical research and statistical methods in the urology literature. J Urol. 2005;174(4 Pt 1):1374–9.View ArticlePubMedGoogle Scholar
  37. Rudan I, Tomaskovic L, Boschi-Pinto C, Campbell H. Global estimate of the incidence of clinical pneumonia among children under five years of age. Bull World Health Organ. 2004;82(12):895–903.PubMedGoogle Scholar
  38. Kramer MS, Kakuma R. Optimal duration of exclusive breastfeeding. Cochrane Database Syst Rev. 2002;1(1):CD003517.Google Scholar
  39. Lewin S, Munabi Babigumira S, Glenton C, Daniels K, Bosch Capblanch X, Van Wyk BE, et al. Lay health workers in primary and community health care for maternal and child health and the management of infectious diseases. Cochrane Database Syst Rev. 2010;3(3):CD004015.Google Scholar
  40. Dewey KG, Adu Afarwuah S. Systematic review of the efficacy and effectiveness of complementary feeding interventions in developing countries. Maternal Child Nutr. 2008;4:24–85.View ArticleGoogle Scholar
  41. Imdad A, Sadiq K, Bhutta ZA. Evidence-based prevention of childhood malnutrition. Curr Opin Clin Nutr Metab Care. 2011;14(3):276.View ArticlePubMedGoogle Scholar
  42. Shi L, Zhang J. Recent evidence of the effectiveness of educational interventions for improving complementary feeding practices in developing countries. J Trop Pediatr. 2010;57(2):91.View ArticlePubMedGoogle Scholar
  43. Lagarde M, Haines A, Palmer N. Conditional cash transfers for improving uptake of health interventions in low-and middle-income countries. JAMA. 2007;298(16):1900.View ArticlePubMedGoogle Scholar
  44. Lagarde M, Haines A, Palmer N. The impact of conditional cash transfers on health outcomes and use of health services in low and middle income countries. Cochrane Database Syst Rev. 2009;4(4):CD008137.Google Scholar
  45. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78.View ArticlePubMedPubMed CentralGoogle Scholar
  46. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.
  47. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.View ArticlePubMedPubMed CentralGoogle Scholar
  48. Moher D, Tricco AC. Issues related to the conduct of systematic reviews: a focus on the nutrition field. Am J Clin Nutr. 2008;88(5):1191–9.PubMedGoogle Scholar
  49. Lichtenstein AH, Elizabeth AY, Joseph L. Application of systematic review methodology to the field of nutrition. J Nutr. 2008;12(138):2297–306.View ArticleGoogle Scholar

Copyright

© Salam et al.; licensee BioMed Central. 2015

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.