Evaluation of the relative validity of a Web-based food frequency questionnaire used to assess Soy Isoflavones and nutrient intake in adolescents

Abstract

Background

Dietary assessment during adolescence is crucial in determining how adolescents’ consumption habits potentially affect current and long-term health. However, assessment methods for adolescents need to be relevant to their emerging technological culture. We developed a web-based food frequency questionnaire (webFFQ) to assess the habitual soy isoflavones and nutrient intake of adolescents. Our purpose was to validate this webFFQ against multiple 1-day photograph-assisted food records (FR).

Methods

Adolescents aged 12–18 years (n = 70) attending middle and high schools completed the webFFQ and provided six 1-day FR. Fifteen participants were excluded due to improbable energy intake. Statistical agreements were determined using Wilcoxon signed-ranks test, Pearson’s bivariate correlations of normalized values with energy-adjustment and correction for attenuation, cross-classification, and Bland-Altman plots.

Results

Based on n = 55, the webFFQ had higher intake estimates for all isoflavones and most nutrients compared to the FR. Energy-adjusted and deattenuated correlations were moderately strong for total isoflavones (r = 0.67), daidzein (r = 0.63), genistein (r = 0.64), and glycitein (r = 0.54). They ranged from a high of 0.82 (animal proteins) to a low of 0.11 (vitamin B12) for nutrients. Cross-classification agreement up to within 1 quartile ranged from 99.0 % (vegetable protein) to 61.8 % (vitamin B12) with low gross misclassifications (0.0–12.7 %). Bland-Altman plots for the isoflavones showed consistent overestimation and wide variation but nevertheless good agreement between the two methods.

Conclusions

The webFFQ is relatively valid in ranking adolescents according to their isoflavones and nutrient intake. However, while it significantly overestimates the absolute intake of most nutrients, results are comparable to other food frequency questionnaires developed for adolescents.

Background

Most chronic diseases in adults originate from lifestyle practices and experiences during childhood and adolescence [1–3]. Meal-skipping [4–6] and snacking on high-calorie and low nutrient-dense foods are more common during adolescence, and these could be detrimental to their health [4–7]. Assessment of diet during this life stage is crucial in determining how consumption habits potentially affect current and long-term health. However, assessment of diet among the youth is laden with challenges. This group may have difficulty identifying or naming the foods they eat, conceptualizing food portion sizes, and recalling their intake due to limited food vocabulary; or, lack the patience, motivation, and perseverance to engage in dietary assessment activity [8, 9].

Despite their shortcomings, particularly recall bias and the need to be culture- or population-specific [10], food frequency questionnaires (FFQ) continue to be the mainstay in the dietary assessment of large study cohorts and in determining diet-and-health outcome relationships in adolescents [11]. FFQs are low-cost, time-efficient, easy to administer, can assess long-term or habitual intake, and entail less respondent burden compared to other dietary assessment methods.

A few validated FFQs for adolescents residing in the United States exist, mostly in the paper format [12–18]. Since a large percentage of adolescents spend a significant amount of time with technological devices, i.e., computers and mobile phones [19], the current technological revolution and emerging culture for young populations should be considered in the design of dietary assessment tools. Tools that tap into the technological skills of adolescents may keep them engaged and interested in dietary assessment activities [9]. In recent years, digital dietary assessment methods have been developed in an effort to improve the accuracy of food and nutrient consumption estimates [20] and decrease respondent burden in quantifying food intake [21]. Thus far, a few online FFQs have been developed for adolescents [22–25]. However, none of the existing FFQs, paper-based or online, met our need to assess the isoflavone intake of a multi-ethnic adolescent group, ~25 % of whom are vegetarians and highly exposed to soy-containing foods.

We conducted the Teen Food and Development Study (TeenFADS) to investigate if soy isoflavones intake is associated with the health and development of adolescents. To make the dietary assessment more engaging and to improve compliance, we developed a web-based FFQ (webFFQ). Considering the context of ubiquitous use of mobile phones with a camera, the use of technology becomes more affordable in epidemiological studies if personal gadgets with unlimited texting and calling are employed. Thus, we chose multiple 1-day photograph-assisted food records (FR) as the reference comparison method and designed it such that participants used their own mobile phones to record their intake in text form and digital photographs. The purpose of this study was to validate the semi-quantitative webFFQ we developed for TeenFADS to estimate the habitual consumption of soy isoflavones and selected nutrients using multiple 1-day FR as the reference method.

Methods

Study design

The TeenFADS is a cross-sectional study that investigated the associations between dietary intake and health and pubertal development of adolescents. Data were collected using an online questionnaire that consisted of several sections including a dietary assessment section (webFFQ) and school visits for the anthropometric measurements. This current study was designed to validate the webFFQ using multiple 1-day FR as reference method. Participants completed the webFFQ, underwent a one-on-one training to do FR, and then provided 6 1-day FR over 2–3 months. They utilized their personal mobile phones for their FR. The Institutional Review Boards of Loma Linda University (LLU) and Andrews University approved the study protocol and the informed consent process for both parental permission and assent of the adolescent participant.

Study participants

The TeenFADS involved adolescents aged 12–18 years who attended middle and high schools near Adventist universities in southern California and Michigan. Convenience sampling was done to recruit volunteers for the validation sub-study from among those who completed the webFFQ and attended the anthropometry clinic (n = 601). A question at the completion of the web-based questionnaire asked respondents if they were willing to participate in a sub-study. Those who responded positively were contacted and participants were selected based on the following criteria: (1) owned a mobile phone with good camera resolution; (2) had unlimited texting/calling; (3) after their one-on-one training, demonstrated through a return demonstration test the ability to follow video and/or printed instructions on how to take digital photographs correctly; and (4) could text messages and digital photographs to an email account set up for the study. Initially, 108 adolescents volunteered but of these, 23 did not meet the first two requirements. Of those who passed the training (n = 85), eight did not proceed with the study due to a busy schedule, five either lost or did not receive their fiducial markers and could not proceed with the return demonstration, and two lost interest in the study before it started. Thus, a total of 70 participants entered the study.

Web-based food frequency questionnaire

The webFFQ is a 151-item self-administered, semi-quantitative dietary assessment questionnaire that was designed to assess the soy intake and habitual diet of adolescents. The food list was initially composed of foods from a pilot questionnaire constructed at LLU that was administered among adolescents to determine their soy foods intake. Foods deemed commonly eaten based on interviews of multiethnic adolescents were added to this list. Registered Dietitians (RDs) who work with adolescents reviewed the list to ensure that the food names used were appropriate for this population. A comprehensive list composed of 151 food items, 36 of which were soy-containing, were included in the final version (see Additional file 1: Table S1). Food items were grouped into eight categories that constituted eight screens in the webFFQ. The webFFQ was then pre-tested on 25 adolescents of the same demographic characteristics but were not participants in the study. This was done to determine clarity, interpretation, length of completion time, and if respondents would use the pop-up feature for additional information about an item if they needed further clarification. No conceptual difficulties in filling out the webFFQ were reported and the majority of participants did not find it necessary to seek additional information about the food items. Technical issues with use of specific browsers were resolved before administering the webFFQ.

Foods in the webFFQ are categorized as convenience foods (32 items), protein-rich foods (29 items), starches/cereals (17 items), vegetables/fruits (21 items), dairy products (10 items), beverages (24 items), snacks/sweets (11 items), and soups/legumes (7 items). These food groups are divided over eight screens (see Additional file 2: Figure S1). Food items are arranged in such a way that various types of a specific food are grouped (e.g., dairy milk: regular, low-fat or non-fat). Protein-rich foods such as meats also have their meat alternative counterparts (e.g., chicken, vegechicken) (see Additional file 3: Figure S2). More information about the food and examples pop up when the pointer is hovered over the food name (see Additional file 4: Figure S3). Fixed portion sizes are based on familiar measuring devices, e.g., cup, tablespoon, 12-fluid ounce can, and others.

On the webFFQ, respondents are asked to self-report the frequency of their intake during the past month. Frequency of intake is selected from a drop-down list of frequency categories which are: never/rarely, 1–3 times per month, once per week, 2–4 times per week, 5–6 times per week, once per day, 2–3 times per day, and 4 or more times per day (see Additional file 4: Figure S3). For seasonal foods, e.g., certain fruits, “when in season” is included as frequency of intake category. To ensure that no item is skipped, respondents are not allowed to move on to the next food category until all items on the screen are completely filled out. However, they can go back to make changes to their responses in any food category if desired.

The current variety of meat analog brands has various formulations. More specifically, some contain more gluten than soy protein or contain no soy protein at all. To differentiate between these formulations and more accurately determine soy and isoflavones intake, a separate window of different brand names for meat analogs appears for respondents to choose from when a frequency of “once per week” or greater is chosen. Frequency of “once per week” or greater for meat intake also shows a separate window where respondents choose the type of meat eaten (beef, chicken, turkey, lamb or pork). It takes approximately 25 min to complete the webFFQ.

Food records with digital photographs

Multiple day food records provide a better measure of food and/or nutrient intake and, thus, is commonly used as a criterion reference measure in evaluating or validating FFQs [26, 27]. This dietary assessment method is often paper-based and recording is done on consecutive days. Weighing or taking a photograph of the foods before and after they are eaten are some of the approaches used to improve the accuracy of food records. As the criterion measure for this study, we determined the feasibility of using personal mobile phones for photographing foods to be consumed and recording intake as part of the food recording process among our young study participants. This method is described in a forthcoming article [28]. Instead of consecutive days of recording, the protocol was designed to reduce the burden associated with continuous food recording which may result in deviation from usual intake [27] or boredom, and to make the procedure more attainable among adolescents.

Before the data collection, participants were trained and then tested through a return demonstration of a one-day FR that was then evaluated to determine readiness, or additional training, for the study. During the trial runs, a few of the participants were excluded for losing their fiducial marker. Laminating the fiducial marker, adding a hole on one side and inserting a 2-foot long 1-inch wide ribbon through the hole resolved the problem of misplacing or losing the marker. Participants were randomly scheduled for six 1-day food recording days over a 2- to 3-month period. All five weekdays (Monday to Friday) and one weekend day (Sunday) were covered in scheduling the recording days.

The FR had two components: (1) digital photographs before- and after-intake of a meal that captures a fiducial marker for size approximation of objects in the digital photograph; and (2) text message that captures the time of intake, meal type (breakfast, snack, lunch, or dinner), and the foods eaten with corresponding amounts. Participants used their personal mobile phones to take digital photographs and record their intake. They sent their text report and photographs of their intake to a central location (i.e., the email address for the study). Reports from each participant (all digital photographs and texts sent during their reporting day) were reviewed and collated into food records the following day by trained research assistants. The amount of intake reported in the text was compared with the before- and after-eating digital photographs (after-eating digital photographs were only sent if there were leftovers) to determine if amount reported was estimated accurately or needed to be adjusted. If reports were incomplete, unclear, have discrepant information or other issues, the participant was contacted for clarification. Otherwise, the FR was cleared and logged on as completed.

Isoflavones and nutrient intake determination

Dietary intake reports on both the webFFQ and FR were coded using the Nutrition Data System for Research (NDS-R) software version 2012 developed by the Nutrition Coordinating Center, University of Minnesota, Minneapolis, MN, for nutrient composition and analysis. The NDS-R has a comprehensive database of about 18,000 foods—which reflects variations according to preparation methods and component ingredients—and 163 nutrients/nutrient ratios/other food components derived from the USDA database and several other sources [29]. The comprehensive meat analogs database of the Adventist Health Study-2 [30] was utilized in determining the nutrient profile of meat analogs. Frequency values of average consumption during the past month were converted into frequency of intake per day, e.g., frequency of 1–3 per month = 2/30 = 0.067, and so on. Thus, factors used were 0 for “never/rarely”, 0.067 for “1–3 times per month”, 0.143 for “once per week”, 0.429 for “2–4 times per week”, 0.786 for “5–6 times per week”, 1.0 for “once per day”, 2.5 for “2-3 times per day”, and 5.0 for “≥4 times per day”. Nutrient amounts were relative to the fixed portion/serving sizes in the webFFQ. For vegetables that are eaten either raw or cooked, the average of raw and cooked nutrient components was used. For mixed foods, recipes were created by trained RDs and nutrient composition was based on the proportion of ingredients in the recipe. For single food items where several types exist, the NDS-R software computes nutrient composition for the most commonly eaten type (the “default” food), a determination made using nationally representative market research data [29]. Nutrient and isoflavones intake per day was computed afterwards using the product-sum method [27]. Daily FRs were coded on NDS-R separately for each participant.

Statistical methods

All 70 participants selected for this validation sub-study completed six recording days. However, 15 participants had to be excluded from the analyses: four due to improbable intake on the webFFQ which we predefined as >4500 kcal or <1000 kcal for boys and >3500 kcal or <900 kcal for girls, and 11 due to poor or non-compliance with the protocol in food recording. Thus, the analyzable data was based on information from 55 participants.

Most of the nutrients and isoflavones had skewed distributions. Absolute median intake estimates from the webFFQ and the FR were compared using Wilcoxon signed-ranks test. Intake values were either log- or square root-transformed to achieve normality before Pearson’s bivariate correlation analyses were performed. Correction for within-person variation in multiple FR measurements was performed using the formula r _c  = r _o [1 + (S_w²/nS_b²)]^½ [31] where r _c  = corrected correlation coefficient, r _o  = crude correlation coefficient between FFQ and mean of the multiple food records, S_w² = within-person variance of the multiple food records, S_b² = estimate of the between person variance in the reference method (food records), and n = number of repeated measures of the food records. Using this method for deattenuation or correction of the correlation coefficients creates conditions in which normally distributed errors for r _c cannot be assumed. Thus, instead of the traditional asymptotic methods to determine confidence intervals about r _c , we computed ‘distribution-free’, non-parametric 95 % confidence intervals using the bias-corrected and accelerated (BCa) bootstrap re-sampling method [32], with each confidence interval determined from the distribution of r _c ’s from 2000 samples.

Analytical tests were performed without energy adjustment and with energy adjustment using the residual method. The ability of the webFFQ to rank respondents according to intake was tested by determining classification agreement between quartiles of consumption levels according to webFFQ responses and the mean of the multiple FR. Proportions of exact agreement, within 1 quartile agreement, and gross misclassification were also estimated. Additional graphical determination of agreement with the reference method using Bland-Altman plots was done for the soy isoflavones (total, daidzein, genistein, and glycitein). Total isoflavones was computed as the sum of daidzein, genistein, and glycitein. Analyses were performed using the Statistical Analysis System (SAS) statistical software package version 9.4 (SAS Institute Inc., Cary, NC, USA) and R version 3.1.2 [33].

Results

Participants in the study included 33 girls and 22 boys with a mean age of 15.3 (standard deviation, SD = 1.7) years. Forty-six percent of the participants were Caucasians and the mean body mass index (BMI) z-score was 0.49 (SD = 1.03). The analyzable group (n = 55) had similar characteristics as the entire study population except for the proportion of vegetarians (see Table 1). Those who were excluded did not significantly differ from the analyzable group (not shown).

Table 1 Demographic characteristics of the study population

Table 2 shows the comparison of isoflavones and nutrient intake estimates between the webFFQ and the average of six 1-day FR using the Wilcoxon signed ranks test before and after energy adjustment. For most of the nutrients, estimates according to the webFFQ were higher compared to FR but not significantly for trans fats, carbohydrate, vitamin E and caffeine even after energy adjustment. WebFFQ estimates of isoflavones were about 300 % more than the FR estimates.

Table 2 Comparison^a of reported soy isoflavones and nutrient intake on the web-based food frequency questionnaire with the reference method (food records with digital photographs) without energy adjustment and with energy adjustment (n = 55)

Table 3 shows the correlation coefficients before and after adjustment for energy. Deattenuated correlations for energy-adjusted soy isoflavones ranged from 0.63 to 0.67. In general, energy adjustment improved the correlation values except for folate, vitamin B12, calcium, iron, and zinc. Deattenuation after energy-adjustment further improved the correlations between the two methods for most of the nutrients except zinc, calcium, folate and iron–which remained the same or increased very slightly–and vitamin B12, which decreased to a non-significant value that was also the lowest correlation. Deattenuated energy-adjusted correlations between the two methods were weak (i.e., r < 0.35) for vitamin B12 (r = 0.11) and zinc (r = 0.26); moderate (i.e., r between 0.35 and 0.50) for calcium, folate, vitamin C, iron, vitamin E, retinol, and protein; and strong (r > 0.50) for beta carotene, total fat, saturated fatty acids (SFA), monounsaturated fatty acids (MUFA), polyunsaturated fatty acids (PUFA), trans fats, animal protein, vegetable protein, and dietary fiber. Corrections for measurement error in the FR showed that within-person variances were higher than between-person variances, an indication that this group of adolescents have a wide variation in their day-to-day intake.

Table 3 Validity correlations between the webFFQ and FR estimates of nutrients and soy isoflavones, before and after energy adjustment (n = 55 adolescents)

To further determine if the webFFQ provides a valid ranking of isoflavones and nutrient intake, agreement between quartile rankings on both methods was performed through cross-classification analysis (Table 4). Exact matches were moderate for soy isoflavones (38–49 %) and high for matches within 1 quartile (80–84 %). For the nutrients, exact match was lowest for vitamin B12 (25.5 %) and highest for animal protein (52.7 %) while matches within one quartile ranged from 61.8 % (vitamin B12) to 99.0 % (vegetable protein). Gross misclassifications ranged from 0 % (total isoflavones, animal and vegetable proteins, and vitamin E) to 12.7 % (zinc). Results of the cross-classification analyses were consistent with the correlation analyses results.

Table 4 Cross-classification of the ranked quartile nutrient and isoflavones intake estimates between the webFFQ and FR (n = 55)

Figure 1 shows the Bland-Altman graphs for total isoflavones, daidzein, genistein, and glycitein. The difference between estimates on the webFFQ and FR was plotted against the mean of the estimates of the two methods. All graphs show that the mean difference line is above zero, which indicates that the webFFQ consistently overestimated intake relative to FR. While a few of the dots appear closer to zero, more dots are scattered both above and below the mean difference between the two methods but very few of these dots are outside the limits of agreement (shaded space). The distribution of the dots also distinctly show non-consumers—shown as a few dots flocked at 0,0—whereas the majority of the dots are spread out which indicates a larger proportion of this group were consumers with a wide variation in isoflavones intake (Fig. 1).

Fig. 1

Bland-Altman graphs for total isoflavones, daidzein, genistein, and glycitein intake of the group that reported intake of soy-containing foods (n = 55). The graphs plot the difference between the energy-adjusted values for the two methods (web-based food frequency questionnaire [webFFQ] and food record with digital photograph [FR]) against the average of the two methods. Shaded part of the graph indicates the limits of agreement

Discussion

The webFFQ is a comprehensive semi-quantitative questionnaire that was developed to assess isoflavones and nutrient intake of adolescents in a population presumably exposed to soy foods. We sought to evaluate the performance of this FFQ by comparing its absolute intake estimates, degree of agreement, and ranking ability with six days of 1-day FR as the reference method on soy isoflavones (total, daidzein, genistein, and glycitein), 20 nutrients, and caffeine. Overall, the webFFQ performed equivalently well as the FR in ranking the intake estimates for isoflavones and most of the nutrients. Corrected energy-adjusted correlations between the webFFQ and FR were moderately strong (r is between 0.50 and 0.70) for all the isoflavones and ranged from moderate (r is between 0.3 and 0.5) to strong (r > 0.5) for 18 of the 20 nutrients. Vitamin B12 and zinc were the only nutrients with weak correlations. Degree of agreement within one quartile was within the acceptable range (62 % for vitamin B12 to 99 % for vegetable protein) and ranking ability was congruent for a substantial proportion (>80 %) of the participants specifically for SFA, total protein, animal protein, vegetable protein, dietary fiber and the isoflavones (total isoflavones, daidzein, genistein, glycitein). However, compared to the FR, the webFFQ had significantly higher estimates of absolute intakes of energy, isoflavones and all nutrients, except carbohydrates, vitamin E, trans fats and caffeine.

In view of the findings regarding the use of technology in the dietary assessment of young people [9], we found a high completion rate of the webFFQ (only 2 % of the participants did not complete). This could be attributed to the privacy, confidentiality of responses, and the convenience of filling out the questionnaire at home, at school, or where access to a computer and the internet was available.

Written food records and 24-h recalls are usually employed as standards in evaluating and validating FFQs. Given that digital photographs of foods eaten were obtained, using FR as the reference method allowed triangulation when evaluating portion size estimates and in identifying carelessly omitted food items in the texted food report. Considering that digital photographs improve energy and macronutrient intake estimates when combined with dietary food records [34], FR is certainly an appropriate alternative to the conventional written food record for this age group. Moreover, the FR afforded a cost-effective technology-based comparison method given that participants were willing to use their personal phone with a built-in camera and unlimited texting.

Relative to two other web-based [24, 25] and six paper-based [35–40] FFQs for adolescents which were validated against either multiple 24-h recalls or food records, our webFFQ shares similar overestimation bias for most nutrients. Of these eight FFQs, all have higher estimates for dietary fiber, 7 overestimated energy and fat and 6 overestimated calcium. Out of seven FFQs that validated carbohydrate and protein, 6 overestimated these nutrients while 3 have higher estimates for iron. Three of the 4 FFQs that validated vitamin C and retinol have higher estimates for these nutrients; and, out of three FFQs, two overestimated magnesium and saturated fat while one overestimated folate and monounsaturated fat.

Under-reporting of true intake in dietary food records [41, 42] coupled with overestimation tendencies during completion of lengthy FFQs may explain the lack of accuracy in the absolute intake estimates in this current study. Overestimation bias may also be attributed to inherent errors associated with the methods or the nature of the study population itself. Bias associated with under-reporting—by either a deliberate or careless omission of foods, or misreporting portions eaten, e.g., to avoid taking additional digital photographs for second helpings—and over-reporting in the webFFQ may explain the discrepancies between the two methods. Further investigation is needed to determine the sources of these biases.

One strength of this study was the use of a comparison method considered least likely to have similar errors inherent in FFQs, specifically reliance on memory [27]. Another strength was that the methods used were age-appropriate, less burdensome and less time-consuming for the respondent compared to the traditional paper-based food recording. In addition, digital photographs that accompanied the food reports allowed for adjustment of miscalculated portion sizes and identification of missed foods, considering that two-dimensional images could be more accurate at estimating food volume and in identifying cooking methods than 3-dimensional images [43]. Another strength was the human intervention in collating the final dietary food record. The follow-up calls from our research nutritionists within the next 48 h after the reporting day further ascertained the accuracy of the reports and the details needed for food components not apparently seen in the pictures.

Our study has a number of limitations. In our attempt to reduce respondent burden, our intermittent collection of 1-day food records may have not captured the inherent daily variation in intake compared to that of continuous food recording [27]. However, our results showed higher within-person than between-person variances in the 6 1-day food records. This indicates that intermittent food records can still catch daily variation in intake. Another limitation, common to most validity studies, is the use of convenience sampling and a small sample size. For this reason, we collected six FRs per participant to improve the precision of the comparison method in capturing habitual intake. Another limitation is the lack of a biochemical marker to assess isoflavones intake. However, it was already demonstrated that FFQ isoflavones estimates are significantly correlated with urinary isoflavonoids in a cohort of adults that includes parents of some of these adolescents [30]. The use of digital photographs in food records made this method less burdensome to respondents; however, managing the texts/digital photographs and quality control of the digital photograph-assisted FR on the part of the research team was still time-consuming [28]. Likewise, although human intervention in ascertaining the accuracy of food records submitted by participants is a strength, it is still subject to several errors, i.e., judgment, inter-evaluators, and coding. On the other hand, these limitations would not differ from managing food records collected through conventional methods.

Recall of dietary intake is more challenging for adolescents than adults so using the most recent past and shorter time coverage may reduce recall bias in adolescents. This also reduces the burden of quantifying and averaging intake based on fixed portion sizes. However, this approach has its adjoining disadvantages, including the reduced ability to capture seasonal variations in the diet. We were not able to do a repeat administration of our webFFQ to determine if reported intakes are reproducible. A reproducibility study could have established confidence in the ability of our webFFQ to measure usual intake especially when the FFQ was designed to measure intake of the most recent past.

Conclusions

Compared to multiple 1-day food record with digital photographs, the 151-item webFFQ we developed for the TeenFADS is relatively valid in ranking adolescents according to their intake of soy isoflavones and nutrients. However, this web-based tool significantly overestimates adolescents’ absolute intakes of soy isoflavones and most of the selected nutrients. Overall, the webFFQ may be used to classify adolescents in this population according to their dietary intake.