The Effect of Presentation Media on the Validity of an Asthma

The Effect of Presentation Media on the Validity of an Asthma Identification Survey Instrument
Roger Pottanat, M.D.5; Mark Graves, M.D.3; Anne McLaughlin, M.D.2; Mark Graves, Jr.; James Brokaw, PhD, MPH1; Bruce Shelton, M.D.4; Majed Koleilat, M.D.2; James Connors, M.D.4 and Courtney Kiefer. 1Medical Student Affairs, Indiana University of Medicine, Indianapolis, Indiana, United States, 46202;
2Allergy and Immunology, Welborn Clinic; 3Internal Medicine, Welborn Clinic; 4Pediatrics, Welborn Clinic and 5Clinical Research Center, Welborn Clinic, Evansville, Indiana, United States, 47714.
ABSTRACT
BACKGROUND
According to the CDC the prevalence of asthma among U.S. children increased from 3.6% in
1980 to 5.8% in 2003.9 It is now the third leading cause of hospitalizations among persons under
18 years, exceeded only by pneumonia and injury.9 It is estimated that approximately 15% of
school aged children have undiagnosed asthma.1 As many as an estimated 25% of children in
underperforming schools have undiagnosed asthma.4 In 1998 asthma cost the United States
approximately eleven billion dollars2, but a large percentage of that could be saved if a suitable
identification method is put in place to reduce emergency room visits for untreated asthma. 3
Symptom based questionnaires remain the cornerstone of asthma prevalence surveys because
they obtain valid data on large numbers of people, with high response rates, in a manner that is
comparable across the social groups, regions or countries being compared. 5 The goal of this
project is to create a survey method that could be integrated into the annual school health
screening tests. With children developing both emotionally and intellectually at different
rates, current paper asthma surveys face literacy and psychological barriers that impede accurate
pediatric responses. The three barriers of communication addressed in this project are: “failure
to be able to understand the mechanics of the words, i.e. cannot read, failure to understand the
meaning of the words, i.e. lack of understanding the meaning of the words, and failure to
understand the subtleties of the word in context.”5 There is also a psychological influence of
questioning a child about a medical condition.6 Schools already have in place annual screening
for commonly undiagnosed childhood diseases. “Schools are one of the few sites in which almost
all children gather and thus are available for screening or identification programs. Schools offer
the opportunity to evaluate a large number of children and to identify treatable diseases, such as
asthma. School-based asthma and allergy case identification programs require a validated
screening tool that provides sufficient sensitivity to identify most of the cases, while limiting the
number of referrals of children who do not have asthma or allergy.” 8 We believe that by changing
the survey into an in-school, interactive computer program, children would be more engaged and
give more accurate responses.
DESIGN
SUMMARY
Question Comparison among Asthma vs. Healthy
0.60
0.50
0.40
0.30
0.20
0.10
Child Flat
0.00
Child Read
Summary
Our interactive child survey proved to be the best screening tool of the three surveys.
Improvement in screening efficacy was only seen when exclusively analyzing data from
children in the first through fifth grades. The survey is an effective method of screening
large number of children simultaneously, regardless of their age or school environment.
The web based design of this survey reduces the need for supervision or paperwork.
Because the interactive survey was the best among our three, we could effectively
screen children by using that alone. That survey takes an average of five to ten minutes
for each child. Children that show a possibility of asthma can be directed to an allergist
for testing and further management. We will continue to gather data for the remainder
of the school year and the summer. After publication of this study, the Vanderburgh
County School Corporation has agreed to conduct the second phase of our study within
their school system. If you are interested in collaborating on such a project please
contact us and be sure to ask for one of our INFO CD's.
Child Interactive
ANALYSIS
Question Comparison among Asthmatic Children
ACKNOWLEDGEMENTS
Parent Flat
The questions on noisy or wheezy breathing, coughing while active and chest tightness proved
to have the greatest potential in differentiating between asthmatics and healthy children when
answered by children. There was an increase in identification power when the children
answered the question about their inability to stop coughing.
1.20
RECEIVER OPERATING CURVE
1.00
1.0
0.80
0.60
0.40
0.20
0.00
Child Flat
Child Read
Child…
Parent Flat
The question about chest tightness showed the greatest potential for identifying which children had
asthma. This question, along with the questions concerning noisy breathing, coughing with
activity, and coughing that is difficult to stop, had a greater reliability when answered by the children
compared to their parent's answers.
0.2
1.00
0.90
0.80
REFERENECES
0.0
0.2
0.4
0.6
0.8
PEDIATRICS
Dr. James Connors, Linda Rodgers, Dr. D.
Bruce Shelton, Judy Ford, Jayme Rainer,
Julie Myerrose (not in picture)
Greg A. Folz and Dr. David Christeson
Welborn Clinic Administration
CHILD
(Interactive
Survey)
AUC = 0.919
0
Clinical
Research
Center
Discussion
PARENT (Flat
Survey)
AUC = 0.949
0.4
Allergy &
Immunology
Dr. Mark Graves, Dr. Anne McLaughlin, Dr. Majed
Koleilat , Dr. Roger Pottanat
CHILD (Read
to - Survey)
AUC = 0.903
0.6
False +
Question Comparison among Healthy Children
Internal
Medicine
CHILD (Flat
Survey)
AUC = 0.916
0.8
True + (Sensitivity)
Background: Adding an asthma screening tool to the school health curriculum would keep many
children from developing the manifestations of untreated asthma, from poor school performance
to emergency room visits. Literacy barriers, identified as failure to be able to understand the
mechanics of the words, failure to understand the meaning of the words, and failure to
understand the subtleties of the questions in a context, serve to decrease the validity of current
student asthma questionnaires.
Objective: To assess whether the integration of a previously validated student questionnaire into
an interactive, web based, computer program can enhance the validity of the survey.
Design/Methods: Based on a previously validated 7 question survey of children (Redline et
al), an interactive, web-based questionnaire program (asthmatest.org) was developed that
mitigates the 3 barriers discussed above. The test group consisted of patients selected from both
departments of pediatrics and allergy and immunology. The subjects use the program, learn about
asthma and take surveys about their potential asthma while parents complete another survey.
The pediatricians and pediatric allergists provide the diagnosis of asthma or healthy.
Results: Five questions were very predictive of asthma, however, a composite survey score was
a better prediction of asthma. A post education composite score was comparable to the parent
test for asthma prediction.
Conclusions: Asthmatest.org is an accurate child screening tool comparable to a parent
questionnaire. A school based screening program could be accomplished as the survey requires
no paperwork, can be run simultaneously on multiple computers, stores and tracks results, and
minimizes adult supervision.
WELBORN CLINIC
1
(1 - Specificity)
The AUC values of each ROC analysis for the different surveys show that our child
surveys were quite close in their abilities to determine the asthma status of the child.
When compared to the parent survey, none of our child surveys proved to be as
effective. The interactive survey did prove to be the best among our child surveys.
DISCUSSION
1. Glasgow GJ, Ponsonby AL, et al. Asthma screening as part of a routine school health assessment in the Australian
Capital Territory. The Medical Journal of Australia 2001; 174: 384-388.
2. National Heart, Lung, and Blood Institute. “Data Fact Sheet on Asthma Statistics.” National Institutes of Health.
Bethesda, MD. 55 (1997):798.
3. Asthma in America. October 2005. 25 Oct. 2005 http://www.asthmainamerica.com/children_index.html
4. Taras, Howard and William Potts-Datema. “Childhood Asthma and Student Performance at School.” Journal of
Science Health 75.8 (2005): 296-313.
5. Brokaw, James “The Effect of Presentation Media on the Validity of an Asthma Identification Survey Instrument.” IRB
Submission. 24 Oct. 2005.
6. Pearce et al. Measuring Asthma Prevalence. Asthma Epidemiology: Principles and Methods 1998; 75-113. Oxford
Univ Press.
7. Schuman, Howard and Jacqueline Scott. “Problems in the Use of Survey Questions to Measure Public Opinion.”
Science 236 (1987): 957-960.
8. Redline Et. Al. “Development and Validation of School-Based Asthma and Allergy Screening Questionnaires in a 4-City
Study.” Annals of Allergy and Asthma Immunology 93 (2004): 36-48.
9. Eder et al. The Asthma Epidemic. The New England Journal of Medicine 355 (2006):2226-35
CONTACT INFORMATION
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
Child Flat
Child Read
Child Interactive
Parent Flat
All but the question concerning chest tightness had a good negative predictive value, in ruling out
children with potential asthma, whether answered by the child or the parent.
Our analysis showed that the expected increase in efficacy of the surveys did not occur
as the presentation media changed with each survey. One reason for this might be that
almost all of the asthmatic children were well managed cases who had their asthma
under control. For publication we will try to determine a new formula that will correct for
this situation. Another reason for this finding might be the length of time needed for
each child to perform the survey and the fact that the questions are repetitive. Many
children did express annoyance at having to repeat their answers three times.
Since our target screening population is younger students an ROC analysis was also
run for the 37 students that were in the first through fifth grades. This data did show a
progressive change in the efficacy of the surveys as evidenced by the following data:
PARENT (Flat Survey)
AUC = 0.975
CHILD (Flat Survey)
AUC = 0.933
CHILD (Read to - Survey) AUC = 0.943
CHILD (Interactive Survey) AUC = 0.955
Dr. Roger Pottanat
&
Dr. Anne McLaughlin
Welborn Clinic
421 Chestnut St.
Evansville, IN 47713
812--426
812
426--9459
Printed by