Development of a Performance Assessment of Executive

Development of a Performance Assessment of Executive
Function: The Children’s Kitchen Task Assessment
Kristy Rocke, Paige Hays, Dorothy Edwards, Christine Berg
KEY WORDS
• cognition
• executive function
• pediatrics
• performance assessment
OBJECTIVE. This study developed and explored the validity of the Children’s Kitchen Task Assessment
(CKTA), a performance assessment of executive function.
METHOD. The development of the CKTA is described. Children were given the CKTA and neuropsychological assessments of executive functioning. Parents completed the Parent Behavior Rating Inventory of Executive
Function (Parent BRIEF).
RESULTS. Forty-nine typically developing children ages 8 to 12 years participated in this study. Interrater
reliability and internal consistency were established. Preliminary evidence of discriminant validity was reflected
in significant differences on neuropsychological tests and the Parent BRIEF between high- and low-scoring
CKTA groups. Age-related differences in CKTA performance further supported discriminant validity. Support for
concurrent validity was observed in moderate correlations with established neuropsychological tests.
CONCLUSION. Preliminary results suggest the CKTA is a valid performance assessment that evaluates the
level of cognitive assistance children require to complete a challenging functional activity.
Rocke, K., Hays, P., Edwards, D., & Berg, C. (2008). Development of a performance assessment of executive function: The
Children’s Kitchen Task Assessment. American Journal of Occupational Therapy, 62, 528–537.
Kristy Rocke, OTD, OTR/L, is a graduate of the
Washington University School of Medicine Program in
Occupational Therapy, St. Louis, MO.
Paige Hays, MS, OTR/L, is a graduate of the
Washington University School of Medicine Program in
Occupational Therapy, St. Louis, MO.
Dorothy Edwards, PhD, is Associate Professor,
Kinesiology–Occupational Therapy, Neurology, and
Medicine, University of Wisconsin–Madison, Madison,
WI.
Christine Berg, PhD, OTR/L, is Instructor, Program
in Occupational Therapy and Neurology, Washington
University School of Medicine, 4444 Forest Park Avenue,
St. Louis, MO 63108; [email protected]
P
articipation in everyday life activities requires executive functioning skills to
formulate goals, to plan strategies to achieve those goals, and to self-evaluate
during these activities (Lezak, 1982). Lezak (1982) described executive functioning
as essential for social and constructive performance that is required for productive
life. Because executive function is vital for the performance and maintenance of
occupations across the lifespan, occupational therapists must evaluate and address
these performance issues (Katz & Hartman-Maeir, 2005). Children and adolescents
with executive function deficits often experience difficulties with participation in
everyday, meaningful activities (Biederman et al., 2004).
Developmental psychologists have written extensively about the assessment and
treatment of dysexecutive syndromes in children with various diagnoses (e.g., attention deficit/hyperactivity disorder, autism, epilepsy, traumatic brain injury, and
sickle cell disease; Geurts, Verte, Oosterlaan, Roeyers, & Sergeant, 2004; Gioia,
Isquith, Kenworthy, & Barton, 2002; Levin & Hanten, 2005; Parrish et al., 2007;
Wang et al., 2001). Traditionally, neuropsychological assessment of children is
based on evaluation of the subcomponents of cognition and executive function such
as memory, attention, inhibition, and verbal or spatial skills. Neuropsychological
assessments provide valuable information about the level of cognitive functioning
in a specific area; however, the ecological validity, or the degree to which an assessment can evaluate or be related to performance in everyday activities, has been
questioned (Burgess, Alderman, Evans, Emslie, & Wilson, 1998; Chaytor &
Schmitter-Edgecombe, 2003; Manchester, Priestly, & Jackson, 2004). Direct assessment of functional activities is thought to demonstrate better ecological validity and
528
Downloaded From: http://ajot.aota.org/ on 08/22/2014 Terms of Use: http://AOTA.org/terms
September/October 2008, Volume 62, Number 5
thus provide a more accurate indicator of performance in a
wide range of everyday activities and contexts (Alderman,
Burgess, Knight, & Henman, 2003; Baum & Edwards,
1993; Burgess, 2000; Burgess et al., 1998; Channon &
Crawford, 1999; Goel, Grafman, Tajik, Gana, & Danto,
1997; Shallice & Burgess, 1991). Direct observation of performance in context allows an occupational therapist to view
the child’s strengths and weaknesses and to identify factors
that facilitate or inhibit performance. Currently, few reliable
and valid standardized performance assessments measure
executive function, particularly for children and adolescents;
thus, development of additional performance assessments
for the use of pediatric occupational therapists is needed.
The most common performance assessments are the
Kitchen Task Assessment (KTA; Baum & Edwards, 1993),
the Executive Functioning Performance Test (EFPT; Baum
et al., 2008), the Assessment of Motor and Processing Skills
(AMPS; Fisher, 2003), and the School AMPS (Fingerhut,
Madill, Darrah, Hodge, & Warren, 2002). The KTA and
EFPT were designed to objectively measure the level of assistance adults require to perform a specific instrumental activity
of daily living (IADL; Baum & Edwards, 1993; Baum,
Edwards, Hahn, & Morrison, 2003). The KTA objectively
measures the level of cognitive assistance a person needs during a cooking task by assessing the cognitive or executive
function elements of task performance. These elements
include initiation, organization, sequencing and planning,
judgment and safety, and completion (Baum & Edwards,
1993, 2003). The EFPT uses the same format and scoring
system as the KTA with an expanded group of IADLs, including the KTA cooking task, bill paying, medication management, and making a telephone call. Both the KTA and the
EFPT provide detailed information for the development of
intervention strategies for clients with performance deficits
(Baum & Edwards, 2003; Baum et al., 2008). The AMPS
and School AMPS evaluate motor and processing skills during a variety of daily living tasks in home and school environments, but unlike the KTA and EFPT, the AMPS and School
AMPS do not evaluate specific executive function skills or
determine the level of assistance required to complete the task
(Fingerhut et al., 2002; Fisher, 2003).
Determining cognitive capacity and the level of assistance needed for successful task completion are essential for
the promotion of performance and development in children
with cognitive impairments (Vygotsky, 1978). Therefore, a
pediatric assessment tool documenting the level of cognitive
assistance needed to support successful performance would
be beneficial. This information can be used to educate parents or teachers about the level and types of assistance needed
for a child to be successful in everyday activities. The goal of
the current study was to modify the KTA for use with chil-
dren. This pediatric version of the KTA is named the
Children’s Kitchen Task Assessment (CKTA). This article
describes the development of the CKTA and examines concurrent and discriminant validity of the CKTA in a sample
of typically developing African-American children.
Development of the CKTA
The CKTA is an adaptation of the KTA, a valid and wellestablished performance assessment of adult performance
during a simple cooking task of making stovetop oatmeal
(Baum & Edwards, 1993). The CKTA uses the same
approach used to develop and validate the KTA and EFPT.
The CKTA uses a safe and age-appropriate activity for children (making play dough) and an expanded version of the
scoring and cueing system validated for the KTA and EFPT.
The testing and scoring procedures are described in detail in
Table 1 and the Appendix.
The first step in the development of the CKTA was to
identify a child-friendly and motivating functional activity
that requires executive functioning skills to complete. The
essential elements of executive function are similar across the
lifespan (Zelazo, Craik, & Booth, 2004); therefore, those
elements were considered when selecting an appropriate
activity. Several different activities were considered as part of
the development of the CKTA, including making microwave popcorn, macaroni and cheese, and play dough.
Making play dough was chosen because it is a novel, ageappropriate, and goal-directed activity for most children (see
Table 1). The novelty and uniqueness of the play dough
recipe reduced the likelihood of procedural memory or prior
experience aiding in the completion of the task.
Table 1. Components of the Children’s Kitchen Task Assessment
INITIATION: Taking recipe out of the box and turning to appropriate page
ORGANIZATION: Gathering appropriate materials
PLANNING/SEQUENCING: Following the recipe
1. Measure 1 cup of flour. Put in bowl.
2. Measure ¼ cup of salt. Put in bowl.
3. Measure 1½ teaspoons of cream of tartar. Put in bowl.
4. Measure 1½ teaspoon of oil. Put in bowl.
5. Mix with a spoon. Set bowl aside
6. Measure ½ cup of water. STOP! Do not add to bowl!
7. Add 3 drops of food coloring into the water. Do not add to bowl!
8. Heat the water in the microwave for 1 min.
9. Add the water to the bowl.
10. Mix for 1 min.
11. Let play dough sit for 1 min.
12. Mix the play dough with your hands. Make the play dough into a ball.
JUDGMENT/SAFETY: Ensuring the play dough is safe to touch. Using safe
judgment while carrying water and throughout other steps.
COMPLETION: Putting the play dough in a bag. Recognizing task is
completed.
The American Journal of Occupational Therapy
Downloaded From: http://ajot.aota.org/ on 08/22/2014 Terms of Use: http://AOTA.org/terms
529
After selecting an appropriate activity, a play dough
recipe was created. The recipe needed to require cooking to
test the element of judgment and safety, but stovetop cooking was not thought to be accessible or safe for this age range.
Heating water in a microwave was determined to be an
appropriate cooking element to assess judgment and safety.
To minimize the language and reading demands, the directions were broken down into simple words, and pictures
were added to accompany the written recipe. Steps of the
recipe were determined to be at a reading level below the first
grade, as determined by the Flesch-Kincaid Grade Level
(Flesch, 1948). Before administration of the CKTA to the
current study sample, the recipe was pilot tested with 6
children ages 6 to 10 years who had various levels of reading
skills to determine the ease of following the written and picture recipe. The recipe and materials were revised on the
basis of the performance and feedback from these children.
After completing the recipe, the CKTA scoring system
was adapted to create a uniform means of evaluating the level
of assistance required to complete the activity. As in the
KTA, standard cueing guidelines were created for each step
of the recipe to ensure consistent cueing across testers. A
consistent and reliable cueing system is vital because the
amount and level of cues determine the children’s total score
and level of assistance required to complete the activity. Like
the KTA, the participant is scored on the basis of the number
and type of cues needed to successfully complete each task
component. Assistance starts with nonspecific verbal guidance and progresses through a standard series of increasingly
more specific and direct cues until the participant has completed the step. The examiner waits 10 s before cueing the
child to determine whether the child self-corrects. The type
of cue given is scored from 0 to 5, beginning with no cues
(0), progressing to general verbal guidance (1), gesture guidance (2), direct verbal assistance (3), physical assistance (4),
and, finally, doing for the participant (5) (see the Appendix;
Baum et al., 2003).
The CKTA uses the components of executive function
defined in the KTA, which include initiation, organization,
planning and sequencing, judgment and safety, and completion (see Table 1; Baum & Edwards, 1993; Baum et al.,
2003). Initiation was defined as the ability to start the activity; this component includes retrieving the recipe book and
opening it to the appropriate page. Organization was determined by the ability to find the necessary materials throughout the task and manage the workspace. Planning and
sequencing was the largest component of the assessment and
consisted of following and completing the steps of the recipe.
Children were given a score for each step of the recipe and
the sum of the steps created a total planning and sequencing
score. Judgment and safety was defined as the ability to dem-
onstrate safe behaviors and appropriate judgment throughout the assessment. Specific tasks within the play dough
recipe were created that required judgment and safety (e.g.,
safely heating water in the microwave and pouring it into
the bowl). Finally, completion was scored when the child
finished the task by putting the play dough in a bag and
acknowledging completion.
The CKTA score form reflects a detailed scoring system
that captures the level of assistance (cues) required to complete each step of the recipe. Scoring is based on the number
and level of cues required to complete each step and section
(initiation, organization, planning and sequencing, judgment and safety, and completion) within the task. The total
weighted score reflects the number and level of cues needed
to successfully complete the task. Total weighted score on
the CKTA can range from 0 to 400. A higher score indicates
a higher level of assistance required to complete the task and
therefore indicates lower executive function skills. Built into
this task is the learning effect of corrective cueing. Within
the sequenced steps of the recipe, three of the steps are immediately repeated to allow for learning: measuring 1 cup of
flour followed by measuring ¼ cup of salt; measuring 1½
teaspoons of cream of tartar followed by 1½ teaspoons of oil;
using the timer to stir for 1 min followed by allowing the
play dough to cool for 1 min. In addition, the tester can
make note of self-talk and the number of times a child needs
to reread an instruction, reflecting the child’s strategies for
memory.
After the scoring system was finalized, pre- and postassessment questionnaires were developed to obtain additional
information from the children. The preassessment questionnaire addressed the children’s previous cooking experience,
microwave use, past experience making play dough, and a
prediction of the amount of help they would require to make
play dough. During this questionnaire, children were asked
to read a recipe card (“Measure 1 cup of flour”) and explain
how to follow this directive. This activity was to ensure that
the children understood the words and the concept of following the recipe directions. The testers explained any words
or concepts that the child did not understand. The postassessment questionnaire evaluated self-awareness regarding
performance while making the play dough. This was determined by asking participants to reflect on their task performance, the amount of help required, and anything they
would do differently next time. The administration of the
entire CKTA takes approximately 15 to 20 min.
Analyzing occupations within the context of everyday
life contributes to our understanding of the role of executive
function ability and performance of complex everyday life
tasks (Katz & Hartman-Maeir, 2005). Occupational therapists have greatly contributed to this area of practice with the
530
Downloaded From: http://ajot.aota.org/ on 08/22/2014 Terms of Use: http://AOTA.org/terms
September/October 2008, Volume 62, Number 5
development of performance assessments for adults, such as
the KTA and the EFPT, that are designed to measure the
level of cognitive assistance required to complete a complex
activity (Baum & Edwards, 1993; Goverover et al., 2005;
Katz, Tadmor, Felzen, & Hartman-Maeir, 2007). These
performance assessments of complex real-world tasks complement the information obtained from neuropsychological
tests (Manchester et al., 2004). Such tests are needed because
neuropsychological assessments may lack ecological validity,
the ability to accurately predict function in real-world environments (Alderman et al., 2003). Performance tests provide
valuable information for treatment planning, as well as a
perspective on normal developmental changes on task
performance.
Method
Participants
Participants in this study were recruited from a local elementary school, community acquaintances, and siblings from a
larger ongoing study evaluating children with sickle cell disease compared with their age-, gender-, and race-matched
peers. Because sickle cell disease primarily affects the African
American population, the age-, gender-, and race-matched
control group from which children were recruited includes
typically developing African American children. Inclusion
criteria for this sample thus consisted of typically developing
African American children between ages 8 and 12 years.
Exclusion criteria were diagnosis of a learning disorder, mental retardation, physical impairment, communication disorder, or receipt of special education services.
Validation Measures
Behavior Rating Inventory of Executive Function (Parent
Form). The Behavior Rating Inventory of Executive Function
(BRIEF; Gioia, Isquith, Guy, & Kenworthy, 2000) is a
standardized parent questionnaire designed to assess realworld behaviors in children related to executive functioning
in the home. The assessment consists of eight subscales:
inhibit, shift, emotional control, initiate, working memory,
plan and organize, organization of materials, and monitor.
The first three subscales combine to form the Behavioral
Regulation Index (BRI), and the last five subscales combine
to form the Metacognition Index (MI). The BRI and MI
combine into one Global Executive Composite score. The
normative sample of the Parent BRIEF includes 11.9%
African Americans (Gioia et al., 2000).
Delis–Kaplan Executive Function System Color-Word
Interference Test. The Delis–Kaplan Executive Function
System (D–KEFS) Color-Word Interference Test (Delis,
Kaplan, & Kramer, 2001) is a standardized neuropsychological assessment that measures a person’s ability to inhibit
overlearned responses. This assessment is divided into four
conditions: naming color patches, reading words printed in
black ink, naming the ink color of printed words, and switching between naming the ink color and reading the words.
Participants are scored on the basis of the time taken to
complete each condition and the number of errors made
during each condition.
D–KEFS Sorting Test. The D–KEFS Sorting Test (Delis
et al., 2001) is a standardized neuropsychological assessment
that measures concept formation and problem-solving skills.
The test has two conditions: free sorting and sort recognition. During the free sorting test, the participants are asked
to sort six cards into two categories in as many ways as possible and to describe the concepts used to make the sort.
During the sort recognition test, the tester sorts the same
six cards and asks the participant to identify and describe
the sorted groups. Age-adjusted scaled scores are available
for people ages 8 to 89 years. Normative data for the D–
KFES battery was consistent with the 2000 U.S. data in
terms of race/ethnicity (Delis et al., 2001; Homack, Lee, &
Riccio, 2005).
Wechsler Intelligence Scale for Children–IV, Digit Span.
The Wechsler Intelligence Scale for Children–IV (WISC–
IV) Digit Span (Wechsler, 2003) is a standardized measure
of working memory in children. The children are read a
series of numbers and asked to repeat the numbers. As the
participant correctly recalls the numbers, the amount of
numbers given increases until the participant is unable to
correctly recite the numbers. Next, the participants are read
numbers and asked to repeat the numbers in reverse order.
Once again, as the participant correctly recalls the numbers
backward, the amount of numbers given increases until the
participant is unable to correctly recall the numbers. The
participants are scored on the basis of the number of correctly recalled series of numbers. Normative data provide
age-adjusted norms and scaled scores for children ages 6 to
16 (Wechsler, 2003).
Additional Measures
Parents and guardians completed a demographic questionnaire to identify whether their children had any known
medical conditions or developmental disabilities. Additional
information collected included participants’ history of grade
retention or special education services and any additional
diagnoses, such as attention disorders, learning disabilities,
or autism. The Hollingshead Four Factor Index of Social
Status score (Hollingshead, 1975) is derived from information provided by parents on the demographic questionnaire.
Parental occupations and education levels are scaled and
The American Journal of Occupational Therapy
Downloaded From: http://ajot.aota.org/ on 08/22/2014 Terms of Use: http://AOTA.org/terms
531
combined. The number of adults contributing to the child’s
daily care is acknowledged, resulting in a classification of
social status between 1 and 5, with 1 being the lowest and 5
being the highest (Hollingshead, 1975).
Procedures
The study was approved by the Washington University
School of Medicine institutional review board, and all parents, guardians, and participants provided informed consent
or assent before beginning testing. Occupational therapy
graduate students and faculty members were trained in the
entire assessment battery and administered all assessments.
All testers were trained to criterion and were required to
achieve 90% or greater interrater reliability on CKTA total
scores before being allowed to administer the test to research
participants. No specific order of testing was used.
Administration of both the CKTA and the neuropsychological test battery took less than 1 hr.
Most participants were recruited from a local elementary school and were tested at their school. Other participants were recruited from the community using similar
methods and were tested at the St. Louis Children’s Hospital
General Clinical Research Center or at the Washington
University Program in Occupational Therapy. All children
assented to participate before data collection. The children
completed the CKTA and the neuropsychological assessments. All participants were given a $5 certificate to a local
restaurant for participating in the study. Parents completed
the demographic questionnaire in an interview format.
Children tested at the elementary school were sent home
with the Parent BRIEF. Parents were called after testing to
complete the demographic questionnaire and to notify them
that the Parent BRIEF had been sent home for them to fill
out and return to the school nurse. Parents of children
tested elsewhere completed the BRIEF during the testing
session.
Data Analysis
All statistical analyses were completed with SPSS for
Windows version 13.0 (SPSS, Inc., Chicago). Descriptive
statistics were computed for all variables. Student’s t tests
were used to test for differences between groups for normally
distributed continuous variables, and chi-square analyses
were used for dichotomous variables. Interrater reliability
was computed using intraclass correlation coefficients
(ICCs). Internal consistency was evaluated with Cronbach’s
alpha statistic.
Discriminant validity was examined in two ways. First,
differences by age were tested with a one-way analysis of
variance (ANOVA; Age Group × Total Score). Second, two
groups were created on the basis of total CKTA scores and
compared for differences between the groups on the neuropsychological tests and Parent BRIEF. A median split
(MacCallum, Zhang, Preacher, & Rucker, 2002) was used
to create a dichotomous measure of performance on the
CKTA. On the basis of the median total score, scores of ≤10
were considered a “good” outcome, whereas scores of ≥11
were indicative of “poor” performance in this sample.
Spearman rho correlation coefficients were used to examine
associations among study measures. A priori alpha levels of
≤.05 were required for significance.
Results
All participants were African American. The children ranged
in age from 8 years, 10 months to 12 years, 6 months (M =
10.4 ± 1.12 years) and were enrolled in second to seventh
grade. Most of the participants were female (69%), and
almost half (49%) lived in single-parent homes. The mean
socioeconomic status score was 2.71 (SD = 1.25), indicative
of lower middle class, although scores varied from 1 (lowest)
to 5 (highest) within the sample. The demographic characteristics of the sample are presented in Table 2. Forty-two of
the 49 participants came from a local elementary school.
Interrater reliability of the CKTA had been established
as part of an ongoing larger study. Twenty-two participants
Table 2. Demographic Characteristics of the Sample
Participants
(N = 49)
Variable
Age (years)
Socioeconomic status
M
SD
10.4
1.12
2.71
1.25
n
%
Male
15
31
Female
34
69
Single parent
24
49
Two parent
23
47
2
4
2nd
1
2
3rd
14
29
4th
12
24
5th
18
37
6th
3
6
7th
1
2
Gender
Living situation
Other relative
Grade
532
Downloaded From: http://ajot.aota.org/ on 08/22/2014 Terms of Use: http://AOTA.org/terms
September/October 2008, Volume 62, Number 5
were videotaped and scored independently by two blinded
raters. The ICC for the total score of the test was .98. Internal
consistency of the CKTA was found to be moderate with a
Cronbach’s alpha of .68.
All participants completed the CKTA. The scores are
presented in Table 3. Several sets of scores have been computed. First, a total score and total number of cues were created from the combination of the scores of each section of
the test (initiation, organization, planning and sequencing,
judgment and safety, and completion). The total cues for
each section of the assessment were recorded. Section scores
reflect the number and level of cues needed to successfully
complete each section of the test. The children tested
required <1 cue for each of the following: initiation, organization, judgment and safety, and completion. The mean
score for planning and sequencing was also found to be <1
cue per step averaged over the 12 steps. The mean total score
of all participants was 11.84 ± 8.80, and the number of cues
for the test as a whole averaged 9.02 ± 5.89.
Discriminant validity was examined by comparing performance across age groups using a one-way ANOVA to
determine whether the test scores increased with age. The
mean scores are shown by 1-year age increments in Figure
1. Performance significantly improved as age increased
(F[4,45] = 3.83, p < .008), suggesting that the CKTA is
sensitive to the expected age-related differences in executive
ability observed in typically developing children.
Performance on the neuropsychological tests and Parent
BRIEF scores were used to examine the discriminant validity
of the CKTA. Thirty-seven Parent BRIEF forms were
returned and were included in this analysis. The BRIEF
provides eight subscale scores. Parent scores on the eight
subscales of the BRIEF were all within the average range of
the standardized scores for each subtest. The Color-Word
Interference and Sorting subtests of the D–KEFS battery and
Table 3. Total and Component Scores on the Children’s Kitchen
Task Assessment
Participants
(N = 49)
Variable
M
SD
Range
11.84
8.80
0–39
9.02
5.89
0–27
Initiation
0.27
0.53
0–2
Organization
0.12
0.48
0–4
Planning/sequencingb
0.70
0.46
0–2
Judgment/safety
0.12
0.60
0–4
Completion
0.10
0.31
0–1
Total score
Total number of cues
Component scoresa
Component scores from total number of cues.
a
Indicates mean number of cues for all steps.
b
Figure 1. Children’s Kitchen Task Assessment total scores and total
cues by age.
WISC–IV Digit Span subtest were used to assess executive
function. Performance on the Color-Word Interference and
Sorting subtests was within the average range based on comparison to the normative values for these tests. Children in
this sample demonstrated poorer performance on the Digit
Span backwards, a test of working memory, when compared
with the normative values.
To further explore the discriminant validity of the
CKTA, we used a median split of the total score to create
high- and low-scoring groups. We then compared the performance of high- and low-­scoring CKTA groups on the
Parent BRIEF and three neuropsychological tests. Participants
with lower scores on the CKTA required less assistance to
complete the task, and children with higher scores on the
CKTA required more assistance to complete the task. Chisquare analyses were used to compare groups for CKTA
differences by gender and socioeconomic status. No significant differences were found between the high- and low-scoring CKTA groups for gender (χ2 = 2.08, p > .05, [1]) or
socioeconomic status (χ2 = 3.42, p > .05, [4]).
Group comparisons were computed using t tests corrected for multiple comparisons to determine differences
between high- and low-scoring CKTA groups’ performance
on the BRIEF and the neuropsychological assessments
(Tables 4 and 5). The group with higher CKTA scores demonstrated higher scores on the Parent BRIEF, indicating that
children who required more assistance to complete the
CKTA were also reported by parents as having demonstrated
more characteristics of executive dysfunction. Significant
differences between the groups were found for the BRIEF
Inhibition subtest score (p < .003) and BRI score (p < .01).
Two scores on the Parent BRIEF approached significance:
Initiation subtest (p < .06) and the Global Executive
Composite (p < .08). Children with better scores on the
CKTA performed better on all three neuropsychological
tests. Significant differences were found between the two
The American Journal of Occupational Therapy
Downloaded From: http://ajot.aota.org/ on 08/22/2014 Terms of Use: http://AOTA.org/terms
533
Table 4. Comparison Between Children’s Kitchen Task Assessment
High- and Low-Scoring Groups on Parent Behavior Rating Inventory
of Executive Function
High-Score
Group
(n = 17)
Low-Score
Group
(n = 20)
M
SD
M
SD
p
Inhibition
46.55
6.40
54.82
9.53
.003
Plan/organize
47.12
7.20
45.65
7.14
.53
Organization of materials
45.53
8.01
46.15
6.94
Monitor performance
47.76
8.99
45.10
Shifting attention
47.05
8.54
Initiation
43.95
Working memory
47.80
Behavioral regulation index
Table 5. Comparison Between Children’s Kitchen Task Assessment
High- and Low-Scoring Groups on Neuropsychological Tests
High-Score
Group
(n = 24)
Low-Score
Group
(n = 25)
M
SD
M
SD
p
Condition 3
10.20
2.96
9.79
3.30
.65
Condition 4
10.60
2.29
9.50
3.60
.20
Total errors—Condition 3
8.13
3.79
7.12
3.72
.35
.80
Total errors—Condition 4
7.76
4.04
7.96
3.74
.86
8.97
.37
Composite scaled score
9.60
2.23
8.67
2.51
.85
49.47
9.32
.42
D–KEFS Card Sorting Testa
11.01
49.94
8.08
.06
Confirmed correct sorts
9.64
2.25
8.71
2.68
.03
10.03
50.82
9.91
.36
Description
9.80
2.50
9.04
2.46
.29
45.70
7.47
52.76
9.90
.01
Sort recognition
8.16
3.51
7.29
2.79
.34
Metacognition index
45.30
8.26
49.12
8.43
.17
WISC–IV Digit Spana
Global executive composite
45.30
7.90
49.94
8.02
.08
Forward
8.36
1.85
8.04
1.90
.35
Backward
5.88
1.59
6.83
1.63
.04
Total scaled
8.88
2.60
9.71
2.48
.41
Variable
Subtest or indexa
Normative scaled mean equals 50 (SD = 10).
a
groups on the D–KEFS Confirmed Correct Card Sorts (p <
.03) and WISC–IV Digit Span backwards (p < .04).
As part of the preliminary examination of the concurrent validity of the CKTA, we examined the relationships
between the results of the CKTA and the results of the
Parent BRIEF and neuropsychological tests using Spearman
correlation coefficients. All of the coefficients were in the low
to moderate range. For example, low to moderate correlations were found between the total score of the CKTA and
scores on the WISC–IV Digit Span backward (rs = –.31)
and with the Inhibition subscale of the BRIEF (rs = .34).
The organization section of the CKTA correlated significantly with the Plan and Organize subscale of the BRIEF
(rs = .38). The judgment and safety section of the CKTA
correlated significantly with the Monitor subscale of the
BRIEF (rs = –.35). The CKTA planning/sequencing score
correlated significantly with the scores on the D–KEFS
Confirmed Correct Card Sorts (rs = –.33) and with the Total
Description Score (rs = –.30).
Discussion
We developed the CKTA as a modification of the KTA, an
assessment used to measure executive function skills in adults
and determine the level of assistance required for successful
completion of a simple cooking task (Baum & Edwards,
1993). The CKTA was developed to meet the need for a
reliable and valid performance assessment of executive function capabilities in children. The purpose of this study was
to examine the reliability and validity of the CKTA in a
sample of typically developing African American children
between ages 8 and 12.
D–KEFS Color-Word Interference Testa
Note. D–KEFS = Delis–Kaplan Executive Function System; WISC–IV =
Wechsler Intelligence Scale for Children–IV.
Normative scaled mean equals 10 (SD = 2.5).
a
Children are given the task of making play dough from a
recipe; however, the children must plan an approach to the task
and carry out their plan. The implicit rule would be that when
one step is completed, the child moves to the next step in a linear
fashion, managing the materials and workspace as he or she
proceeds. The role of the tester is to prompt, after waiting a set
time for the child to self-correct, when the child is struggling
with a step. Although the CKTA is divided into components
of executive functioning, the integration of numerous executive
function skills must be used to complete the task; therefore,
detection of specific deficits is not the purpose of the CKTA.
Our study provides preliminary evidence of the reliability
and validity of the CKTA and establishes the CKTA as a clinically useful tool to evaluate children’s performance during a
challenging real-life activity. We found high levels of interrater
reliability and moderate internal consistency when the test is
administered and scored by trained testers. In addition, the
discriminant validity of the CKTA is supported by the incremental improvement in performance seen as age increases, as
well as by the ability to discriminate between high- and lowscoring participants.
Developmental theory suggests that executive function
skills develop continuously as children mature throughout
childhood (Anderson, 2002; Zelazo et al., 2004). The role of
social interaction in the development of cognitive skills in
children is supported by Vygotsky’s (1978) theory and the use
of scaffolding. Problem solving is learned within a social context when children receive support, or “scaffolding,” from
adults who, knowing a child’s current cognitive abilities, cue
534
Downloaded From: http://ajot.aota.org/ on 08/22/2014 Terms of Use: http://AOTA.org/terms
September/October 2008, Volume 62, Number 5
the child to a higher potential level of performance (Vygotsky,
1978; Wertsch, 1985). Typically, as children’s cognitive skills
mature through this guidance, the amount of support required
to complete a task lessens until the child is able to successfully
problem solve a cognitively challenging task independently.
In this study, this developmental process is reflected in the
greater need for cues and therefore higher total scores on the
CKTA in the younger participants.
Further support of discriminant validity is seen in the
significant differences in the test scores between high- and
low-scoring CKTA groups. Children in the high-scoring
CKTA group had poorer performance on the D–KEFS
Sorting Test, indicating that they had more difficulty with
abstract reasoning, cognitive flexibility, and initiation of
problem-solving strategies. This group also did less well on
the WISC–IV Digit Span backwards, indicating more difficulty with working memory. Significant differences on the
Parent BRIEF support the validity of the CKTA. Although
most of the children in this study scored within the normal
range on the BRIEF, the parents of children in the high­scoring CKTA group noted more problems with initiation,
behavioral regulation, and inhibition.
The construct validity of the CKTA is supported by the
correlations with the Parent BRIEF, parts of the D–KEFS
battery, and the WISC–IV Digit Span. The correlations of
the CKTA with the neuropsychological measures observed in
this sample are consistent with the findings of previous studies
examining the relationship between performance assessments
of executive function and traditional neuropsychological measures in adults (Burgess et al., 2006). The moderate correlations between performance measures and neuropsychological
tests have been observed in a variety of adult populations
(Burgess et al., 1998; Chaytor & Schmitter-Edgecombe, 2003;
Shallice & Burgess, 1991). The authors of these studies have
argued for the construct validity of performance measures,
providing the evidence that there are higher correlation coefficients observed between the performance assessments and
the “executive measures” and lower coefficients observed
between performance assessments and measures of other cognitive functions such as memory.
We observed a similar pattern. For example, the CKTA
scores were more highly correlated with the WISC–IV Digit
Span backwards, which is thought to measure working memory, than with the WISC–IV Digit Span forwards, which
measures short-term memory. Godefroy (2003) suggested that
differences in test content account for the observed low to
moderate correlation coefficients among the measures. The
neuropsychological tests are designed to assess isolated cognitive functions, whereas the performance measures require
simultaneous integration and synthesis of several cognitive
operations (Burgess et al., 2006).
A multitask directive from a parent, such as “go clean
your room,” or from the teacher, such as “put everything
away and get ready for recess,” requires the integration of
many executive function skills from the child. Teachers and
parents are familiar with the child who has difficulty initiating the task independently, staying focused on the task,
sequencing the steps to complete the task, using judgment
and safety awareness, or organizing materials. When therapists are able to reliably identify the amount and type of
cueing required to enhance task performance, they can
instruct parents and teachers on how to support a child’s
performance similar to what Baum and Edwards (2003) have
done for older adults with dementia and their caregivers. By
objectively identifying the level of support children require
for successful participation in meaningful activities through
the CKTA, parents, teachers, and therapists will be able to
create a supportive learning environment that enhances a
child’s capacity for occupational success.
Strengths, Limitations, and Future Directions
The African American sample is a strength of this preliminary
study and a valuable contribution to the literature for this
understudied population. However, the generalizability of
this study’s findings may be limited until testing of a more
representative sample is completed. Future research should
focus on assessing a variety of races and ethnicities of children
with the CKTA to establish cross-cultural validity of the
assessment. Moreover, the utility of the CKTA should be
addressed by studying the performance of children with a
variety of disabilities. Throughout the ongoing development
of this assessment, modifications have been made to strengthen
the CKTA for future research and clinical practice.
Conclusion
The development of the CKTA created a motivating and childfriendly assessment of executive functioning that demonstrated
validity as a pediatric performance assessment tool. Use of the
CKTA allows pediatric occupation-based practitioners to determine the level of cognitive assistance children require to successfully complete a functional activity, and assessment results
can immediately guide treatment and inform parents and
teachers of cognitive strategies to improve children’s performance and participation in everyday activities. s
Acknowledgments
We thank the participants and their families; Adams
Elementary School in St. Louis, MO; Surilla Shaw, RN; and
The American Journal of Occupational Therapy
Downloaded From: http://ajot.aota.org/ on 08/22/2014 Terms of Use: http://AOTA.org/terms
535
the graduate students in the Berg Research Lab. We acknowledge the Saint Louis Children’s Hospital General Clinical
Research Center (EMMA1 RR0036) for their support of
this project.
References
Alderman, N., Burgess, P. W., Knight, C., & Henman, C.
(2003). Ecological validity of a simplified version of the
multiple errands shopping test. Journal of the International
Neuropsychological Society, 9, 31–44.
Anderson, P. (2002). Assessment and development of executive
function (EF) during childhood. Child Neuropsychology, 8,
71–82.
Baum, C. M., Connor, L. T., Morrison, T., Hahn, M., Dromerick,
A. W., & Edwards, D. F. (2008). Reliability, validity, and
clinical utility of the Executive Function Performance Test:
A measure of executive function in a sample of people
with stroke. American Journal of Occupational Therapy, 62,
446–455.
Baum, C., & Edwards, D. F. (1993). Cognitive performance in
senile dementia of the Alzheimer’s type: The Kitchen Task
Assessment. American Journal of Occupational Therapy, 47,
431–436.
Baum, C. M., & Edwards, D. F. (2003). What persons with
Alzheimer’s disease can do: A tool for communication about
daily activities. Alzheimer’s Care Quarterly, 4, 108–118.
Baum, C. M., Edwards, D. F., Hahn, M., & Morrison, T. (2003).
Revised Kitchen Task Assessment (KTA). Unpublished manuscript, Washington University School of Medicine.
Biederman, J., Monuteaux, M. C., Doyle, A. E., Seidman, L. J.,
Wilens, T. E., Ferrero, F., et al. (2004). Impact of executive
function deficits and attention deficit/hyperactivity disorder
on academic outcomes in children. Journal of Consulting and
Clinical Psychology, 72, 757–766.
Burgess, P. W. (2000). Strategy application disorder: The role
of the frontal lobes in human multitasking. Psychological
Research, 63, 279–288.
Burgess, P. W., Alderman, N., Evans, J., Emslie, H., & Wilson,
B. A. (1998). The ecological validity of tests of executive function. Journal of the International Neuropsychological Society, 4,
547–558.
Burgess, P. W., Alderman, N., Forbes, C., Costello, A., Coates,
L. M., Dawson, D., et al. (2006). The case for the development and use of ecologically valid measures of executive function in experimental and clinical neuropsychology. Journal of
the International Neuropsychological Society, 12, 194–209.
Channon, S., & Crawford, S. (1999). Problem-solving in real-lifetype situations: The effects of anterior and posterior lesions of
performance. Neuropsychologia, 37, 757–770.
Chaytor, N., & Schmitter-Edgecombe, M. (2003). The ecological
validity of neuropsychological tests: A review of the literature on everyday cognitive skills. Neuropsychology Review, 13,
181–197.
Delis, D. C., Kaplan, E., & Kramer, J. H. (2001). D–KEFS executive function system: Examiner’s manual. San Antonio, TX:
Psychological Corporation.
Flesch, R. (1948). A new readability yardstick. Journal of Applied
Psychology, 32, 221–233.
Fingerhut, P., Madill, H., Darrah, J., Hodge, M., & Warren, S.
(2002). Brief report: Classroom-based assessment: Validation
for the School AMPS. American Journal of Occupational
Therapy, 56, 210–213.
Fisher, A. G. (2003). Assessment of Motor and Process Skills. Vol. 1:
Development, standardization, and administration manual (5th
ed.). Fort Collins, CO: Three Star Press.
Geurts, H. M., Verte, S., Oosterlaan, J., Roeyers, H., & Sergeant,
J. A. (2004). How specific are executive functioning deficits
in attention deficit hyperactivity disorder and autism? Journal
of Child Psychology and Psychiatry, 45, 836–854.
Gioia, G. A., Isquith, P. K., Guy, S. C., & Kenworthy, L. (2000).
Behavior Rating Inventory of Executive Function. Odessa, FL:
Psychological Assessment Resources.
Gioia, G. A., Isquith, P. K., Kenworthy, L., & Barton, R. I. (2002).
Profiles of everyday executive function in acquired and developmental disorders. Child Neuropsychology, 8, 121–137.
Godefroy, O. (2003). Frontal syndromes and disorders of executive functions. Journal of Neurology, 250, 1–6.
Goel, V., Grafman, J., Tajik, J., Gana, S., & Danto, D. (1997). A
study of the performance of patients with frontal lobe lesions
in a financial planning task. Brain, 120, 1805–1822.
Goverover, Y., Kalmar, J., Gaudino-Goering, E., Shawaryn, M.,
Moore, N. B., Halper, J., et al. (2005). The relation between
subjective and objective measures of everyday life activities in
persons with multiple sclerosis. Archives of Physical Medicine
and Rehabilitation, 86, 2303–2308.
Hollingshead, A. B. (1975). Four factor index of social status.
Unpublished working paper, Yale University.
Homack, S., Lee, D., & Riccio, C. A. (2005). Test review: Delis–
Kaplan Executive Function System. Journal of Clinical and
Experimental Neuropsychology, 27, 599–609.
Katz, N., & Hartman-Maeir, A. H. (2005). Higher-level cognitive functions: Awareness and executive functions enabling
engagement in occupation. In N. Katz (Ed.), Cognition and
occupation across the life span (2nd ed., pp. 3–25). Bethesda,
MD: AOTA Press.
Katz, N., Tadmor, I., Felzen, B., & Hartman-Maeir, A. (2007).
Validity of the executive function performance test in individuals with schizophrenia. Occupational Therapy Journal of
Research, 27, 1–8.
Levin, H., & Hanten, G. (2005). Executive functions after traumatic brain injury in children. Pediatric Neurology, 33(2),
79–93.
Lezak, M. D. (1982). The problem of assessing executive functions. International Journal of Psychology, 17, 281–297.
MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D.
(2002). On the practice of dichotomization of quantitative
variables. Psychological Methods, 7, 19–40.
Manchester, D., Priestly, N., & Jackson, H. (2004). The assessment of executive functions coming out of the office. Brain
Injury, 18, 1067–1081.
Parrish, J., Geary, E., Jones, J., Seth, R., Hermann, B., &
Seidenberg, M. (2007). Executive functioning in childhood epilepsy: Parent-report and cognitive assessment.
Developmental Medicine and Child Neurology, 49, 412–416.
536
Downloaded From: http://ajot.aota.org/ on 08/22/2014 Terms of Use: http://AOTA.org/terms
September/October 2008, Volume 62, Number 5
Shallice, T., & Burgess, P. W. (1991). Deficits in strategy application
following frontal lobe damage in man. Brain, 114, 727–741.
Vygotsky, L. S. (1978). Mind in society: The development of higher
psychological processes. Cambridge, MA: Harvard University
Press.
Wang, W., Enos, L., Gallagher, D., Thompson, R., Guarini, L.,
Vichinsky, E., et al. (2001). Neuropsychological performance
in school-aged children with sickle cell disease: A report
from the Cooperative Study of Sickle Cell Disease. Journal of
Pediatrics, 139, 391–397.
Wechsler, D. (2003). Wechsler Intelligence Scale for Children (4th
ed.). San Antonio, TX: Psychological Corporation.
Wertsch, J. V. (1985). Vygotsky and the social formation of the mind.
Cambridge, MA: Harvard University Press.
Zelazo, P. D., Craik, F. M., & Booth, L. (2004). Executive function across the life span. Acta Psychologica, 115, 167–183.
Appendix.
Levels of Cueing for the Children’s Kitchen Task Assessment
No cues required (0): The participant requires no help or reassurance, does not ask questions for clarification, and goes
directly to task. Self-talk is acceptable. The participant independently completes the step.
Verbal guidance (1): The participant requires prompting with an open-ended question or an affirmation that will help
him or her move on. Verbal guidance is in the form of a question, not a direct instruction; for example, “What should
you do now?” “What is the next step?” “What else do you need?” “What does the recipe say?”
Gesture guidance (2): The participant requires gesture prompting. The tester is not physically involved with the task.
Tester makes a gesture that mimics the action that is necessary to complete the subtask or makes a movement that guides
the participant back on task; for example, the tester may move his or her hands in a stirring motion, points to where the
participant may find the item, or points to the recipe picture.
Direct verbal assistance (3): The participant requires a direct phrase or command. The tester is telling the participant
what to do. The tester delivers a one-step command, so that the tester is cueing the participant to take the action; for
example, saying, “You need 1 cup of flour,” “Put the salt into the bowl,” or “Use the timer.”
Physical assistance (4): Tester is physically assisting the participant with the step but not doing the step for the participant.
The tester may hold the bowl while the participant adds the ingredients, retrieve a necessary item for the participant,
and so forth.
Do for participant (5): The tester is required to do the step for the participant. Participant is unable to complete the task.
The tester only completes the specific step the participant is demonstrating difficulty with. Participant continues with
the next step.
The American Journal of Occupational Therapy
Downloaded From: http://ajot.aota.org/ on 08/22/2014 Terms of Use: http://AOTA.org/terms
537