PAPELUCHO: A MODEL FOR DEVELOPING INCLUSIVE DIGITAL

PAPELUCHO: A MODEL FOR DEVELOPING INCLUSIVE DIGITAL BOOKS FOR DEAF
CHILDREN
Authors: Ricardo Rosas, Soledad Véliz, Rodrigo Arroyo, María Ignacia Sánchez, Marcelo
Pizarro, Andrés David Aparicio
Affiliation: Center for the Development of Inclusion Technologies (CEDETi UC), Pontificia
Universidad Católica de Chile; Santiago, Chile.
Abstract
In general, deaf and hard of hearing people (D/HH) must overcome several obstacles to
acquire the reading-writing code. In this context, CEDETi UC created a model for developing
inclusive digital books for deaf children which was first implemented through an adaptation of
the title “Papelucho”. The purpose of this media is to support the reading comprehension of
D/HH children. This study explores the reading comprehension of deaf students aged 9 to 14
who experience a book in three modes: text only, Sign Language only, and multimedia (text,
images, and Sign Language). The research analyses how each type of content and their
interaction enhance reading comprehension in Chilean D/HH children. Variables that affect
reading skills, such as vocabulary, were also considered. The hypothesis advanced is that
this type of multimedia mode allows deaf children to approach literacy more effectively.
Introduction
For deaf people, the acquisition of the reading-writing code is fraught with difficulties. In this
context, CEDETi UC created a model for developing inclusive digital books for deaf children
which started with the implementation of an adapted version of “Papelucho”, a traditional
Chilean book about the adventures of an 8-year-old child. This book, which includes the
adapted text, Sign Language, images, and audio, allows D/HH children to explore it in a
multimodal way. Users can access the digital book for free (http://papelucho.cedeti.cl/).
Figure 1. Papelucho’s accessible multimedia book
Although reading and writing is an essential ability for individual development, many deaf or
hard of hearing people (D/HH) have not achieved reading comprehension proficiency by the
time they graduate from high school (J. L. Luckner & Handley, 2008). These difficulties of
D/HH students can be explained, in part, by limited access to written language learning
opportunities and to the fact that acquiring the written code involves learning a different
language from the one they learn naturally: Sign Language (Meier, 1990; Supalla, 1991).
Some studies on D/HH people attribute reading comprehension performance to many
variables and difficulties such as reading skills, vocabulary, or problems with syntax
processing (Kelly & Gaustad, 2007; Paul, 1998). Limited research has been conducted, both
in general and with D/HH subjects, to determine the impact in comprehension when the text
1
is not exclusively constructed by written words (Nikolaraizi, Vekiri, & Easterbrooks, 2013).
We know that Sign Language, when used along with reading, increases the literacy levels of
deaf children (Wilbur, 2000).
D/HH students have been considered strong visual learners (Dowaliby, 1999). Observations
in educational settings reveal greater visual sensitivity in this group (Marschark & Hauser,
2012). Therefore, D/HH people, whether they use oral or Sign Language, and whether they
have better or similar visual skills compared with hearing people, must be ensured
unrestricted access to visual or non-verbal information (Nikolaraizi et al., 2013) in the form of
either Sign Language or images, which can be used as an asset to support their reading
process. In this context, the present study highlights that a carefully designed multimodal
device, which conveys information through images, text, and videos that complement each
other, can support reading comprehension in D/HH students.
It has been argued that multimodality can affect learning, which occurs when students
receive information presented in more than one mode: verbal and non-verbal (Mayer, 1997).
Research in multimodality applied to narrative texts suggests that, when knowledge is
represented and communicated by multiple semiotic ways, meaning is distributed through
those modalities. Therefore, different aspects of the text's meaning are carried by each
modality, complementing and affecting one another (Daly & Unsworth, 2011). Multimedia
tools have provided some evidence that, when text, pictures, videos in Sign Language,
graphics, and animation are effectively combined, a positive impact is generated, which
supports the comprehension of D/HH students (Mich, Pianta, & Mana, 2013). Previous
research has assessed the impact on comprehension of different stimuli combinations such
as Sign Language+Images, Sign Language+Text, and Sign Language+Text+Images
(Gentry, Chinn, & Moulton, 2004).
In this document, an accessible multimedia book will be understood as a digital resource that
delivers information using verbal (text and Sign Language) and non-verbal (static images)
modes, and which involves different presentation modes (images, videos, and text).
A connection has been established between reading comprehension and vocabulary
knowledge in D/HH children (Loeterman, Paul, & Donahue, 2002). Deaf children are
reportedly delayed in their acquisition of vocabulary knowledge when compared to their
hearing peers, which affects their comprehension levels (Allen, 1994; J. Luckner & Cooke,
2010).
This research explores the relation between variables related to the text, in this case a
multimodal device, and those associated with the reader, controlling for relevant individual
variables such as vocabulary.
Method
The creation of this accessible multimedia book model consisted in the development of an
innovative tool that allows technological mediation to stimulate literacy and comprehension
in an inclusive way. In the book, users find images, adapted text, Sign Language videos, and
audio.
In order to understand the impact of reading in different modalities (text only, Sign
Language, or multimodality, which includes text+Sign Language+images), a study was
carried out at a Chilean Sign Language (ChSL)/Spanish bilingual school for D/HH children
located in Santiago, Chile. It was an analytic study with repeated measures that sought to
establish the relationship between modality of presentation and reading comprehension in
D/HH children.
2
28 children aged 9 to 14 students at a special education school were invited to the study.
The participants had no additional documented special educational needs and their
caretakers were asked to sign an informed consent. Six participants were removed from the
sample due to application complications and non-compliance with the inclusion criteria. The
final sample consisted of 22 children.
This research involves a digital platform hosting the same content in three stimuli
combinations: text only (TXT), in which the content was written in Spanish; Chilean Sign
Language only (SL), in which the content was expressed through videos with a Chilean Sign
Language interpreter; and the full version (FULL), in which the content was expressed both
through text with Chilean Sign Language (ChSL) videos and through images with ChSL
videos. The FULL version is different from the TXT and SL versions because it allows the
reader to choose between two illustrated versions, with text or ChSL, of the same content.
All three versions (TXT, SL and FULL) are based on the same text.
The participants were randomly assigned to 6 groups, each of which was exposed to three
versions (TXT, SL, and FULL) of the same story. The same comprehension questions were
asked to participants after each exposure to the content. For example, children in the first
group would read the story in the following order: 1) TXT, 2) questions, 3) ChSL, 4)
questions, 5) FULL, and 6) questions. The final design in Table 2 is intended to control for
the effect of reading in different formats.
Table 1. Frequencies by Group
Group
Frequency
TXT-SL-FULL
4
TXT-FULL-SL
5
SL-TXT-FULL
3
SL-FULL-TXT
2
FULL-TXT-SL
4
FULL-SL-TXT
4
Total
22
Percent
18.2
22.7
13.6
9.1
18.2
18.2
100.0
Cumulative Percent
18.2
40.9
54.5
63.6
81.8
100.0
100.0
The research team developed a Reading Comprehension Scale based on several aspects of
reading comprehension and which included multiple types of questions. Expert
recommendations about the questions were incorporated and 21 questions were
consolidated: eight open-ended questions and thirteen multiple-choice questions. Multiplechoice questions have four options, one of which is the correct answer, while open-ended
questions allow the participants to answer in their preferent way of comunication within a
time limit and are scored using a rubric.
Vocabulary comprehension was assessed as a covariable for reading comprehension. A
Multimodal Response Picture Vocabulary Test was constructed to assess vocabulary in 4
dimensions: writing, signing, fingerspelling, and reading. The first three indicators are
examined using 54 items and two trials. In each of them, the subject is presented with an
image and has to indicate what it represents by fingerspelling, signing, and writing. The
reading indicator is assessed using 54 items and two trials, each composed of 4 images and
one word. Participants have to match the image to the word.
Results
The reliability of all instruments was excellent (𝛼=0.928). A score for the Reading
Comprehension Scale was computed for each subject in each modality. As expected, the
score for FULL is the highest and the score for TXT is the lowest.
3
Table 2. Mean scores for each modality
Modality
M
SD
TXT
SL
FULL
9.77
10.5
5
12.0
0
4.60
4.15
CI (95%)
Lower
7.85
8.82
Upper
11.69
12.28
4.27
10.22
13.78
A paired samples t-test was used to compare the Reading Comprehension Scale between
modalities. There was a significant difference between the scores of SL and FULL; t(21)=2.35, p=0.029. The difference between the scores of TXT and FULL was also significant;
t(21)=-2.831, p=0.010. In both cases the score of FULL was higher, which suggests that this
modality provided better support for reading comprehension, regardless of where in the
sequence it was presented to the subject.
To account for a carry-over effect between successive applications of the Reading
Comprehension Scale, we computed the mean scores by modality for each group (Table 4).
Table 3. Mean Modality Scores in Reading Comprehension by Group
Group
TXT
SL
FULL
TXT-SL-FULL
9.00
12.00
13.75
TXT-FULL-SL
6.80
9.40
12.00
SL-TXT-FULL
11.67
10.00
13.67
SL-FULL-TXT
13.50
12.00
14.00
FULL-TXT-SL
10.25
11.00
11.50
FULL-SL-TXT
10.50
9.75
8.50
Because the sample size is too small to perform meaningful statistical tests, we ranked the
modality scores within each group from top to bottom to see what pattern emerged. While
the TXT and SL modalities are distributed among the middle and bottom spots of the ranking
in all groups, the FULL modality consistently performs better (in all groups except for one).
This seems to indicate that the general comparison between modality scores is valid and
that carry-over effects are not important.
Table 4. Ranking of Mean Modality Scores in Reading Comprehension by Group
Group
Top
Medium
Bottom
TXT-SL-FULL
FULL
SL
TXT
TXT-FULL-SL
FULL
SL
TXT
SL-TXT-FULL
FULL
TXT
SL
SL-FULL-TXT
FULL
TXT
SL
FULL-TXT-SL
FULL
SL
TXT
FULL-SL-TXT
TXT
SL
FULL
To explore the effect of vocabulary in the reading comprehension task, a correlational
analysis was performed. A composite vocabulary score was calculated for each participant
as the sum of the z-scores of each scale in the Multimodal Response Picture Vocabulary
Test. The correlations between the reading comprehension scores for each modality and the
4
composite vocabulary score were calculated (Table 6). They were all significant, positive,
and large, which suggests that vocabulary does indeed support reading comprehension.
Table 5. Correlations Between Vocabulary Scores and Reading Comprehension by Modality
Vocabulary
RCS TXT
RCS LS
RCS FULL
Vocabulary
1
RCS TXT
.598**
1
RCS LS
.597**
.621**
1
RCS FULL
.703**
.762**
.656**
1
** Indicates significance at the .01 level (bilateral).
Conclusions
The present study tried to ascertain the compared efficacy of two monomodal conditions—
text (TXT) and Chilean Sign Language videos (SL)—and one multimodal condition (FULL),
controlling for vocabulary and order of presentation, and the impact of these three conditions
on reading comprehension in Chilean D/HH children.
All the evidence reviewed suggests that multimodality allows D/HH children to have a better
understanding of the text. Results showed that, when exposed to multimodality (FULL),
comprehension scores were higher than for any of the monomodal conditions. This proves
the positive impact of the information supplied by the conjunction of images, adapted text,
and Sign Language. These results were achieved regardless of the order of presentation of
the condition.
We explored the relationship between vocabulary and reading comprehension, a topic which
has already been examined in D/HH students (Loeterman, Paul & Donahue, 2002; Luckner
& Cooke, 2010). A significant and positive correlation was observed between the results of
the Vocabulary Test (Multimodal Response Picture Vocabulary Test) and those of the
Reading Comprehension Scale.
Considering the results above, the hypothesis advanced in this study can be confirmed.
D/HH people, when exposed to a book, achieve greater reading comprehension if the book
is presented in a multimodal format, consisting of adapted textual information, meaningful
images, and Sign Language. These results are highly relevant to the consideration of
successful strategies for D/HH education. The reading comprehension difficulties of D/HH
children may not only be a matter of reading comprehension, but also a result of the
modalities in which the content is presented to them.
It is important to take some limitations of this research into account. The sample size was
small (22 D/HH students) due to the difficulties associated with contacting D/HH students. A
larger sample would allow a more detailed analysis. Also, the sample presented great
diversity in relation to several variables which are known to affect the education of D/HH
children.
References
Allen, T. (1994). Who are the deaf and hard-of-hearing students leaving high school and
entering postsecondary education? 1994. Retrieved from
http://research.gallaudet.edu/AnnualSurvey/whodeaf.php
5
Daly, A., & Unsworth, L. (2011). Analysis and Comprehension of Multimodal Texts. Journal
of Language and Literacy, 34(1), 61–80.
Dowaliby, F. (1999). Adjunct aids in instructional prose: A multimedia study with deaf college
students. Lang, H., 4, 270–282.
Gentry, M. M., Chinn, K. M., & Moulton, R. D. (2004). Effectiveness of Multimedia Reading
Materials When Used With Children Who Are Deaf. American Annals of the Deaf,
149(5), 394–403. doi:10.1353/aad.2005.0012
Kelly, R. R., & Gaustad, M. G. (2007). Deaf college students’ mathematical skills relative to
morphological knowledge, reading level, and language proficiency. Journal of Deaf
Studies and Deaf Education, 12(1), 25–37.
Loeterman, M., Paul, P. V., & Donahue, S. (2002). Reading and deaf children. Reading
Online, 5(6).
Luckner, J., & Cooke, C. (2010). A Summary of the Vocabulary Research With Students
Who Are Deaf or Hard of Hearing. American Annals of the Deaf, 155(1), 38–67.
Luckner, J. L., & Handley, C. M. (2008). A summary of the reading comprehension research
undertaken with students who are deaf or hard of hearing. American Annals of the
Deaf, 153(1), 6–36.
Marschark, M., & Hauser, P. C. (2012). How deaf children learn. New York: Oxford
University Press.
Mayer, R. (1997). Multimedia Learning: Are We Asking the Right Questions? Educational
Psychologist, 32(1), 1–19.
Meier, R. (1990). Person deixis in American Sign Language. In G. Morgan & B. Wool (Eds.),
Theoretical issues in sign language research, Vol. 1: Linguistics (pp. 175–90). Chicago:
University of Chicago Press.
Mich, O., Pianta, E., & Mana, N. (2013). Interactive stories and exercises with dynamic
feedback for improving reading comprehension skills in deaf children. Computers &
Education, 65, 34–44. doi:10.1016/j.compedu.2013.01.016
Nikolaraizi, M., Vekiri, I., & Easterbrooks, S. R. (2013). Investigating deaf students’ use of
visual multimedia resources in reading comprehension. American Annals of the Deaf,
157(5), 458–73. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/23505841
Paul, P. (1998). Literacy and deafness: The de- velopment of reading, writing, and literate
thought. Needham Heights, MA: Allyn & Bacon.
Supalla, S. (1991). Manually Coded English: The modality question in signed language
development. In P. Siple & S. Fischer (Eds.), Theoretical issues in sign language
research (Volume 2: Acquisition). Chicago: University of Chicago Press.
Wilbur, R. (2000). The Use of ASL to Support the Development of English and Literacy. Deaf
Studies and Deaf Education, 5(1), 81–104.
6