User testing the CBeebies website: a more child

MA Early Childhood Studies
Roehampton University
Olivia Dickinson
User testing the CBeebies website:
a more child-centred approach?
This dissertation is submitted in part fulfilment of the
MA Early Childhood Studies
Roehampton University
September 2006
Abstract
This study investigates the feasibility of a more child-centred approach when user testing
the CBeebies website. The focus of the study has been on the web production team and
how we can work together to improve the user testing we do. In order to find out if a
more child-centred approach is feasible, I investigated how the team understand the
principles and purposes of user testing and how they understand the young children who
use the CBeebies website.
The literature review shows that there is very little literature which focuses on how young
children use the internet or how they use a computer. Observation and assessment
methods in early years’ settings, and literature about children’s minds and empowering
young children are more useful for moving towards a more child-centred approach.
The research methodology was qualitative. I interviewed the web production team and
then conducted two group discussions to find out how to improve user testing and explore
the feasibility of a child-centred approach.
The study finds that the team understand the criteria for a successful user-testing session
but often those criteria are problematic when user testing with young children. Their
understanding of the audience is from a Piagetian child development perspective, which
hinders the use of methods to empower children in the user-testing sessions. While a
child-centred approach is feasible it is not to the exclusion of other approaches.
A direct outcome of the study has been a working group which is establishing a new way
of user testing the CBeebies website. The working group will adapt some child-centred
theories for user testing but still value the existing ‘theories in use’ (Argyris and Schön in
Moon, 1999: 40) of the team. This will contribute to the growing body of research about
user testing with children within the HCI community.
2
Contents
Abstract……………………………………………………………………………………2
Acknowledgements………………………………………………………………………..4
Introduction………………………………………………………………………………..5
Literature Review………………………………………………………………………….9
Research Methodology…………………………………………………………………..25
Discussion: Analysis and Evaluation…………………………………………………….36
Conclusion……………………………………………………………………………….56
Bibliography…………………………………………………………………..................61
Appendix A: final questionnaire for the interviews……………………………………...65
Appendix B: example of bolding key themes in interview write-up…………………….66
Appendix C: write-up of interviews for Liz and Debra………………………………….69
Appendix D: ideas sheet for second group discussion…………………………………...72
Appendix E: list of themes and groups for second group discussion……………………74
Appendix F: second group discussion write-up………………………………………….75
Appendix G: final project definition document for user testing working group……..….79
3
Acknowledgements
Thanks to all those at BBC Children’s who have participated, contributed or supported
me:
Helen Baker
Rachel Bardill
Emma Bullen
Christina Colbeck
Sandrine Dordé
Stephanie Gauld
Nicola George
Eimer Heekin
Sue Howes
Katherine Monk
Aidan O’Brien
Joanne Patterson
Alan Robinson
Rebecca Shallcross
Helen Stephens
Debra Trayner
Cecilia Weiss
Liz Wilton
Many thanks to Gee-Kay Wong and Ana Joly for stimulating and helpful conversations.
Thanks too to Debra Trayner, Liz Wilton, Leo Sen, Philip Buckley, Stephanie Gauld,
Dave Howell and Lewis Ward for conversations over the last year or so that have helped
me to refine my ideas.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Thank you to those at Roehampton University who have inspired and supported me
throughout the MA: Elise Alexander, Peter Elfer, Hiroko Fumoto and Shirley Maxwell.
Thank you particularly to Elise for her judicious comments at all times.
Thank you to Deborah Susman for her thorough proof-reading.
My greatest thanks to Stephen – thank you for being my 21st century helpmeet.
4
Introduction
I am a producer for the CBeebies website at the BBC. The CBeebies website
(bbc.co.uk/cbeebies) offers interactive games, stories, songs and other online activities
for children aged between eighteen months and six years. I have worked for CBeebies for
over four years and as part of my job, my colleagues and I have done regular user testing
with the target audience for CBeebies’ interactive services (for online and interactive
television). When we go user testing we visit local nurseries and schools, and
occasionally homes, and watch young children as they play on the CBeebies website. We
also sometimes user test with parents and children together, and occasionally parents
alone. The main goal of the testing has always been to ensure that the content we are
creating is of the highest quality and puts the user (the audience of young children) at the
centre of the interactive experience. The positive feedback we receive from the usertesting sessions often confirms the high quality of the content while any negative
feedback or observations are used to improve content. The many users of the CBeebies
site who return again and again to play online with CBeebies show how worthwhile and
useful the user testing is.
However, over the last two years I noticed that the user testing we were doing was no
longer as satisfactory as it had been in the past, both the user-testing sessions themselves
and the results we were getting. It was often hard to quantify what we had found, there
was a sense of frustration in some of the sessions about how we interacted with the
children and some of the team commented that they were no longer learning anything
new when they went user testing.
From March to May 2005 I observed a baby using the Tavistock Clinic’s method of
psychoanalytic infant observation (Rustin, 1989) as part of the Children Under Three
module for my MA in Early Childhood Studies. I observed Mara on four separate
occasions from age three-and-a-half weeks to eleven-and-a-half weeks. I found my
observations of Mara to be rewarding, offering a contrast to the observations of children
under six for user testing that I was finding frustrating. This inspired me to rethink how
5
we observe children when we go user testing. I wanted a user-testing session to be as
rewarding and informative as one of the observations of baby Mara. As user testing was
often the team’s only chance to be in touch with our young users I was keen to ensure
that contact and experience was as positive and rewarding as possible.
Would an observational model like the one from the Tavistock Clinic, which was
designed specifically for observing babies and children under two, be more suitable for
our user-testing sessions than the methods we were currently using, based upon
user-testing methodology for adult users? Moreover, might other methods of observation
and assessment designed specifically for young children and early years’ settings (Abbot
and Gillen, 1997; Sylva, Roy and Painter, 1980; Pascal and Bertram, 1997; Carr, 2001;
Clark and Moss, 2001) be more suitable as well?
In the summer of 2005 I set out to find out how user testing for young children under six
could be changed and maybe improved. I had originally hoped to try out different
methods of observation and assessment for the early years, alongside the team’s more
orthodox user-testing methods. However, user testing cannot be conducted alone, it is
always a team exercise as you need at least one facilitator and one observer. Even though
I wanted to try out new observation methods that might change the roles of the traditional
facilitator and observer, I knew those methods would not result in user testing becoming
a solitary activity. Moreover, whatever I might find out alone it was essential to share it
with the team to ensure that the new method was understood and practiced by everyone. I
therefore realised early on that the main focus needed to shift from adapting different
methods to working with the team to come up with a new way to approach user testing. I
wanted to improve user testing not only as an experience for the children with whom we
user test, but also for the CBeebies web production team themselves. Once I could
establish how the team understood both user testing and the audience of young children, I
could work out with them the best approach for a new way of user testing. While my
reading and ideas from my Early Childhood MA were informing my thinking,
particularly around listening to young children and empowering them, I wanted to work
6
inclusively so that all the team’s opinions were heard; it was not just about listening to
the children but listening to the team too.
At times I have wondered if it would have been simpler to have dictated the new process,
using all the research I had done and knowledge I had gained, rather than democratising
the process so that the team were consulted at every step. This became particularly
pertinent towards the end of the project when Ana, a student completing an MSc in
Digital Television, joined my department at the BBC on work experience. Ana’s
particular focus was on young children’s use and response to interactive television. While
at BBC Children’s she developed some prototypes for the CBeebies interactive TV
service and then tested them at her university’s user-testing lab with children aged four
and five. I have found it inspiring to share ideas and new ways of doing user testing with
young children with Ana, as well as refreshing and reassuring to compare our
experiences. Ana has worked completely alone in developing and testing the prototypes,
which contrasts to my approach. As an outsider to the department who is only with us
temporarily she does not need to consult at every stage and make sure everyone involved
agrees with what she is proposing. She has done a finite piece of research that will
contribute to the next development of the interactive TV service for CBeebies and along
the way has highlighted different methods of user testing with young children.
Denscombe refers to how the participatory aspect of action research democratises it in
some way, showing ‘respect for practitioner knowledge’ (2003: 77). At the very
beginning of the project I expected to carry out action research. That was ultimately not
practicable, but through the paradigms I have worked within I have retained that
aspiration to action research. Those paradigms include my belief that no researcher can
participate in society without adhering to a particular paradigm or world view,
consciously or not; that theory informs practice and practice informs theory; and that
everyone’s experiences are valid and should be valued, through the feminist paradigm of
equality between men and women. Moreover, my approach of listening to the team about
their experiences and trying to see how those experiences can feed into the theory and
practice of the user testing we do with young children reflects the child-centred
7
methodologies that have been so crucial to my overall project. Listening is at the basis of
the work of Donaldson (1978 and 1987), Tizard and Hughes (1986), Paley (1986), Carr
(2001) and Clark and Moss (2001). I have not yet had the chance to apply those listening
methods to the user-testing sessions in order to empower young children but I hope by
having listened to the team I can empower them to go away and listen to the children
themselves.
8
Literature Review
Introduction
The literature review covers four different areas of knowledge and research, all of which
contribute to an understanding of what might be needed to improve user testing with
children aged six and under. I look at definitions of usability and user testing, and how
they are both applied to adult users of different interfaces and websites. I then review the
literature that is specific to children and usability within a web and information and
communication technology (ICT) context. There is a long tradition of observation and
assessment in early years’ settings, some of which originally inspired how CBeebies user
testing could be improved, so I selectively review the relevant literature, comparing and
contrasting different approaches and their long-term goals. Finally, I turn to the children
with whom we user test and so in my fourth area I review literature and findings about
children’s capabilities, their minds, how they learn, and how that has been used to
empower children.
Literature about usability and user testing within web design context
Usability within the context of ergonomic requirements for office work with visual
display terminals has been defined since 1998 under ISO 9241-11 as ‘the extent to which
a product can be used by specified users to achieve specified goals with effectiveness,
efficiency and satisfaction in a specified context of use’ (UsabilityNet, 2006). This is a
definition which has been used by Human-Computer Interaction (HCI) researchers in
usability studies (see Sim, MacFarlane and Read, 2006: 237). However, there is also a
slightly more recent definition within software engineering under ISO 9126-1 that
combines usability as being essential for interface design and to meet users’ needs (ISO,
2001; UsabilityNet, 2006). Usability under the second international standard is defined as
‘the capability of the software product to be understood, learned, used and attractive to
the user, when used under specified conditions’ (UsabilityNet, 2006). This second
definition is most familiar to web developers and designers who practise user-centred
design, always thinking about the needs of the user when creating a website. Both Krug
and Kuniavsky sum this up very simply:
9
‘Usability really just means making sure that something works well: that a person
… can use the thing … for its intended purpose without getting hopelessly
frustrated.’ (Krug, 2006: 5)
‘Ultimately, usability is good design.’ (Kuniavsky, 2003: 20)
Thus usability is designing something (in the context of this study, a website or an online
game or application) that can be used easily by the people for whom it is made (again, for
this study, children aged six and under).
In order to ensure that a website is working well and can be understood, that it can be
used and is also attractive to the user, user testing or usability testing (the two terms are
interchangeable) should be carried out during the development of the site and once it has
been launched for public use (or ‘gone live’). Nielsen identifies just three components of
user testing: representative users, users to perform representative tasks and people from
the website to observe what the users do without helping them in any way (Nielsen,
2003). You can do the user testing in a usability lab with a one-way mirror, cameras and
microphones but it is not essential: it can equally be done in a home or office setting with
a few users and at least one observer. Kuniavsky (2003) covers other ways that you can
get feedback about the usability of a site or product and users’ needs, such as cardsorting, surveys, focus groups and usability diaries. Even with the detail of the different
approaches from Kuniavsky, this all sounds very simple and achievable. However, is it
that simple when user testing with young children?
Both Nielsen and Kuniavsky are writing about user testing with adult users. Kuniavsky
refers fleetingly to teenagers twice (Kuniavksy, 2003: 112, 532) but does not discuss how
user testing might need to be adapted for teenagers, let alone children under thirteen or
much younger children aged six and under. Nielsen has done one usability study on how
children aged six to twelve use websites (Nielsen, 2002) which is discussed below. Krug,
a third example of a ‘usability expert’ in the world of web development and HCI, does
not refer to children or teenagers at all, though he does stress that it is a good idea to get
10
to know your users and test with them, and that it is important not make assumptions
based on your (adult) experiences of being a web user (Krug, 2006: 125-129). I now turn
to specific literature about children’s ICT and internet use in order to find out more about
children and usability.
Literature about children and usability within a web and ICT context
Literature about how children in the early years in the UK use computers is based around
what they learn from their encounters with ICT and how it can benefit or delay their
development. While it is important to review literature about young children and
usability, there are not many written accounts of such research studies; very little
literature focuses on how pre-school children use the internet or how they use a computer.
Therefore I begin in this section by examining the general guidance for early years’
practitioners when planning children’s learning around ICT and then look at how a few
researchers have tried to apply that guidance (from the QCA) to young children’s use of
different forms of ICT at home and at nursery.
‘Curriculum Guidance for the Foundation Stage’ takes an umbrella approach to ICT –
children are expected to develop technological literacy, gain information and
communication skills from ICT and start to understand control technology (SirajBlatchford and Whitebread, 2003: 1; QCA/ DfEE, 2000). Within the general area of
learning ‘Knowledge and Understanding of the World’, specific areas of learning about
ICT range from ‘stimulate all children’s interests in ICT and other technology’ (QCA/
DfEE, 2000: 93) to ‘complete a simple program on the computer and/or perform simple
functions on ICT apparatus’ (ibid: 92). In their review of international research evidence
examining ways in which ICT is used in pre-school settings, Plowman and Stephen
(2003: 150) allude to this ‘broad definition of ICT’ but highlight that ‘most of the
literature currently focuses on ICT defined as computers’ and that there is ‘a scarcity of
good quality research findings on using ICT with pre-school children’. Brooker and SirajBlatchford support this, commenting that ‘there is still a relatively small body of
empirical evidence on the effects of their use by children aged three and four’ (2002:
253).
11
McPake, Stephen, Plowman, Downey and Sime have tried to go some way to correcting
this, looking at the home experiences of pre-school Scottish children using ICT (McPake
et al, 2006), covering interactive TV, DVD players, home internet-use and video games.
However that study is still only looking at how ICT in a general sense can benefit a
child’s learning, not at how a child uses a particular ICT application at a given time (or
even how that application can be made better for the child). Brooker and SirajBlatchford’s study of computer-use in an inner-city nursery (2002) looks at how children
interact with a particular ICT application over a given time (the nursery computer over
four months). However, as with McPake et al, the study is most interested in the effect
the computer and other ICT has on the children’s development and learning, as well as
whether home ICT experiences are an advantage or disadvantage for then using the
nursery computer.
The literature therefore suggests that ICT is viewed only as something to be used with a
child, to benefit the child or to see how it affects a child’s cognitive, linguistic or social
development. As stated above, there is little that goes one step back and focuses on how
pre-school children use the internet or how they use a PC. Even though some of these
researchers may be looking unwittingly at the usability of certain ICT applications and
software, their focus is not on how to make the ICT better for the child or how to conduct
a study to find that out. For literature about how children use the internet or a computer
and how to conduct research around that, we need to look at usability studies of slightly
older children, some over the age of six but most of at least age eight.
Researchers at Microsoft have published guidelines for usability testing with children
(Hanna, Risden and Alexander, 1997 and Hanna, Risden, Czerwinski and Alexander,
1998). In both of these articles the researchers discuss ways to adapt traditional adult
usability testing for children. However while they do address specific issues for children
under six they rarely focus just on that age group: the guidelines cover children from age
two to fourteen. The guidelines are also specific to Microsoft – they assume the provision
of a usability lab and suggest ways to make the environment more appealing to children
12
(at the BBC I do have access to a usability lab but it is more common to user test with
children in their own homes, the office, schools or early years’ settings). In the second
article (Hanna et al, 1998) the focus is on how to make usability central to the design
process of creating computer products for children. Thus while they give age-specific
recommendations and examples (differentiating between three and six-year-olds’
different understanding; reading text alone is not age-appropriate for four-year-olds;
asking parents to rate products as well for the under-fives; card-sorting is only
appropriate for children of eight or over) the emphasis is still on making adjustments to
traditional user-testing methods for all children, not just those under six. This is probably
because one aim of the article is to persuade product development teams of the
importance of having usability engineers working alongside them. So they play down
how hard it might be to user-test with children: ‘children can participate in traditional
laboratory usability tests with only minor adjustments in procedure’ (1998: 7).
Markopoulos and Bekker (2003b) take issue with the advice from Hanna et al (1997) as
‘it is not very clear on what evidence it is based and thus we conclude it describes the
personal experience of the researchers involved’. While personal anecdote can be useful
as a starting point for anyone setting out to do usability testing with children, the main
limitations of their advice is that they give the impression there is no need to delineate
between the different ages of children. The first article (Hanna et al, 1997) promises
much, setting out sections about children from pre-school, elementary school and middle
school but ultimately the authors do not follow this through and give only cursory
examples of different ages, sometimes not being clear to what age they are referring.
Nielsen’s usability study on how children use websites supports how important it is to
delineate between the different ages. His study (Nielsen, 2002) user-tested websites with
children aged six to twelve. This means he has missed the age of CBeebies users,
children aged six and under, but some of the findings around the youngest children in his
group of users should surely be illuminating. However, while he finds that ‘extensive text
was problematic for young children who are just beginning to read’ (2002: 2), when he
then summarises the differences between testing with adult and child users he finds that
‘half of our young users were willing to read instructions’ (2002: 3). There is obviously a
13
problem with grouping all primary school-age children together – presumably the half
who wanted to read instructions came from the older half. Nielsen himself highlights how
grouping children together can be problematic, as there were big differences between
boys and girls in the testing (it was mostly girls who were willing to read instructions).
As with the Microsoft articles, Nielsen’s study is helpful as a starting point but is not
specific enough for doing usability testing with children under six.
Markopoulos and Bekker (2003a) highlight how ‘limited methodological advice has been
published to date regarding the adaptation of usability testing procedures for children of
different ages … considering the importance of the topic and its practical utility there are
surprisingly few research results published to guide the practitioner’ (2003a: 146). The
guidelines published by the Nielsen Norman Group for websites for children following
Nielsen’s usability testing with children aged six to twelve (Nielsen, 2002) are a ‘sign of
maturation for this field’ (Markopoulos and Bekker, 2003a: 149). They also credit Hanna
et al with both the articles already cited (1997 and 1998). However Markopoulos and
Bekker’s emphasis is on how usability testing needs adapting for the particular needs of
children, as well as other special target user groups.
Druin’s research into the role of children in designing new technologies has led to the
setting up of a scheme at her HCI lab at the University of Maryland where children are
design partners with adult research teams (Druin, 2002: 13). Her review of how the role
of children has changed in the HCI community since the 1970s includes a diagram of an
onion representing the four roles children may have in the design of new technologies, as
user, tester, informant and design partner:
14
(Druin, 2002: 3)
The detailed information she gives of each role, from a historical and practical
perspective, seems promising. In the user testing CBeebies does currently, children would
be seen as user, tester and informant, depending on the stage of development of a game or
website. However, as Druin does not really discuss children under the age of five, this
conclusion is only reached from extrapolation. She does allude to adults observing
children and how it is ‘particularly common for research with young children (ages 4 and
younger)’ (2002: 5) but most of her study and motivation seems to be tailored to children
of seven or above. Moreover, the overall feeling of her work is that it is not specifically
looking at usability and user-centred design for children but at the many different ways
children can interact with different technologies. This is shown by the myriad examples
she gives of research methods for children as users (2002: 5-7). When she discusses
children as design partners, she states that ‘the impact that technology has on children
may not be as significant as the impact children can have on the technology design
process’ (2002: 12). Thus her greatest interest lies in developing the design process and
making sure children are equal stakeholders with adults in the design of new
technologies; the emphasis is on the relationship and process, not the end product. This
applies even for the younger children as the emphasis in the article about Swedish
children aged five to seven who were design partners was on how they ‘changed from
being learners and critics to being inventors and design partners’ (2002: 21). No mention
is made of what they invented or designed. For that reason, Druin’s approach needs to be
treated carefully. While it is useful for CBeebies user testing as it may help to make the
15
process of user testing more enjoyable, in that it focuses more on the child as user, the
user testing still needs to feed into the end product, benefiting all users of the website.
In Sim, MacFarlane and Read’s empirical study of fun, usability and learning in
educational software, with children age seven and eight, they highlight how ‘there is no
clearly established methodology for evaluating software for children’ (2006: 236). Their
approach seems to reflect the studies from Brooker and Siraj-Blatchford (2002),
Plowman and Stephen (2003) and McPake et al (2006). Even though they use
observational methods and a ‘fun sorter’ to assess usability and fun of software, they also
measure the learning effect of the software on children. The software was for Key Stage
One Science, which would be studied by children at the very top end of the CBeebies
user profile. However, the children in the study had already taken their SATs in Science
and so ‘were about one year older than the age range for which the products were
intended’ (2006: 240). This study is of some value in suggesting ways for younger
children to evaluate their experience of using software or interactive products (the ‘fun
sorter’ helps children to map how much fun, learning and ease of use they thought the
software had, (2006: 242)) but does not give much insight into how to tailor a usability
study for children under the age of six. The observational methods they use are just free
note-taking with two researchers, with one observing the child and one looking at
usability issues (with some inevitable cross-over, 2006: 241). The children themselves
are that bit older than CBeebies users, so are used to performing ‘tests’ in a school
environment with an adult who is unfamiliar to them.
Markopoulos, Barendregt, Brus and Plakman (2005) suggest a new way for evaluating
interactive products. Their method uses the parent as facilitator and is most suited to
children aged four to six. While the research is in its early days (‘evidence for the
effectiveness of the method is very small’ and ‘it is quite hard to recruit parents and
children to participate’ (2005: 38)) it at least takes into consideration the importance of
children interacting with products in their own habitat (i.e. at home) and allows the
testing to take place over a few days, thus mimicking how they genuinely might use a
product or website if they were new to it. In a similar way, Barendregt (2006) has tried to
16
find ways to tailor user testing to young children. She tried using picture cards from the
Picture Exchange Communication System (PECS) for children aged five to seven to
identify usability problems. She found that ‘the problem identifying picture cards are a
good addition to the thinking-aloud method’ (2006: 132). The picture cards used in
addition to thinking-aloud methods allow children to ‘express more problems than with
standard thinking-aloud’ (ibid). Barendregt gives ‘some new guidelines for usability
testing of computer games with children’ based on experimental studies, rather than on
the personal experience of Hanna et al (2006: 136). However, while these two studies are
helpful to give hints and tips for how to improve user testing for the CBeebies users, they
are both small scale and neither attempt to user test with children under four years. Both
studies are more useful than Hanna et al (1997 and 1998) and Nielsen (2002) as they are
focusing on how methods need to be adapted very specifically for younger children.
Overall, none of the literature about children and usability focuses on a child-centred
approach specifically for children under the age of six. I now review literature about
observing and assessing children in early years’ settings as it more commonly puts
children at the centre.
Literature about observation and assessment of children in early years’ settings
There are many different approaches to observing pre-school children, all of them
designed specifically for understanding children’s behaviour, emotions and experiences.
They also all have different ways of recording what is observed or what is discovered as a
result of the observations. What links all of them is that they put children at the heart of
the observations. They are also linked through the observations being carried out to fulfil
some form of assessment, whether it be assessing the child, the setting, the staff of an
early years’ setting (or parents and other carers) or all three. This is particularly pertinent
for user testing as none of the three of the child, the setting or the staff is ever being
assessed. Instead, the value and usability of the website or online game to the child is
being assessed. The methods of observation discussed below are helpful to offer new
ways of approaching user testing for children under six, as their focus is on the same age
group of children who use the CBeebies website. This is in contrast to traditional
usability methods which are designed around adult users. However as the majority of the
17
observation methods are forms of assessment they need to be treated with some care
when seeing if they can be adapted or adopted for user testing for children under six.
The emphasis in the Tavistock method of psychoanalytic infant observation is on a
synthesis of behaviour and experience, so that there is always a whole picture of the baby
and its developing relationships, emotions and sensations. Unlike ‘empirical research
methods in child development psychology ... [the observations] do not aim to select
specific attributes of behaviour (perceptual or cognitive skills, recognition or memory, for
example) for discrete study...’ (Rustin, 1989: 55, 54). By then encouraging the observers
to record what they see in a ‘literal and factual’ way (ibid: 52), using non-theoretical
language, interpretation, reflection or analysis is left for later. This method of recording
what is seen is similar to user testing, where observers are encouraged to just write down
what users do and say, without any interpretation. Discussion among all stakeholders
once the results are recorded is the time for interpretation and conclusions. Rustin
highlights how psychoanalytic infant observation contrasts to most other forms of
studying children from the fields of child psychology and child development in that it is
holistic and subjective (focusing on the feelings of the observer and the observed). It also
contrasts to other forms of child observation as its focus is on the child at that moment,
rather than how the child changes or develops over time.
The ‘multi-method’ approach of the research project ‘Educare for the Under Threes’ at
Manchester Metropolitan University focused on very young children (Abbott and Gillen,
1997). That focus on children under three could be helpful for CBeebies user testing, but
the project’s primary aim was to assess the contributions from ‘educarers’ in a setting,
thus ensuring good practice in early years’ settings for under threes. So one of the
researchers observed ‘the interactions between staff and children, children and their
peers, and staff with other adults including parents’ (Griffin, 1997: 36). Thus the ‘multimethod’ approach is looking more at adult-child interaction and is therefore not
immediately appropriate to be adapted for user testing.
18
Sylva, Roy and Painter (1980) explain that in the target child technique from the Oxford
Preschool Research Project ‘observational research is undertaken in vivo as children go
about their everyday activities at preschool’ (1980: 8). It therefore does not find out how
they get on when they are older, unlike longitudinal studies such as Head Start in the
USA, nor does it base its findings on anecdote (as per Susan Isaacs). Target child
researchers note key behaviour of one child every 30 seconds over a 20 minute period,
trying to answer the question ‘does preschool experience matter?’ (Sylva, Roy and
Painter, 1980: 9). By using the research findings to answer that question, the target child
researchers are assessing the setting and staff as well as the child. They want to know
how a child adapts his or her behaviour to being in a pre-school setting and thus the
observations for their study can only take place in that setting. The target child method is
similar to the multi-method approach in that both want to get an evaluation of the child’s
experience at pre-school out of the observations, though the observation techniques
differ. While that means that both methods could be useful for user testing as they look at
a child’s experience ‘in the moment’, they are less useful once it is clear that they are
assessing the setting and staff as well.
The Effective Early Learning (EEL) Project explicitly aims ‘to create a methodology for
evaluating and developing the quality of early learning ... for 3 and 4 year olds in the UK’
(Pascal and Bertram, 1997: 1). The EEL Project evaluated the quality of early learning
within a setting using Laevers’ Child Involvement Scale and Adult Engagement Scale
(ibid: 11), together with Vygotsky’s theories on the social context of children’s learning.
In the EEL Project ‘process’, participants for each setting are expected to ‘improve the
quality of provision’ once they have evaluated their setting (ibid: 11). The Child
Involvement Scale is in the spirit of psychoanalytic infant observation and the target child
approach: ‘it attempts to measure the processes of learning, rather than to concentrate on
outcomes’ (Pascal and Bertram, 1997: 11). So even though the scale is measuring how
well a child learns, it does not then test what the child has learnt. This is similar to what
user testing tries to achieve – observing how well a child interacts with a website or
online game but not necessarily then testing them on what they have experienced. By
using the Adult Engagement Scale alongside the Child Involvement Scale the researchers
19
make clear that they are assessing the staff and thus the setting as well, as the staff’s
interactions with the children contribute to determining the quality of the setting. Thus a
common purpose of at least three of these methods for observing young children is the
assessment of quality.
As Dahlberg, Moss and Pence (1999: 5) point out, ‘in the 1990s, the concept of quality in
the early childhood field ... has been questioned or problematized.’ Both Vernon and
Smith (1994) and Dahlberg, Moss and Pence give potted histories of the concept of
quality, particularly since 1945 and within the realm of early childhood. There is a
growing body of literature that addresses this problem (e.g. Elfer and Wedge, 1996;
Selleck and Griffin, 1996) and it seems intrinsic to any discussion of the value of
assessment and observation. Moreover there is plenty of literature about the value itself
of observing and assessing young children (Drummond and Nutbrown, 1996; Carr 2001;
Drummond, 2003). Most of these discussions focus on the assessment of the child which
brings us back to the question – what or who is being assessed through the observations?
Ackers (1997) highlights how observation and assessment became interchangeable in the
multi-method project: ‘if we are to see observation and assessment as a priority in the
educare of young children then perhaps as educarers we need to consider ... making time
to record, analyse, discuss and share our observations’ (1997: 85). Both Drummond and
Nutbrown (1996) and Carr (2001) highlight that ‘we could not proceed with the “how” of
assessment until we confronted “what” was to be assessed’ (Carr, 2001: xiii). This is not
quite the same as the difference between what or who is being assessed. While Carr looks
what children can do in their day-to-day lives as they become learners and then how to
assess that learning, some of the observation and assessment methods discussed look
more at the difference between assessing a setting, a child and an adult carer or educator.
In user testing, the child is never supposed to be ‘what’ is being assessed, it is always the
website or the online game. This leaves a potential gap around the child – will we ever
find out what children think, know and can do when they use the CBeebies website if we
persist in the usability model of testing the website and not the user? I am therefore
20
interested in exploring children’s capabilities and so now review literature about what
children can do and how we can empower children to use those capabilities.
Literature about children’s capabilities, their minds and how they learn, and how
that has been used to empower children
The main themes in literature that explores how children can often do much more than
we (society, educators, institutions, even parents) think they can, are listening and
language. From Donaldson in 1978 to Carr and Clark and Moss in 2001, researchers have
listened to young children, recorded what they say and then found out much more about
how children think than is sometimes supposed, how children perceive the world around
them and how they make sense of it. Carr and Clark and Moss have taken that one step
further and enabled children to record for themselves what and how they think.
Donaldson rejects ‘certain features of Jean Piaget’s theory of intellectual development’
(1987: 9) when she shows that young children can use language for thinking in ‘real-life’
contexts. Thus she establishes that children under seven are actually quite good at
appreciating the point of view of another person in a literal sense, through Hughes’
adaptation of Piaget’s three mountains task (Donaldson, 1987: 20-23). If the situation
makes sense to the child or uses familiar situations, characters and language (policemen,
naughty boy, hiding) then they find it easier to understand the task in hand and the
alternate point of view. This therefore challenges the long-held view that children are
unable to ‘decentre’ and are highly ‘egocentric’ (Donaldson, 1987: 18-19). Donaldson
goes on to show that children respond when language is changed, for example when a
statement is modified with an adjective in the case of the ‘steps’ between teddy, the chair
and the table (1987: 45-46). Notwithstanding that even when tested on a group of five
graduates, two of whom also have PhDs, I found that Piaget’s test of sub-classes was
only properly understood when McGarrigle’s adaptations for young children were used
(‘more black cows or more sleeping cows’, not ‘more black cows or more cows’,
Donaldson, 1987: 44), Donaldson shows that young children are capable of reasoning
when they can understand what they are being asked. Donaldson looks in some detail at
the demands of school on young children, and the effect on their thinking and reasoning.
21
However the majority of CBeebies users are not yet at formal school and so for the
purposes of this study it is significant enough that she finds ‘by the time they come to
school, all normal children can show skills as thinkers and language-users to a degree
which must compel our respect’ (1987: 121). Therefore, young children’s intellectual
development must not be underestimated when we are observing children as users of the
website.
Donaldson’s findings have been backed up by Tizard and Hughes’ study of 30 four-yearold girls from middle-class and working-class backgrounds at home and school (Tizard
and Hughes, 1986). By studying the conversations the girls had with their mothers they
found that the girls used language to understand new ideas and were persistent and
logical thinkers. These ‘passages of intellectual search’ showed that the young children
were ‘powerful and determined thinkers in their own right’ (1986: 9). Both Donaldson
and Tizard and Hughes are helpful when thinking about the environments in which we
user test (home, nursery, school) and the language we use with the children.
Paley (1986) recounts how she changed her approach as a kindergarten teacher in the
early 1980s to become a listener; she learnt that discussion blossomed when she kept
quiet and did not express her own point of view. From this, she built up a technique for
children to tell their own stories, culminating in the children she taught taking it in turns
to tell stories each day. Paley found a way to empower the young children in her care,
enabling them to find their own voice. She showed too that each child had their own
opinions and fears, something which can often be ignored. Carr (2001) also uses stories
to empower children. Like Paley, Carr has her own story about listening to the children in
her classroom when she was a kindergarten teacher and how that prompted her to develop
an alternative model of assessment to the one usually accepted by students, teachers and
parents (Carr, 2001: 2).
The ‘Learning Stories’ approach Carr has developed as a way to assess children’s
learning enables children to participate in their own learning and assessment. Children
can be fully involved in discussing and documenting what they have learnt, while early
22
years’ practitioners describe children’s learning and make decisions about where to go
next in the child’s learning. The learning dispositions that form the framework for the
Learning Stories are all about children as active learners, as subjects in society: ‘learning
dispositions are about responsive and reciprocal relationships between the individual and
the environment’ (Carr, 2001: 22). A child is a ‘learner-in-action’ (2001: 5) – being
ready, being willing and being able [to learn]. Their learning story is then told through
the five domains of the learning dispositions: taking an interest, being involved,
persisting with difficulty or uncertainty, communicating with others and taking
responsibility (2001: 23).
The Mosaic approach (Clark and Moss, 2001) is similar to Carr’s Learning Stories in that
it finds ways to listen to young children and then document children’s thoughts and
experiences. The approach includes the idea of ‘children as experts in their own lives’
(2001: 6) and advocates a way of listening that allows children and adults to be coconstructors of meaning (2001:1). This approach reflects Dahlberg, Moss and Pence’s
argument that early childhood institutions need to look beyond providing pre-specified
outcomes with a view of the child as an empty vessel and instead look towards a
‘discourse of meaning making’ (Dahlberg, Moss and Pence, 1999). Central to all three
approaches (Carr’s Learning Stories, the Mosaic approach and Dahlberg, Moss and
Pence’s ideas) is the idea that children are capable of being the architect of their own
lives, as long as they are given the means to communicate and record what they do. Thus
for user testing, where the children are sometimes seen as objects to observe, perhaps
even ‘empty vessels’ absorbing the online content, new approaches are possible where
the child’s perspective and opinion can be recorded, documented and discussed (with the
child and the adults).
Now that I have evaluated the literature that contributes to an understanding of what
might be needed to improve user testing with children aged six and under I will conclude
this chapter with the three research questions that I am considering at every stage of the
project. The three questions are:
23
1. What is the web production team’s understanding of the principles and purposes
of user testing?
2. What do the web production team understand about the audience when they go
user testing?
3. How feasible is a more child-centred approach when the web production team go
user testing?
The first research question refers back to the literature about usability and user testing
within a web design context, as well as sometimes to the literature that is specific to
children and usability. The second research question refers to the fourth area of literature,
about children’s capabilities, their minds and how they learn, and how that has been used
to empower children. The second research question also refers at time to the third area of
literature, about observation and assessment of children in early years’ settings. For the
third research question, all of the literature is relevant as it is the key question of my
research project, how adult usability testing methods can be combined with child-centred
methods that empower children to create a new and improved way of user testing for
children aged six and under. The first and second research questions build up to the third
– it is essential I know how the team understand the audience and give them a good basis
of the principles and purposes of user testing before I can even broach the subject of
making the user testing we do more child-centred.
24
Research Methodology
The parameters of my research methodology: qualitative research, the feminist
paradigm and the child-centred view
‘Social theory informs our thinking which, in turn, assists us in making research
decisions and sense of the world around us. Our experiences of doing research
and its findings, in its turn, influences our theorizing; there is a constant
relationship that exists between social research and social theory.’
(May, 1997: 28)
This description from May of the ‘mutual interdependence’ between theory and research
is the starting point for the research methodology I have used. Within May’s explanation
of how ‘we need theory and theory needs research’ (1997: 30) there is the assertion that
no researcher can participate in society without adhering to a particular paradigm or
world view, consciously or not. Thus when I am at work at the BBC I am not overtly
applying a paradigm to the research I do with my team. Yet when I come to analyse it
and decide upon the different methods I am going to use in order to find out different
things, a particular paradigm – the way I view society and the way I live my life – comes
into play.
The overall approach of this study is qualitative; the reasons why that is the most
appropriate approach for the research will become apparent as I explain each step of the
methodology. However, I am also applying a feminist paradigm throughout the research.
The feminist paradigm I am applying is more accurately described as a variety of feminist
perspectives. I will give some examples of different feminist perspectives, such as
standpoint feminism, the critique of disengagement and not separating reason from
emotion, and show how they apply to the research I have carried out. Some of them have
their parallels in child-centred research. All of those perspectives though are not only
linked by feminism but also by the mutual dependence of theory informing practice and
practice informing theory. When I, as a researcher, participate in society I am adhering to
25
a feminist paradigm or world view, consciously or not. Thus those feminist theories
inform my practice.
While in social research the distinction is often made between qualitative and quantitative
methods, Pring’s assertion of the false dualism between those two schools of thought is
useful to understand how and why I have chosen an overarching feminist approach.
Pring’s false dualism argument is that a researcher can be working within the political
arithmetic tradition, which uses a quantitative approach to produce hard data, but that
they do so ‘without commitment to a thoroughly determinist position or to a rejection of
the distinctively personal way in which the social world and experience is interpreted by
each individual’ (Pring, 2004: 97). Thus a quantitative researcher can take a feminist
approach when interpreting results by relating to the subjects of the research and not
insisting upon separating reason and emotion (May, 1997: 20). I am guided by the belief
that theory informs practice and practice informs theory and so I cannot reject theory and
let the facts speak for themselves (May, 1997: 11). Nor can I assume that facts exist out
there about the social world, independent of interpretation, as empiricists might. In that
sense, a qualitative approach may be the only option. But it is important to remember that
within feminist perspectives both quantitative and qualitative methodologies are valid.
While some feminist researchers may reject any form of science as it does not take into
account the many different versions of social reality, there are still feminist scientists and
feminist researchers who apply hard quantitative methods.
The basics of usability are to put the user at the centre of the design of a website or
application. By involving the user in that way, with the emphasis on the user’s
experience, usability methodology tentatively alludes to some of the aims of standpoint
feminism. Standpoint feminism takes women’s experiences as a starting point but the
paradigm is only complete once women are actively involved in the theorising about
those experiences (May, 1997: 22). Nielsen (2002 and 2003), Krug (2006) and others
would be unlikely to see themselves as working within a feminist paradigm; indeed many
feminist researchers would be likely to label usability researchers as working within
‘male-science’, reflecting hierarchical society and methods, with the usability researcher
26
becoming expert in people’s computer and internet use. But by always involving the user
and placing primacy on the user’s real-life experience, usability researchers have taken
one step towards standpoint feminism. Druin’s (2002: 13) emphasis on the user as
design-partner goes one step further, as she favours a democratic and participatory way
for applications to be designed and developed.
Of course, both Druin and I focus on the child user in our research, not on women’s
experiences of websites. However I would argue that by putting the child at the centre of
the research, many child-centred researchers are unconsciously applying different
feminisms. Within feminist criticisms of research, women are seen as marginalised in
society; their experiences and world-view are not the dominant one. Child-centred
researchers similarly tackle how children are marginalised and silenced in society. While
some elements of feminism prioritise women’s experiences within society, sometimes
above men’s, others focus on the equality of everyone within society, irrespective of
gender, class, age or race. From that perspective, everyone’s experiences are valid and
there should not be one dominant view. Pring’s assertion that the distinctions within
paradigms are as important as the distinctions between them (Pring, 2004: 48) is pertinent
here. The child-centred theories and methodologies of researchers like Carr (2001) and
Clark and Moss (2001) may not seem feminist at first glance but their aims and outcomes
can be seen as part of one overarching feminist paradigm. In the research I have done at
work with my team I have referred to and discussed child-centred theories with them, but
for myself, I have applied a feminist paradigm when analysing the interviews, discussions
and interpretation of child-centred theory.
In my research project my team at work are central to my methodology. As I am a
member of the team, I am involved with them and identify with them; just by virtue of
who I am in relation to them, I take a feminist stance of the critique of disengagement
(May, 1997: 20). While the team have been involved in the process of finding new ways
to user test the CBeebies website with children under six they have not been involved
every step of the way in critically evaluating what works and what doesn’t. They gave
their feedback about the current process when I interviewed each of them individually
27
and when I conducted the group discussions with them. The smaller ‘working group’ I set
up as a result of those group discussions is more involved in actually formulating a new
process. Therefore, I am not doing action research with the whole team, though the
potential outcomes (new or improved practice for young children) may be similar. The
methods I use with the smaller working group are more akin to action research. The
emphasis with the team is on practical outcomes and their input into the research will
affect the theories and methodologies I then apply at later stages of research.
Before I started my research my team and I thought we were already applying a particular
methodology when we went user testing. We based our user testing on the usability
methods advocated by Nielsen (2002), Kuniavsky (2003), Krug (2006) and others. We
also adapted those methods as we were aware that most usability methods were designed
for adults. So we used expertise from colleagues in the department who had done user
testing with children and took advice from usability managers and behavioural scientists
across the BBC. We referred to the article from Microsoft about user testing with
children (Hanna et al, 1997) and the 70 design guidelines for websites for children
published by the Nielsen Norman Group (Nielsen and Gilutz, 2002). Thus we had a
methodology of sorts, even to the extent that we had a departmental document to refer to
about usability, user testing and child development (Ward, Howell, Wilton, Winter,
Dickinson and Gauld, 2004). So from the very beginning of my research, the principles
of usability gave me a foundation and starting point. These principles are based on
qualitative methods and analysis. Some quantitative methods can be used to analyse
usability test results (see Nielsen, 2004) but overall usability studies and user testing are
qualitative in both methods and analysis.
The other methods I have chosen for my research project reflect the evolution of my
thinking as well as the evolution of the project itself. I will shortly explain in detail why I
have chosen each method. When explaining these choices, I am keeping in mind how the
methods fit within the parameters I have explained above: qualitative research, the
feminist paradigm and the child-centred view.
28
My methods of research
I began the project with the intention of trying out different observation methods at the
different user-testing sessions we do each month, motivated originally by the Tavistock
method of psychoanalytic infant observation (see Rustin, 1989). I eliminated this as an
effective methodology early on but the actual process of attending the sessions shaped the
subsequent stages of the project. As we take it in turns to go user testing each month in
pairs, it was unusual for the same person (me) to go to every user-testing session over
three or four months. At first I was there as a third person observing the children’s
interaction with the website but I was also able to observe the differing approaches to
user testing from my colleagues. I then compared the different reports written about each
session. These reports and the discussions I had with my colleagues after each session
gave me insights into subjective interpretations of the findings. More significantly,
attending the sessions showed me that the methodology advocated within the department
for user testing was not always being followed and that sometimes my colleagues were
justified in not following it. Therefore, I realised after just two or three sessions that my
original plan to try out different methods of observation while my colleagues carried on
with more traditional user testing was just not possible. I had been aware that our
methodology was flawed in some ways because we were having to adapt adult usability
methods for testing sites with children under six but the continuity of attending every
session highlighted how we as a team needed to rethink what user testing means for
young children.
I therefore decided to interview everyone individually at the BBC who does or has done
user testing for the CBeebies website. I wanted to find out their exact thoughts on why
they thought we did user testing, how they saw the whole user testing process and how
they thought user testing could be changed. While I had support from my manager and
the lead designer to make changes to the user testing process, I wanted to be sure that
everyone was consulted, felt involved with those potential changes and had a chance to
have their say.
29
The final list of twelve questions (see appendix A) was carefully ordered to progress from
general thoughts on user testing to the user-testing process itself and then to how user
testing could be ‘made better’. I interviewed fifteen of my colleagues. Four were
designers, nine were from the editorial team (one editor, three producers, six researchers)
and one was from the TV production side of CBeebies, who conducts focus groups to test
new CBeebies television programmes. O’Connell Davidson and Layder refer to how
difficult it can be to tell whether an interviewee is telling the truth and being completely
open (2004: 116). By giving my colleagues the option to use pseudonyms I gave them the
opportunity to be as open as they liked within a working environment, in case they had
any concerns about making comments about other colleagues or current working
practices. Equally I stressed that the final report I would write was designated mainly for
the university rather than our department. As they all chose to use their own names they
appeared to have very few concerns, although one was indecisive and only decided to
definitely use her own name near the end of the interview. If the department as a whole
wants to use my findings I will need to ask permission again from my colleagues to allow
others access to the interview findings.
The interviews themselves were semi-structured: there was a script of twelve open
questions but I probed further if need be and a dialogue built up between me and the
interviewee. Therefore my interview method was qualitative. May refers to the three
necessary conditions for the successful completion of interviews, from Moser and Kalton:
accessibility, cognition and motivation (May, 1997: 115). All fifteen interviews were
conducted successfully, with only a few problems around accessibility and cognition; all
the interviewees understood what was expected of them and as they had all been user
testing were able to provide the information I was seeking. Three of the questions were
sometimes taken more literally than I expected and did not yield as much information as I
was seeking. The answers to the fourth and fifth questions (‘Think back to the last usertesting session you went on. What did you think worked well?’ and ‘What didn’t work
well?’) sometimes focused on the game or website that had been tested, rather than on
how the session itself was conducted. So, a game ‘worked well’ if it had been positively
received by the children. I tried to seek clarification by asking questions about the session
30
itself but it was interesting how closely a child’s response to a game was seen to affect
the success of a session by at least four of the interviewees. Question eight, ‘Do the
children learn anything new?’ had the shortest set of answers. I did not probe further with
this question and will explain why in my analysis of the interviews.
The third necessary condition, motivation, was sometimes problematic. It was difficult to
phrase any probing questions in an open a way as possible. It was sometimes hard to ask
for clarification without a closed question. Moreover, in my desire to make each
interviewee’s participation as valued as possible I found myself agreeing with comments
more than I expected and making approving comments (‘that’s really good’, ‘that must
have been rewarding’). I also clarified what they said by repeating comments back but in
my own words. While the interviews were semi-structured and worked as a conversation,
if I were to do them again I would try to be more neutral in my responses and repeat back
exactly what they were saying to me if I needed clarification, rather than framing their
comments within my own belief system.
I recorded and transcribed every interview. I found that as I knew all the interviewees
very well, both the interviewees and I were sometimes making assumptions about what
we each understood from the other. Thus points that seemed clear during the interview
were actually incomplete thoughts once I listened again to the audio. I had sometimes let
the interviews progress too much as conversations, (see Denscombe , 2003: 164) so I did
not always find out (or remember) as much detail as I had wanted. This approach was
consistent across all fifteen interviews, irrespective of my working relationship with them
or their gender (just two were men). Of the fifteen, the designers and producers (seven
people), I saw as my peers, in the sense that I work alongside them but do not manage
them or report to them. Of the six researchers, I have been responsible for managing five
of them at one time or another so I was aware of my more senior status to them. The
editor of the CBeebies website is my manager. For both the researchers and the editor I
felt that the interviews were open and honest but of course as we are not on a totally
equal footing at work some opinions may have been more muted than with my colleagues
who are peers.
31
When writing up the interviews I ordered the document by the questions rather than the
interviewees so I could compare what each person said for each question. I then went
through and put into bold text the points that seemed most important (see appendix B as
an example). I then extracted those phrases in bold text and put them all into a separate
document, so that each question had clear themes beneath it. I summarised those points in
another document, which I used to brief Liz, the lead designer and Debra, the senior
designer (both usability experts and both of whom I had interviewed) on the findings so
far (see appendix C).
I chose to have a meeting with Liz and Debra before scheduling a group discussion with
all the interviewees primarily to find out how we could begin to return to the principles of
usability in the user-testing sessions (i.e. a return to the original methodology we were
supposed to be following). The meeting clarified two things. One was that the existing
methodology did need rethinking for children under six – while it was being followed
successfully for user testing for the CBBC website (for children aged seven to twelve) we
needed to have our own methodology for user testing the CBeebies website. The other
was that to apply child-centred theories to the new methodology may be harder than I had
expected with my colleagues. When we discussed question six, ‘What do you learn when
you go user testing? (Do you learn anything new?)’ we talked about having high and low
expectations of the audience and its capabilities, partly in the context of HumanComputer Interaction and partly in the context of child development. Within HCI we
discussed how the emphasis tends to be from a marketing perspective (what can children
do that will ensure the success of this site and therefore make money for advertisers and
commercial companies). Within child development, the two designers talked mostly
about the mental capabilities of children, referring to the usability document that contains
basic child development milestones (Ward et al, 2004) from birth to six years. As they
did not seem to think beyond that document in our discussion, that informed the structure
and plan of the first group discussion and the expectations I had of my colleagues.
32
I originally planned just one group discussion but we ran out of time to fit in everything I
wanted to discuss so I then scheduled a follow-up. In that first group discussion I decided
to focus on our audience of young children and on how we are successful as a team
making online content for children for one of the most popular UK websites for young
children. I did that to ensure that the overall feeling was positive and also to try to get
them to look at the children who use the website in a different way than they might have
done before. I was therefore sharing for the first time how child-centred theories were
guiding me and could help us in our user testing. When talking to the two designers there
had been a feeling that the three of us could implement changes which the team would
have to go along with. But I was keen to make sure, as with the interviews, that everyone
felt involved and that they were initiating any changes. In that way I was applying my
overall paradigm of feminism as I saw everyone’s perspectives as valid, there was no
‘right’ way to change the user testing. I was also applying standpoint feminism as I
wanted to involve all of the team in the changes, using their experiences of user testing as
a starting point.
In the discussion, after I had asked everyone to think about what is really important about
CBeebies and what is really important about what we do as a team, I divided them into
three smaller groups, chosen by job role and known team dynamics: two groups of three
and one group of four (three people were absent and I did not join any of the smaller
groups). In these smaller groups I asked them to think about ‘what is a child’ and write
down their thoughts on post-it notes of a particular colour. I asked them to particularly
think about real children they knew and about what children can do overall, not just
referring to computer use. I then showed a video of children of five and six-years-old
making fences and chairs out of wood, using adult carpentry tools (Carr, 1998). I also
handed out photocopies of ‘The amusement park for birds’ from Reggio Emilia
(Malaguzzi et al, 1987). I then asked the groups to carry on writing down ‘what is a
child’ and ‘what can a child do’, but on different coloured post-it notes from before.
When I asked each group to feed back to the group as a whole I was interested to see how
much difference there was between the two sets of post-it notes. I also asked them to talk
33
about what they were thinking before and after seeing the video. The final step in the first
group discussion was to write down, in their three small groups, their thoughts about the
children they meet when they go user testing, on a third set of differently coloured post-it
notes. After they fed back those thoughts we had already run over time so I had to
postpone the last two steps I had planned for another week and another group discussion.
The group had already started to talk about the shortcomings of the user-testing process,
once they started thinking about the children they user tested with. I asked them to save
those thoughts for next time.
The second group discussion took place six weeks later, much later than I had planned,
due to illness and the difficulty of getting everyone in one place at one time. I asked them
in the interim to look at the user testing and child development document we used in the
department (Ward et al, 2004), in order to think about what a user-testing methodology
‘ought’ to be but also, more importantly, to refresh their minds about child development
and what children can do. I had had a good response to the Learning Stories video (Carr,
1998) and some of the group were interested in how children could be involved in talking
about and planning their learning activities (even though I had offered the video as an
example of what children can do, rather than as an example of Carr’s ‘four Ds’ of
Learning Stories). Therefore I decided to be more overt than before about the childcentred theories. I put together a document outlining different approaches to user-testing
with children, together with some assessment techniques (target child and Learning
Stories) and a reference to how young children’s thinking and reasoning skills are
sometimes underestimated (see appendix D). I also chose to include ‘the role of the child’
in the user-testing sessions in the group discussion.
In the second group discussion I gave them the list of themes from the interviews (see
appendix E). I again divided the group into three smaller groups. As before, I chose
carefully who was in each group, dividing up different job roles. I asked each group to
come up with solutions to each theme and then share those ideas with the larger group. I
gave out the document I had prepared about different approaches (appendix D) for them
to refer to while coming up with solutions. I also gave them the Reggio Emilia photocopy
34
again (Malaguzzi et al, 1987) and wanted them to keep in mind what a child can do
throughout their discussions. I had wanted to show the video again (Carr, 1998) but
unfortunately did not have it available that day.
When I listened to each group as they discussed their assigned themes and tried to come
up with solutions, I had a general feeling of having heard it all before. I had chosen to
give them the themes from the interviews in order to avoid repetition but discussing it in
groups just meant the same comments and complaints came up. I had asked them to come
up with action points of what we could actually do for each theme and was hoping the
document about different approaches would help them with that. There were some very
good points when they all fed back but overall the document had been ignored and the
solutions were more for general guidance than specific ideas (with some notable
exceptions which I document in the next chapter). I was disappointed as I had hoped for
more specific action points.
I finished the discussion by suggesting we set up a working group of a few of us in order
to actually put their ideas into practice. It had become evident that to keep discussing in a
large group was impractical. I asked for volunteers for the group and four people put
themselves forward that day. The group is now led by me and made up of two designers,
two researchers and one producer from interactive TV (who has not been involved in any
of the discussions so far but needs to establish a way to user test interactive TV content
with children under six). There is also one client-side developer as I requested someone
from the technical team. On the periphery is the senior designer, who will advise about
usability best practice, and another researcher who has expertise in usability for children
with special needs. The other outcome of the group discussion was that we cancelled the
regular user-testing sessions for the time being, or at least over the summer, in order to
provide us with the space to develop a process that works for us and the children.
35
Discussion: Analysis and Evaluation
Research questions
1. What is the web production team’s understanding of the principles and purposes
of user testing?
2. What do the web production team understand about the audience when they go
user testing?
3. How feasible is a more child-centred approach when the web production team go
user testing?
Introduction
Throughout this chapter in which I discuss and interpret the findings from the interviews
and group discussions, I am keeping in mind the metaphor of two threads entwining,
coming undone and then entwining again, perhaps less tightly – imagine it as a double
helix. The two threads represent themes that have run throughout the project as well as
those that inspired it at the beginning. The first is respect for usability (reflecting the
principles and purposes of user testing from the first research question) and the second is
putting the child at the centre (reflecting the child-centred approach of the third research
question). The second research question, to find out how the web production team
understand the audience when they go user testing, is represented by the rungs of the
double helix, while the first and third research questions are the twisted sides. Finding out
how the team understand young children when they go user testing is crucial to joining
up and keeping entwined the two dominant themes of the whole project. At various times
the threads have become separated, harking back to their ostensible original lack of
connection, reflecting how usability is not always compatible with empowering children
and putting them at the centre.
A key question at each stage of the project has been ‘where is the child?’. While that
question reflects both the third research question and the second thread, it is most
relevant when the child is ‘missing’ from thoughts, discussions or theories. If the child
disappears then the second thread, of putting the child at the centre, completely unravels
36
from the first thread and the rungs joining the two together. Disappointingly for me, the
main themes or concepts from the findings of the project are often focused around the
first thread and thus the child has got lost. The child was lost at different times and for
different reasons, but sometimes because of my chosen approach. I therefore look very
closely not only at what I said to the team but also what I did not say. Was I always
giving the team a chance to entwine those two threads together themselves? Equally, I am
aware that what the team have not said is sometimes as important as what they have said.
As discussed in the previous chapter, underlying the whole project is the principle that
theory informs research or practice and research or practice informs theory. As I interpret
the findings of the project, this principle becomes more significant, coming to the fore
more than I had expected. It reflects the double helix metaphor too, in that you cannot
have one thread without the other. Of course, overarching the whole study is a feminist
paradigm. Feminism in terms of prioritising women’s experiences above men’s in society
is not applicable here, instead it is the feminist paradigm of equality of everyone within
society, irrespective of gender, class, age or race. Reflecting standpoint feminism,
everyone’s experiences are valid and there should not be one dominant view. This
reflects the child-centred approach but also acknowledges how the experiences of the
team themselves are just as important as the experiences of the children. Those
experiences of the team have fed into the theory and practice of the end project, showing
the mutual dependence of theory informing practice and practice informing theory.
Denscombe (2003: 272) refers to ‘early coding’ and ‘new insights’ in the process of
analysing qualitative data such as interview transcripts. In my discussion I am aware of
how sometimes the early coding and analysis has needed rethinking in light of new
insights. I have a good ‘audit trail’ (Denscombe, 2003: 274), recording my thoughts and
decisions around the interviews and the two group discussions. That trail and its decisions
are dependent to some extent upon the document I prepared for Liz, the lead designer,
and Debra, the senior designer. The document is the culmination of the initial stage of
coding and analysis of the interviews, summarising the key themes I had found,
organised question by question (see appendix C). New insights around the interviews
37
only came about once I could compare my colleagues’ responses to what was discussed
in the two group discussions. Those insights are less well-coded than the original bolding
and extracting of themes that I did for the interviews. Also, by organising the analysis of
the interviews by each interview question rather than by another method, such as by
interviewee, the three research questions or any other overarching theme, it is not always
clear from the original analysis how the different questions relate to each other and have
connecting themes. There are obviously connecting themes between the answers to the
different interview questions and those themes informed how I structured the group
discussions. Thus some of the initial coding contributed to the evolution of some of the
later insights but also had to be rethought as new themes or connections emerged.
The interviews
When I asked the second, third and ninth questions in the interviews I was mainly trying
to find answers to the first research question. I was trying to improve my understanding
of what user testing meant to the team. I asked:
•
What do you think is the most important part of user testing?
•
What do you see as your role in user testing?
•
How do you use your user-testing findings?
When I analysed the answers to those questions my response was not a child-centred one.
When I summarised the main points for Liz and Debra I judged what the team had said in
light of the methodology they ‘ought’ to be applying. When I had decided to interview
everyone I had done so on the premise that the methodology advocated within the
department for user testing with young children was not always being followed and that
sometimes my colleagues were justified in not following it. However, I did not look at
their answers in those terms. I let the first thread, of respect for usability, dominate my
analysis, rather than acknowledging how the overall analysis needed the two threads
intertwined in order to contribute to how the user testing could be changed. One reason
for this may have been my own confusion about what was a valid methodology for user
testing with children under six. I had obviously been influenced by usability experts such
as Krug (2006), Kuniavsky (2003) and Nielsen (2003), despite the fact that none of them
addresses user testing with young children. Part of my original plan was also to return to
38
the basics of usability and user-centred design with the team and then apply child-centred
theories and practices from that point.
As I show below with examples from the second, third and ninth questions, I kept that
aim too far at the front of my mind once the interviews had been conducted, but once I
gained new insights in later analysis the two threads sometimes remained entwined. The
examples also show how I sometimes missed what my colleagues were really saying by
focusing too much on the first thread in my analysis for Liz and Debra. Moreover,
sometimes I needed to look at answers to other interview questions to find the answer to
one particular interview question. Cecilia herself alerts me to this during her response to
the very first question, ‘Why do you think we do user testing?’:
“I think – no, that’s answering a different question! [I say she can carry on
nevertheless] I think one of the drawbacks of the way we do our user testing right
now is the fact that a lot of it is done when something is finished. So we learn
what we did wrong or, not what we did wrong, but what we could do better for
next time and take that information with us. But I would like to do more user
testing at the halfway stage of any content – and that’s my plan from now on.”
She sets up very early on what her belief is about user testing and that runs throughout
the interview. However, in her later answers she does not repeat her plan to always user
test halfway, presumably as she has already told me about it.
When I summarised the team’s answers to question two, ‘what do you think is the most
important part of user testing’, I said there was a split in opinion between listening or
observing being most important and being prepared or organised. I also acknowledged
there was a significant minority who favoured talking to the children but I wondered if
that minority were the least experienced in user testing. However, evaluating the answers
from the perspective of the second research question, seven or eight of my colleagues
think that the most important part of user testing is paying full attention to the child,
listening to what they say and observing what they do. Indeed, some of them favour
39
talking to the children and asking them what they like, which shows a connection to the
child and putting the child at the very centre.
I grouped the minority who favoured talking to the children with those who had
mentioned guiding the children in the sessions. I was aware that within traditional user
testing the facilitator in no way guides what the user does, returning every ‘what do I do
here’ question with a question of their own: ‘I don’t know, why not have a go and see?’
or ‘What do you think you do?’. Therefore I was alert to any suggestion from the team
that they might be guiding the children and somehow adversely affecting the results we
were getting. In fact, only two people, Eimer a researcher and Aidan a designer,
mentioned guiding the children in their answers to the second question and both of them
made clear that it was of secondary importance:
Eimer: “I think it’s actually getting kids to sit down in front of the PC and seeing
firstly how … their reaction to it. It is generally what they do when they’re in
front of that game or that story. And then secondly, as the adult… just kind of
guiding them through it but I think the initial, what their understanding and
competency is, is very important.”
Aidan: “I always quite like just sitting back and trying not to help kids too much,
and just seeing how they find their way about and if it’s really… if it’s simple to
use, just observe them first of all, maybe give them like a hint to … ‘start there’,
maybe but then just really see how they go from there… just observe and maybe
give guidance after that.”
I then asked him how much the session would be split into in terms of observation and
guidance.
Aidan: “Probably because of our age group I mean… I would think a tiny bit of
guidance to kick things off and then if they get stuck, suggestions, what do you
think you would do here… but leave things quite open.”
40
In my analysis of the second interview question I was partly influenced by the answers
for the next question, ‘what do you see as your role in user testing?’. Eimer and Aidan
again mentioned guiding the children, as did Rachel, the editor of the CBeebies website.
Others talked about “talking them through it”, “directing them” and “dictating the
session”. Some of the team therefore did see their role as being a guide when they were
with the children and I summarised it in that way for Liz and Debra.
The other comment I pull out for analysis of question two is: ‘Also emphasis on
environment and being at ease – no distractions.’ This was mentioned by three of the
respondents. I had been expecting some answers along those lines as we had often
discussed at work how a busy computer class or a cramped nursery were not ideal
environments in which to user test. The comments about environment reflected both how
the team were thinking about the children and how they understood some of the
principles and purposes of user testing. Not all usability experts recommend using a usertesting lab – Nielsen is keen to show how it can be done in a home or office setting with a
few users and at least one observer. However he does stress that in those settings you
must ‘close the door to keep out distractions’ (Nielsen, 2003). Nikki and Helen, both
producers, really focused on how the child should be feeling:
Nikki: “It’s essential to have children in a safe, relaxed environment, to try and
get a good rapport with them to start with, which is often difficult with time
limitations.”
Helen: “The most difficult thing I find is when a child isn’t relaxed – so I suppose
the answer to that would be… making sure the child is relaxed, you know, be
comfortable… putting them at ease, either in an environment they feel
comfortable in, or… you know.”
Their comments show how they are identifying with the children when they go user
testing and thinking about how the sessions affect them.
41
For question three I was concerned that some of the team were confusing being a
facilitator and being an observer, using a comment from one of the producers, Cecilia, as
an example:
Cecilia: “But the observing is not done in a stand-back watching way, it is done
very proactively, which is why it’s good to have two of you, because you both
take notes.”
I then highlighted how only the designers (four of the interviewees) and Rachel, the
editor of the CBeebies website, suggested how editorial and design could take it in turns
to be facilitator and observer. I was concerned that the editorial team were not
understanding the principles and purposes of user testing as they were happy for more
than one person to take notes, for the adults to be proactive and involved with the
children and for the difference between facilitator and observer to become muddled. Yet
those very responses were verbalising the concerns that had prompted me to interview the
team. Their comments confirmed that the methodology we thought we were applying was
not working because of the age of the children we were user testing. The answers to
question three also tell me many things about how the production team understand the
audience of children aged six and under when they go user testing, and thus answers the
second research question.
Looking at the answers to question nine in light of the second and third research
questions gives one crucial insight. The child has been lost by most of the team. I
summed up the majority response to question nine as: ‘Overwhelmingly about feeding
into next or future projects (eight-nine respondents)’. I then expressed surprise at this
response as I had always believed user testing was about the specific site or game being
tested in one particular session, and how the user experience of that particular site or
game can be improved. I was thinking more of the principles of usability and user testing,
not the reality of how the team really do use their findings. While some of my colleagues
may aspire to make changes at the halfway stage of a game’s creation, often there is not
42
enough time within the production process or they do not have the authority or resources
to instigate that change. Liz, Debra and I discussed how the researchers particularly are
not empowered to make changes. However, there was doubt from Liz and Debra in our
discussion that findings from user testing actually did feed into future projects – in their
experience, it usually only happened by chance. The three of us were thinking too much
about how findings ‘ought’ to be documented and interpreted, and how the timing of user
testing in a project is vital, following the user-testing methodologies of Krug (2006),
Kuniavsky (2003) and Nielsen (2003). Along with most of the team we had also lost the
child-centred focus.
A few members of the team are thinking of the child though, and the benefit to the whole
audience of what is gained from a user-testing session. Therefore the rungs of the double
helix are still precariously in place.
Katherine: “But I like to think I would use them [her findings], especially from
the last thing I learnt, that they like to be interactive, they like to hear noises, and
see things react to something they do. Even like a really simple thing like
listening to those animal noises which, you know, is so basic, and seeing the pig
put his face in a load of mud and be covered in mud… they just really seemed to
love that. I think that’s… I like to think that I would use what I’ve learnt.”
While this comment is partly about feeding into future projects, Katherine, a researcher,
is also very much thinking about the children she met at her last user-testing session. She
is telling me how she understands the audience and how she can think of what the child
might want from both the content we produce and the user-testing sessions. She is also
telling me what she learns when she goes user testing, which I had asked in question six.
In this response she gives me more detail than before, as for question six she talked most
about learning how competent a child is and how their age and ability can affect their
response to a game. Helen also talks about the role of the child in her response to
question nine, suggesting an ideal approach would be to have children feeding into a
43
project from the very beginning, telling us what they like about a particular television
programme or character and how we can reflect that on the website.
I was disappointed with the responses from the whole team to questions seven and eight.
The questions were:
•
How do you think the children find user testing?
•
Do the children learn anything new?
Question eight had the shortest set of answers and question seven was not far behind (the
first and seventh questions had about the same length of responses and were the second
shortest set of answers). For question seven, I summarised that most of the team think the
children enjoy user testing with a significant minority emphasising how the children were
sometimes shy and could find it intimidating and tiring. Question eight made them think
most about the child’s role, more than for question seven, but they mostly focused on the
children just finding out that there are new games to play on the CBeebies website and
learning computer skills. I had been hoping for answers that showed how the team put the
child at the centre in user testing. I was thinking mostly of the second thread, putting the
child at the centre, and the second and third research questions, both when composing
and when analysing the answers to these two questions.
If I had asked ‘What do the children learn’, without the emphasis on learning anything
‘new’ I may have got more detailed and in-depth answers. By using the word ‘new’ the
team seemed to think about what new content we had produced recently, as shown by
comments by Sandrine, a researcher, and Rachel, the editor:
Sandrine: “No. Only thing I can think of they learn is when we show them ‘check,
have you played that game before? That’s our new game…’. The only thing they
can learn is what’s available…. broaden a bit their horizons.”
Rachel: “Yeah, learn about new pieces of content, we’re not really there to teach
them, there to observe. They might learn about a new piece of content but they
don’t develop a skill, they don’t develop anything in that respect.”
44
The use of the word ‘learn’ made some of the respondents think about teaching (as with
Rachel’s comment above, three others spoke about teaching and educational value).
While a lot of our content is produced with the six early learning goals in mind (see
QCA/ DfEE, 2000) and thus offers ‘soft learning’ this did not seem to be at the forefront
of any of the interviewees’ minds, though to be fair many made the point that not much
can be learnt in a half hour session. I had not phrased the question well enough to find out
what I actually wanted to know, namely how much the team think of children as active
learners throughout their daily lives.
When composing question eight I had had the different observation and assessment
methods for young children at the back of my mind, particularly Carr’s Learning Stories.
As I had not yet introduced any of the team to her methods and ideas at the time of the
interviews I did not feel it was appropriate to start expounding upon learning dispositions
to them when I asked that question. However, Cecilia’s answers to questions six and
eight show that she sees us, the team, as constantly learning something new, whereas if
user testing is ‘perfect’ then the children won’t necessarily learn anything:
Question six: ‘What do you learn when you go user testing? (Do you learn
anything new?)’
“I think we always learn something new, we really do ... To go in and say ‘I knew
what was going to happen and it was a waste of my time’ you should go and get
another job.”
Question eight: ‘Do the children learn anything new?’
“If our user testing was really, really good and perfect and in the home … in
many ways then they themselves wouldn’t because they may be doing something
that they do a lot or they do fairly frequently, unless we’re testing a game at the
halfway stage.”
45
Cecilia then goes on to say that children do learn things to help them count, encourage
them to clap and play a tune, but she thinks it is a moot point whether they learn that
during our testing or playing the same game after we’ve gone.
In our discussion about the main findings from the interviews, Liz and Debra suggested
that maybe the majority of the team saw themselves in a nurturing role when user testing
and thus made assumptions about how the children are not capable and need help. I partly
agreed that the team take on a nurturing role, as some of them do, and therefore
underestimate the audience, not giving them as much independence as they might like.
The lack of information and insight I had got for questions seven and eight from the team
supported that angle to some extent. I had hoped to discuss with Liz and Debra how
children’s capabilities are more far-reaching than the basic child-development milestones
outlined in our internal document (Ward et al, 2004), as well as looking at the role of
expectation of users within user testing generally. But they focused on the accepted
knowledge we already had and I found it hard to move the discussion beyond that.
The first group discussion
In my plan for the group discussion I wanted to build up to thinking about the children
with whom we go user testing. I wanted the team to focus on the second thread, putting
the child at the centre, while my focus was on the second and third research questions.
Moreover, while I was trying to challenge the belief that user testing is only testing the
website and not the user, putting the child at the centre is actually one of the principles
and purposes of user testing. As user testing is crucial for user-centred design, the first
research question was also present throughout that first group discussion.
I chose not to feed back what had been said in the interviews and instead took them down
a different path for a while to see what the response then was. We first explored the value
and strength of both the CBeebies brand and the CBeebies web production team before
talking about the children whom we meet. This was important because they were all
acknowledging that there were problems with the current user-testing process: I wanted
to reassure them that these were not problems per se with the team and the brand. When I
46
then tried to make sure the team were really thinking about the children they meet when
they go user testing, I was trying to challenge any preconceptions they might have about
how competent young children are. I took that approach because of the disappointing
responses I had received from the team for questions seven and eight and the apparent
lack of understanding from Debra and Liz about the high and low expectations of the
young audience. But of course, thinking about early coding versus new insights
(Denscombe, 2003: 272), the team had given me more than I realised about how they
understand the audience and how feasible it would be to take a more child-centred
approach to user testing. It was just they hadn’t given me all of that information in the
answers to questions seven and eight. I also made the assumption that if Debra and Liz
were unable to look beyond basic child development information then the rest of the team
were not doing so either.
In my original planning for the group discussion I was keen to share with the team the
four Ds of assessment from Carr: Describing, Discussing, Documenting and Deciding
(Carr, 2001: 101). It offered a good process to suit what we do in user testing, particularly
the writing up and following through of the user-testing findings, plus it was designed
originally for assessing young children. I wanted the main focus of the discussion to be
about children – general vignettes of children and specific examples of those with whom
we user test. This seemed important not only for the reasons above about preconceptions
of the audience but also in order to wholly understand Carr’s Learning Stories the team
needed to think from a more child-centred position. So while the discussion was building
up to finding solutions for which the four Ds might be appropriate, I did not want
methods of documenting to dominate the discussion. Thus when I showed part of the
video about Learning Stories I chose the extract carefully. I was using the video as an
example of what young children can do (make gates, saw up wood), not as a lesson in the
four Ds of assessment.
There were two significant findings from the first group discussion. The first was around
the difference between the responses when I asked the team to think about ‘what is a
child’ twice during the group discussion and then about the children they meet when they
47
go user testing. Liz, the lead designer, commented that the second set of post-its, those
written after they had watched the video, were ‘more advanced’ in terms of what children
could do than the first set of post-its. This was certainly true as comments included
‘creating and making things’, ‘decisive’ and ‘capable/ intelligent’. The first set had a
mixture of active and passive comments, and positive and negative impressions of
children. There was mention of them being creative, curious, always learning, unselfconscious, liking to build things. There was also mention of them not being dextrous, of
being cautious, of getting easily frustrated, not understanding things, being self-absorbed.
There was nothing overtly negative in the second set of post-its, although while they were
watching the video itself one comment was ‘scary!’. However, there was an increased
reference to the role of adults in a child’s world. Out of 22 comments, eight were
somehow related to adults. These were those eight responses:
Role-play
Mimic adult behaviour
Inspired by their parents
Social/ cultural perceptions
Social pressure by adults on what kids can/ can’t do
Perception from adults what kids can do
Copying adults
Pick up on adult attitudes
While ‘social/ cultural perceptions’ may not seem to mention adults in a child’s world I
have included it in this list as once everyone had fed back about their second set of postits, the discussion was mainly about what is imposed on a child by an adult and what is
socially acceptable. All three groups mentioned how the children were copying or
mimicking adults and I challenged the group on this as in the video we had not seen any
adults making gates or sawing up wood – it was all being done by the children. They said
they viewed the children as aspiring to be like adults. They then acknowledged how they
had low expectations of children but that that is often to do with society’s expectations
and portrayals of children.
48
When the group told each other about the children they meet when they go user testing
and wrote examples down on a third set of post-its, they told me how they did not refer
back to the other sets of post-its. To them, the third activity was a specific request and
task but the other two activities were not tasks in the same way. This explains to some
extent why most of the comments about the real children they encounter when they go
user testing did not reflect any of their previous thinking. After watching the video they
said children are ‘proud to show you what they’ve done’, but in referring to a user-testing
session they said ‘they like encouragement’. From the video, they commented that
children are ‘not hindered by reality’ and are ‘capable/ intelligent’ but the comments
about user testing emphasised how children need boundaries and have limited
capabilities:
Apprehensive and scared
Not very dextrous
Need structure
Very short attention span
Easily distracted unless they really like what they’re doing
They like encouragement
Need more positive reinforcement
Can get demoralised
Unpredictable
Computer literate yet like security/ praise of easy games (e.g. Teletubbies)
One of the group summed up their response to the first group discussion by saying that
‘the [CBeebies] website breaks cultural perceptions but the user brings you back to
reality’. In the same way, it seemed the video had broken their cultural perceptions but
thinking about real children had brought them back to reality.
I therefore received mixed messages in answer to the second research question, how the
team understand the audience when they go user testing. Before we had talked about the
49
children they meet when they go user testing, they all seemed inspired and motivated by
the video and really enjoyed the discussion about challenging society’s expectations
about children. Once we did talk about the real children they meet, that possibility
became more muted. If they were not able to see the young children they meet as capable
and intelligent then I was not sure they would be able to apply a child-centred approach
to user testing. While they were thinking deeply about the audience and trying to put the
child at the centre their inability to make that leap to empower children meant I was not
convinced that a child-centred approach along the lines of Learning Stories (Carr, 2001),
the Mosaic approach (Clark and Moss, 2001) or even Paley (1986) was feasible.
Moreover, even when they were identifying the abstract child, they still saw that child
from a Piagetian view. They regarded young children as unable to decentre and generally
‘egocentric’, so only thinking of their own point of view. The first set of post-its included
these comments:
‘Selfish (meaning self-absorbed, not learnt to think outside self)’
‘Difficult to see others’ point of view’
Donaldson (1987: 20-23) has shown that this is not true of children of four, five and six
years old, while Astington (1993) asserts that between the ages of two and five, children
begin to have insights into their own mental life and that of others, so they are able to
infer other people's thoughts, feelings and perceptions from words and actions.
The team’s reaction to the video, the second significant finding from the first group
discussion, showed that a new user-testing model was possible, though not necessarily a
child-centred one. This informed my plan for the second group discussion. Even though I
had carefully chosen the extract so that there was very little about the four Ds of
assessment, there was enough within the voice-over about the gate-making activity to get
the team interested in it. They were all intrigued by the talking and planning and wanted
to see the talking in process (the ‘Discussing’) with the children. In the first group
discussion we ran out of time to reach the point at which we were trying to find solutions
to the problems of user testing. I had hoped that by the time we reached that point that I
50
could bring in the four Ds as an option to help us, or even find that the group were
moving towards them without realising (talking about the importance of documenting
what we find, interpreting it as a group, making decisions etc.).
The second group discussion
As documented in the methodology chapter, there was a much longer time gap between
the first and second group discussions than I had originally planned, giving me time to
reflect on the first discussion and make changes accordingly. I tried to harness the team’s
interest in the Learning Stories assessment process, particularly about how the
practitioners involved the children in their own assessment, to feed into thinking about a
new model of user testing. The positive reaction to the video (Carr, 1998) seemed to
show that the team needed new ideas to inspire them for finding a new way to approach
user-testing. Those ideas could enable them to put the child back at the centre and enable
me to establish the feasibility of a more child-centred approach when we go user testing.
However, both those ideas (see appendix D) and my plan for the discussion show how the
double helix of respect for usability, putting the child at the centre and how the team
understand the young audience is delicately entwined.
The sheet I prepared for the second group discussion (appendix D) pulled out key ideas
from the literature I had been reading. The ideas were not all explicitly about putting the
child at the centre but they were acknowledging the role of the child in different ways.
Most importantly though, the ideas were all new to the team and offering new ways of
looking at the user testing we do: I wanted to enable them to find alternative ways of
doing user testing. I was trying to combine the two threads so that even though I saw the
second thread, putting the child at the centre, at the fore, I was still maintaining respect
for usability (the first thread). I did this in two ways. The first was to give neutral
explanations of each idea or alternative method, so that I was not judging what we
currently did. The second was to show how the Learning Stories could be adapted for our
needs around user testing, not just suggesting we lift the assessment method wholesale.
Thus while one of the key parts of Carr’s Learning Stories is the involvement of the
51
child, in my explanation of it I highlighted the documenting side and how we could adapt
it for our needs.
By offering the sheet of ideas I was trying to follow the principle that theory informs
research or practice and research or practice informs theory. I wanted to see if the
theories I had found useful could inform the actual practice of user testing with the
CBeebies team. But even though I tried to portray the ideas neutrally I was still
unconsciously applying my own world view by which ideas I chose to include and how I
described them. Indeed, by hoping that others’ theories and research could inform our
practice and research I was already applying my own particular paradigm to the
discussion, my belief that no researcher can participate in society without adhering to a
particular paradigm or world view, consciously or not.
While I had intended the second group discussion to be a chance for the team to entwine
the two threads together themselves, combining respect for usability with putting the
child at the centre, the structure and focus of the group discussion eventually denied the
presence of the child. As I said in my introduction to this chapter, if the child disappears
then the second thread, of putting the child at the centre, completely unravels from the
first thread and the rungs joining the two together in the double helix. I asked the group to
keep in mind what young children can do throughout the brainstorming and discussion
and therefore handed out the Reggio Emilia photocopy again (Malaguzzi et al, 1987),
though I was unable to re-show the video of children making gates. I also tried to focus
on the second thread and the third research question by recapping what we had discussed
in the first group discussion. However from the point at which I introduced the seven
main themes from the interviews, the discussion focused on the first research question
and the first thread. Even though I added the role of the child to those seven themes (see
appendix E) I was still asking predominantly for solutions to problems that were about
the basic principles and purposes of user testing.
When I introduced the eight themes for discussion I asked the group if they had anything
to add. Sandrine suggested stress on the child during the sessions. I said I felt that fell
52
into a few of the areas – role of the child, environment, timing, our role and
communication – and so I thought it would be covered when we looked at each of those
areas. In retrospect, I wonder if it should have been included as part of the role of the
child. Sandrine was obviously thinking solely about the child and was therefore putting
the child at the centre, something I was trying to achieve through the inclusion of the role
of the child within the discussion.
In the write-up for each group (see appendix F) there is no evidence that anyone had
incorporated any of the ideas I had given them, nor the thoughts from the previous group
discussion, the Learning Stories video (Carr, 1998) or the Reggio Emilia photocopy
(Malaguzzi et al, 1987). This may have been because I did not really give the team a
chance to read over the sheet in detail before they divided into their three groups to
discuss the eight themes, though they could refer to the sheet throughout the discussion.
On that basis, I felt I had misjudged how ready the team were to use theory to inform
practice. However, as I show below, using examples from Group C’s discussion, that
misjudgement arose for two reasons. The first was emphasising respect for usability
above putting the child at the centre in the plan of the group discussion and the second
was focusing on initial analysis of the second group discussion above new insights. This
second reason reflects the analysis of the interviews – sometimes I needed to look in the
less obvious places for answers to the second and third research questions.
Group C discussed ‘your role’ (i.e. the team’s) and the role of the child, as well as
documenting of results. The theme of ‘your role’ produced more answers to research
question three and the second thread than the theme of the role of the child. For example,
points around ‘your role’ included:
We have to ask lots of questions as they don’t tend to give up how they are
feeling or what the experience is like for them freely.
Also need to encourage them so they can have the best experience possible.
Children cannot always read instructions or hear audio on laptop.
53
These points show how the team would apply a child-centred approach to user testing,
and not necessarily follow the ‘traditional’ principles and purposes of user testing. In
contrast, the points for the role of the child are not as active and do not put the child at the
centre in the same way. They go some way to answering the second research question
about how the team understand the audience when they go user testing:
To help us find out how well a piece of content had been made or how it could be
improved.
To express their honest opinion about how they feel when using the piece of
content.
They are the truth and evidence of what we do.
They are the window into our audience, their lives, what they like and dislike.
They need to have some degree of computer literacy to get the most out of the
content.
I had included the role of the child in order to make sure a child-centred approach could
be considered within the discussion, alongside some of the child-centred ideas and
theories I had offered (appendix D). While I had then focused more on the principles and
purposes of user testing through the seven themes from the interviews, Group C had
thought about the child, just again, as with the interviews, not where I was necessarily
looking for the child.
Argyris and Schön define two forms of theory of practice – espoused theories and
theories in use (Moon, 1999: 40). Espoused theories are part of the public perception of a
particular profession, seen to formally guide action, while theories in use are the patterns
learned and developed in day-to-day work. This division reflects the user-testing
methodologies that the web production team apply to their work. The espoused theories
are both the methodology of adult usability testing and the methodology advocated in the
internal document about user testing with children (Ward et al, 2004). The theories in use
are the actual methods the team use when in a user-testing session with children under
six, adapting the methodology that we have within the department to suit each child or
54
each session. In their three small groups the team talked about their own experiences of
user testing, using those experiences to come up with solutions. When they shared their
experiences of user testing, whether with each other or with me in the interviews, they
were showing how their own theories in use guide them day-to-day and inform their
practice. This is similar to the principle that theory informs research or practice and vice
versa. The practice of user testing (the experiences of the team) was informing the theory
(the adapted methodology the team were trying to apply to each session without
necessarily realising, their theories in use). The team’s theories in use were more
informed by child-centred practices and theories than it initially appeared. By applying
those theories in use the team showed how they were ready to use theory to inform
practice, they were just not necessarily ready to apply new theories to inform their
practice.
While I had felt at the end of the second group discussion that the second thread had
become completely unravelled from the first thread and wanted to go some way to
repairing that, the actual methods that the team applied and their theories in use showed
that the two threads were still loosely entwined. I suggested a working group be set up in
order to take the action points from the second group discussion forward and to make the
process of applying change to the user testing more manageable. The specific ideas that
had come out of the discussion needed a process behind them (how to build relationships
with schools and nurseries; how to engage teachers and practitioners; create example
scripts for user-testing sessions; decide the generic form for documenting the findings)
and specific individuals to work on them. I also wondered if with fewer people it would
be easier to investigate the team’s attitude to a more child-centred approach, as well as
applying new theories to their practice. I was keen to move on from what I saw as a
‘talking shop’ of repeated ideas and complaints and actually do something that would
make a difference. The team had shown me how their experiences informed their theories
in use when they went user testing. I wanted to build upon that, particularly within the
paradigm of standpoint feminism, where everyone’s experiences are valid and there is not
one dominant view.
55
Conclusion
I started this project over a year ago, in summer 2005. My emphasis then was on trying
out different observational methods for young children in the user-testing sessions. The
focus of the thesis and the methodology applied has obviously changed since then but at
the heart still stands the original motivation, to find a different and more effective way to
conduct user-testing sessions with children under six. In my original plan I had hoped to
have a new way of user testing up and running by the spring of 2006; I now hope to have
a new way of user testing in place in the autumn of 2006. The user-testing working group
I set up after the second group discussion in May is currently on track to deliver a new
approach by October. The amount of time it has taken to reach different points
throughout the project has always been longer than I anticipated. Therefore one learning
outcome for me has been realising that taking an inclusive approach to everyone in a
team requires a flexible schedule and attitude. I have particularly reflected on that
inclusive approach at the end of the project after comparing my approach to the work
done by Ana, the Masters student on work experience in my department at the BBC.
When I saw the apparent ease with which Ana could produce one piece of finite research,
I was tempted to be more didactic than I have been. However, I have made sure I have
kept true to my original paradigms and methodologies. Those paradigms include my
belief that no researcher can participate in society without adhering to a particular
paradigm or world view, consciously or not; that theory informs practice and practice
informs theory; and that everyone’s experiences are valid and should be valued, through
the feminist paradigm of equality between men and women. In Dewey’s definition of
reflective action, the first necessary attitude is open-mindedness: ‘an active desire to
listen to more sides than one, to give full attention to alternative possibilities, and to
recognize the possibility of error even in beliefs that are dearest to us’ (Zeichner and
Liston, 1996: 10). I have sometimes wondered if I have been too open-minded, too
willing to listen to everyone and consider alternatives, not actually making any decisions
or producing concrete outcomes. Equally, as shown with my analysis of what I did and
what I found out, I have at times been less open-minded than is ideal for reflective action.
56
I have perhaps been more of a sophisticated believer than a critical believer (Wright Mills
in Zeichner and Liston, 1996: 10), wanting to find out opposing points of view in order to
refute them rather than engaging with those opposing points of view to show up the flaws
in my own belief system.
Another aspect to consider is my own status within the team. I have given reasons above
for not taking a didactic approach and for being led by particular paradigms. However,
even if I had decided to wholeheartedly embrace either didacticism or action research for
the whole project I would have had to gain approval from my managers and other more
senior people within the department to take either approach. As the majority of the people
I interviewed are junior to me or are my peers I have felt able to lead them in the group
discussions and in the working group. Working in a small team has also contributed to
the democratisation of the process – we have a strong tradition of our research feeding
into each others’ ideas and evolving into processes. But, ultimately, we are part of a
bigger department and the final decision on exactly how CBeebies user testing might
change needs to be considered within that. Though I hope that the new process from the
working group can be rigorously applied and tested to prove what works best, and then
disseminated to the team and department, we have already identified that we have at least
five project stakeholders (teams and individuals) and three sponsors (see appendix G,
project definition document for the working group). The department is also undergoing
management and structural changes as I write so those stakeholders and sponsors are
changing, as is the make-up of the web production team itself. Therefore there are a
number of unknown factors contributing to the risks of the project, though my status and
role will remain the same (leading the working group but not having ultimate say on the
new process).
To conclude now with a return to the three research questions:
1. What is the web production team’s understanding of the principles and purposes
of user testing?
2. What do the web production team understand about the audience when they go
user testing?
57
3. How feasible is a more child-centred approach when the web production team go
user testing?
The three research questions are obviously intrinsically linked, whether through the first
two questions building up to the third or through the metaphor of the double helix, where
the second research question is the rungs of the double helix and the first and third
research questions are the twisted sides. In the previous chapter I have evaluated my
methods and my findings in detail, showing when and how the two threads of the double
helix sometimes became separated. I have also shown when and how the two threads
have been entwined or at least loosely twisted together. However, I have sometimes
focused on the second and third research questions to the disadvantage of the first
research question. I have most often focused on it when evaluating my own analysis of
the team’s responses in the interviews or the group discussions, becoming frustrated
when the first thread of respect for usability seems to take over the research, analysis or
discussion.
The findings from the second group discussion showed most clearly what the web
production team understand about the principles and purposes of user testing (appendix
F). The seven themes from the interviews shaped the format of the second group
discussion and those seven themes were decided upon after discussion with Liz and
Debra. They reflected the frustrations the team had expressed to me – they were
effectively seven different areas that everyone saw as problematic across user testing and
which needed solutions. While I had some concerns that the focus upon them meant that
we lost the focus on the child in the second group discussion, their origination from the
individual interviews shows how the team understood the criteria for a user testing
process that adheres to ‘traditional’ adult user testing as advocated by Krug (2006),
Kuniavsky (2003) and Nielsen (2003).
Throughout my evaluation the second research question has been crucial to keeping the
two dominant themes of the whole project entwined. The web production team
understand different things about the audience when they go user testing, shown from the
58
varied responses in the interviews and the contrasting comments in the first group
discussion. When in the first group discussion they listed capabilities of real children that
were hindered by society, they showed that they did not necessarily believe the
empowering descriptions they had used for the second set of post-its could exist in
reality. As one of the team actually said, ‘the user brings you back to reality’. Indeed,
while they may have suggested ‘more advanced’ capabilities after watching the Learning
Stories video (Carr, 1998), ultimately they understand the audience from a Piagetian child
development perspective. In the second group discussion this did not change – although
there was evidence that a more child-centred approach in user testing was feasible, this
was based around willingness to listen to the children (‘To express their honest opinion
about how they feel when using the piece of content’; ‘They are the truth and evidence of
what we do’) rather than re-evaluating their capabilities.
The third research question has been the main focus and point of the project. The
feasibility of a more child-centred approach has decreased and increased throughout the
project, depending on what information the team gave me about what they understood of
the audience and how they understood some of the child-centred ideas I was presenting to
them. Through the basic principle of user-centred design the web production team do
think of the child user throughout their working lives. In order for the user testing itself to
be more child-centred and therefore empowering the children within the sessions, the
team do need to rethink some of their conceptions of real children. Overall, the team are
often thinking of the child when they go user testing and thinking from the child’s
perspective. However, because of the constraints of what user testing is ultimately for and
what it is trying to achieve (good usability and well-designed websites for all users),
CBeebies user testing cannot completely embrace such child-centred approaches as Carr
(2001) and Clark and Moss (2001). Therefore a more child-centred approach is definitely
feasible but not to the exclusion of other approaches.
I have set up the working group in direct response to the third research question, giving a
small group of people a chance to combine all their ideas and knowledge about user
testing and the audience to produce a new way of doing user testing for CBeebies
59
Interactive. The documented audit trail I have of the interviews, the group discussions
and now the working group meetings will feed into that new process. Moreover, by
learning to value the methodology that the team were already applying, their ‘theories in
use’ (Argyris and Schön in Moon, 1999: 40), the new process can be a continuation of
what some of them were already doing. If the working group succeeds in its objectives
then the two threads will remain entwined, ensuring that new team members can
confidently climb the rungs of the double helix, wholly understanding the audience and
delivering an enjoyable user-testing experience for both the CBeebies team and the
children with whom we user test.
60
Bibliography
Abbott, L. and Gillen, J. (eds.) (1997) Educare for the Under Threes - identifying need
and opportunity, Manchester: Manchester Metropolitan University
Ackers, J. (1997) ‘The ways in which children’s learning and development are assessed
and the extent to which this influences provision’ in Abbott, L. and J. Gillen,
(eds.) Educare for the Under Threes - identifying need and opportunity,
Manchester: Manchester Metropolitan University
Astington, J. W. (1993) The Child’s Discovery of the Mind, Cambridge: Harvard
University Press
Barendregt, W. (2006) ‘Evaluating fun and usability in computer games with children’
PhD thesis, University of Eindhoven
Brooker, L. and J. Siraj-Blatchford (2002) ‘ “Click on Miaow!”: how children of three
and four years experience the nursery computer’ in Contemporary Issues in Early
Childhood Vol. 3, no. 2 pp.251-272
Carr, M. (1998) Assessing Children’s Learning in Early Childhood Settings: What to
assess? – Why assess? - How to assess? (video) Wellington: NZCER
Carr, M. (2001) Assessment in Early Childhood Settings: Learning Stories, London: Paul
Chapman
Clark, A. and P. Moss (2001) Listening to Young Children: The Mosaic approach,
London: National Children’s Bureau and Joseph Rowntree Foundation
Dahlberg, G., P. Moss and A. Pence (1999) Beyond Quality in Early Childhood
Education and Care: Postmodern Perspectives, London: RoutledgeFalmer
Denscombe, M. (2003) The Good Research Guide for small-scale social research
projects, Maidenhead: Open University Press and McGraw-Hill Education
(second edition)
Donaldson, M. (1987) Children’s Minds, London: Fontana (first published 1978)
Druin, A. (2002) ‘The role of children in the design of new technology’ in Behaviour and
Information Technology Vol. 21, no.1, pp.1-25
Drummond, M. J. (2003) Assessing Children’s Learning, London: David Fulton (first
published 1993)
Drummond, M. J. and C. Nutbrown (1996) ‘Observing and assessing young children’ in
61
Pugh, G., (ed.). Contemporary Issues in the Early Years, London: Paul Chapman
and the National Children’s Bureau (second edition)
Elfer, P. and D. Wedge (1996) ‘Defining, measuring and supporting quality’ in Pugh, G.,
(ed.) Contemporary Issues in the Early Years, London: Paul Chapman and the
National Children’s Bureau (second edition)
Griffin, B. (1997) ‘The contribution made by educarers to the social construction of
identity’ in Abbott, L. and J. Gillen, (eds.) Educare for the Under Threes –
identifying need and opportunity, Manchester: Manchester Metropolitan
University
Hanna, L., K. Risden and K. Alexander (1997) ‘Guidelines for Usability Testing with
Children’ in interactions, Vol. 4 issue 5, pp9-14
Available online http://www.microsoft.com/usability/UEPostings/p9-hanna.pdf
Hanna, L., K. Risden,, M. Czerwinski, and K. Alexander (1998) ‘The role of usability
research in designing children’s computer products’ in Druin, A. (ed.) The Design
of Children’s Technology, San Francisco: Morgan Kaufmann
Available online http://research.microsoft.com/users/marycz/druin98.htm
Harms, T. and R. M. Clifford (1980) Early Childhood Environment Rating Scale, New
York: Teachers College Press
ISO (1998) Ergonomic requirements for office work with visual display terminals
(VDTs) - Part 11: Guidance on usability (ISO 9241-11)
ISO (2001) Software engineering - Product quality - Part 1: Quality model (ISO 9126-1)
Kuniavsky, M. (2003) Observing the User Experience: A Practitioner’s Guide to User
Research, San Francisco: Morgan Kaufmann
Krug, S. (2006) Don’t Make Me Think! A Common Sense Approach to Web Usability,
Berkeley: New Riders (second edition)
Malaguzzi, L. et al. (1987) The Hundred Languages of Children, Reggio Emilia,
Italy: Department of Education.
Markopoulos, P. and M. Bekker (2003a) Editorial to ‘Interaction design and children’ in
Interacting with Computers, Vol. 15, issue 2, pp141-149
Markopoulos, P. and M. Bekker (2003b) ‘On the assessment of usability testing methods
for children’ in Interacting with Computers, Vol. 15, issue 2, pp.227-243
Markopoulos, P., W. Barendregt, M. Brus and I. Plakman (2005) ‘The Parent
62
Evaluator Method’ Proceedings Workshop on Children Computer Interaction:
Methodological Research, p. 36-38, at Interact 2005, Rome, September 2005.
May, T. (1997) Social Research: issues, methods and process, Buckingham: Open
University (second edition)
McPake, J., S. Downey, L. Plowman, D. Sime and C. Stephen (under review, 2006).
‘Already at a disadvantage? The impact of socio-economic disadvantage on
young children’s developing ICT competences in the home’ in Contemporary
Issues in Early Childhood
Moon, J. (1999) Reflection in Learning and Professional Development, London: Kogan
Page
Nielsen, J. (2002) ‘Kids’ Corner: Website Usability for Children’
http://www.useit.com/alertbox/20020414.html
Nielsen, J. (2003) ‘Usability 101: Introduction to Usability’
http://www.useit.com/alertbox/20030825.html
Nielsen, J. (2004) ‘Risks of Quantitative Studies’
http://www.useit.com/alertbox/20040301.html
Nielsen, J. and S. Gilutz (2002) ‘Usability of Websites for Children: 70 Design
Guidelines’, Fremont: Nielsen Norman Group
Available online http://www.nngroup.com/reports/kids
O’Connell Davidson, J. and D. Layder (1994) Methods, Sex and Madness, London:
Routledge
Paley, V.G. (1986) ‘On listening to what the children say’ in Harvard Educational
Review Vol. 56 no. 2 pp.122-131
Pascal, C. and T. Bertram, (eds.) (1997) Effective Early Learning: Case Studies in
Improvement, London: Paul Chapman
Plowman, L. and C. Stephen (2003) ‘A “benign addition”? Research on ICT and preschool children’ in Journal of Computer Assisted Learning 19 pp.149-164
Pring, R. (2004) Philosophy of Educational Research, London: Continuum (first
published 2000)
QCA/ DfEE (2000) ‘Curriculum Guidance for the Foundation Stage’ London: QCA/
DfEE
Rustin, M. (1989) ‘Observing Infants: Reflections on Methods’ in Miller L., M. Rustin,
63
M. Rustin and J. Shuttleworth, (eds.) Closely Observed Infants, London:
Duckworth
Selleck, D. and S. Griffin (1996) ‘Quality for the under threes’ in Pugh, G., (ed.).
Contemporary Issues in the Early Years, London: Paul Chapman and the National
Children’s Bureau (second edition)
Sim, G., S. MacFarlane and J. Read (2006) ‘All work and no play: measuring fun,
usability and learning in software for children’ in Computers & Education 46
pp.235-248
Siraj-Blatchford, J. and D. Whitebread (2003) Supporting Information and
Communications Technology in the Early Years, Maidenhead: Open University
Press and McGraw-Hill Education
Sylva, K., C. Roy and M. Painter (1980) Childwatching at Playgroup and Nursery
School, London: Grant McIntyre
Tizard, B. and M. Hughes (1986) Young Children Learning: Talking and Thinking at
Home and School, London: Fontana (first published 1984)
UsabilityNet, (2006) http://www.usabilitynet.org/tools/r_international.htm#9241-11
Vernon, J. and C. Smith (1994) Day Nurseries at a Crossroads, London: National
Children’s Bureau
Ward, L., D. Howell, L. Wilton, G. Winter, O. Dickinson and S. Gauld (2004) Usability:
CBeebies and CBBC Interactive, Unpublished internal document, BBC
Zeichner, K. and D. Liston (1996) Reflective Teaching: An Introduction, New Jersey:
Lawrence Erlbaum Associates
64
Appendix A
Questions about user testing for the team
1. Why do you think we do user testing?
2. What do you think is the most important part of user testing?
3. What do you see as your role in user testing?
4. Think back to the last user-testing session you went on. What did you think
worked well?
5. What didn’t work well?
6. What do you learn when you go user testing? (Do you learn anything new?)
7. How do you think the children find user testing?
8. Do the children learn anything new?
9. How do you use your user-testing findings?
10. If we didn’t go user testing, how do you think it would affect your work, the
CBeebies site and our output?
11. What one thing would make user testing better, for you?
12. Is there anything you would like to add that we haven’t discussed about user
testing?
65
Appendix B
Interviews with the team about user testing –
notes from their answers for question one
Why do you think we do user testing?
Emma
Research into how children use games, how the games are working, whether the
children enjoy them. Also look at gender and navigation.
Jo
We can’t assume that we know everything and just because we think something’s good
they’ll think it’s good.
You might think it’s easy to remember what you were like as a child and what you liked
as a child and what would appeal but if you have to do it to make sure that the people
who it’s intended for get out of it what they’re supposed to get out of it. Make it
worthwhile.
Katherine
Make sure content is accessible for children and find out what they like and can and
can’t do and might need help with.
Also to know that what we produce is doing what it’s supposed to do.
Christina
To get an accurate view on how correct our content is and aimed at the right level.
What works well and doesn’t work so well. Trying to find gaps for the content we have.
Nikki
To ensure that we can provide most accurate content for our users, our audience.
Accurate: suitable for age range, capabilities, style of learning, enjoyment, requirements.
Eimer
To gauge how children cope with new content that we’re producing. How young
children use the computer from various different backgrounds and if they’re able to
do so on their on own or with parent or carer alongside them.
And also to see what’s popular – what works and what doesn’t. What sort of things? If a
child struggles with something or lacks interest or really captures their imagination, so
pull something more into frame than something else.
Helen
Particularly for our age group essential to see if they can actually do it as can’t make
assumptions.
Also has to be quite instinctive as they can’t read instructions. Games and content need
to be instinctive and can’t tell until you test.
Last three years – skills increased enormously, bigger change than any other age group.
66
First started user testing reception class can’t do it, now most can.
Sandrine
To see if we’re going the right way when we create either new sites or new pieces of
content. Does it make sense for the child, do we make sense for the child.
Navigation particularly – arrows at bottom, does that make sense for them.
To keep in touch with a child’s ability in terms of technical skills – mouse, keyboard.
And very important with children with disabilities as not expert.
Cecilia
In order to see how well our content is received by the audience. To pick up any small
technical points such as ease of playing a game, using the mouse, scrolling, looking
around.
One of the drawbacks right now is that a lot of it is done when something is finished. We
learn what we did wrong or what we could do better next time and take that information
with us.
Like to do at halfway stage of content – and that’s my plan from now on.
At the moment apply to next project and not to current project.
Other facts: very fact of being with audience on regular basis puts you on their
wavelength and also with adults who work with children (i.e. in school or nursery).
Really important to watch how they behave in nursery before even doing user
testing.
Your own children – can be spot on and can be a drawback.
User testing always always valid.
Regularly so always in contact with audience.
Aidan
I haven’t done a lot. But from my point of view from when have done it, usability of a
game, children can use it, age it’s aimed at, functionality, if it works, easy to get
story across, that it works basically.
User testing done – couple of away days, couple of time at Canberra.
Alan
Basically to keep in touch with our audience. Downstairs don’t do any user testing
from what I’ve seen – packed schedules. Now doing a bit more.
New from word go built it into our system.
Really important to keep in constant contact with the audience and see how they respond
to things. Pick up a lot. We’re so competent with the computer we assume that a lot of
kids are and they’re not e.g. kids in White City area don’t have computers at home.
Quite shocking some of their computer skills. Have kids in from some of our producers.
Important to do it across the country – Newcastle, Bristol, Wales. Welsh sites and
content.
What we do, though we don’t get a lot done, is a start and that’s important.
Rachel (recording not great)
67
Age group we work with. How they think and feel, and opinions. Remove ourselves from
cocoon. How they operate and how innovative they can be as well – and what opinions
they bring up. Involved at every stage of developing a game or application or tool.
Liz
Ascertain that everything we’re building and designing is from a user-centred
approach. In terms of checking we’re always putting the user first and thinking about
their needs and what they’re (we’re?) trying to achieve, trying to do when they come to
our websites.
Trusting our own experience or judgement about websites isn’t necessarily a good idea
(though have a lot of internal knowledge) because we’re ‘super users’ and our audience
are normal, every day people. Situation quite different to sitting in the office – they’re at
home, with children, children in school or upstairs in bedroom using it depending on age
group so in order to get a ‘real life’ experience then need to do user testing.
Usability point of view – obviously strive to do our best, that everything works, but only
true test is to put it in front of people. See if they can do it, can find things, without that
prior knowledge we all have.
Debra
Us personally? – questions designed for team? Rephrase to ‘why do you think CBeebies
and/ or CBBC do user testing’
Mainly to see if what we are doing… difference between should and why people think.
In theory we do it to make sure that what we’re doing is what the users want and
that they can actually navigate – two key points.
Should and..? Whether results are applied in a consistent, analytical way (from the user
testing).
In theory to check what users want and can navigate around it.
Sue:
Q: Why do you have the focus groups?
To check that the content we’re putting out and the programmes are working really,
that the children enjoy them, giving them something they want. Also seeing where
the gaps are, seeing what else they’re interested in, that gap with the older ones, what
they’re interested in.
68
Appendix C
Questions about user testing for the team
Brief write-up for Liz and Debra on Thursday
1. Why do you think we do user testing?
Mostly agree with each other. Focus on child alone on PC or using it with an adult –
brought up by K and Ei. i.e. ‘find out what they might need help with’ – is this a valid
part of our user testing? What role do we play as adult partner? Is there any point doing
testing with children on their own?
New: important to watch children away from PC as well.
2. What do you think is the most important part of user testing?
Split between listening/ observing and being prepared and organised.
Also significant minority (mostly newer people, less experienced?) who favour talking to
the children, asking them what they like, guiding them to some extent.
Also emphasis on environment and being at ease – no distractions.
3. What do you see as your role in user testing?
Split again? Emphasis from editorial team on being proactive – being there as a guide,
talking to them, ensuring they have a good experience. Confusing being facilitator and
being observer? (i.e. Cecilia and ‘Observing not done in a stand back watching way but
done in a proactively way so good to have two of you.’)
Only designers and Rachel suggesting having similar roles, taking it in turns, gathering
data, feeding back etc.
4. Think back to the last user-testing session you went on. What did you think
worked well?
5. What didn’t work well?
Interesting focus on content – i.e. answering question ‘what game did the children
respond well to’. But then backed up by ‘if they respond well you feel much better…’
Equipment – technical bit etc.
Environment – distractions, like home, at ease
Ratios of adults and children
Planning – only coming from design team and Rachel (consult experts)
New:
Talk to the adults before we see the children
Need children with certain competence with the mouse/ computer – agree?
69
Communication with teachers/ staff
6. What do you learn when you go user testing? (Do you learn anything new?)
No real themes?
Significant number of answers about what you learn about running a session/ how to user
test/ interacting with children
A few about needing a purpose to learn anything worthwhile? (Sandrine and Aidan)
How to improve it for next time/ feed into future projects
High and low expectations of the audience and its capabilities – look at for child-centred
approach. But also see if any literature on having certain expectations of users as part of
usability?
Not all agree always learn something new.
7. How do you think the children find user testing?
Most think they enjoy it with significant minority emphasising being shy/ it being
intimidating and tiring.
Is this something we ever ask about adults and user testing?
8. Do the children learn anything new?
Shortest set of answers.
This made them think most about the child’s role, more than the previous question.
General agreement that only learning things around what’s out there to do and play, plus
some computer skills but not really.
Maybe as said new they couldn’t focus on the learning process – otherwise an indictment
of our content?
9. How do you use your user-testing findings?
Overwhelmingly about feeding into next or future projects (eight - nine respondents).
Surprised by this as not how I see it but am I in minority? Views on what user testing is
‘for’? i.e. isn’t it about that piece of content and how I can improve that particular
experience?
Scheduling and time – how reports are written.
Not always make immediate changes but if important do try.
10. If we didn’t go user testing, how do you think it would affect your work, the
CBeebies site and our output?
Slight split. Most agree it’s vital but Sandrine, Liz and Debra emphasise how it needs to
be done properly to be worthwhile (and Aidan?).
Thoughts on Sandrine’s comment about evolution of user testing – OK as it is at mo for
‘new researchers’ but not for ‘advanced researchers’. Think is getting at point that it is
70
mostly audience connection and learning basics of our age groups’ competence. Thoughts
from L and D?
11. What one thing would make user testing better, for you?
Time and scheduling
Planning and preparation
Following through – completion
New idea:
User testing not for me, it’s for the kids. Therefore make it more free and easy. Again,
who is it for?
12. Is there anything you would like to add that we haven’t discussed about user
testing?
Timing, scheduling, planning
New ideas:
New independent projects stemming from findings (K, plus H and talk to children about
ideas at planning stage)
Discussion (part of follow through?)
Evolution (see above)
Communication between editorial and design
Putting it actually into practice
71
Appendix D
Ideas that may be useful while discussing what and how you want to change about user
testing (or for later once have action points)
Margaret Carr and Learning Stories (video just seen as well)
Inspired by Te Whairiki (pronounced Te Fareeki) curriculum in New Zealand
4 D’s:
Describing – defining learning but also can be observations of an activity
Discussing – talking with staff (us and early years’ staff, perhaps?), the children and
families (i.e. parents in user testing?) in order to develop, confirm or question an
interpretation
Documenting – recording an assessment but equally, could we use it to record an
observation or user testing experience? You can use text, pictures, photos or a
combination
Deciding – what to do next? Informal or formal planning
‘Learning Story’ observation sheets are used as part of Documenting – something to be
adapted for us? Uses five ‘learning dispositions’ from NZ:
• Finding something of interest
• Being involved
• Engaging with challenge and persisting with difficulty
• Communicating with others/ expressing a point of view
• Taking responsibility
Usability diaries (Kuniavsky and other usability experts)
Ask users to keep a dairy over several days, a week or a month of how they use a website
or the internet
Parent as evaluator (Markopoulos et al)
Tried out with four to six year olds. Parents learn first what to do with a game/ website/
CD-Rom etc. and then follow instructions to become the facilitator with the child. Can be
combined with usability diaries and can be filmed. It takes a few days or weeks but may
be more ‘natural’ to a child – i.e. more as they would normally use something at home
with a parent.
Picture cards (Barendregt)
PECS images used for children aged five to seven to communicate how hard or easy they
found tasks on the computer, how much they liked something etc. Early days and only
one piece of research but quite useful to help children think aloud and articulate what
they thought.
Using webcams of children’s faces together with stationary camera of whole room?
Eye-tracking software/ video
72
Target child observation method (Sylva, Roy and Painter)
Record key behaviour every 30 seconds for 20 minutes in different columns, with no
interpretation:
Activity
|
Language
|
Task |
Social
Adapt for our needs while user testing?
Children’s ability to think and reason at home and school (Donaldson and Tizard and
Hughes)
Donaldson (1978): ‘by the time they come to school, all normal children can show skills
as thinkers and language-users to a degree which must compel our respect’. Donaldson
then finds children have to find new ways of thinking and reasoning once they are at
school.
Tizard and Hughes (1984): four-year-olds learn most and think most at home, away from
school and nursery (irrespective of parental job/ background).
Fun sorter (Sim, MacFarlane and Read) with seven and eight year olds
73
Appendix E
Group A
•
•
Time, timing, scheduling (incl. what user testing is for – when and for how long)
Communication with children and staff/ parents/ carers
Group B
•
•
•
Planning and preparation
Equipment/ technical glitches
Environment
Group C
•
•
•
Sharing findings, documenting, writing up (incl. communication in team and
between editorial and design)
Your role – in parental role, as facilitator, as observer
The role of the child
74
Appendix F
Group discussion part two.
Group A
•
Time, timing, scheduling (incl. what user testing is for – when and for how long)
Problems
Unplanned
Teacher not engaged with process – engaging teachers in objectives is hard
Using only London schools
Budget to allocate from outset
Anarchy
10-11.15 – break time
Fit too much into testing
Solutions
Value in unprompted testing
Aims of testing established
Planning in part way through a game or application
Testing early in design process then test again
Plan reasonable testing milestones into project plan
Build in to budget funds to go around country to focus test
Two types of testing – getting to know audience and product testing
Build relationship with schools
•
Communication with children and staff/ parents/ carers
Very good briefing of objectives of testing
External ‘best’ contacts
Usability – external agencies good for the range of children you use
Database of best teachers
Have sufficient people to help
Time to build rapport
Research – visit/ watch them play
Warm up sessions with schools
Home visits in their own environment
Take care about ‘feeding’ them (or teaching? Can’t read writing, sorry!)
iCan school good example of communication
Prepare your technology
Brief teacher about type of child we need
Presents and fun important
Other points
75
Scotland can do testing in Scotland for us
Timing and role – exhausting as in active role?
Engage the child – stickers and presents
Two types of testing – could combine?
What about communication in the session itself? i.e. working out what tasks we do etc.
Need to prepare more than we do currently
Need time to write up
Maintaining all results, updating
Make changes for external and internal projects, part of bigger, fewer, better
Share results in team meeting (NB therefore design and tech need to be invited)
Group B
•
•
•
Planning and preparation
Equipment/ technical glitches
Environment
Planning and Preparation
Set objectives: what do we want to get out of this?
Break it down into realistic chunks – organise more than one session if necessary.
Have a script or planned format to refer during session.
Build prototypes/scenarios (print/flash).
Do a practice run through.
Ensure you have a good mix and balance of skills in the team.
Plan what roles each person will take on (note taking/facilitator etc.).
Equipment & Tech glitches
Choose locations that have computers – this will relax the kids as they will be familiar
with using them. Move away from going to places locally just for convenience.
Phone the school or nursery first and ensure they have correct software i.e. Flash. Ask
them to double check if uncertain.
Liaise with tech team to ensure someone ‘back at base’ is on hand in case of any live tech
problems.
Environments
Real spaces, nursery and home, with familiar , real set-up
Reduce numbers and distractions: 30 kids is not productive!
Be more prescriptive about what we need, take control of the sessions i.e. we would like
6 children, 3 boys, 3 girls, for 45mins, need 3 computers with flash 5 etc.…
Create a friendly environment, let the children know how helpful they are being and that
there is no right or wrong.
76
Group C
Sharing findings, documenting, writing up (incl. communication in team and between
editorial and design)
Your role – in parental role, as facilitator, as observer
The role of the child
1. Our Role
We have to gain users’ trust but we also have to get them to do what we want them to.
We often have to act as negotiators between groups of children who may be taking turns
or playing together.
We have to ask lots of questions as they don’t tend to give up how they are feeling or
what the experience is like for them freely.
Also need to encourage them so they can have the best experience possible.
We are the outsiders in a different environment so can take us time to settle into our
surroundings – get organised, make sure everything is working.
Don’t always see children in a natural place where they are uninfluenced from their peers
or the familiar people around them.
We are the active observers. Children cannot always read instructions or hear audio on
laptop.
It’s important we also have a non-active observer, facilitator, who can take notes while
the other tester supports the children.
We must make sure we test with all types of children, from different abilities,
backgrounds, gender, ages and from different parts of the country. The advantage of
testing with older children is that they can often be observed better as they don’t need as
much help.
The broader the testee, the more we know how the content is used, how well it is used
and how it could be improved.
At the same time it is important to encourage the child by not giving someone of their
ability a piece of content that will be too hard for them to test or understand – although
this is hard to know in advance if the child is a new tester.
2. Role of Child
To help us find out how well a piece of content had been made or how it could be
improved.
To express their honest opinion about how they feel when using the piece of content.
They are the truth and evidence of what we do.
Must test similar content produced by our competitors at the same time as testing our
own content, to see how it compares.
They are the window into our audience, their lives, what they like and dislike.
They can help us improve our content for all.
They need to have some degree of computer literacy to get the most out of the content.
77
3. Documentation
We could film testing sessions to keep a record – not always practical though.
We could create a generic form that gets filled in during or upon return of testing – this
could include all relevant details about the child – age, background, ability, gender, what
was tested and how they got on.
This would help in subsequent sessions – if the same or a similar game was being tested
we would know what things we may need to look out for.
It would also help in the production of new games – we could see what elements children
found difficult or what they really enjoyed.
78
Appendix G
Project Definition Document
Project name: CBeebies User Testing
Deliverables: a new user testing process in place for the CBeebies interactive services
with supporting documentation on the Children's Interactive intranet (ciki)
Objectives:
Primary:
To look at the current problems of CBeebies user testing and find a way to fix them
To improve the user testing experience for both the CBeebies team and the children with
whom we go user testing
Secondary:
To evaluate whether child-centred theories can form the basis for CBeebies user testing
To evaluate how useful ‘traditional’ usability theories are for children under six – can we
maintain or improve upon user-centred design for our audience?
To maintain the best online content for children under six in the UK through effective
user testing
Scope:
1. Inside: user testing of the CBeebies website (broken down into testing of Flash
games and navigation), CBeebies interactive TV, CBeebies WAP service and
PDA, Video on demand and IPTV. User testing with children and parents together
and children on their own.
2. Outside: user testing of CBBC services, user testing of CBeebies Grown-ups site
and WAP service.
Assumptions:
User testing of CBeebies needs improving - agreed by design team, editor and editorial
team
CBBC has a user testing process in place that works for older children (6+)
Risks:
Ideas to improve the process do not actually work for the audience
Not enough time is set aside to concentrate on the project by managers
Design, editorial and tech do not agree how to set up new process
Project manager has particular aims that the group do not buy into
External influences from marketing, scheduling within BBC Children's
Take too academic approach, need to be really hands-on
Think only within BBC - need to learn from competitors and toy manufacturers
Key stakeholders:
Head of Children's Interactive and On Demand
Editor of CBeebies Interactive
CBeebies interactive team
79
Design team lead and design team
Interactive TV team
Tech team
Project Sponsors:
Design Lead, BBC Children’s Interactive (Liz)
Editor of CBeebies website (Rachel)
Senior Producer, Children’s iTV (Anthony)
Project Manager:
Olivia Dickinson
Team Members:
Leo Sen
Joanne Patterson
Aidan O'Brien
Katherine Monk
Helen Stephens
Dave Howell
Debra Trayner - usability expert
Sandrine Dordé - special needs expert
Key Milestones:
Identify key roles of each team member
Identify key problems of user testing – already done in group discussion, need to go over
Identify solutions to those problems – again, already done to some extent in group
discussion
Set up panel of ten children with different abilities, background and ages - need to
identify what personas we want
Make contact with competitors and toy manufacturers?
Try out solutions - practical, hands-on - go user testing
Approve or discard solutions
Agree new process
Try out new process
Finalise process and share with stakeholders
Write guide to new process
Apply new process, with stakeholders
Budget, resources and constraints:
Currently no budget but two options:
Is money available from Marketing, Communication and Presentation?
could use some of redesign user testing budget?
Limited time - need to negotiate time with line managers?
Project approval date: October 2006? (do want to set a date as point of project is short
term to put new process in place)
80