Ceneval + TAO, improving the developers and test takers plataform

TAO Days 2013
Bern, Switzerland
October 1, 2013
Ceneval + TAO, improving the developers
and test takers plataform
César Chávez Álvarez
Francisco M. Otero Flores
Contents
• 
Who are we?
• 
Ceneval’s platform
• 
Ceneval + TAO : Improving our content
2 Who are we?
3 • 
Ceneval is the National Center for the Evaluation of Higher Education.
• 
Is a Mexican non-profit institution founded in 1994 by the National
Association of Universities and Higher Education Institutions (ANUIES).
• 
An institution committed to promote the quality of education by
developing valid, reliable, and relevant assessments.
• 
Ceneval´s activities are focused on the evaluation of knowledge and
skills acquired in the teaching-learning process at different levels in
formal and informal education.
• 
Ceneval has a 20-member Board of Trustees integrated by Principals/
Presidents of public and private universities, Presidents of the two
Universities Associations and the Heads of Professional or Trade
Associations.
4 Who we are in numbers
Tests in operation
>250
Experts’ groups in charge of instruments
>90
Members of experts’ groups
>900
Teachers writing & reviewing items
Thousands
Test takers per year
>3 millions
Test administrations per year
>2,100
High schools & universities served each year
>1,500
Online test administrations events per year
>170
Staff working full time
540
Government financial support
$0.00
5 Ceneval’s tests
Ceneval’s tests can be grouped into four categories:
• 
Admissions or entrance tests (EXANI) for high school,
undergraduate programs and postgraduate programs (basically
masters degree programs)
• 
Higher Education Exit Assessments (EGEL) for undergraduate
programs (40)
• 
Generic knowledge and skills assessments
• 
On demand assessments
6 Ceneval’s plataform
7 • 
Since 2002, Ceneval started a project which main purpose was to
create a set of tools to automatize most of the tasks required by
the two productions lines of the Centre: items development and
test administration.
• 
The first developments were the website, the secure items banks
and the online registry. Over the years, new needs required new
solutions.
• 
Nowadays, Ceneval’s platform consists of seven modules: portal,
registry, testing, developers banks, security banks and BPM for
test management.
8 Ceneval’s platform modules (1)
• 
Ceneval’s website, which broadcast its services and
most important activities.
• 
Based on BPM technology, this module integrates the
working flow to administer more than 2,100 test
administrations events requested by our customers
each year.
• 
This module allows test takers to register to a specific
test session.
9 Ceneval’s platform modules (2)
• 
This module is open to all item developers and is
designed to capture and validate technical and
grammatically new items that potentially will be used in
a test form.
Annually this module adds more than 40,000 new items, which are validated by
almost 1,000 different experts
• 
This module can only be accessed within Ceneval’s
facilities, it stores items suitable to be pilot tested in a
form, builds new forms, and stores psychometrical data.
Currently, this module stores more than 300,000 items in 256 test pool
• 
Integrated by a group of sub-modules, this module
allows the candidates to take a test.
10 Platform’s connectivity
- Test Information
- Student guides
- Test takers scores
Students
Administrations
requests
Items contents
& logs
Assessments’
forms
Psychometrical data & planning
11 12 Developers banks
(e-BRAE)
• 
This module is open to all item developers and is designed to capture and validate,
technical and grammatically new items that could potentially be used.
• 
In order to automate the item’s development process, the module defines:
ü  A working flow for item development attached to international standards
ü  Seven types of users profiles that may access one or more banks
ü  An item’s log which tracks its changes along the process, and also provides valuable
evidence at every stage
ü  A system’s log which tracks every user activity
ü  More than 30 different functions to administer the items and its content
ü  17 different types of reports
ü  An automated connectivity with the security banks module when the item has
reached the end of the working flow
ü  The items contents and users passwords are encrypted at the database
• 
Annually this module adds more than 40,000 items developed and validated by almost
1,000 users in almost 256 different banks
13 Item’s life-cycle
Expert content create a new item +
Experts content review +
Grammar & Spelling review Pilot test +
Multiple choice
14 15 Item’s life-cycle
Pilot test +
Psychometrical analysis +
Ready for an Assessment form Multiple choice
16 Security banks
(e-BASE)
• 
This module can only be accessed within Ceneval’s facilities, and its purposes
are:
ü Store the items cataloged at e-BRAE as suitable to be piloted in an
assessment form
ü Build assessments forms according to the technical specifications, that may
be administered by paper and pencil or online
ü Store the psychometric data obtained during one or more tests
administrations
• 
This module was the first developed, and shares many features with the eBRAE, although most of the items administration functionality is restricted.
• 
Nowadays, the module stores more than 300,000 items for 256 banks
17 Increasing tendencies and new requirements
• 
In the last 10 years, the Mexican government encouraged public and private educational
institutions to use widely the ICT as the mexican society transforms quickly into an
information society. This government’s initiative has:
•  Enable the construction of more computer classrooms
•  The development or implementation of computer software as a supporting tool for
teachers
•  Increase the demand on bigger and low cost of broadband services
• 
The tendency at the urban institutions is to spread the use of computers for testing
purposes, in order to have quick feedback and “low” the administration costs.
• 
Ceneval´s commitment to the educational institutions is to develop new test incorporating
innovative computer-based items intended to measure knowledge, skills and abilities that
are difficult to evaluate using only multiple-choice items.
• 
The new schema is intended to represent authentic, real word tasks in a competence
evaluation framework.
18 Meeting our clients demands
• 
During the last quarter of 2011, Ceneval had a group of systems aligned with the
organization requirements; allowing the development of new item, forms and administer test
based only on single multiple choice items.
• 
At that moment, we were standing in a crossroad:
•  Continue with our system and develop new functionality
•  Acquire licensing for software that meet our requirements
• 
In our research, we meet a lot of providers and have a lot of remote meetings, with more or
less the same results:
•  Our software must be installed in your servers (so my items will no longer live with me?)
• 
• 
We don’t meet your requirements, but you can change your processes to meet my
system’s? (yes, indeed, when do we start)
We charge you for a user account (>2,000 item developers accounts + thousands of
reviewers) = $$$$$
19 Ceneval + TAO: Improving content
20 • 
Almost accepting that we might grant the demands of the new software providers, we
found TAO; a socratic midpoint to our crossroads because:
•  Can live in our data centers
•  We might have as many accounts as we need
•  As a GNU, we can have access to the source code
• 
In November 2012, TAO visited us in our headquarters at Mexico City with three
objectives:
•  Made a feature presentation of TAO to the organization
•  Showed TAO the items’ workflow: processes, tasks, reports
•  How TAO capabilities and Ceneval’s needs could be meet?
DON’T THROW AWAY ANYTHING,
USE WHAT YOU NEED!
21 Platform’s connectivity
- Test Information
- Student guides
- Test takers scores
December 2013
Items contents
& logs
May 2013
+ TAO :: Items
editor (QTI)
May 2013
+ TAO :: Items
editor (QTI)
+ XHTML Content
Assessments’
forms
Psychometrical data & planning
22 Item’s life-cycle
+
Expert content create a new item +
Experts content review +
Grammar & Spelling review Pilot test +
Multiple choice
Innovative
Items
(QTI)
1) 
2) 
3) 
4) 
5) 
6) 
7) 
8) 
9) 
TEXT
Choice
Inline choice
Associate
Order
Match
Gap match
Text entry
Extended text
Hottext
GRAPHIC
10)  Hotspot
11)  Graphic order
12)  Graphic associate
13)  Graphic gap
14)  Select point
15)  Slider
16)  Upload documents
23 Item’s life-cycle
+
Pilot test +
Ready for an Assessment form Psychometrical analysis +
Multiple choice
Innovative
Items
(QTI)
1) 
2) 
3) 
4) 
5) 
6) 
7) 
8) 
9) 
TEXT
Choice
Inline choice
Associate
Order
Match
Gap match
Text entry
Extended text
Hottext
GRAPHIC
10)  Hotspot
11)  Graphic order
12)  Graphic associate
13)  Graphic gap
14)  Select point
15)  Slider
16)  Upload documents
24 User interface
25 Some other adjustments…
In order to implement this new assessment process, it is necessary:
• 
• 
Ceneval’s system (easy part)
• 
• 
• 
Update manuals and training courses
Adjust the item’s life-cycle
Spread the benefits of this type of evaluation among users
Item development process (difficult part)
• 
• 
• 
Update the policies and standards through the results of the pilot study to take
place next January
Include new roles and tasks
Retrain item developers and item reviewers
26 27