Score Interpretation Guide

i
NOCTI Score Reports
NOCTI’s assessment package includes a comprehensive score report. Our standard score report
summarizes each participant’s written and/or performance assessment results. Normative data is
provided at the nation, state, and site (local) level. Criterion-referenced cut scores are provided
for the Job Ready written and performance assessments. Educators can use this data to recognize
student achievement, promote continuous program and instructional improvement, and meet
regulatory requirements. Score reports are available in functional formats, including downloads
that allow schools to easily make use of the data.
An overview and sample reports are provided in this publication. We hope that you will find this
information useful as you implement a new assessment program or continue to manage an
existing program.
In addition to this guide, existing clients can access numerous resources available at the Client
Services Center, and our customer service specialists are available by phone or online to clarify
any additional reporting questions you may have.
Our goal is to provide useful scoring information for
students, teachers, administrators, and parents.
Table of Contents
Score Interpretation Guide Overview ..............................................................................................1
Individual Score Report ...................................................................................................................2
Figure 1 – Sample Individual Score Report .........................................................................2
Group Score Report .........................................................................................................................3
Figure 2 – Sample Group Score Report ...............................................................................3
Analysis of Scores............................................................................................................................4
Figure 3 – Sample Analysis of Scores .................................................................................4
Specialized and Customized Reports ...............................................................................................5
Figure 4 – Sample from a Task Link Report .......................................................................5
Figure 5 – Sample Demographic Analysis Report ..............................................................6
Figure 6 – Sample Comparison to Cut Score Report...........................................................6
Figure 7 – Sample Year-to-Year Comparison Report .........................................................7
Figure 8 – Sample Integrated Academics Report ................................................................8
General Information about NOCTI Score Reports ..........................................................................9
Scoring Terminology .....................................................................................................................10
Score Interpretation Guide Overview
The next several pages present examples of the score reports included in NOCTI’s standard
assessment package. A percentage score for each duty area and/or performance job as well as
total scores are provided for every participant. Comparative data is also provided for the group,
site, state, and nation. Below is an explanation of each data category.
Pre-test/Post-test – A total post-test score is reported for each participant as well as for each
duty area and/or performance job. Pre-test scores are reported, when applicable, and the
percentage change from pre-test to post-test administration is included at the duty level and for
the total score.
% of Change – Compares the pre-test score to the post-test score. This is the percentage of gain
or loss for the named participant. If a pre-test was not administered, the pre-test and percent of
change columns will both be blank.
Group – Group scores reflect the average of the entire testing group for the test code and test
date referenced.
Site Cumulative – The site cumulative data is the average of all participants who have taken an
assessment at a particular location since the assessment was released for use and includes the
current testing group.
State – State data represents the average score for all participants in a particular state who have
taken an assessment since it was released for use.*
Nation – This data refers to the average score for all participants who took this assessment
anywhere in the nation.*
*NOCTI computes state and national normative data in January and July so these scores
represent the average of all participants who have taken this assessment prior to January or July
of the current year. For example, score reports generated and viewed in May of the year will
reflect computed averages from January and will remain the same until the averages are
computed in July.
1
Individual Score Report
Score reports are provided for each participant (Figure 1) and include scores for each duty
area/performance job as well as a total score. Participants that complete both a pre- and post-test
will receive pre- and post-test score information along with the percentage of change. Pre/post
comparative data is a powerful tool for teachers and school administrators to assist in identifying
and correcting the factors that impact student learning and performance. This report can be
shared with participants and their parents and can also be included in a student portfolio.
Sample Individual Score Report – Figure 1
2
Group Score Report
The group score report (Figure 2) is a summary of the assessment results for all participants
within the particular test group. Participants are listed alphabetically with individual scores
displayed for each duty area/performance job and a total score. Group averages can be used to
identify strength and gap areas in the program.
Sample Group Score Report – Figure 2
3
Analysis of Scores
The analysis of scores (Figure 3) provides useful comparative data that can be helpful in making
inferences about the observed assessment results. Group, site, state, and national averages are
displayed for both total assessment and each duty area/performance job. State and national
averages are updated in January and July each year. Group and site averages are updated with
every assessment administration.
Sample Analysis of Scores – Figure 3
Criterion-Referenced Cut Score
The criterion-referenced cut score was established with a national group of subject matter
experts. The criterion-referenced cut score sets a point on the test at which a test taker could be
considered a "minimally competent candidate", and/or one who is sufficiently competent for a
beginning position in the chosen field.
The criterion-referenced cut scores are included on the standard score reports for Job Ready and
Pathway tests. These cut scores are presented as an alternative for client use, but as always,
NOCTI does NOT mandate the use of any particular cut scores. Clients may still set their own
cut scores or use a normative method as needed/desired. In addition, some states/schools have
established cut scores in place. Those should be used unless otherwise instructed.
4
Specialized and Customized Reports
NOCTI offers both specialized and customized reports that provide additional or more detailed
information. Examples of the types of specialized and customized reports that can be created are
listed on the following pages. Costs of the reports vary based on the type of information
requested.
Task Link Report – Provides in-depth information on an individual test taker’s performance on
the various tasks within a duty area. By utilizing a Task Link Report (Figure 4), teachers can
easily identify strengths and areas for improvement by task for the individual test takers. Task
Link Reports are available for all Job Ready assessments.
Sample from a Task Link Report – Figure 4
5
Demographic Analysis Report (Figure 5) - Prepared based on NOCTI’s demographic
questions which include gender, ethnic category, age, and grade point average. Additional
demographic questions can be added to meet the needs of a particular school such as
designations for students involved in Tech Prep, articulation, dual credit, Career and Technical
Student Organizations (CTSOs) and work-based learning options. These reports can also
incorporate comparison to cut score by demographic group if desired.
Sample Demographic Analysis Report – Figure 5
Comparison to Cut Score Report (Figure 6) - Provides the percentage of test taker scores
meeting the cut score if a cut score has been determined for an assessment. The data can be
displayed by the test title at a specific school, district, or state level. This report is customized to
specifically meet the needs of the school, district, or state.
Sample Comparison to Cut Score Report – Figure 6
6
Year-to-Year Comparison Report (Figure 7) - Being able to measure year-to-year gains is an
important piece of an assessment program. Below is an example of a two-year comparison by
test title and school. State and national averages are also provided for comparative purposes.
Introductory text, tables and charts can be designed around the specific needs of the testing
program.
NOCTI works with a number of states that request year-end reporting. Reports are modified
annually to include the data considered most important and include comparisons of instructional
programs, test titles, regions, educational reference groups, and demographic designations.
Sample Year-to-Year Comparison Report – Figure 7
7
Integrated Academics Report (Figure 8) - The Integrated Academics Report shows when
students respond correctly to items that are linked to academic standards. This information
provides clues to develop strategies for formative assistance to students. It is intended as a guide
to point out areas of overlap so that concepts can be reinforced and as a guide for upcoming
instruction.
The example below shows the student performance on the ten items in the Carpentry assessment
which have been identified as addressing the Math standard referenced. The number of items
answered correctly is displayed, as well as the percentage correct of those possible, and the
average for the testing group. One conclusion based on this example could be that more
emphasis may need to be placed on Geometry concepts.
Sample Integrated Academics Report- Figure 8
Site: Sample Site
Site Code: XXXX
8
General Information about NOCTI Score Reports
Score reports become available approximately two weeks from the time online tests are released
to NOCTI or answer sheets are received at the NOCTI headquarters. Score reports remain in the
Client Services Center for one year from the test date. Saving score reports to your local drive
ensures that the data will be available after the year has lapsed.
NOCTI score reports are designed to provide valuable data. Each score report has both a “View”
option and an “Excel” option. The “View” option provides a pdf document that can be printed or
saved to your local drive or other media. Selecting the “Excel” option opens the score report in a
spreadsheet format, which allows for manipulation of data as needed.
9
Scoring Terminology
This glossary provides definitions of terms used in this text. For many of the terms, multiple
definitions can be found in the literature. Please note that technical usage may differ from
common usage.
Accountability - Reporting of student, program, or institutional data for the purpose of program
improvement.
Assessment Blueprint - Assessment blueprints provide a list of the competencies for each
assessment with a summary of the skill areas covered, sample questions from the written test, a
sample job from the performance test, and a percentage breakdown of the test content. The
assessment blueprints are available for download at www.nocti.org.
Assessment (Summative) - An assessment measure of achievement at the end of an
instructional unit, course of study, or program.
Authentic (Performance) Assessment - Assessment technique that involves the gathering of
data through systematic observation of a behavior, process, or product and the evaluation of
those based on a clearly articulated set of performance criteria, which serves as the basis for
evaluation judgments.
Cognitive (Multiple Choice) Assessment - The process of systematically gathering test scores
and related data in order to make judgments about an individual’s ability to perform various
mental activities involved in the processing, acquisition, retention, conceptualization, and
organization of sensory, perceptual, verbal, spatial, and psychomotor information. Also
“written” test.
Competency - Demonstration of the ability to perform a specific task or achieve specified
criteria.
Competency Listing - (Also critical competencies, core competencies, standards, duties, and
tasks.) Terms used interchangeably on NOCTI web site, referring to the broad categories and
particular skills covered on a specific assessment.
Criteria - The set of standards used to evaluate or rate an individual’s performance of a process
or the resulting product. With respect to NOCTI performance assessments, criteria are provided
with defined levels of expected performance.
Criterion-Referenced Assessment - An assessment that measures student progress toward
specific curriculum goals or standards. Scores are reported as comparisons to predefined
acceptable levels of performance rather than through comparisons to other students.
10
Scoring Terminology (Continued)
Cut Score - An established score which an individual needs to achieve on an assessment to
demonstrate minimal competency.
Demographics - Characteristics of human populations (e.g., gender, ethnicity, etc.).
Duty - This is an arbitrary division of a job (or occupational area) consisting of related tasks that
are grouped for descriptive purposes. Duties are usually stated as general areas of responsibility.
Item - (Also “question”) a term used to refer to a question on a multiple choice assessment or a
job and task in a performance assessment.
“N” - On NOCTI assessment reports, stands for “number”, and equals the number of
participants included in each of the groups. For example on Figure 3, for the group, N=10, and
for the Site Cumulative, N=38.
Performance (Psychomotor) Assessment - Assessment technique that involves the gathering of
data through systematic observation of a behavior, process, or product and the evaluation of
those based on a clearly articulated set of performance criteria, which serves as the basis for
evaluation judgments. Spatial and psychomotor skills are evaluated as they pertain to the
criteria. Also called “hands-on” or “practical” assessment.
Post-Test - An assessment measurement administered after program completion, usually for the
purpose of documenting attainment of program-related objectives or comparing the score to an
earlier pre-test measure on the same content.
Pre-Test - An assessment measurement administered prior to course initiation, usually for the
purpose of identifying existing skills and/or knowledge or for comparing post-test measurement
of the same content.
Standard Deviation - The square root of the deviance, used as a measure of dispersion or spread
among a group of scores. Represents the number of points an individual scored above or below
the national average.
Standard Error of Measurement - An estimate of the amount of variation to be expected in a
particular score on a particular test. This statistic provides a range of scores within which a test
subject's true score is likely to fall.
Task - A meaningful unit of work activity generally performed within a limited period of time.
A task must be logical and necessary to achieving a single objective or output.
Task Link - Links the individual’s performance to the various tasks within a competency that is
used to evaluate specific strengths and weaknesses.
11