Exam Breaches - American Association of Veterinary State Boards

AAVSB 2008 Annual Meeting & Conference
Exam Breaches: Impact
on Boards and Candidates
Newport Beach, CA
September 20, 2008
Jennifer Ancona Semko
Partner, Baker & McKenzie
Baker & McKenzie International is a Swiss Verein with member law firms around the world. In accordance with the common terminology used in
professional service organizations, reference to a “partner” means a person who is a partner, or equivalent, in such a law firm. Similarly, reference
to an “office” means an office of any such law firm.
Exam Breaches: A Problem for Everyone
• Test Developers and
Administrators
• Licensing Boards
• Candidates
• Vendors
g
• Universities/Programs
• Test Prep Companies
•
•
•
•
•
•
•
Licensed Professionals
General Public
Innocent Bystanders
Immigration Authorities
Governments
Employers
Recruiters
Sometimes cheating is obvious . . .
. . .or is it?
But anytime someone knowingly obtains
an unfair
f i advantage
d t
...
. . . it’s unethical.
Recent Notable Security Breaches
• Licensing Exams: NABP suspends NAPLEX in August 2007
after Univ. of Georgia professor accused of distributing
“
“recalled”
ll d” test
t t questions
ti
• Certifications: Operator of Oregon-based web site buys
answers to Microsoft,
i
f Cisco,
i
Oracle
l andd other
h IT certification
ifi i
exams from Pakistan; makes $700,000+ in about 9 months
before apprehended by FBI
• Admissions tests: Ohio web site operator makes $317,000
selling copies of the GMAT before being sued in 2007
Recent Notable Security Breaches
(cont d)
(cont’d)
• Occupational Certifications: Proctor takes $500 bribe for
advance copy of National Comm’n for the Certification of
C
Crane
O
Operators
t exam (2005)
• Military: Investigation by The Boston Globe in June 2007
reveals that hundreds of thousands of Army soldiers have
downloaded answers to military skills exams
• Security: Private security guards at North Carolina nuclear
plant admit that their employer routinely gave them answers
to state certification exams; San Diego city employees
accusedd off using
i study
t d guides
id with
ith exam answers to
t cheat
h t on
DHS exam on disaster response
Security Breach Case Studies:
• Physical Therapy exam: Overseas use of
recalled test items,
items criminal prosecution
“recalled”
• American Dental Association: Score
invalidation leads to candidate lawsuits
• GMAT: Multi-million dollar judgment
against “braindump”
braindump site and strong message
to candidates
FSBPT’s Experience in the Philippines
• “Recalled” items used
by Manila test prep
companies
• Criminal
C i i l proceedings
di
initiated
• Exam
E
results
lt
invalidated, licenses
denied or revoked
Discovery of the Problem
• Fall 2005: Postings on Physical Therapy Forum web site
identifyy St. Louis Review Center (“SLRC”)
(
) and its apparent
pp
use of recalled test items
– “Apparently, what they do is this: they compile all questions they can
remember after the test”
• FSBPT retains private investigative firm, which infiltrates
SLRC and RCAMP courses in Manila
– Customers receive comprehensive compilation of test questions
recalled by past test takers
• Philippines simultaneously experiences its own
domestic cheating scandal . . .
– IInitiate
i i criminal
i i l case against
i 17 individuals
i di id l andd 3 review
i
centers linked to leak of test questions; President suspends
thousands of nursing licenses
– Catholic bishop publicly renounces “culture of cheating”
in Philippines
Sample Recalled Items vs. Real Items
A patient with chronic obstructive pulmonary
disease is seen at home for a follow-up
physical therapy visit. His wife informs
the physical therapist that the patient has
been very confused and
sleepy the last several days. She has also
noticed that his breathing has been rapid
and shallow, and that his lips have been
slightly
li h l blue.
bl The
Th patient
i is
i MOST likely
lik l
experiencing which of the
following conditions?
• 1. Metabolic acidosis
• 2.
2 Respiratory acidosis
• 3. Metabolic alkalosis
• 4. Respiratory alkalosis
Sample 2
To successfully perform the test demonstrated in the photograph, the physical
therapist should FIRST ensure adequate performance of which of the
following muscles?
1. Supraspinatus
2 S
2.
Subscapularis
b
l i
3. Posterior deltoid
4. Upper trapezius
Use of Recalled Items . . .
• Is it CHEATING?
• Is it ILLEGAL?
• Is it a PROBLEM?
Is it cheating?
Is it illegal?
• U.S. copyright laws protect the creative expression
found in test questions and multiple choice answers
• All NPTE items are registered with the U.S. copyright office
• A recent federal court ruling against a bar exam prep
company makes clear that:
– Even multiple-choice items testing factual issues can be copyrighted
– Copycat items do not have to be identical to be infringing
– It is dangerous to brag that you prepare students with test items similar
to the real thing
. . . and sharing test items breaches a
candidates’ contract with the FSBPT
Candidates know they shouldn’t do it . . .
• “Warning: Never dare to attempt to bring the
recalled questions in Guam (even in your checked
i luggage
in
l
or your carry on bag).
b ) Immigration
I
i ti
officers are aware that you will be taking the
exam, and theyy sometimes ask about how yyou
prepare for the NPTE!”
Is it a problem? Yes, for obvious reasons.
Cost to the FSBPT, NPTE and Profession
•
Several hundred item bank items
deemed too compromised to be
used
d again
i
•
Estimated cost of replacing these
items is in the hundreds of
thousands
•
Cost to the integrity of the exam
and the profession not
quantifiable
What can be done about it?
Criminal Proceedings in Manila: Impact on
T t Prep
Test
P
Companies
C
i
• National Bureau of
Investigation infiltrates
SLRC and confirms
FSBPT’s allegations
ll
i
• Judge grants search
warrants authorizing raids
Results of the Raid
•
Agents seize: documents, books,
enrollment forms, receipts, computer
CPU, flash drive, test pprepp materials
•
Instructor/owner tells NBI agents he
does not use recalled questions
•
Seized materials include additional
recalled items not previously
obtained during private investigation
•
Receipts identify more than 1,500
individuals who have taken the
course
Judge Says NPTE Items Copyrightable
• February 20, 2007: One defendant moves to quash the search
warrants, arguing NPTE questions are not entitled to
copyright
i ht protection
t ti
– After multiple hearings and extensive briefing, the Philippine court in
mid-July rejects this argument
One defendant has requested settlement.
The remaining defendants only recently
appeared to defend themselves . . .
Word spreads regarding the raid
• Numerous internet bulletin
boards and anonymous
submission to FSBPT
applaud the effort to combat
the use of recalled items
• Philippines media publicize
the story
Simultaneous Political Efforts
• Meetings with Department of
Commerce and U.S. Trade
Representative officials
• Meetings with Minister of
Philippines Embassy in
W hi t
Washington
• Philippines officials attend
FSBPT annual meeting
• FSBPT attorneys meet with
government officials in
Philippines
Impact on Candidates
• Guam test site closed in October 2006
• Score reports of all Philippines-educated candidates
temporarily held for about one month pending Board
investigation and Caveon analysis
• Compromised items removed March 1, 2007: pass rate for
inferred SLRC participants plummeted from approximately
77% to 20%
CAVEON Forensic Analysis
• Conducted on 23,512 test results
from March 2005 through June 5,
2007
• Results suggest strongly that
candidates with access to recalled
i
items
received
i d unfair
f i advantage
d
• “Filter” set to identify
performances whose likelihood of
occurring by chance was 1 in
10,000 or smaller
Three Statistical indices
•
Extreme Difference (Gain Scores): Compared how candidates performed
on compromised items vs. noncompromised items. Substantially higher
scores on compromised items suggest prior exposure and unfair advantage.
advantage
•
Extreme Similarity (Collusion): Identified candidates whose responses are
similar to at least one other test taker, suggesting possibility of prior
k
knowledge
l d off test
t t content.
t t
•
Extreme Agreement (Exposure Likelihood): Computed the odds that each
candidate is an SLRC or tainted review course graduate based on pattern
of responses. Looked for agreement with the atypical response pattern
observed for SLRC participants, as compared to the pattern for non-SLRC
participants.
Individuals Identified
• When the 3 analyses are applied so as to
identify scores with a 1 in 10,000
10 000 (or less)
chance of occurring by chance, there were 20
individuals who appear aberrant under all 3
statistical measures
• All 20 of these individuals are Philippineseducated
Individuals Identified
• The odds of appearing on all 3 lists vary by
i di id l but
individual,
b t are att least
l t 1 in
i 1 trillion
t illi
• For example, the odds of one particular
pp
g on all 3 lists by
y chance
individual appearing
were . . .
1 in
21 000 000 000 000 000 000 000 000
21,000,000,000,000,000,000,000,000
Invalidation of Scores
• In August 2007, the FSBPT Board approved the
invalidation of the NPTE scores of the 20 individuals
• 15 have re-taken the exam – 5 failed, 10 passed
(
(some
after
f multiple
l i l attempts))
• 14 jurisdictions affected
– 17 candidates already licensed (some in multiple states)
Licensure Decisions
• 1 jurisdiction suspended licenses pending hearings
• 1 candidate voluntarily surrendered license
• 1 jurisdiction required re-registration for exam within 90 days
– 2 candidates passed and kept licenses
– 1 candidate did not re-register and license was revoked
– 1 candidate re-registered and failed; licensure status TBD
• 1 jurisdiction will not renew license when it expires without
valid passing score
• Pending applications: 1 license granted after successful retest; one application on hold pending passing score
• 1 candidate retained counsel, then registered to re-take
American Dental Association
• After a 2007 investigation, the ADA invalidates the
2006 Part I national dental exam scores of 13 UCLA
dental students accused of sharing recalled test
questions
– performance of the students was not in question
– Students required to wait two years before retaking the
exam
• January 2008 – the 13 students sue the ADA, while
also appealing to the ADA Joint Commission on
National Dental Exams
Student allegations:
•
•
•
•
Branded as cheaters, denied access to
competitive post-graduate programs;
lik l to
likely
t default
d f lt on student
t d t loans
l
Given no advance notice of penalty,
description of evidence, or
meaningful right to appeal
Did everything required by candidate
agreement
Want injunction
j
stopping
pp g the
invalidation, monetary damages
(including punitive damages) and
attorney’s fees
Interesting Questions Raised:
• How much leeway did the ADA give itself in
the candidate agreements?
• How deferential would the court be to an
invalidation decision based on post
post-exam
exam
conduct?
• Is the ADA a state actor that owes the students
constitutional due process?
GMAT “Braindump” Suit
• Federal Court in Virginia awarded GMAC $2.3
million judgment against Chinese national and U.S.US
based web site selling pirated GMAT test questions
(www.scoretop.com)
• Court also allowed seizure of company assets
– Hard drive with data re customers
– Web site itself
Visitors to www.scoretop.com now see:
Issues for the Testing Entity
• How to detect or confirm the breach?
• Has
H exam iintegrity
i been
b
compromised?
i d? If so, how
h to
remedy?
• Who
Wh is
i culpable?
l bl ? Legal
L l recourse??
• Are scores valid? Do we have the right to
invalidate? If so,
so what is our legal liability?
• How to prevent this from happening again?
Issues for Licensing Boards
• Does an invalidated score = invalidated
li
license?
?
• Can we revoke a license? Suspend? Deny
licensure?
• Is p
public protection
p
at issue?
• Is a disciplinary action warranted?
Issues for Candidates
• How will this affect my license/application?
M job?
My
j b? My
M immigration
i
i ti status?
t t ?
• If I unknowingly participated, should I be held
responsible?
• What is my
y legal
g recourse,, and is it worth the
fight?
Recommendations – Be Proactive
• Get the tools you need now:
– E
Evaluate
l t candidate
did t agreements
t to
t ensure maximum
i
flexibility
– Developp a response
p
ppolicyy before
f
the breach
– Consider whether enabling legislation is needed (e.g.,
authorizing suspension or temporary licensure)
– Develop a monitoring/detection program
AAVSB 2008 Annual Meeting & Conference
Jennifer Ancona Semko
Partner, Baker & McKenzie