Basic Item and Exam Analysis Focused Discussion (Session 1079)

Basic Item Analysis
By
Bob McMullen, MS, PA-C
Associate Professor
AT Still University
Mesa, AZ
[email protected]
2011 PAEA Annual
Education Forum
Objectives
At the end of this session, the participant will be able to:
1. Define the following terms as they apply to excam and item analysis:
a. Stem, answer and distractors
b. Item difficulty and discrimination
c. Distractor analysis
d. Reliability and validity
2. Describe how each of the above metrics is obtained from Scantron or LMS
software and/or spreadsheet functions
3. Describe how each of the above metrics can be analyzed and compared
Disclaimers
The speaker doesn’t receive any
money from anybody except his wifeand she doesn’t get anything from
Scantron or any other companies
mentioned in this talk.
Step 1- Write a Test
Open-book, closedbook or
combination??
Parts of a MCQ
Title
Title: Lid Lag
Stem
Lid lag is associated with which condition?
Correct
Answer
a. Narrow-angle glaucoma
*b. Hyperthyroidism
c. Presbyopia
d. Hypothyroidism
e. Open-angle glaucoma
Distractors
Step 2: Give the Test
Note to Self:
Mention
importance of
scratch paper
Step 3: Grade Test
Image from Scantron.com
Step 4: Print the Analysis
Image from Scantron.com
Step 4a: Examine the Analysis
Screen capture from Blackboard Learning System (WebCT)
Step 4a: Examine the Analysis
Screen capture from Blackboard Learning System
(WebCT)
Step 5: Determine Exam Mean
Image from Scantron.com
Step 5: Determine Exam Mean
Screen capture from Blackboard Learning System
(WebCT)
Step 6: Examine Question Difficulty
• Difficulty is the percentage of
students that got the question correct
• It is denoted as “p”
Step 6: Examine Question Difficulty
Highlight questions that may be too difficult
Too difficult defined by number
of answer choices:
2 alternatives (true/false)= .75 or 75%
3 alternative MCQ= .67 or 67%
4 alternative MCQ= .63 or 63%
5 alternative MCQ= .60 or 60%
Step 6: Examine Question Difficulty
Highlight questions that may be too difficult
Too difficult defined by number
of answer choices:
2 alternatives (true/false)= .75 or 75%
3 alternative MCQ= .67 or 67%
4 alternative MCQ= .63 or 63%
5 alternative MCQ= .60 or 60%
Step 7: Locate or Calculate
Response Distribution
Step 7: Locate or Calculate Response Distribution
Screen capture from Blackboard Learning System (WebCT)
Step 7: Locate or Calculate
Response Distribution
Step 8: Transfer Difficulty,
Discrimination, and Response
Distribution values to Each Question
P=54%
r=44
Step 8: Transfer Difficulty,
Discrimination, and Response
Distribution values to Each Question
r=44
•
•
•
Discrimination is the
item’s ability to
discriminate between
students who did well on
the exam and those that
didn’t
It is the difference
between the difficulty for
the upper part of the
group and the lower part
It is denoted as “r”
Step 8: Transfer Difficulty,
Discrimination, and Response
Distribution values to Each Question
r=44
•
•
•
•
40-100= Excellent item
25-39= Good item
0-24= Usually
unacceptable- might be
approved
Negative= Unacceptable,
check item for error
Step 8: Transfer Difficulty,
Discrimination, and Response
Distribution values to Each Question
Step 9: Examine Each Question with SME
Title: 00-001-013-1
2. Which of the following is NOT a feature of P-glycoproteins in the
intestine?
a. Act as extruding transporters
*b. Enhance absorption of drugs from GI lumen
c. Remove substances from enterocytes
d. Have inhibitors and inducers
e. Act on a substrate
14
38
15
2
2
P=54%
r=44
•
•
•
•
Difficulty?
Discriminator?
Response Distribution?
Validity?
•
•
•
•
40-100=
Excellent item
25-39= Good
item
0-24= Usually
unacceptablemight be
approved
Negative=
Unacceptable,
check item for
error
Step 9: Examine Each Question with SME
Title: 00-002-003-5
3. What single factor affects drug binding to receptors the most?
*a. Drug concentration
b. Drug solubility
c. Drug Molecular size
d. Receptor lipophilicity
e. Receptor polarity
33
29
0
2
7
P=47%
r=28
•
•
•
•
Difficulty?
Discriminator?
Response Distribution?
Validity?
•
•
•
•
40-100=
Excellent item
25-39= Good
item
0-24= Usually
unacceptablemight be
approved
Negative=
Unacceptable,
check item for
error
Step 9: Examine Each Question with SME
38
4
27
2
0
Title: 01.001.005-1
6. A patient presents with severe abdominal pain and "burning"
epigastric discomfort. Endoscopy reveals benign ulcers in the antral
mucosa of the stomach. Which of the following medications will
provide the fastest, albeit probably the briefest, relief of the
discomfort with just a single dose?
*a. Magnesium hydroxide
b. Cimetidine
c. Misoprostol
•
d. Omeprazole
e. Dicyclomine
•
P=54%
r=39
•
•
•
•
Difficulty?
Discriminator?
Response Distribution?
Validity?
•
•
40-100=
Excellent item
25-39= Good
item
0-24= Usually
unacceptablemight be
approved
Negative=
Unacceptable,
check item for
error
Step 9: Examine Each Question with SME
0
11
0
42
18
Title: Pharm-00.004.003-4
9. A 42-year-old female presents with acute left ankle pain which
she describes as 3/10. After a thorough physical exam and X-rays,
you diagnose a mild ankle sprain. Her history includes GERD for
which she takes omeprazole daily. In addition to heat/ice,
crutches, elevation and physical therapy, which of the following
medications is the most appropriate management for this
patient?
a. Strong opioid agonist
b. Weak opioid agonist
c. Mixed opioid agonist-antagonist
*d. APAP
e. NSAID
P=59%
r=-5.56
•
•
•
•
Difficulty?
Discriminator?
Response Distribution?
Validity?
•
•
•
•
40-100=
Excellent item
25-39= Good
item
0-24= Usually
unacceptablemight be
approved
Negative=
Unacceptable,
check item for
error
Step 9: Examine Each Question with SME
Title: 01.009.003-25
11. Which of the following classes of drugs used in constipation
works by providing indigestible, hydrophilic colloids that absorb
water, forming an emollient gel that distends the colon and
promotes peristalsis?
a. Stool softeners
b. Osmotic laxatives
*c. Bulk-forming laxatives
d. Stool lubricants
2
24
43
2
P=61%
r=28
•
•
•
•
Difficulty?
Discriminator?
Response Distribution?
Validity?
•
•
•
•
40-100=
Excellent item
25-39= Good
item
0-24= Usually
unacceptablemight be
approved
Negative=
Unacceptable,
check item for
error
Step 9: Examine Each Question with SME
Title: 02.003.001-25
15. Which of following classes of antidepressants slow cardiac
conduction and are the most cardiotoxic?
a. SSRIs
b. MAOIs
*c. TCAs
d. SNRIs
e. Atypical antidepressants
0
33
33
2
3
P=47%
r=33
•
•
•
•
Difficulty?
Discriminator?
Response Distribution?
Validity?
•
•
•
•
40-100=
Excellent item
25-39= Good
item
0-24= Usually
unacceptablemight be
approved
Negative=
Unacceptable,
check item for
error
Step 10: Check other questions’
Discrimination values
r=-5.56
r=0
r=0
Step 10: Check other questions’
Discrimination values
64
0
2
4
1
•
•
•
•
Title: 00-001-006-2
1. Which of the following drug administration
routes would be expected to take the longest in
terms of achieving a high serum drug
concentration?
*a. Oral
b. Intravenous
c. Subcutaneous
d. Intramuscular
e. Inhalation
P=90%
r=-5.56
Difficulty?
Discriminator?
Response Distribution?
Validity?
•
•
•
•
40-100=
Excellent item
25-39= Good
item
0-24= Usually
unacceptablemight be
approved
Negative=
Unacceptable,
check item for
error
Step 10: Check other questions’
Discrimination values
Title: 00.005.002-4
5. Why is the location of the patient when the drug is dispensed
important in the topic of antimicrobial resistance?
a. Antibiotics may have different levels of effectiveness in different
climates
b. Antibiotics dispensed to a patient standing in WalMart are likely to
not work
c. Patients in out-patient settings are more likely to finish their
prescription
*d. Patients in in-patient settings have much higher compliance rates
than out-patient
e. Patients from New York will not respond to treatment provided in
Arizona
9
0
2
60
0
•
•
•
•
Difficulty?
Discriminator?
Response Distribution?
Validity?
P=85%
r=0
•
•
•
•
40-100=
Excellent item
25-39= Good
item
0-24= Usually
unacceptablemight be
approved
Negative=
Unacceptable,
check item for
error
Step 10: Check other questions’
Discrimination values
Title: Pharm-00.004.008-25
10. A patient in your surgery rotation uses fentanyl (Duragesic) for
chronic pain. She now also has additional moderate pain from the
minor surgery your team just performed. Of the following
analgesic options, which would be the BEST choice to manage this
patient's pain?
*a. Hydrocodone/APAP (Lortab)
b. Acetaminophen (Tylenol)
c. Hydromorphone (Dilaudid)
d. Naproxen (Naprosyn)
e. Celecoxib (Celebrex)
45
8
10
7
1
P=63%
r=0
•
•
•
•
Difficulty?
Discriminator?
Response Distribution?
Validity?
•
•
•
•
40-100=
Excellent item
25-39= Good
item
0-24= Usually
unacceptablemight be
approved
Negative=
Unacceptable,
check item for
error
Step 11: Regrade Exam and Re-evaluate
Image from Scantron.com
References
Flynn, M. K., & Reese, J. L. (1988). Development and evaluation of classroom tests: A
practical application. Journal of Nursing Education, 27(2), 61-65.
Jenkins, H. M., & Michael, M. M. (1986). Using and interpreting item analysis data...
Valuable tool in test construction. Nurse Educator, 11(1), 10-14.
Kuang, H. P. (1952). A critical evaluation of the relative efficiency of three techniques
in item analysis. Educational and Psychological Measurement, 12, 248-266.
McGahee, T. W., & Ball, J. (2009). How to read and really use an item analysis. Nurse
Educator, 34(4), 166-171.
Wood, R. (1977). Multiple choice: A state of the art report. Evaluation in Education:
International Progress, 1(3), 191-280.