CREOG Exam Scores – What do they really mean? I have no

1/26/2014
CREOG Exam Scores – What do they really mean?
Peter Hong, M.D.
Residency Program Director
Nassau University Medical Center
East Meadow, N.Y. [email protected]
Precis
• CREOG scores are under appreciated. Converting scores to percentiles allow for comparison of a resident’s/training program’s standing with their peers in areas of knowledge depth, progress of learning & risk of failing written Board exams. Intrepretation basis explained, tools provided. Learning Objectives
•Advise residents of their test results in comparison with their peer residents nationally
•Monitor resident’s progress & national standing in learning over time
•Easily convert scores to percentiles
•Assess failing risk on written Board exam
I have no conflicts of interest
1
1/26/2014
Administer the exam as instructed
It’s the only meaningful opportunity to assess your residents knowledge foundation
Major Test Categories & Breakdown
2012
ITEMS
2013
ITEMS
General Considerations
29 (9%)
24 (8%)
Primary and Preventive
Ambulatory Health Care
50 (15%)
50 (16%)
Obstetrics
75 (23%)
75 (24%)
Gynecology
76 (23%)
74 (23%)
Reproductive
Endocrinology
42 (13%)
40 (13%)
Oncology
37 (11%)
42(13%)
Genomics
14 (4%)
13 (4%)
323
318
CREOG QUESTION
CATEGORY
TOTAL EXAMINATION
Common Beliefs & Misconceptions
• Test Scores are linearly related to # questions answered correctly
• Similar scores in subsequent years will happen if I don’t read
• Occasional reading will significantly increase my CREOG scores
• Test scores provide little or unknown value in risk assessment towards failing ABOG written exam
Each of the above points are fallacies 2
1/26/2014
From Exam to Score
To CREOG for grading & analysis
Exam Results
Comparative 3 Digit Score
Where to focus on the Report?
There are about 1100 residents in training in each level Nationally
Life Distributes in a Bell Shape Curve
• Stature – there are skinny people, there are obese people
• Height – there are short people, there are tall people
• IQ – there are morons, there are bright • In each of the above, there are also average people which account for the majority 3
1/26/2014
Life Distributes in a Bell Shape Curve
Stature
Height
IQ
Skinny
Short
Moron
Obese
Tall
Bright
Average
What about CREOG comparative 3 digit scores? CREOG Scores distributes in a bell shape curve also
This test score is 200
These 550 Residents know less than the residents who scored at this line
50% of test takers scored BELOW this level
50% of test takers scored ABOVE this level
These 550 Residents know more than the residents who scored at this line
This test score is 180
16 % of test takers scored BELOW this line
84 % of test takers scored ABOVE this line
These 924 Residents know more than the residents who scored above this line
176 Residents
4
1/26/2014
This test score is 160
2 % of test takers scored BELOW this line
98 % of test takers scored ABOVE this line
These 1067 Residents know more than the residents who scored above this line
22 Residents
Facts to Consider
• CREOG is the only standardize national exam administered to residents
• CREOG assesses resident education
• ABOG assesses ability to practice
• Both CREOG & ABOG ultimately tests knowledge
• Test are designed to be discriminators, (i.e. there are passers & failures (e.g. ABOG, USMLE, COMLEX)
• About 15% of ABOG test takers fail every year Counseling residents post CREOG
Program B – given result printout, advised to read more
Program A – given result printout, advised to do better next year
Resident
Program C – given result printout, did you know that your 160 places you in the lower 2% of test takers and that the lower 15% of ABOGers fail. How much medical school debt do you have? Remediate!
5
1/26/2014
Charting Resident CREOG Performance
0.025
0.025
Resident A PGY 4
0.02
0.02
0.015
0.015
Resident A PGY 3
Resident A PGY 2
0.01
0.01
0.005
0.005
0
0
140
160
180
200
220
240
260
Resident A PGY 1
Score
%ile
PGY 4 2013 CREOG
192
36
PGY 3 2012 CREOG
194
40
PGY 2 2011 CREOG
175
13
PGY 1 2010 CREOG
177
14
Normal Curve with 3 Digit Comparative Scores
0.025
0.025
0.02
0.02
0.015
0.015
0.01
0.01
0.005
0.005
0
0
140
160
180
200
220
240
260
National CREOG Distribution
15% Line
National Mean
Conversion of Comparison Score to Percentile
*Partial List
6
1/26/2014
Does the Same Score Each Year Mean No Growth in Knowledge?
Medical Knowledge
PGY 4
PGY 3
PGY 2
PGY 1
Medical
Student
Does the Same Score Each Year Mean No Growth in Knowledge?
CREOG Score
Percent Correct
Resident B
PGY 1 2009 CREOG
195
PGY 2 2010 CREOG
195
PGY 3 2011 CREOG
193
55
4%
59
6%
65
• To maintain the same 3 digit score each successive year – resident needs to answer 4‐6% more questions right than previous year (i.e. 12‐19 new facts). Assuming 320 CREOG questions and 1% is about 3‐4 questions.
How Much Work is Involved in Improving CREOG Scores?
• Each year the average resident increases knowledge by 12‐19 new facts.
• To improve a comparative 3 digit score, the resident must do better than previously better scoring residents from previous years.
• The number of residents that they must outperform is dependent where their CREOG score is.
7
1/26/2014
How Many Residents Need to be Outperformed to Improve CREOG Scores
374
154
140 160
180
200
220
240 260
To go from 180 to 200, Resident A has to do better than 374 residents the next time.
To go from 160 to 180, Resident C has to do better than 154 residents the next time.
Additional Knowledge Needed to Improve CREOG Scores
20 Point Jump
140 160 180 200 220 240 260
Requires 32‐50 more correct answers than the previous year.
Who to Remediate?
Remediation Line
0.025
0.025
0.02
0.02
0.015
0.015
0.01
0.01
0.005
0.005
0
0
140
160
180
200
220
240
National CREOG
Distribution
15% Line
260
• Where do you draw the line at which everyone below is placed on remediation?
• Draw it high, then you work harder to remediate more residents
• Draw it low then you may jeopardize the weak resident passing ABOG •
•
•
•
180 ‐ 16%
182 – 19%
185 – 24%
190 – 33%
Scores are not linear
8
1/26/2014
Who to Remediate?
• SEM – This is an index of the examination’s precision in measuring your overall ability as represented by the score reported. If you were to repeat the CREOG In‐Training Examination 100 times, your score would fall in the range of +/‐ 1 SEM on 68 out of 100 times.
Who to Remediate?
Remediation Line
0.025
0.025
0.02
0.02
0.015
0.015
0.01
0.01
0.005
0.005
0
0
140
160
180
200
220
240
National CREOG
Distribution
15% Line
260
•
•
•
•
180 ‐ 16%
182 – 19%
185 – 24%
190 – 33%
Consider everyone below this
9
Scores & Percentiles
2Score
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
Percentile
0.1%
0.2%
0.3%
0.4%
0.5%
0.7%
0.8%
0.9%
1.0%
1.1%
1.2%
1.3%
1.4%
1.5%
1.6%
1.8%
1.9%
2.0%
2.1%
2.2%
2.3%
3.0%
3.7%
4.3%
5.0%
5.7%
6.4%
7.1%
7.7%
8.4%
9.1%
9.8%
10.5%
11.1%
11.8 %
12.5%
13.2%
13.9%
14.5%
15.2%
2Score
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
Percentile
15.9%
17.6%
19.3%
21.0%
22.7%
24.4%
26.1%
27.8%
29.5%
31.2%
32.9%
34.6%
36.3%
38%
39.7%
41.4%
43.1%
44.8%
46.5%
48.2%
50%
51.7%
53.4%
55.1%
56.8%
58.5%
60.2%
61.9%
63.6%
65.3%
67%
68.7%
70.4%
72.1%
73.8%
75.5%
77.2%
78.9%
80.6%
82.3%
2Score
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
Percentile
84.1%
84.8%
85.5%
86.1%
86.8%
87.5%
88.1%
88.9%
89.5%
90.2%
91%
91.6%
92.3%
92.9%
93.6%
94.3%
95%
95.7%
96.3%
97%
97.7%
97.8%
97.9%
98%
98.1%
98.2%
98.4%
98.5%
98.6%
98.7%
98.8%
98.9%
99%
99.1%
99.2%
99.3%
99.5%
99.6%
99.7%
99.8%
99.9%
CREOG Comparative 3 Digit Score Graph 0.025
0.025
0.02
0.02
0.015
0.015
0.01
0.01
0.005
0.005
0
0
140
160
180
200
220
240
260
National CREOG Distribution
15% Line
National Mean