Analysis of Differential Prediction of Law School Performance by

LSAT TECHNICAL REPORT SERIES
 Analysis of Differential Prediction of Law School
Performance by Racial/Ethnic Subgroups Based on
2008–2010 Entering Law School Classes
Deborah A. Suto
Lynne L. Norton
Lynda M. Reese
 Law School Admission Council
LSAT Technical Report 12-02
October 2012
A Publication of the Law School Admission Council
The Law School Admission Council (LSAC) is a nonprofit corporation that provides unique, state-of-theart admission products and services to ease the admission process for law schools and their applicants
worldwide. More than 200 law schools in the United States, Canada, and Australia are members of the
Council and benefit from LSAC's services.
© 2012 by Law School Admission Council, Inc.
LSAT, The Official LSAT PrepTest, The Official LSAT SuperPrep, ItemWise, and LSAC are registered
marks of the Law School Admission Council, Inc. Law School Forums, Credential Assembly Service,
CAS, LLM Credential Assembly Service, and LLM CAS are service marks of the Law School Admission
Council, Inc. 10 Actual, Official LSAT PrepTests; 10 More Actual, Official LSAT PrepTests; The Next 10
Actual, Official LSAT PrepTests; 10 New Actual, Official LSAT PrepTests with Comparative Reading; The
New Whole Law School Package; ABA-LSAC Official Guide to ABA-Approved Law Schools; Whole Test
2
Prep Packages; The Official LSAT Handbook; ACES ; ADMIT-LLM; FlexApp; Candidate Referral Service;
DiscoverLaw.org; Law School Admission Test; and Law School Admission Council are trademarks of the
Law School Admission Council, Inc.
All rights reserved. No part of this work, including information, data, or other portions of the work
published in electronic form, may be reproduced or transmitted in any form or by any means, electronic or
mechanical, including photocopying and recording, or by any information storage and retrieval system
without permission of the publisher. For information, write: Communications, Law School Admission
Council, 662 Penn Street, PO Box 40, Newtown PA, 18940-0040.
LSAC fees, policies, and procedures relating to, but not limited to, test registration, test administration,
test score reporting, misconduct and irregularities, Credential Assembly Service (CAS), and other matters
may change without notice at any time. Up-to-date LSAC policies and procedures are available at
LSAC.org.
Table of Contents
Executive Summary ........................................................................................................1
Introduction .....................................................................................................................2
Methods ...........................................................................................................................3
Sample ...................................................................................................................3
LSAT Version .........................................................................................................4
Variables Used in the Study ...................................................................................5
Analysis Methods ...................................................................................................5
Results .............................................................................................................................6
Descriptive Statistics ..............................................................................................6
Predictive Validity ................................................................................................. 16
Predicting First-Year Averages ............................................................................. 16
Conclusions ................................................................................................................... 23
References ..................................................................................................................... 24
i
Executive Summary
In the law school admission process, it is essential that the criteria used for
admission are fair to all subgroups in the applicant population. One method used to
evaluate the fairness of the admission process is to compare the predicted and actual
first-year averages (FYAs) within individual law schools for various subgroups of the
applicant population. The current study was designed to address questions of
differential prediction of law school grades for various racial/ethnic subgroups.
The sample used in this study was drawn from the 2008, 2009, and 2010 entering
law school classes, using data that were available from the Law School Admission
Council (LSAC)-sponsored correlation studies. The study examined results for three
racial/ethnic minority subgroups and the nonminority (White) subgroup. Data were
analyzed from 178 law schools, each of which over the 3-year period enrolled 10 or
more first-year students who identified themselves as Asian American, Black, or Latino
and 10 or more first-year students who identified themselves as White.
Statistical analyses were used to predict FYAs using Law School Admission Test
(LSAT) score alone, undergraduate grade point average (UGPA) alone, and the best
predictive combination of LSAT score and UGPA. Analyses were carried out separately
for all individual law schools included in the study, resulting in three prediction equations
for each law school.
The results of the analyses indicate that FYA tended to be, on average, slightly
overpredicted (i.e., predicted FYAs exceeded actual FYAs) for all three of the
racial/ethnic minority subgroups studied here, with Black law students exhibiting the
most overprediction and Asian American law students exhibiting the least
overprediction. The combination of both LSAT score and UGPA provided the least
amount of overprediction for racial/ethnic minority subgroups on the school level
compared to the use of either single predictor alone. Overall, these results do not
support the concern that LSAT score alone or the combination of LSAT score and
UGPA may contribute to unfair admission decisions for the racial/ethnic subgroups
studied here.
While considering the results of this study, the reader should keep in mind that they
refer only to subgroup behavior and not to individuals. For example, while results may
suggest that UGPAs alone may overpredict FYAs for Black law students on average,
the performance of many individual Black law students may be underpredicted based
solely on their UGPAs.
Finally, it is worth repeating that the average amount of overprediction or
underprediction of FYAs found for the four racial/ethnic subgroups studied was very
slight, regardless of the prediction equation that was used. In other words, this study
provided no evidence that LSAT score, UGPA, or the combination of those two predictor
variables unfairly predict future law school performance for any racial/ethnic subgroup.
1
Introduction
Over the past two decades, the proportion of racial/ethnic minority students in law
schools has increased, due in part to efforts to increase diversity among law school
students and, ultimately, within the legal profession. Between the academic years
spanning from 1988–1989 to 2009–2010, Asian American representation among firstyear law school students rose from 3.0% to 7.7%; Black representation among first-year
law school students rose from 5.7% to 7.3%; and Latino representation among first-year
law school students rose from 3.4% to 6.5% (Law School Admission Council and the
American Bar Association Section of Legal Education and Admissions to the Bar, 2010).
While the representation of these racial/ethnic subgroups in first-year law school
classes has increased, the difference between minority and nonminority performance on
the Law School Admission Test (LSAT) remains significant. These differences in
average LSAT scores continue to raise questions about the validity of the test for
racial/ethnic minority test takers and about possible differences in prediction of law
school performance as a consequence of relying in whole or in part on LSAT scores.
One method used to evaluate the fairness of the LSAT is to compare the predicted and
actual first-year averages (FYAs) for each law school for various subgroups of the
applicant population. If one subgroup of the applicant population experiences either
significantly more overprediction (i.e., average predicted FYA greater than average
actual FYA) or significantly more underprediction (i.e., average predicted FYA less than
average actual FYA) than some other subgroup, then differential prediction is said to
occur. The purpose of the current study is to address questions of differential prediction
for four student subgroups (three racial/ethnic minority subgroups and the nonminority
[White] subgroup) based on data from the 2008, 2009, and 2010 first-year classes.
Indeed, these questions are not new to research sponsored by the Law School
Admission Council (LSAC), nor are they unique to the LSAT or to the law school
admission process. Several studies using LSAT data to investigate questions of
differential subgroup validity have been sponsored previously by LSAC (Anthony & Liu,
2003; Linn & Hastings, 1984; Norton, Suto, & Reese, 2006; Norton, Suto, & Reese,
2009; Powers, 1977; Schrader & Pitcher, 1976a, 1976b; Stilwell & Pashley, 2003;
Stilwell, Reese, & Pashley, 1998; Wightman & Muller, 1990). Differential prediction has
also been the subject of research studies for other admission-testing programs such as
the SAT (e.g., Breland, 1979; Mattern, Patterson, & Kobrin (2012); Mattern, Patterson,
Shaw, Kobrin, & Barbuti, 2008; Ramist, Lewis, & McCamley-Jenkins, 1994; Sackett et
al., 2012; Shaw, Kobrin, Patterson, & Mattern (2012); Stricker, Rock, & Burton, 1991;
Willingham, Lewis, Morgan, & Ramist, 1990); the ACT (e.g., Noble, 2003; Noble,
Crouse, & Schulz, 1996); the GMAT (e.g., Braun & Jones, 1981); and the GRE (e.g.,
Burton & Wang, 2005). Numerous studies focusing on the same questions in the arena
of employment testing have also been reported (e.g., Houston & Novick, 1987; National
Research Council, 1989; Schmidt & Hunter, 1981). Most of these studies concluded
that, although there is evidence of differential prediction for racial/ethnic minorities, there
is no evidence of test bias against those subgroups. That is, the use of the majority
regression model or the pooled regression model tends to overpredict (or at least not
underpredict) minority performance on the criterion variable.
2
The present study is part of an ongoing monitoring effort designed to address the
following question: Does either of the traditional predictors of first-year law school
performance—LSAT score and undergraduate grade point average (UGPA), or the
combination of both—result in differential prediction for racial/ethnic minority applicants?
Earlier studies looked at similar questions. The Wightman and Muller (1990) study
included data from Hispanic law students as well as from Black and Mexican American
law students. That study provided limited representation among law schools, including
data from 51 schools that had a sufficient number of Black students, 7 schools that had
a sufficient number of Mexican American students, and 13 schools that had a sufficient
number of Hispanic students. The Anthony and Liu (2003) study included data from
Asian American students as well as from Black and Latino students. That study
provided broad representation among law schools, including data from 118 schools that
had a sufficient number of Asian American students, 142 schools that had a sufficient
number of Black students, and 108 schools that had a sufficient number of Latino
students. The Stilwell and Pashley (2003) study was similar to the Anthony and Liu
(2003) study and was based on a sample that displayed a slight increase in school
representation. That study included data from 128 schools that had a sufficient number
of Asian American students, 147 schools that had a sufficient number of Black students,
and 111 schools that had a sufficient number of Latino students. The Norton et al.
(2006) study analyzed data from students who entered law school in 2002, 2003, and
2004. That study included data from 147 schools that had a sufficient number of Asian
American students, 148 schools that had a sufficient number of Black students, and 124
schools that had a sufficient number of Latino students. The Norton et al. (2009) study
analyzed data from students who entered law school in 2005, 2006, and 2007.
Reflecting increased racial/ethnic minority enrollments, that study included data from
148 schools that had a sufficient number of Asian American students, 150 schools that
had a sufficient number of Black students, and 133 schools that had a sufficient number
of Latino students. The current study examines data for 2008, 2009, and 2010 law
school entering classes. Continued increases in racial/ethnic minority enrollments were
evident: The sample for this study comprised 157 schools that had a sufficient number
of Asian American students, 153 schools that had a sufficient number of Black students,
and 151 schools that had a sufficient number of Latino students.
Methods
Sample
The sample used in this study was drawn from the 2008, 2009, and 2010 entering
law school classes. With a few exceptions, the data closely mirror the data that were
used for the 2011 LSAT Correlation Studies. In general, this study includes law schools
that participated in the 2011 LSAT Correlation Studies and had sufficient racial/ethnic
minority and nonminority data available for the combined years of classes. Canadian
law schools were excluded from this report because they did not participate in LSAC’s
Credential Assembly Service (CAS). Three additional law schools were also excluded
from this report because of insufficient numbers of nonminority students and evidence
3
that preselection effects distorted their results. The data from each participating ABAapproved law school were combined across years to ensure stability in the analysis and
to increase the representation of the law school. In instances where a school
experienced a change in grading scale or did not participate every year in the
correlation studies, less than 3 years of data were available. While 182 ABA-approved
and Canadian law schools participated in the 2011 LSAT Correlation Studies, 188
schools were represented in this study. The 188 schools included 176 ABA-approved
schools that were participants in 2008, 184 ABA-approved schools that participated in
2009, and 176 ABA-approved schools that participated in 2010. Of the 188 schools, 165
had data for all three entering classes, 18 had data for two of the three classes, and 5
had data for a single class. The total pool included approximately 109,871 law school
students across the three entering classes.
The current study focused on Asian American, Black, and Latino student subgroups.
The racial/ethnic identity used for the correlation study data was based on a selfreported description code provided by students. Note that for the 2008 and 2009
entering classes, the Latino subgroup consisted of students who reported their ethnicity
as either Hispanic or Mexican American. For the 2010 entering class, the
Hispanic/Latino category is displayed as “Latino,” and the new categories of Native
Hawaiian/Other Pacific Islander and Asian are combined and displayed as “Asian
American.”
Data were analyzed separately for each law school that had 10 or more students
from at least one of the racial/ethnic minority subgroups of interest and 10 or more
students from the nonminority subgroup. Among the schools included in the sample,
157 had 10 or more students in both the Asian American and White student subgroups,
153 schools had 10 or more students in both the Black and White student subgroups,
and 151 schools had 10 or more students in both the Latino and White student
subgroups. Overall, 178 individual law schools had a sufficient number of students in
one or more of the racial/ethnic minority subgroups of interest.
LSAT Version
All students whose data were used in this study were tested with the version of the
LSAT given in 2010 or earlier. That version of the test included four 35-minute sections
and one 35-minute variable section containing material that is used to pretest new
questions or pre-equate new test forms. The variable section did not contribute to the
test taker’s score. The specific item-type makeup was as follows:
Item Type
Reading Comprehension
Logical Reasoning A
Logical Reasoning B
Analytical Reasoning
No. of Items
26–28
24–26
24–26
22–23
4
Time
35 minutes
35 minutes
35 minutes
35 minutes
The total number of scored items on a form ranged from 100 to 102. A single score
was derived from the sum of the total number of questions answered correctly across
the four scored sections and then equated and reported on an LSAT scale ranging from
120 to 180. A writing sample was administered at the end of the test. Beginning in June
2005, the writing sample was extended from 30 minutes to 35 minutes. This writing
assessment was not scored by LSAC, but copies of the writing sample were sent to all
law schools to which the test taker applied.
Variables Used in the Study
The variables analyzed in this study are the same ones that are currently used in the
correlation studies: FYA, UGPA, and LSAT score. LSAT score and UGPA are the
predictor variables (i.e., the variables that are used to predict performance in the first
year of law school). FYA—the measure of performance in the first year of law school—
is the criterion variable, or the variable that is predicted using LSAT score and UGPA.
Only students for whom data were available on each of the three variables were
included in this study.
Additional operational details related to these three variables are given below:



First-year average. This variable is the average grade earned by the student in
the first year of law school. FYA is provided for each student by the individual law
schools. Different law schools use different scales for first-year grades. In order
to maintain the confidentiality of the individual schools and to allow direct
comparison across law schools, FYA values were transformed to a scale having
a mean of 50 and a standard deviation of 10. Results presented in this report are
on the transformed 50/10 scale.
Undergraduate grade point average. The average grade earned by each student
during his or her undergraduate study is computed by the CAS, according to
CAS procedures. Grades computed in this manner are expressed on a scale
from 0.00 to 4.33. The UGPAs used in these studies are the same as those used
in the correlation studies carried out for individual law schools.
LSAT scores. Only LSAT scores reported on the 120–180 scale were used in this
study. For students with multiple LSAT scores, a single arithmetic average (i.e.,
mean) of the multiple scores was used. If any student took the test more than
three times, only the most recent three scores were averaged.
Analysis Methods
This study was undertaken to evaluate the fairness and appropriateness of using
LSAT score and UGPA to predict performance in law school for racial/ethnic minority
students from a single prediction equation. This equation is developed from the
combined data of all racial/ethnic minority and nonminority students. In other words, the
study sought to evaluate the potential for differential prediction across Asian American,
Black, Latino, and White law school student subgroups. The methods used to predict
5
FYA in the current study were the same as those used in the ongoing predictive validity
studies for individual schools that participate in the correlation studies. As in the
correlation studies, data were pooled from the 3 years under investigation in order to
achieve stable results within schools. The evaluation included descriptive, validity, and
predictive analyses. These analyses were carried out separately for each law school
that had a sufficient number of students in the racial/ethnic minority and nonminority
subgroups. The following is a more detailed description of these analyses:



The descriptive analysis presents statistics comparing racial/ethnic minority and
nonminority student populations.
The validity analysis computes and compares the correlation coefficients for the
different subgroups. Correlation coefficients are used to measure the linear
relationship between the predictor variables (LSAT score and UGPA) and the
criterion variable (FYA). Correlation values were calculated separately for each
law school included in the study and averaged across schools.
For the predictive analysis, three separate least-squares regression calculations
were used to predict FYA using LSAT score alone, UGPA alone, and a linear
combination of LSAT score and UGPA as predictors. The difference between the
predicted and actual FYA was calculated for each student within a law school
and averaged for each subgroup at each school.
Results
The results from this study are presented in three parts. The first part includes
descriptive data about the racial/ethnic minority and nonminority first-year students. The
second part reports the validity coefficients between the predictor variables and FYA for
each subgroup included in the analyses. In the third part, the results of applying the
prediction equations derived using the total group data are reported for the separate
subgroups.
Descriptive Statistics
Descriptive statistics for the sample of students within the law schools used in this
study are presented in Tables 1–5 and Figures 1–3. These data provide information
about the number and proportion of racial/ethnic minority and nonminority students and
the size of the racial/ethnic minority subgroups among the law schools included in this
study. The tables and figures allow for the comparison of LSAT score, UGPA, and FYA
between racial/ethnic minority and nonminority student subgroups.
Table 1 describes the overall racial/ethnic subgroup breakdown among the 182
schools that participated in the 2011 LSAT Correlation Studies. Table 2 provides similar
information for the 188 schools with data available for the current study. A comparison
of the two tables shows that this study is very representative of the 2011 LSAT
Correlation Studies sample. Of the 109,871 students at the 188 schools represented
across the 3 years, 8,341 (7.6%) were Asian American; 7,562 (6.9%) were Black; 6,732
6
(6.1%) were Latino; and 79,659 (72.5%) were White. The analyses included 178
schools that met the sample size requirements. A total of 107,794 students were
represented, of whom 8,176 (7.6%) were Asian American; 7,367 (6.8%) were Black;
6,520 (6.0%) were Latino; and 77,830 (72.2%) were White. The percentages of
racial/ethnic minorities among schools included in this study are similar to the
percentages reported by the Law School Admission Council and American Bar
Association (2010) for all law schools.
Note that LSAC revised its method of race/ethnicity data collection beginning with
the 2009–2010 testing year. The categories from which test takers may choose to
describe themselves were updated. In addition, the category “Other” was dropped
starting with this testing year. Test takers were also permitted to choose more than one
race/ethnicity category, resulting in the category “Multiple Ethnicities.” Those who chose
not to respond to the optional demographic questions are included in the respective “No
Response” subgroups. For schools participating in the 2011 LSAT Correlation Studies,
the changes to ethnic coding caused an increase in “No Response” rates in 2010 to
2,412 (6.3%) from 255 (0.8%) in 2008 and 226 (0.6%) in 2009. Overall, for the pooled
data in Table 1, the “No Response” rate is 2,893 (2.7%).
7
TABLE 1
Number and percentage of Asian American, Black, Latino, and White first-year students among schools that were included in the 2011 LSAT
Correlation Studies
Asian American
Black
Latino
White
Entering
No. of
n
n
n
n
Class
Total
Schools
%
%
%
%
2008
33,584
172
2,597
7.7
2,272
6.8
1,945
5.8
24,532
73.0
2009
36,919
179
2,800
7.6
2,508
6.8
2,270
6.1
26,917
72.9
2010
38,226
182
2,835
7.4
2,530
6.6
2,220
5.8
27,687
72.4
Pooled data
108,729
182
8,232
7.6
7,310
6.7
6,435
5.9
79,136
72.8
Note: For the 2008 and 2009 entering classes, the Hispanic and Mexican American categories are combined and displayed as “Latino.” For the
2010 entering class, the Hispanic/Latino category is displayed as “Latino,” and the new categories of Native Hawaiian/Other Pacific Islander and
Asian are combined and displayed as “Asian American.”
TABLE 2
Number and percentage of Asian American, Black, Latino, and White first-year students among schools that were included in the current study
Asian American
Black
Latino
White
Entering
No. of
n
n
n
n
Class
Total
Schools
%
%
%
%
2008
34,546
176
2,727
7.9
2,410
7.0
2,102
6.1
25,028
72.4
2009
37,905
184
2,870
7.6
2,633
6.9
2,414
6.4
27,532
72.6
2010
37,420
176
2,744
7.3
2,519
6.7
2,216
5.9
27,099
72.4
Pooled data
109,871
188
8,341
7.6
7,562
6.9
6,732
6.1
79,659
72.5
Analysis sample
107,794
178
8,176
7.6
7,367
6.8
6,520
6.0
77,830
72.2
Note: For the 2008 and 2009 entering classes, the Hispanic and Mexican American categories are combined and displayed as “Latino.” For the
2010 entering class, the Hispanic/Latino category is displayed as “Latino,” and the new categories of Native Hawaiian/Other Pacific Islander and
Asian are combined and displayed as “Asian American.”
8
Table 3 provides a distribution of law schools by percentage of subgroup enrollment
across the 188 law schools. Table 3 reveals that Asian American, Black, and Latino
student subgroups have a representation of 0–10% in most of the law schools included
in this study. Consequently, it also displays quite clearly that most of the law schools
included in this study are made up primarily of White students.
TABLE 3
Distribution of law schools by percentage of subgroup enrollment
Asian American
Black
Latino
White
% of Subgroup
n
n
n
n
Enrollment
%
%
%
%
0–10
79.8%
150
89.9%
169
88.8%
167
0.5%
1
11–20
16.5%
31
6.9%
13
8.5%
16
0.5%
1
21–30
3.2%
6
0.5%
1
1.6%
3
0.5%
1
31–40
0%
0
0%
0
1.1%
2
1.6%
3
41–50
0%
0
1.1%
2
0%
0
2.1%
4
51–60
0.5%
1
1.1%
2
0%
0
9.0%
17
61–70
0%
0
0%
0
0%
0
17.0%
32
71–80
0%
0
0%
0
0%
0
34.0%
64
81–90
0%
0
0.5%
1
0%
0
32.4%
61
91–100
0%
0
0%
0
0%
0
2.1%
4
Note: For the 2008 and 2009 entering classes, the Hispanic and Mexican American categories are
combined and displayed as “Latino.” For the 2010 entering class, the Hispanic/Latino category is
displayed as “Latino,” and the new categories of Native Hawaiian/Other Pacific Islander and Asian are
combined and displayed as “Asian American.”
Table 4 indicates that the sample included 157 law schools that met the sample size
requirements for Asian American and White students, 153 schools that met the
requirements for Black and White students, and 151 schools that met the requirements
for Latino and White students. For one or more of the three racial/ethnic minority
subgroups of interest, 178 individual schools had sufficient data available for the
analysis sample used in this study. This number was greater than that for earlier
studies. The greatest increase occurred in the number of schools included for the Latino
and White comparison. In this study period, 18 more schools met the sample
requirements for this comparison as compared to the previous study period (Norton et
al., 2009).
TABLE 4
Summary of the number of included law schools by size of racial/ethnic minority subgroup
Size of Subgroup
Race/Ethnicity
10–29
30–49
50–74
75–99
100 or More
Total
Asian American
65
37
20
11
24
157
Black
70
40
24
8
11
153
Latino
69
37
22
11
12
151
Note: For the 2008 and 2009 entering classes, the Hispanic and Mexican American categories are
combined and displayed as “Latino.” For the 2010 entering class, the Hispanic/Latino category is
displayed as “Latino,” and the new categories of Native Hawaiian/Other Pacific Islander and Asian are
combined and displayed as “Asian American.”
9
Table 5 provides descriptive statistics by racial/ethnic subgroup for the three
variables used in the study. The data indicate that all three variables tend to be highest
for the White subgroup, followed by the Asian American, Latino, and Black subgroups.
These results are consistent with those for LSAT taker populations. The standard
deviations for FYAs demonstrate the opposite pattern, with the greatest amount of
variation among the Black subgroup and the least variation among the White subgroup.
The variation in LSAT scores is highest for the Asian American and White subgroups.
TABLE 5
Descriptive statistics of study variables for included law schools
Mean LSAT
Mean UGPA
Mean FYA
SD
SD
SD
Race/Ethnicity
Schools
Mean
Mean
Mean
Asian American
157
156.54
5.68
3.30
0.22
47.07
2.13
Black
153
150.54
4.21
3.20
0.19
42.17
3.18
Latino
151
154.26
4.81
3.30
0.20
45.70
2.41
White
178
157.59
5.35
3.41
0.18
51.50
0.78
Note: For the 2008 and 2009 entering classes, the Hispanic and Mexican American categories are
combined and displayed as “Latino.” For the 2010 entering class, the Hispanic/Latino category is
displayed as “Latino,” and the new categories of Native Hawaiian/Other Pacific Islander and Asian are
combined and displayed as “Asian American.”
Figure 1a presents the differences between mean LSAT score for Black and White
students. To analyze the data presented in this figure, the mean LSAT score was first
calculated separately for each subgroup at each school. The difference between the
means for the two subgroups being compared (i.e., LSAT mean for White students
minus LSAT mean for Black students) was then determined. Figure 1a summarizes the
number of schools displaying each LSAT mean difference observed. Figures 1b and 1c
present similar analyses of mean differences in UGPA and FYA for the Black and White
student subgroups, respectively. Figures 1a–1c reveal that White students tend to
outperform Black students on each of the predictor variables (LSAT score and UGPA)
and on the criterion variable (FYA in law school).
10
FIGURE 1a. Frequency distribution of differences between LSAT means for the Black and White student
subgroups at the participating law schools
FIGURE 1b. Frequency distribution of differences between UGPA means for the Black and White student
subgroups at the participating law schools
11
FIGURE 1c. Frequency distribution of differences between FYA means for the Black and White student
subgroups at the participating law schools
Figures 2a–2c present similar analyses of the mean differences between the
performance of White and Latino law students on LSAT score, UGPA, and FYA,
respectively. The pattern of LSAT score, UGPA, and FYA differences observed for the
Latino subgroup is similar to, but less extreme than, those observed for the Black
subgroup.
12
FIGURE 2a. Frequency distribution of differences between LSAT means for the Latino and White student
subgroups at the participating law schools
FIGURE 2b. Frequency distribution of differences between UGPA means for the Latino and White student
subgroups at the participating law schools
13
FIGURE 2c. Frequency distribution of differences between FYA means for the Latino and White student
subgroups at the participating law schools
Figures 3a–3c present results comparing the performance of White and Asian
American law students on these variables. Differences reported for the Asian American
subgroup tend to be smaller, but in the same direction as those reported for the other
two minority subgroups.
FIGURE 3a. Frequency distribution of differences between LSAT means for the Asian American and
White student subgroups at the participating law schools
14
FIGURE 3b. Frequency distribution of differences between UGPA means for the Asian American and
White student subgroups at the participating law schools
FIGURE 3c. Frequency distribution of differences between FYA means for the Asian American and White
student subgroups at the participating law schools
15
Predictive Validity
The relationships between the predictor variables (LSAT score and UGPA) and the
criterion variable (FYA) are measured through the computation of correlation
coefficients. Correlation coefficients can range in value from −1 to 1, where 1 represents
a perfect positive linear relationship. For each school with a sufficient number of
racial/ethnic minority students, the correlation coefficients for each subgroup were
calculated separately by law school and averaged for all schools included in the
comparison sample. Table 6 provides the correlations between the predictors—both
alone and in combination—and FYA. For both racial/ethnic minority and nonminority
subgroups, the combination of LSAT score and UGPA produces higher correlations
than either predictor alone. The correlation coefficient for the combination of LSAT
score and UGPA is highest for the Latino subgroup, with the other subgroups yielding
slightly lower values. LSAT is a better predictor than UGPA for all subgroups. While
these results are lower than those reported for the 2011 LSAT Correlation Studies, the
ranges of scores and grades for the subgroups are narrower than for the complete firstyear classes. This restriction of range contributes to understated correlation coefficients
for these subsets of data.
TABLE 6
Correlations of LSAT score, UGPA, and the combination of LSAT score and UGPA with FYA by
racial/ethnic subgroup
Correlation
Race/Ethnicity
Schools
Students
LSAT
UGPA
LSAT & UGPA
Asian American
157
8,176
0.26
0.19
0.41
Black
153
7,367
0.30
0.17
0.43
Latino
151
6,520
0.31
0.17
0.45
White
178
77,830
0.26
0.24
0.40
Note: For the 2008 and 2009 entering classes, the Hispanic and Mexican American categories are
combined and displayed as “Latino.” For the 2010 entering class, the Hispanic/Latino category is
displayed as “Latino,” and the new categories of Native Hawaiian/Other Pacific Islander and Asian are
combined and displayed as “Asian American.”
Predicting First-Year Averages
The primary research question addressed by this study was whether or not LSAT
score alone, UGPA alone, and the combination of both predictor variables differentially
predicted FYA for racial/ethnic minority law school students as compared to nonminority
law school students. One method to address this question is to determine how
accurately LSAT score, UGPA, and a combination of both predictor variables predict
performance in law school for various racial/ethnic subgroups. Figures 4–6 examine the
differences between predicted and actual FYAs for the Asian American, Black, Latino,
and White subgroups. Using least-squares regression, separate equations were derived
to predict law school FYA for the total group of law school students within each
individual law school for LSAT score alone, UGPA alone, and the combination of both
predictor variables. Differences between predicted and actual FYA were then calculated
for each subgroup based on each regression equation. For the difference calculation,
the mean actual FYA earned by students at a participating school was subtracted from
16
the mean predicted FYA for students at the school. A resulting negative difference
indicates that the regression equation underpredicted the mean (or average)
performance of a subgroup in a law school, while a positive value indicates that the
regression equation overpredicted the mean performance of a subgroup in a law school.
Because each school has a distinct grading scale, a conversion was made to allow
comparisons across law schools and to preserve the confidentiality of the school-level
data. FYAs were converted to a scale where the mean for the total group of students at
each school was set to 50 and the standard deviation to 10. Distributions of the
differences between predicted and actual FYA means are presented graphically for
each prediction equation for each subgroup. To further summarize these data, the
weighted average of the mean residuals between predicted and actual FYA for each
prediction equation/subgroup combination is provided in Table 7.
TABLE 7
Summary of mean residuals between predicted and actual FYA by racial/ethnic
minority subgroup
Race/Ethnicity
LSAT
UGPA
LSAT & UGPA
Asian American
2.36
2.15
1.80
Black
2.52
5.55
0.98
Latino
1.83
3.46
1.16
Note: For the 2008 and 2009 entering classes, the Hispanic and Mexican American
categories are combined and displayed as “Latino.” For the 2010 entering class,
the Hispanic/Latino category is displayed as “Latino,” and the new categories of
Native Hawaiian/Other Pacific Islander and Asian are combined and displayed as
“Asian American.”
Figures 4a–4c show the differences between predicted and actual FYA means for
White and Black law students using LSAT score alone, UGPA alone, and the
combination of LSAT score and UGPA as the predictor variables, respectively. Figure
4a reveals that when FYA is estimated from a regression equation based on data for all
students, LSAT alone emerges as a fairly accurate predictor of law school performance
for the White subgroup, but tends to slightly overpredict the performance of Black
students on average. Table 7 shows that the mean residual for the Black subgroup
using LSAT alone as the predictor in this study is 2.52 points, indicating a modest
amount of overprediction on average. Figure 4b reveals that UGPA alone overpredicts
the performance of Black law school students to a greater extent than LSAT alone does
and slightly underpredicts the performance of White law students in this study. The
mean residual observed for the Black subgroup using UGPA alone as the predictor is
5.55 points. Finally, the prediction equation combining LSAT score and UGPA (see
Figure 4c) resulted in the most accurate prediction of law school performance for Black
students among the prediction equations studied here. The mean residual observed
here for the Black subgroup using the combination of LSAT score and UGPA as the
predictor variables is 0.98 points. This indicates that, on average, the overprediction
produced by this equation is very slight.
17
FIGURE 4a. Frequency distributions of differences between predicted and actual FYA means for Black
and White student subgroups at participating law schools using LSAT as the predictor variable
FIGURE 4b. Frequency distributions of differences between predicted and actual FYA means for Black
and White student subgroups at participating law schools using UGPA as the predictor variable
18
FIGURE 4c. Frequency distributions of differences between predicted and actual FYA means for Black
and White student subgroups at participating law schools using LSAT score and UGPA as the predictor
variables
The results for the Latino subgroup, presented in Figures 5a–5c, reveal a pattern
similar to the results observed for the Black subgroup, though performance of this
subgroup is overpredicted by the single predictors to a lesser extent than that observed
for the Black students. The mean residuals observed for the Latino subgroup using
LSAT alone, UGPA alone, and the combination of LSAT score and UGPA as predictor
variables were 1.83, 3.46, and 1.16, respectively (see Table 7). As was the case with
the results reported for the Black subgroup, the combination of LSAT score and UGPA
emerges as the most accurate predictor of law school performance for Latino students,
and the use of UGPA alone to predict FYA results in the most overprediction among the
equations studied here.
19
FIGURE 5a. Frequency distributions of differences between predicted and actual FYA means for Latino
and White student subgroups at participating law schools using LSAT as the predictor variable
FIGURE 5b. Frequency distributions of differences between predicted and actual FYA means for Latino
and White student subgroups at participating law schools using UGPA as the predictor variable
20
FIGURE 5c. Frequency distributions of differences between predicted and actual FYA means for Latino
and White student subgroups at participating law schools using LSAT score and UGPA as the predictor
variables
Predictive validity results for the Asian American subgroup are presented in Figures
6a–6c. The mean residuals observed for the Asian American subgroup were 2.36 for
the equation using LSAT alone as the predictor, 2.15 for the equation using UGPA
alone as the predictor, and 1.80 for the equation combining LSAT score and UGPA to
predict FYA (see Table 7). These results differ from the pattern observed for the Black
and Latino student subgroups in that the use of LSAT alone results in the most
overprediction of FYA for the Asian American subgroup. Again, the overprediction
observed for all three prediction equations is slight, on average.
FIGURE 6a. Frequency distributions of differences between predicted and actual FYA means for Asian
American and White student subgroups at participating law schools using LSAT as the predictor variable
21
FIGURE 6b. Frequency distributions of differences between predicted and actual FYA means for Asian
American and White student subgroups at participating law schools using UGPA as the predictor variable
FIGURE 6c. Frequency distributions of differences between predicted and actual FYA means for Asian
American and White student subgroups at participating law schools using LSAT score and UGPA as the
predictor variables
22
Conclusions
This study analyzed data from 178 law schools, each of which enrolled 10 or more
first-year students who identified themselves as a member of one of three racial/ethnic
minority subgroups—Asian American, Black, or Latino—and had 10 or more first-year
students who identified themselves as a member of the nonminority subgroup. The
present study, like earlier studies of its kind (Anthony & Liu, 2003; Norton et al., 2006,
2009; Stilwell & Pashley, 2003; Wightman & Muller, 1990), was conducted to determine
whether there is evidence of differential prediction for members of different racial/ethnic
subgroups. More importantly, results of this study were evaluated to determine whether
the current practices used to predict law school performance are unfair to certain
racial/ethnic minority subgroups. Use by admission committees of a regression equation
that systematically excludes members of some racial/ethnic minority subgroup by
underpredicting the performance of its members or overpredicting the performance of
some other subgroup could result in admission decisions that are unfair to certain
racial/ethnic minority subgroup members.
Regression equations derived by combining data for all four racial/ethnic subgroups
included in this study were used to evaluate differential prediction of law school FYA
when LSAT alone, UGPA alone, and the combination of LSAT score and UGPA were
used as predictors. The use of UGPA alone as a predictor variable seemed to produce
the most differential prediction for two of the three racial/ethnic minority subgroups
studied. The exception was the Asian American subgroup, in which the use of LSAT
alone produced slightly higher overprediction than did the use of UGPA alone. The data
confirm that using the combination of LSAT score and UGPA produces the least amount
of differential prediction, as compared to using either predictor variable alone for each of
the subgroups studied. In fact, the overprediction observed when applying this equation
was very small.
While the use of LSAT score and UGPA in combination resulted in the most
accurate prediction of first-year law school performance, it is worth noting that none of
the regression equations studied would serve to systematically exclude individual
members of the three racial/ethnic minority subgroups studied here. Performance of the
racial/ethnic minority students tended to be slightly overpredicted on average rather
than underpredicted for all three of the regression models evaluated. The performance
of White students was slightly underpredicted by the use of UGPA alone, but this
difference was greatly diminished in the regression equation combining LSAT score and
UGPA.
At least two caveats should be remembered when evaluating the results of this
study. First, differences only in average predicted performance were analyzed. That is,
individuals within a subgroup that is overpredicted on average may still be themselves
underpredicted in terms of their individual law school performance. Second, differential
prediction is only one aspect of an overall construct validity evaluation.
23
References
Anthony, L. C., & Liu, M. (2003). Analysis of differential prediction of law school
performance by racial/ethnic subgroups based on the 1996–1998 entering law
school classes (LSAC Technical Report 00-02). Newtown, PA: Law School
Admission Council, Inc.
Braun, H., & Jones, D. (1981). The Graduate Management Admission Test prediction
bias study (Graduate Management Admission Council, Report No. 81-04, and ETS
RR-81-25). Princeton, NJ: Educational Testing Service.
Breland, H. (1979). Population validity and college entrance measures (Research
Monograph No. 8). New York: College Entrance Examination Board.
Burton, N. W., & Wang, M. (2005). Predicting long-term success in graduate school: A
collaborative validity study (Graduate Records Examination Board Research Report
No. 99-14R and Educational Testing Service, RR-05-03). Princeton, NJ: Educational
Testing Service.
Houston, W. M., & Novick, M. R. (1987). Race-based differential prediction in Air Force
technical training programs. Journal of Educational Measurement, 24, 309–320.
Law School Admission Council and the American Bar Association Section of Legal
Education and Admissions to the Bar. (2010). ABA-LSAC official guide to ABAapproved law schools (2011 ed.). Newtown, PA: Law School Admission Council, Inc.
Linn, R. L., & Hastings, C. N. (1984). Group differentiated prediction. Applied
Psychological Measurement, 8, 165–172.
Mattern, K. D., Patterson, B. F., & Kobrin, J. L. (2012). The validity of SAT® scores in
predicting first-year mathematics and English grades (College Board Research
Report 2012-1). New York, NY: The College Board.
Mattern, K. D., Patterson, B. F., Shaw, E. J., Kobrin, J. L., & Barbuti, S. M. (2008).
Differential validity and prediction of the SAT® (College Board Research Report
2008-4). New York, NY: The College Board.
National Research Council. (1989). Differential validity and differential prediction. In J.
A. Hartigan & A. K. Wigdor (Eds.), Fairness in employment testing: Validity
generalization, minority issues, and the general aptitude test battery (pp. 172–188).
Washington, DC: National Academy Press.
Noble, J. (2003). The effects of using ACT composite score and high school average on
college admission decisions for racial/ethnic groups (ACT Research Report 200301). Iowa City, IA: American College Testing.
24
Noble, J., Crouse, J., & Schulz, M. (1996). Differential prediction/impact in course
placement for ethnic and gender groups (ACT Research Report 96-8). Iowa City, IA:
American College Testing.
Norton, L., Suto, D., & Reese, L. (2006). Analysis of differential prediction of law school
performance by racial/ethnic subgroups based on 2002–2004 entering law school
classes (LSAC Technical Report, TR 09-02). Newtown, PA: Law School Admission
Council, Inc.
Norton, L., Suto, D., & Reese, L. (2009). Analysis of differential prediction of law school
performance by racial/ethnic subgroups based on 2005–2007 entering law school
classes (LSAC Technical Report, TR 09-02). Newtown, PA: Law School Admission
Council, Inc.
Ramist, L., Lewis, C., & McCamley-Jenkins, L. (1994). Student group differences in
predicting college grades: Sex, language, and ethnic groups (College Board, CBR93-01, and Educational Testing Service, RR-94-27). Princeton, NJ: Educational
Testing Service.
Powers, D. E. (1977). Comparing predictions of law school performance for Black,
Chicano, and White law students (LSAC Research Report, RR 77-3). Newtown, PA:
Law School Admission Services, Inc.
Sackett, P. R., Kuncel, N. R., Beatty, A. S., Rigdon, J. L., Shen, W., & Kiger, T. B.
(2012). The role of socioeconomic status in SAT-grade relationships and in college
admission decisions. Psychological Science, 23(9), 1000–1007.
Schmidt, F. L., & Hunter, J. E. (1981). Employment testing: Old theories and new
research findings. American Psychologist, 36, 1128–1137.
Schrader, W. B., & Pitcher, B. (1976a). Predicting law school grades for Black American
law students (LSAC Research Report, RR 73-6). Princeton, NJ: Law School
Admission Council, Inc.
Schrader, W. B., & Pitcher, B. (1976b). Prediction of law school grades for Mexican
American and Black American students (LSAC Research Report, RR 74-8).
Princeton, NJ: Law School Admission Council, Inc.
Shaw, E. J., Kobrin, J. L., Patterson, B. F., & Mattern, K. D. (2012). The validity of the
SAT® for predicting cumulative grade point average by college major (College Board
Research Report 2012-6). New York, NY: The College Board.
Stilwell, L. A., & Pashley, P. J. (2003). Analysis of differential prediction of law school
performance by racial/ethnic subgroups based on the 1999–2001 entering law school
classes (LSAC Technical Report, TR 03-03). Newtown, PA: Law School Admission
Council, Inc.
25
Stilwell, L. A., Reese, L. M., & Pashley, P. J. (1998). Analysis of differential prediction
of law school performance by racial/ethnic subgroups (LSAC Technical Report,
TR 98-02). Newtown, PA: Law School Admission Council, Inc.
Stricker, L. J., Rock, D. A., & Burton, N. W. (1991). Sex differences in SAT predictions
of college grades (College Board, CBR-91-02, and Educational Testing Service, RR91-38). Princeton, NJ: Educational Testing Service.
Wightman, L. F., & Muller, D. G. (1990). An analysis of differential validity and
differential prediction for Black, Mexican American, Hispanic, and White law school
students (LSAC Research Report, RR 90-03). Newtown, PA: Law School Admission
Council, Inc.
Willingham, W. W., Lewis, C., Morgan, R., & Ramist, L. (1990). Predicting college
grades: An analysis of institutional trends over two decades. Princeton, NJ:
Educational Testing Service.
26