Partnership and Pedagogy: Admissions, Assessment, and Curricular

Partnership and pedagogy
Admissions, assessment, and curricular reform
f improving
for
i
i diversity
di
i in
i STEM
Dorian A. Canelas, Ph.D.
Department
p
of Chemistry,
y, Duke University
y
Jennifer Hill, Ed.D.
Office of Assessment, Duke University
AAC&U  Atlanta  November 2014
Should we aim to teach to the
average or to the students
average,
at the edges?
Should we aim to teach to the
average or to the students
average,
at the edges?
Nobody is average in every
di
i relevant
l
t tto science
i
dimension
learning!
Credit: Todd Rose
http://www.personalizelearning.com/2013/07/design-learning-from-extremes.html
An analogy:
Fighter jet cockpit design
Credit: Todd Rose, The Myth of Average
http://planetoddity.com/a-peek-inside-fighter-jet-cockpits/
An analogy:
But what if the seats and control locations were more
adjustable, designed “to the edges” of human anatomy
norms, so that a wider range of people could fit properly
and reach the controls?
Credit: Todd Rose, The Myth of Average
http://planetoddity.com/a-peek-inside-fighter-jet-cockpits/
What’s the issue?
General, national consensus that students with less preparation in STEM in
secondary schools have lower rates of persistence in STEM programs in
college.
ll
•
The highest rate of loss of student interest in STEM coursework
occurs during the first year of college.1,2
•
When a student changes his or her mind about pre-health study,
chemistry is cited 4 to 5 times more often than the next two highest
contributors (biology and math) as the subject that contributes to this
decision.3
1. P. A. Daempfle, An Analysis of the High Attrition Rates Among First Year College Science, Math, and Engineering Majors. J. Coll. Student
Retention 5, 37 (2004).
2. E. Seymour, Why Undergraduates Leave the Science. Am. J. Phys. 63, 199 (1995).
3. D. A. Barr, J. Matsui, S. F. Wanat, M. E. Gonzalez, Chemistry Courses as the Turning Point for Premedical Students. Adv. Health Sci. Educ.
15, 45 (2010).
What’s the issue?
Positive predictors of success in the first semester of
college General Chemistry
1
2
3
4
5
6
Relatively high SAT-Math score
Relatively high SAT-Verbal score
Completion of AP Calculus (AB) in high school
Completion of AP Calculus (BC) in high school
Relatively high last high school science grade
p
of high
g school Chemistryy with relatively
y high
g
Completion
emphasis on stoichiometry
7
Completion of AP Chemistry in high school and/or
completion of high school Chemistry course with relatively strong
emphasis on full understanding as opposed to memorization*
* As reported by students in surveys
Summary of data analysis of 1333 students at 12 different universities.
Study by Tai, Sadler & Loehr (2005) J. Res. Sci. Teaching, 42(9), 987-1012.
What’s the issue?
“It
It is clear that if a student has an SAT of 550,
550 we
can assure him that he isn’t going to get an A.”
Miles Pickering
Pickering, Columbia University
Pickering, J. Chem. Educ. 1975, 52(8), 512-514.
“Students with SAT [math] scores under 600
will probably do very poorly...
poorly ”
Marjorie and Lester Andrews, University of Virginia, 1979
“A high SAT-M score does not guarantee a
good grade, but a low score is a strong
indicator of a low grade...”
Harry Spencer, Oberlin College, 1996
Student attrition from STEM
Student attrition from STEM
Central question:
Central question:
Who is unprepared: the students . . . or us?
h i
d h
d
?
Chem31L Study Groups
9
For discussion, part 1
Ability to
Ability to…
Solve quantitative STEM problems
Design an independent project
Conduct a literature search productively
Work collaboratively with a team
Work collaboratively with a team
Where are your students?
students? (Preparation)
Where is your institution?
institution? (Readiness)
For discussion, part 2
Ability to
Ability to…
Maintain an effective lab notebook
Produce a thorough lab report
Interpret and use both quantitative and qualitative information
Give a scientific presentation
Where are your students?
students? (Preparation)
Where is your institution?
institution? (Readiness)
What’s the issue? (At Duke)
Duke Chemistry faculty also observed that students in lowest
quartile
il off SAT-Math
SAT M h scores hhave llowest outcomes (i
(i.e., grades,
d
persistence).
Findings for General Chemistry 1, Fall 2008
Students with a Math SAT score in lowest quartile of their matriculating class:
•
•
•
•
465 students enrolled in g
general chemistry:
y pperformance vs. various factors
analyzed using a multiple regression analysis.
~15% of these students earned grade of A or B
received four times as many grades of D or F as the rest of the class
Comprised 35% of the enrolled students, but accounted for 81% of the D or
F grades
Source: Office of Assessment, Trinity College
“Exploratory Analysis of Likely Predictors of Chem21L Grade, Including the Diagnostic Placement Quiz,”
April 16, 2009.
Key question
How can colleges and universities keep an open gateway to
the science disciplines for the least experienced first year
science students while also maintaining high standards for
all students that challenge the students with the strongest
possible high school backgrounds?
Curriculum innovation
Duke’s response
Goal:
Provide real opportunity
pp
y for success in STEM disciplines
p
(g
(going
g
beyond access)
Challenge:
Students with standardized test scores in the bottom quartile of their
matriculating class have historically fared poorly in gateway science
courses at Duke (low grades,
grades high attrition)
Three-Pronged Strategy:
1) New chemistry course sequence
2) SAGE study groups rather than one-on-one peer tutors
3) Cardea Fellowship Program
Duke’s response: New Chem. course sequence
Course Placement starting fall 2009
First year students have 5 options for initial chemistry placement each fall:
Chem91
No
N
Lab
Chemistry, Technology, and Society
No additional chemistry intended
Chem99D
Introduction to Chemistry
(less than 1 year of HS chem OR SAT Math < 630)
Preparation for Chem101L
Chem101DL
General Chemistry 1: Core Concepts
(1 year of HS chem; no AP or AP < 3)
Chem110DL
Honors General Chemistry
(AP = 4 or 5)
Chem201DL
g
Chemistry
y1
Organic
(AP = 5)
Chem 99D: Course design
• One 75 minute “lecture” per week (Mondays)
• Two 50 minute small ggroup
p discussions per
p week ((Wed. & Fri.))
• Team-based, active learning activities (POGIL, SCALE UP,
PBL)
bl M
i l ti M
th d
• P
Problem
Manipulation
Method
17
Retention results
96
100
81
97
41
59
67
78
50
65
Hall, Curtin-Soydan, and Canelas, J. Chem. Educ. 2014, 91(1), 37-47.
Duke’s response
Develop a framework for the ongoing longitudinal assessment of
student outcomes in chemistry,
y, asking:
g
What is it about chemistry or the way it is taught in college, in
particular, that drives this sort of change in career plans?
Discussion
Institutional resource
What do you already know?
What kind of data would be helpful to explore STEM outcomes?
Who has access to student‐level
Who
has access to student level data at data at
your institution? Who has the resources (software, expertise) to run descriptive and/or inferential statistics on student outcomes?
At Duke
We developed a framework for ongoing longitudinal assessments of
student outcomes in chemistry,
y, asking:
g
What is it about chemistry or the way it is taught in college, in
particular, that drives this sort of change in career plans?
Original hypotheses: adapting the curriculum and sequence of courses
i foundational
in
f
d ti l Chemistry
Ch i t courses will:
ill
a)
be associated with increased retention of students in Chemistry,
STEM, and pre-health tracks, and
b) cause students in the bottom quartile of Math SAT scores to earn
higher grades in Organic Chem 1.
A longitudinal methodology
At Duke
Quick notes:
Q
• Institutional collaborators
• Data sources
p (graduating
(g
g cohorts))
• Studyy sample
• Operationalizing pre-STEM, pre-Health
Study methodology: Changes in enrollment
Original hypotheses: adapting the curriculum and sequence of courses
in foundational Chemistry courses will be associated with increased
Percent o
of enrollments
retention of students in Chemistry, STEM, and pre-health tracks.
20.0%
18.0%
16 0%
16.0%
14.0%
12.0%
10.0%
8 0%
8.0%
6.0%
4.0%
2.0%
0.0%
Student enrollment in Chem foundation courses
Student
enrollment in Chem foundation courses
As proportion of all enrollments in the term
Grads 2012‐
2014: Pre‐
STEM students in lowest quartile of SAT‐
M
Grads 2007‐
2009: Pre
2009: Pre‐
STEM students in lowest quartile of SAT‐
M
term1
term2
term3
term4
term5
term6
term7
term8
Study methodology: Persistence after withdrawal
Original hypotheses: adapting the curriculum and sequence of courses
in foundational Chemistry courses will be associated with increased
retention of students in Chemistry, STEM, and pre-health tracks.
Rate of successful course completion after prior course withdrawal
100%
90%
Pre‐STEM students in lowest quartile SAT‐M:
quartile SAT
M: 2007‐2009
(cohort N = 139)
80%
70%
60%
50%
Pre‐STEM Pre
STEM
students in lowest quartile SAT‐M: 2012‐2014 (cohort N 114)
(cohort N=114)
40%
30%
20%
10%
0%
Gen Chem I
Gen Chem II
Organic Chem I
Organic Chem II
Study methodology
Original hypotheses: adapting the curriculum and sequence of courses
in foundational Chemistry courses will:
a)
be associated with increased retention of students in Chemistry,
STEM, and pre-health tracks, and
b) cause students in the bottom quartile of Math SAT scores to earn
higher grades in Organic Chem 1.
Study methodology: Grades in later courses
Original hypotheses: adapting the curriculum and sequence of courses
in foundational Chemistry courses will cause students in the bottom
quartile of Math SAT scores to earn higher grades in Organic Chem 1.
4.00
Differences in Chemistry course grades, By initial course in Chem. sequence (Students in lowest quartile SAT‐Math)
Students who started Chem. Sequence with General Chem I: 2007‐2009 (N=168)
3.50
3 00
3.00
2.50
Students who started Chem. Sequence with General Chem I: 2012‐2014 (N=78)
2.00
1.50
Students who started Chem. Sequence with Intro. Chem: 2012‐2014 (N=73)
1.00
0 50
0.50
0.00
General Chem I
General Chem II
Organic Chem I
Organic Chem II
40
Study methodology: Graduation outcomes
Outcomes at Graduation: Latin Honors
Percen
nt of graduating cohort
30 0%
30.0%
25.0%
20 0%
20.0%
15.0%
10 0%
10.0%
5.0%
0.0%
25.3%
25.7%
9.9%
5.9%
5.4%
10.3%
17.8%
2007‐2009
2012‐2014
2007‐2009
2012‐2014
2007‐2009
2012‐2014
2012‐2014
All graduates
Graduates in lowest quartile SAT‐M Graduates in lowest quartile Graduates in SAT‐M who started in Gen. lowest quartile Chem. I
SAT‐M who started in Intro
started in Intro. Chem
41
Study methodology: Graduation outcomes
Outcomes at Graduation: Merit Scholarships
Percen
nt of graduating cohort
30 0%
30.0%
25.0%
20 0%
20.0%
15.0%
10 0%
10.0%
5.0%
0.0%
6.2%
9.7%
2.7%
5.9%
4.8%
2.6%
9.6%
2007‐2009
2012‐2014
2007‐2009
2012‐2014
2007‐2009
2012‐2014
2012‐2014
All graduates
Graduates in lowest quartile SAT‐M Graduates in lowest quartile Graduates in SAT‐M who started in Gen. lowest quartile Chem. I
SAT‐M who started in Intro
started in Intro. Chem
42
Study methodology: Graduation outcomes
Outcomes at Graduation: Research courses as proportion of total courseload
Research courses as proportion of total courseload
Perce
ent of total courrse load
30.0%
25.0%
20.0%
15.0%
10.0%
5.0%
0.0%
8.6%
11.1%
9.3%
12.6%
9.0%
11.1%
10.1%
2007‐2009
2012‐2014
2007‐2009
2012‐2014
2007‐2009
2012‐2014
2012‐2014
All graduates
Graduates in lowest quartile SAT‐M Graduates in lowest quartile Graduates in SAT‐M who started in Gen. lowest quartile Chem. I
SAT‐M who started in Intro
started in Intro. Chem
43
Study methodology: Graduation outcomes
Outcomes at Graduation: Grad. with Distinction
Perce
ent of graduating cohort
30.0%
25.0%
20.0%
15.0%
10.0%
5.0%
0.0%
19.7%
24.0%
12.4%
14.3%
17.9%
20.5%
11.0%
2007‐2009
2012‐2014
2007‐2009
2012‐2014
2007‐2009
2012‐2014
2012‐2014
All graduates
Graduates in lowest quartile SAT‐M Graduates in lowest quartile Graduates in SAT‐M who started in Gen. lowest quartile Chem. I
SAT‐M who started in Intro. Chem
44
Study methodology: Academic plans
Chemistry in graduates' academic plans:
First majors
First majors
50%
P
Percent of stud
ents
45%
40%
35%
30%
25%
20%
15%
10%
7%
5%
5%
4%
4%
4%
2012‐2014
2007‐2009
2012‐2014
2012‐2014
0%
2007‐2009
All graduates indicating pre‐STEM at matriculation
Lowest quartile SAT‐M graduates indicating Graduates who started pre‐STEM at matriculation
Chem sequence with Intro. Chem
45
Study methodology: Academic plans
Chemistry in graduates' academic plans:
First minors
First minors
50%
P
Percent of stud
ents
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
23%
22%
17%
25%
37%
2007‐2009
2012‐2014
2007‐2009
2012‐2014
2012‐2014
All graduates indicating pre‐STEM at matriculation
Lowest quartile SAT‐M graduates indicating Graduates who started pre‐STEM at matriculation
Chem sequence with Intro. Chem
47
Study methodology: Academic plans
Chemistry in graduates' academic plans:
Including Chemistry as major OR minor
Including Chemistry as major OR minor
50%
P
Percent of stud
ents
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
34%
32%
27%
33%
44%
2007‐2009
2012‐2014
2007‐2009
2012‐2014
2012‐2014
All graduates indicating pre‐STEM at matriculation
Lowest quartile SAT‐M graduates indicating Graduates who started pre‐STEM at matriculation
Chem sequence with Intro. Chem
49
Study methodology: Inferential stats
Original hypotheses: adapting the curriculum and sequence of courses
in foundational Chemistry courses will be associated with increased
retention of students in Chemistry, STEM, and pre-health tracks.
• Logistic regression model
• Sample:
Control (2007-2009 grad classes) and treatment (2012-2014 grad classes) cohorts, specifically
students who, at matriculation, indicated having academic and professional interests in STEM
• D.V.: Persistence operationalized as the binary variable:
Did student successfully complete the final recommended course in the sequence of Chemistry
foundations courses?
• Independent variables include
• Pre-matriculation markers from Office of Admissions
• Biodemographic information from Office of Admissions, College Board, etc.
• Academic data collected during tenure at Duke
Study methodology: Inferential stats
Original hypotheses: adapting the curriculum and sequence of courses
in foundational Chemistry courses will be associated with increased
retention of students in Chemistry, STEM, and pre-health tracks.
Changes in influence of selected factors (odds ratios): Completing Org. Chem I
• Logistic regression model
All students
• Sample:
Lowest qtr. SAT‐M
2007‐09
2012‐14
2007‐09
2012‐14
0 695
0.695
0 751
0.751
0 456
0.456
0 506
0.506
1.093
Non‐sig
Non‐sig
Control (2007-2009 grad classes) and treatment (2012-2014 grad classes) cohorts, specifically
URM (1=URM)
Non‐sig
Non‐sig
2.014
3.692
students who, at matriculation, indicated having academic and professional interests in STEM
G d (1 M l )
Gender (1=Male)
starting Chem.
•Term of
D.V.:
Persistence operationalized as the binary 1.532
variable:
Did student successfully complete the final recommended
in the
Avg. grade in all Duke Math courses prior to Chem. sequence
1.565 course1.393
sequence
of Chemistry
2.106
Non‐sig
foundations courses?
Number of Duke Math courses prior to Chem. sequence
1.009
0.981
1.578
5.081
Appropriate placement in first Chem. Course (1=correct placement)
6.742
5.872
15.776
15.685
Admissions Rating: Strength of H.S. curriculum
• Pre-matriculation markers from Office of Admissions
SAT‐Math 1.006
1.005 Board, etc.
1.007
• Biodemographic information from Office of Admissions,
College
Student
Student self‐identification as pre‐STEM (1=yes)
identification
as pre
STEM (1=yes)
4 935
10 357
10.357
29 525
29.525
• self
Academic
data
collected
during tenure at Duke 4.935
1.778
0.587
1.716
1.695
AP Math score (AB or BC)
1.600
1.071
1.368
1.235
AP Chem score
1.320
1.287
1.594
1.650
Started Chem. sequence in Intro Chem. (1=yes)
N/A
2.681
N/A
14.353
• Independent variables include
1.005
21 680
21.680
Study methodology: Inferential stats
Original hypotheses: adapting the curriculum and sequence of courses
in foundational Chemistry courses will cause students in the bottom
quartile of Math SAT scores to earn higher grades in Organic Chem 1.
• OLS regression
g
model
• Sample:
Control (2007-2009 grad classes) and treatment (2012-2014 grad classes) cohorts, specifically
students who, at matriculation, indicated having academic and professional interests in STEM
• D.V.:
Grade in first semester Organic Chemistry, on 4-point scale
• Independent
p
variables include
• Pre-matriculation markers from Office of Admissions
• Biodemographic information from Office of Admissions, College Board, etc.
• Academic data collected during tenure at Duke
Study methodology: Inferential stats
Original hypotheses: adapting the curriculum and sequence of courses
in foundational Chemistry courses will cause students in the bottom
quartile of Math SAT scores to earn higher grades in Organic Chem 1.
Changes in influence of selected factors (stb): Grade in Org. Chem I
• OLS regression
g
model ((model selection method: stepwise)
p
)
All students
• Sample:
2007‐09
2012‐14
Lowest qtr. SAT‐M
2007‐09
2012‐14
Control (2007-2009
URM (1=URM)
grad classes) and treatment (2012-2014
grad
classes) cohorts,
‐0.155
‐0.179
Non‐sigspecifically
Non‐sig
students
who, at matriculation, indicated havingg academic
professional
f
interests
Gender
Gender (1=Male)
(1=Male)
Non‐sig and p
Non‐sig
Non‐sig in STEM
Non‐sig
starting Chem.
‐0.085
Non‐sig
•Term of
D.V.:
Avg. grade in all Math courses prior to Chem. sequence
0.376
0.430
Grade in first semester Organic Chemistry, on 4-point
scale
Number of college Math courses prior to Chem. sequence
Non‐sig
Non‐sig
0.748
1.206
Non‐sig
Non‐sig
Non‐sig
Non‐sig
Non‐sig
Non‐sig
Non‐sig
Non‐sig
Office of Admissions
Non‐sig
‐0.215
Non‐sig
• Biodemographic information from Office of Admissions,
College
SAT‐Math 0.104
0.117 Board, etc.
Non‐sig
• Academic datapcollected
tenure at Duke Non‐sigg
( yduring
)
Non‐sigg
Non‐sigg
Student self‐identification as pre‐STEM (1=yes)
Non‐sig
AP Math score (AB or BC)
Non‐sig
Non‐sig
Non‐sig
Non‐sig
AP Chem score
Non‐sig
Non‐sig
Non‐sig
Non‐sig
Started Chem. sequence in Intro Chem. (1=yes)
Non‐sig
Non‐sig
Non‐sig
Non‐sig
•Appropriate placement in first Chem. Course (1=correct placement)
Independent variables include
• Pre-matriculation markers from
Admissions Rating: Strength of H.S. curriculum
Non‐sig
Non‐sigg
Study methodology: Next steps
Original hypotheses: adapting the curriculum and sequence of courses
further
study:courses will:
inFor
foundational
Chemistry
•
•
•
•
•
•
a)
be associated with increased retention of students in Chemistry,
Repeat
analyses by student biodemographic factors
STEM, and pre-health tracks, and
b) causemeasures
students inof
thestudent
bottom quartile
of Math SAT
scores toand
earn
Include
dispositions,
attitudes,
higher grades
in Organic
1.
behaviors,
including
but Chem
not limited
to integration of course
evaluation data;
Explore interaction effects among independent variables;
Resolve SAT/ACT dichotomy (i.e., as it pertains to missing
data);
Look at Medical School application as a dependent variable;
Look at major/minor as a dependent variable
Acknowledgements
Dean Alyssa Perz-Edwards, Ph.D.
Department of Biology
Academic Advising Center
Professor Richard MacPhail, Ph.D.
Department of Chemistry
Matt Serra, Ph.D.
Office of Assessment, Trinity College
Amanda Curtin-Soydan,
y
, Ph.D.
Donna Hall
Academic Resource Center
Discussion
What q
questions do yyou have about this study?
y
What are YOUR next steps for partnerships?