Beyond “Smile Sheets:” Measuring Student Learning in Advising

Beyond “Smile Sheets:” Measuring Student
Learning in Advising Appointments and Events
this is our story
1,800 appointments
80% advisor respected them
88% received timely email response
93% 1st years = advising is great
800 drop-ins
71% seniors = advising is great
Is this an effective advising office?
talk to your neighbor
1,800 appointments
80% advisor respected them
88% received timely email response
93% 1st years = advising is great
800 drop-ins
71% seniors = advising is great
Is this an effective advising office?
what did students learn?
(Bresciani, 2002)
not the complete picture
critical thinking
can read an academic
set goals
advisement report
decision making skills
how to ask for help
choose a major
articulate the purpose of their
liberal education requirements
how do we REALLY know?
forge a new path
move beyond “smile sheets”
(Kirkpatrick & Kirkpatrick, 2006)
(Kirkpatrick & Kirkpatrick, 2006)
learning
satisfaction
assess at the next level
before we jump in...
how students feel about services
93% of seniors rated advising as “good” or “excellent”
what students know, do and/or value
Students can articulate their degree requirements
smile sheets aren’t bad…
just not enough by themselves!
how do I assess student learning?
First Day
how might we measure learning?
let’s go back to the classroom…
interviews
instructor created exams
essays
observations
lab reports
standardized tests
informal class questioning
portfolio & projects
performance based tasks
peer evaluation
self evaluation
how could we do these sorts of things in advising?
before we
jump in…
http://thedailywaster.files.wordpress.com/201
1/10/sharkfin.jpg
Can students
accurately assess
what affects their
growth?
What does the research tell us?
(Bowman & Seifert, 2011; Bowman, 2001; Pascarella, 2001).
Assessment Strategies
advisor as expert
rubrics
direct vs. indirect
pre- and post-tests
advisor as
“expert”
rubric
(Stevens & Levi, 2005)
direct
collecting information that requires students
to display their knowledge and skills
(Palomba & Banta, 1999)
indirect
gathering information through means other
than looking at actual samples of student work
(Palomba & Banta, 1999)
The capital of the United States is:
a)
b)
c)
d)
Chicago, IL
Nashville, TN
San Diego, CA
Washington, DC
The capital of the United States is:
a)
b)
c)
d)
Chicago, IL
Nashville, TN
San Diego, CA
Washington, DC
If a student late to an advising
appointment travels at a 5 minute per
kilometer pace, how long would it
take the student to run a 5K race?
darn. short answer.
I don’t even25
know
minutes.
what a kilometer is.
http://3.bp.blogspot.com/_2bQ43yQwT0/TGYUTDwWeVI/AAAAAAAAHwM/jmD
OTbpXo0M/s1600/Cameron-Diaz+copy.jpg
pre- and post-tests
As a result of this appointment I…
what are they actually learning?
assessment rocks!
Dinner &
Degree
Planning
Workshop
Confidence in Degree Planning Tools
Increased After Appointment
4
3.65
3
3.6
2.7
2.17
Pre
2
Post
1
0
Graduation Plan
APAS Report
Average Assessment Score Increased
on Post-Test
100.0%
91.60%
90.0%
80.0%
70.0%
71.2%
60.0%
50.0%
Pre
40.0%
Post
30.0%
20.0%
10.0%
0.0%
Correct Answers
Freshman
Check-Backs
Learning
Outcomes
315 completed surveys
60%
What are the structured group study
sessions in the SMART Learning
Commons called?
75%
correct!
Which liberal education
requirement(s) does "HIST3429: Latin
American History in Film and Text"
fulfill?
33%
correct!
Which item(s) can be found on your
APAS report?
62%
correct!
post-appointment student assessment
hey, what about us?
100% of students identified campus
resources during the appointment.
98% of students discussed the
adjustment to college with an advisor.
97% of students discussed personal or
career goals with an advisor.
96% of students received helpful
feedback on their graduation plan.
98% of students will be comfortable
contacting their academic advisor in
the future.
What could be changed to improve the
freshman check-back appointment?
120
100
101
80
60
40
20
18
15
15
14
9
9
Structure
Logistics
0
Nothing More time Scheduling Location
Advising
post-appointment advisor assessment
Student vs. Advisor Assessment:
Locating Campus Resources
4
3.11
3.16
Student (Pre)
Student (Post)
3
2.93
2
1
0
Advisor
Average score on pre-assessment: 62%
Student vs. Advisor Assessment:
Academic Program Requirements
4
3.64
3.12
2.92
3
2
1
0
Student (Pre)
Student (Post)
Advisor
Average score on pre-assessment: 55%
Student vs. Advisor Assessment: APAS
Report
4
3.7
3.26
3.03
3
2
1
0
Student (Pre)
Student (Post)
Advisor
Average score on pre-assessment: 61%
Academic Support Resources
250
200
150
Not familiar
Heard of it
100
Used it
50
0
SMART LC
Writing Center
MCAE
SASS
implement in your unit
Now it’s your turn to
think about this!
Yes, there is one more activity
before you get to leave…
Get advice from your neighbor
Lisa Novack
[email protected]
Nikki Letawsky Shultz
[email protected]
References
Bowman, N. (2001). Validity of college self-reported gains at diverse institutions.
Educational Researcher, 40, 22-24.
Bowman, N., & Seifert, T. (2011). Can college students accurately assess what
affects their learning and development? Journal of College Student
Development, 52 (3), 270-290.
Bresciani, M.J. (2002, September). The relationship between outcomes,
measurement. and decisions for continuous improvement. National
Association for Student Personnel Administrators, Inc NetResults EZine.
http://www.naspa.org/netresults/index.cfm.
Kirkpatrick, D. L. and Kirkpatrick J.D. (2006). Evaluating training programs (3rd ed.).
San Francisco, CA: Berrett-Koehler Publishers
Palomba, C & Banta T. (1999). Assessment essentials: Planning, implementing, and
improving assessment in higher education. San Francisco: Jossey Bass.
Pascarella, E. (2001). Using self-reported gains to estimate college impact: A
cautionary tale. Journal of College Student Development, 42(5), 488-492.
Stevens, D. D., & Levi, A. J. (2005). Introduction to rubrics: An assessment tool to
save grading time, convey effective feedback and promote student
learning. Sterling, VA; Stylus.