Learning Analytics – Future possibilities and current

Learning Analytics – Future possibilities
and current limitations.
Examples from Norway, Denmark and Singapore.
Conexus - Release knowledge!
Founded in 2001
100 employees'
Software and content
•
•
•
•
•
We work in:
Norway
Denmark
Singapore
Sweden from 2014
• 250 municipality
costumers
• The Norwegian
ministry of education
• The Norwegian
Directorate for
Education and
Training
• Norwegian research
consul
• Innovation Norway
Funded by:
Concept partners
Norway
• IMTEC
• Gyldendal
• Kommuneforlaget (KS)
• mYouTime
• Microsoft
International
• KMD Education (Denmark)
• Conexus Orient (Singapore)
• Orient Software (Vietnam)
• Knewton (New York)
• IST (Sweden)
In Norway
• The national government, all students and teachers.
• 80% marked
• All big publishers
Evidence Informed Quality Work (EIQW)
The main purpose of the project is to develop and test
processes and technology that supports quality work
based on updated evidence.
Over 30 school and preschool owners participate, together with 100
organizations (preschools, primary schools, upper secondary schools).
Content analyzed in the examples, form:
•
•
•
•
•
•
•
•
•
•
Norwegian government
All SAS providers in Norway
SSB
Gyldendal (publisher)
Cappelen (publisher)
Aschehoug (publisher)
KIKORA
National Center for Mathematics
National Center for Reading
20 Other content providers
Academic feedback from:
•
•
•
•
•
•
•
Erling Lars Dale
Knut Roald
Peter Mortimore
Louise Stoll
Kjell B. Hjertø
Andy Hargreaves
Viviane Robinson
Learning analytics has to develop:
•
•
•
•
•
•
•
•
Learning for all students
Professional Capital
Assessment for learning
Professional learning
communities
21 CENTURY capacities
Evidence informed processes
Student Centered leadership
Open To Learn
Communication
The strategy to implement Learning
analytics must:
1. Not only focus on the easy counted
information
2. Build capacity and evidence informed
processes
3. Improve students and school performance by
assessment for learning
4. Be based on trust in the relation between
government, school leadership, teachers,
students and the community.
Learning Analytics –
possibilities
Learning Analytics – ecosystem
“A tool is only as good as the mindset using it”
(Fullan 2013)
1. Vision, values and direction
1. Establishing goals and expectations
1.1 Uplifting leadership
1.2 Ethical leadership
1.3 Establishing goals
2. Fostering relationships
2. Evidence-informed practise
2.1 Reflective enquiry
2.2 Research-informed practice
2.3 Data-informed practice
3. Professional learning and development
3.1 Learning environment
3.2 Feedback culture
3.3 Collaborative learning
3.4 Collective responsibility
4 Innovation, improvement and change
4.1 Change processes
4.2 Improvement
4.3 Innovation
4.4 Continuous learning and development
5 Local and global responsibility
5.1 Collaboration with the municipality
5.2 Collaboration between school and home
5.3 Collaboration with other schools
5.4 Global awareness
6 Management
6.1 Information Literacy
6.2 Using School Administration Systems (SAS)
6.3 Communicating and sharing information (LMS)
6.4 Evaluating/monitoring (PULS)
6.5 Tracking and follow up of students (VOKAL)
6.6 Capacity assessment and appraisal systems (KAN)
1.1 Clear Purpose
1.2 Raising expectations
2.1Relationships with students
2.2Relationships between students
2.3Relationships with parents
2.4Relationships with community?
3. Engaging students
3.1 Promote passion for the subject
3.2 Recognising different starting points
3.3 Designing challenging learning programs
3.4 Developing students’ responsibility for
learning
3.5 Encouraging well-organized cooperative
learning
3.6 Enhancing learning through technology
4. Assessing learning
4.1 Creating success criteria
4.2 Closely monitoring students’ understanding
4.3 Giving and receiving feedback
4.4 Promoting self-assessment
4.5 Promoting peer assessment
4.6 Tracking progress over time
5. Enquiring into your practice
5.1 Reflective enquiry
5.2 Research-informed practice
5.3 Data-informed practice
6. Committing to collective
responsibility
6.1 Collaborative learning
6.2 Collective responsibility
7. Stepping up to leadership
8. Innovating and changing
8.1 Experiencing change
8.2 Learning about your learning
School leaders need to
know
•
•
•
•
•
•
•
Professional Capital
Assessment for learning
21 CENTURY capacities
Evidence informed processes
Professional learning communities
Student Centered leadership
OTL
Teachers need to know
1. Establishing goals and expectations
1.1 Clear Purpose
1.2 Raising expectations
2. Fostering relationships
2.1Relationships with students
2.2Relationships between students
2.3Relationships with parents
2.4Relationships with community?
3. Engaging students
3.1 Promote passion for the subject
3.2 Recognising different starting points
3.3 Designing challenging learning programs
3.4 Developing students’ responsibility for
learning
3.5 Encouraging well-organized cooperative
learning
3.6 Enhancing learning through technology
4. Assessing learning
4.1 Creating success criteria
4.2 Closely monitoring students’ understanding
4.3 Giving and receiving feedback
4.4 Promoting self-assessment
4.5 Promoting peer assessment
4.6 Tracking progress over time
5. Enquiring into your practice
5.1 Reflective enquiry
5.2 Research-informed practice
5.3 Data-informed practice
6. Committing to collective responsibility
6.1 Collaborative learning
6.2 Collective responsibility
8. Innovating and changing
8.1 Experiencing change
8.2 Learning about your learning
Modell 1 Relation between data
MAP
Results
MAV
Support
from home
Class
climat
Teacher
Self-eff
Motivation
Learning analytics in Norway
Final estimation of fixed effects
Fixed Effect
For INTRCPT1, β
INTRCPT2, γ
TRINN_ME, γ
K_MILJ4, γ
LAERER17, γ
For KJONN slope, β
INTRCPT2, γ
For KARAKT6 slope, β
INTRCPT2, γ
For VEIL3_11 slope, β
INTRCPT2, γ
For HJEM3_09 slope, β
INTRCPT2, γ
Coefficient
t -ratio d.f.
p -value
0
3,78 169,44
0,03
0,65
0,13
2,53
0,33
4,21
00
01
02
03
115 <0.001
115
0,52
115
0,01
115 <0.001
1
10
0,09
2,58 1757
0,01
0,08
4,33 1757 <0.001
0,19
11,97 1757 <0.001
0,14
8,59 1757 <0.001
2
20
• Individnivå (N=1757/55)
– Nr.2. Den nest viktigste faktoren for motivasjonen er
vellykkede veiledningssamtaler med rådgiver (t=11.97,
β=.19***). Jo bedre veiledning, desto bedre motivasjon.
Rådgivningen er den mest signifikante faktoren, men ikke
den sterkeste.
•
3
30
4
40
Final estimation of fixed effects
Fixed Effect
Coefficient
t -ratio
For INTRCPT1, β
INTRCPT2, γ
3,78 166,74
TRINN_ME, γ
-0,02 -0,39
K_MILJ4, γ
0,20
3,80
LAERER17, γ
0,42
5,84
For KJONN slope, β
INTRCPT2, γ
0,10
2,76
For KARAKT6 slope, β
INTRCPT2, γ
0,08
4,78
For VEIL3_11 slope, β
INTRCPT2, γ
0,20 12,63
TRINN_ME, γ
0,02
0,56
LAERER17, γ
0,10
1,98
For HJEM3_09 slope, β
INTRCPT2, γ
0,14
8,84
d.f.
p -value
0
00
01
02
03
115
115
115
115
<0.001
0,70
<0.001
<0.001
1755
0,01
1755
<0.001
1755
1755
1755
<0.001
0,58
0,05
1755
<0.001
1
10
2
20
3
30
31
32
4
40
Hvis man tar med interaksjonen mellom trinn og rådgiver, er det en signifikant
interaksjonseffekt mellom rådgivers veiledning og tilfredshet med lærerne på
motivasjonen (t=1.97; β=.10; p<.05). Jo bedre lærere , desto bedre effekt av
individuell veiledning, ut over den direkte effekten mellom veiledning og
motivasjon.
– Nr.3: Støtte fra hjemmet. Jo bedre hjemmestøtte, desto
høyere motivasjon (t=8.59; β=.14***).
– Nr.5: Kjønn (t=2.58; β=.09**). Kontrollfaktor.
– Nr.6: Karakterer (t=4.43; β=.08***). Kontrollfaktor.
• Gruppenivå (N=115)
– Nr.1: Viktigste faktor: Lærerne (t=4.21; β=.33***).
Lærerne er de som påvirker elevenes motivasjon mest.
•
NB! Elevene ble ikke spurt om lærernes faglige dyktighet (naturlig nok?).
– Nr.4: Klassemiljø (t=2.53; β=.13**).
– Nr.7: Klassetrinn betyr ingen ting for motivasjonen
(kontrollert for de andre faktorene i modellen).
• Konklusjon:
Learning analytics in Norway
Final estimation of fixed effects
Fixed Effect
For INTRCPT1, β
INTRCPT2, γ
Coefficient
t -ratio d.f. p -value
0
00
TRINN_ME, γ
K_MILJ4, γ
LAERER17, γ
For KJONN slope, β
INTRCPT2, γ
For SE8_01 slope, β
INTRCPT2, γ
For MAV6_03 slope, β
INTRCPT2, γ
For MAP3_12 slope, β
INTRCPT2, γ
01
02
03
3,77 218,71
115 <0.001
0,01
0,17
0,23
0,29
4,51
3,92
115
0,77
115 <0.001
115 <0.001
0,10
3,19 1757
0,00
2
20
0,31
11,96 1757 <0.001
0,11
4,05 1757 <0.001
0,22
10,42 1757 <0.001
3
30
4
40
Final estimation of fixed effects
Coefficient
t -ratio
d.f. p -value
3,78 167,03
-0,02 -0,46
0,20
3,78
0,42
5,98
115
115
115
115
<0.001
0,65
<0.001
<0.001
0,13
3,65 1755
<0.001
0,31
0,10
-0,03
11,28 1755
1,68 1755
-0,43 1755
<0.001
0,09
0,67
0,11
3,99 1755
<0.001
0,22
10,03 1755
<0.001
0
00
01
02
03
1
10
2
20
21
22
3
30
40
– Nr.1. Mestringstro (self-efficacy). Den viktigste faktoren for
motivasjonen er mestringstro (t=11.96, β=.31***).
Mestringstro er både den sterkest faktoren og den mest
signifikante faktoren.
•
1
10
Fixed Effect
For INTRCPT1, β
INTRCPT2, γ
TRINN_ME, γ
K_MILJ4, γ
LAERER17, γ
For KJONN slope, β
INTRCPT2, γ
For SE8_01 slope, β
INTRCPT2, γ
K_MILJ4, γ
LAERER17, γ
For MAV6_03 slope, β
INTRCPT2, γ
For MAP3_12 slope, β
INTRCPT2, γ
• Individnivå (N=1757/55)
4
Hvis man tar med interaksjonen mellom lærer og mestringstro, er det en
marginalt signifikant interaksjonseffekt mellom mestringstro og klassemiljø på
motivasjonen (t=1.68; β=.10; p<.10). Jo bedre klasseklima, desto sterkere effekt
har mestringstroen på motivasjonen, ut over den direkte effekten av på
motivasjon.
– Nr.3: Aktiv mestring mot mål (MAP) (t=10.42; β=.22***).
– Nr.5: Perfeksjonistisk mestring (MAV) (t=4.05; β=.11***).
– Nr.6: Kjønn (t=3.19; β=.10**). Kontrollfaktor.
• Gruppenivå (N=115)
– Nr.2: Lærerne (t=3.92; β=.23***). Lærerne er de som
påvirker elevenes motivasjon mest etter mestringstro.
– Nr.4: Klassemiljø (t=4.51; β=.17**).
– Nr.6: Klassetrinn betyr ingen ting for motivasjonen
(kontrollert for de andre faktorene i modellen).
• Konklusjon:
Results and progression
Learning analytics is about: Selection and focus
1
Input
3
Proses
Profil
Actions
Multi 3rd grade
Thegraph shows the amount of money four children spent
in a week. Use the graph to answer the questions.
How much did Ben spend?
Dollar
$
Who spent over $ 500?
John
Eve
Alan
Kate
What is the difference between the one who spent the
most and the one that spent the least?
40$
60$
80$
100$
John
Previous
Next
Eve
Kate
Alan
Math
<header>
Numbers
Algebra
Data
Stats
%
Meassurement
Geometry
<header>
<name>
<updated>
<header>
The result of a test is measured as a percentage of the norm (national average) for each function or each subtest. A score of 100% means,
that the result is identical to the national average. A score of 130% would mean that the student has 30% more right than the national average
<header>
<level>
<esultatet av en test i Kartleggeren måles i prosent av norm (landsgjennomsnittet) for hvert funksjonsområde eller hver deltest. Et resultat på
100 % betyr altså at resultatet er identisk med landsgjennomsnittet. Et resultat på 130 % vil bety at eleven har 30 % mer rett>
<header>
<level>
<esultatet av en test i Kartleggeren måles i prosent av norm (landsgjennomsnittet) for hvert funksjonsområde eller hver deltest. Et resultat på
100 % betyr altså at resultatet er identisk med landsgjennomsnittet. Et resultat på 130 % vil bety at eleven har 30 % mer rett>
<header>
<level>
<esultatet av en test i Kartleggeren måles i prosent av norm (landsgjennomsnittet) for hvert funksjonsområde eller hver deltest. Et resultat på
100 % betyr altså at resultatet er identisk med landsgjennomsnittet. Et resultat på 130 % vil bety at eleven har 30 % mer rett enn
landsgjennomsnittet, mens et resultat på 70 % vil tilsvarende bety at eleven har svart rett på 70 % av det en gjennomsnittselev har gjort.
Kategori 1: > 80 % av norm. Resultatet indikerer at eleven trolig ikke har behov for tilrettelagt opplæringstilbud på dette>
<header>
<header>
<text>
<text>
<header>
<header>
<header>
<page>
<page>
<page>
<chapter link>
<text>
<text>
Math
New data in cooperation with Knewton
Work load
Task level
Goal %
Progression
Test correlation
Students
development tool
•
•
Assessment for learning
21 CENTURY capacities
21 CENTURY capacities
What it means
The term 21st century capacities refers to
a broad set of knowledge, capacities , work
habits, and character traits that are
believed to be critically important to
success in today’s world, particularly in
collegiate programs and modern careers.
Generally speaking, 21st century capacities
can be applied in all academic subject
areas, and in all educational, career, and
civic settings throughout a student’s life.
Do it again
Creative thinking 80%
Creative thinking is the
process we use when we
come up with new ideas.
creative thinking process
can be accidental or
deliberate
Students overview
Students overview
Numbers
Algebra
Math
Data
ENG
Stats
%
Reading
Meassurement
Geometry
Science
Sport
Test-correlation
Progression
Task level
Self-efficacy
Self-assessment
Logged in
Start Page
Supervision
report
Management
areas
School owners report
Display
Teaching
evaluation
Selection
School report
Selection
Year
My reports
NYC
2013
Total report
Information
Management
targets
Quality
description
Export
Dotted
Updated
06.02.14
Total report
Key information
School 1
Indicators
Parents’ education level
Class room management
Motivation and coping
National tests, 5th grade
National tests, 8th grade
National tests, 9th grade
Primary points
Deviation achievement- and examination mark
Proportion who have successfylly completed first year of college
Grade development from middle school to first year of college
School 2
School 3
School 4
Level Quality characteristics - Student Assessment
Level Students find that feedback from the teachers is adapted to their
prior knowledge. Students know what they have achieved and what
4
Indicators
Student survey
- Motivation
to strive for. They utilize the feedback so the learning environment is - learning and
characterized by motivation and student participation. The teachers assessment
have a common understanding of what competence the various
- professional guidance
characters express.
- criteria for self
Level Students and teachers have a common understanding of the
assessment
strategies
that
should
be
used
to
achieve
good
learning.
Teachers
3
engage students in assessing their own work. Final assessment is
based on the total competence goals.
Level The school has a clear focus on assessment, but the feedback is often
general and self-assessment is not set in system. Final assessment is
2
to some extent based on the overall competence goals.
Level The teachers' practice of continuous assessment is unsystematic.
Final assessment is just to a small extent based on the overall
1
competence goals.
Level
Level 4
Level 3
Level 2
Level 1
Quality characteristics - Organizational Learning
The school is involved in learning at all levels. There is a culture of
sharing knowledge and exchanging good practice. Colleagues support
each other and make each other better. The discussions are
characterized by confidence and openness.
The school's development is characterized by collaboration and
reflection on own practice and analysis of their results. The school is in
the forefront of academic and educational development, uses new
research and learn from good examples from other schools.
The school's educational development follow central planning and
control documents. The school's goals and priorities are known to the
staff. The school analyzes its own performance and set targets.
The school tries to systematize the educational development in relation
to central planning and control documents. Individual results are
presented for the staff.
The school's educational development is characterized by randomness,
and the effort of individual initiative.
Indicators
Personnel survey
- Collegial support
- Common goals
- Learning Press
- Extra role behavior
•
•
•
•
•
•
•
•
•
•
Learning Analytics – current limitations
Not all information in the same system (the ecosystem is incomplete)
Not enough standards in the ed-tec industry
The ed-tec industry is not cooperating
The target is not the students learning but the government control need,
or researchers joy in finding correlations
Researchers focus on algorithms or teaching, not the processes in a ecosystem
The governments focus on content and not structures
Professional capital and capacity building is not the target for the tools
and reports
To much data driven control or personalized tasks, not enough evidence
based processes and learning to learn.
Not enough focus on the markers of future performance and formative
assessment (Improve school performance by assessment for learning)
Not enough trust in the relation between government, school
leadership, teachers, students and the community.
Learning Analytics – ecosystem
Yngve Lindvig
Chief Research and Development Officer
Conexus
[email protected]
www.conexus.no
+ 47 90785069
Linkedin: http://no.linkedin.com/pub/yngve-lindvig/30/907/47b/