The Linked World: Working Paper Series

The Linked World: Working Paper Series Impact of ICT on Production of Goods and Services: Measuring the Impact of ICT on Education By Cor‐Jan Jager, Jesse Bos, and Robbin te Velde (Dialogic) About The Linked World Project
Information and communication technology (ICT) has decisively established itself as a general purpose
technology—one that affects an entire economy. Over the past four decades, ICT has spurred dramatic
changes that will continue for the foreseeable future. Harder to predict, though, is the exact nature of
those changes, and how they will play out across societies—in our economies, our cultural relationships
and the way human beings interact.
This notion formed the basic motivation for embarking on the study, which the Telefónica Foundation
agreed to underwrite in 2008. The purpose of The Linked World: How ICT Is Transforming Societies,
Cultures, and Economies is to take stock of our knowledge on what the economic, social, and cultural
impacts of ICT will be. How has it evolved, how much have we been able to quantify or to evaluate in a
qualitative sense, and what does it mean for the challenges and opportunities ahead?
The Linked World: How ICT Is Transforming Societies, Cultures, and Economies is the result of a twoyear global research project led by The Conference Board and underwritten by the Telefónica
Foundation.
This working paper is one of a global series that forms the basis of the book The Linked World: How
ICT Is Transforming Societies, Cultures, and Economies published by the Telefónica Foundation and
Artel Press.
For more information about this project, including video summaries and online forums please go to:
www.ictlinkedworld.com
Measuring the impact of ICT on education By Cor‐Jan Jager, Jesse Bos, and Robbin te Velde (Dialogic) 1
2
Introduction .............................................................................................. 3
1.1
Background .................................................................................................... 3
1.2
Conceptual framework ....................................................................................... 6
1.3
Research approach............................................................................................ 8
Micro and Macro model............................................................................... 10
2.1
2.1.1
3
Constitution of the micro model ............................................................................ 10
Approach ..................................................................................................... 10
2.2
Constitution of the macro model ........................................................................... 15
2.3
Findings of the three micro models and the macro model ................................................ 22
2.3.1
Similar findings in micro and macro model ................................................................ 23
2.3.2
New findings in the macro model .......................................................................... 24
Country studies ........................................................................................ 26
3.1
Introduction .................................................................................................. 26
3.2
USA ........................................................................................................... 27
3.2.1
Context ....................................................................................................... 27
3.2.2
Readiness .................................................................................................... 28
3.2.3
Use ........................................................................................................... 31
3.2.4
Impact ........................................................................................................ 33
3.3
Spain .......................................................................................................... 35
3.3.1
Context ....................................................................................................... 35
3.3.2
Readiness .................................................................................................... 37
3.3.3
Use ........................................................................................................... 41
E‐learning use ........................................................................................................ 42
3.3.4
3.4
4
5
Impact ........................................................................................................ 45
The Netherlands ............................................................................................. 46
3.4.1
Context ....................................................................................................... 46
3.4.2
Readiness .................................................................................................... 47
3.4.3
Use ........................................................................................................... 49
3.4.4
Impact ........................................................................................................ 52
Overall conclusions and implications for further research .......................................... 56
4.1
Conclusions ................................................................................................... 56
4.2
Implications for further research ........................................................................... 59
Literature ............................................................................................... 61
Appendix I: List of developed countries that contribute to the average in the country studies ................... 65
Appendix II: PISA Results ............................................................................................... 66
Appendix III: TIMSS results ............................................................................................. 71
Appendix IV: PIRLS results ............................................................................................... 75
Appendix V: Macro Model results ...................................................................................... 81
2
1
Introduction 1.1 Background This study covers one particular sector of the economy: education. The promise of ICT in education can be described in brief as an opportunity to provide better education to more people cost effectively. Spurred by these apparent promises, countries all around the world have invested massively in ICT in education over the past decade. Thus, it is useful to analyse the impact of ICT on education. We will do so on a global scale, with focus on primary and secondary education. The first main research question for this study is: I. What are the key factors (drivers, enablers and bottlenecks) that determine ICT’s impact on education? ICT was originally expected to raise the quality of education, expand its reach, and lower costs simultaneously. Expectations might have been too high. Many instances of e‐learning boil down to the traditional application of distance learning. The objective in distance learning is not to raise the quality of education but to increase the geographic reach of primary and secondary education in a cost‐effective manner. Especially in the U.S. many studies have been conducted on the impact of distance learning on education performance. Overall, these studies arrive at the sobering conclusion that online learning is as effective as conventional education in terms of academic outcomes. (Smith, Clark, Blomeyer, 2005). Thus, the provision of online education should at least be more cost‐effective than the provision of traditional education. However, we do not pursue the cost‐effectiveness of distance learning. In contrast to education performance, there is little hard data available on the cost‐
effectiveness of the use of ICT in education. Also, distance learning is a particular niche of e‐learning. Instead, we study the impact of ICT on conventional education (regular face‐to‐face teaching of students) with a particular focus on educational performance. Hence the second main research question is: II. How does the introduction of ICT affect the educational performance of students in conventional lower and secondary education? A similar, recently‐published study was performed by the OECD (2010). This study is based on PISA, an international student performance assessment. The main focus of this study was to examine the relationship between technology use and educational performance in science. The student is the central focus and emphasis is placed on generic patterns within the relation of technology use and educational performance. Our study, in contrast, zooms in on specific countries, especially the U.S., Spain and the Netherlands, and attempts to explain some of the differences in educational performance by looking at contextual country characteristics (e.g. policies, technological infrastructure, etc.). Moreover, it combines data from PISA with two other international assessments, PIRLS and TIMSS. A brief overview of relevant similarities and differences in findings will be given in section 4. 3
The availability of ICT, that is, investments in hard infrastructure, is one of the factors that could determine the impact of ICT on education performance. Obviously, the availability of hardware and software alone is not sufficient to improve the quality of education. In general, investments in hard infrastructure should always be accompanied by investments in soft infrastructure, that is, (skills and organizational changes (see for instance Brynjolfsson & Hitt, 2000; 2003)). Thus we should also look into factors such as ICT skills of pupils and teachers and organizational changes. With regard to the latter factor, it should be noted that the impact of ICT is notoriously hard to isolate from overall organizational changes. In the Netherlands, for instance, the use of ICT in education is closely related to the introduction of new ways of teaching with a much stronger focus on the autonomy and independence of pupils (see Box 1). We should also take a number of controlling factors into account, such as the number of traditional class teaching hours. It should be noted that there are other potential justifications for using ICT in schools, an example of which is teaching digital skills to students. These are, however, not the focus of this study. Finally, the overall impact of ICT on educational performance should always be regarded against the background of factors that traditionally have a strong influence on performance regardless of technological and organizational changes, such as the educational background of parents. ICT‐related factors might have an impact on performance, but their overall impact might therefore be modest. A two‐step approach will be followed in this study: (1) A number of large broad‐based, multi‐country datasets will be analyzed to detect potential systematic relationships between available measures of presence and use of ICT and traditional measures of educational performance. Two important limitations of this approach are: (a) relations will be measured by means of partial correlations (with constituted factors). These correlations do not imply causality. (b) Measures of the type of ICT use (at school) are lacking in the available data. (2) Case studies will be conducted for three countries (Netherlands, Spain, U.S.)
in order to focus more specifically on the country data and delve in more detail into available country‐specific (contextual) research to help interpret the results. The selection of these three countries was made pragmatically and is based both on the available data/contextual knowledge of these countries and our desire for a set of countries with sufficient cultural, economic, and geographic diversity to allow an interesting comparison. Box 1. Amadeus College: a Front‐runner in E‐learning The Amadeus Lyceum in Utrecht is one of the front‐runners in the Netherlands in the use of e‐learning in secondary education. The new method of teaching (the New Learning or the Study House) has been rigorously implemented: all class teaching has been abolished. Instead 90‐100 students work independently in a large shared room. If needed, teachers instruct individual students in separate niches that are connected to the main room. 4
Nearly all teaching material is accessible in a web‐based teaching system (ELO or Electronic Learning Environment). Every student obtains a private laptop. These are rented by their parents from the school. The student owns the laptop but is not allowed to bring the laptop home. However, the ELO is accessible from home. The overall performance of the Amadeus Lyceum is on par with the national average. It performs slightly better in foreign languages (albeit somewhat lower in the native language) but underperforms in economics and especially sciences. Final examinations of Amadeus (orange) compared to the national Dutch average (grey)1 1
The number is the average result on the final examination for that particular class. Note that the Dutch grading system is on a 0‐10 scale. To pass an exam, the score should be ≥5,5. 5
1.2 Conceptual framework To describe the impact of ICT on education we use an overall conceptual framework that describes the effect of ICT on society. The model is based on a traditional three staged linear diffusion model, subsequently Readiness, Use, and Impact (Colecchia, 1999; Holland et al., 2004). Impact can be divided further into direct and indirect effects. Whereas direct impacts refer to the performance of the educational system, indirect effects refer to the wider economic and societal effects of the improved performance of the educational system.2 Technological and societal developments are closely intertwined. During all three stages technological developments trigger countervailing societal trends (that is, are met with resistance and resilience). Since Readiness primarily refers to technology and Impact to (changes in) social practices, the strength of these countervailing trends increases along the way. In Figure 1, this general pattern has been visualized. The fact that new technologies usually require a lot of societal adaptations (including overcoming substantial amounts of societal resistance) is probably one of the reasons ICT has not yet lived up to expectations in the domain of education. Figure 1. Overall conceptual model: diffusion of ICT in society Techno logica l po le
Socio-technica l surroundings
Societa l po le
direc t effects
Readiness
Actual use
Impact
ind irec t
effec ts
Technological trends
Technological bottlenecks
Societal trends
Societal resilience / resistance
2
A typical example from literature would be the inverse correlation between the level of (primary and secondary) education of women and the fertility rate (for underlying data, see for instance www.nationmaster.com). 6
The wider social and economic effects of the presumed changes in education due to the use of ICT (such as an increased coverage of the educational system) are not part of this study. We simply do not have the quantitative data to describe these broader indirect effects. What we do have, instead, is a more refined classification of the direct effects. It seems that the use of ICT not only directly influences the performance of students and teachers, but also changes the attitude of pupils and students, and via the changing attitude indirectly influences the performance.3 Since we have micro data on both performance and attitude we can isolate the latter as a separate factor. Figure 2. Causal relations between Use and Impact Attitude
(Readiness)
Use
Performance
Impact
The developments in the specific domain of education (e‐learning) should always be considered against the background of the availability and use of ICT in general. Obviously, widespread use of ICT, as in the U.S. and the Netherlands, might greatly facilitate the use of ICT in education. However, the general use of ICT is in turn influenced by the general characteristics of a country, such as geography (e.g., extensiveness might boost the use of ICT for distance learning), economy (e.g., wide income disparities might negatively impact the overall educational performance of pupils), and culture (e.g., the differences between pupils and teachers affects the adoption of e‐learning). Figure 3 shows the adapted version of the overall conceptual model, including a number of underlying indicators for each component. Readiness and Use are divided into a general and specific (here: e‐learning) component. Impact (that is, direct effects, see before, figure 2) is split into Attitude and Performance. Finally, we have included a sizeable number of control variables that correspond with important background factors, such as the overall investments in education and the intellectual background of parents. We have data at the level of individual students and at the level of countries and will use both types of data in this study. 3
To prevent confusion between the indirect (broader) effects of Impact – which are not covered by this study – and the indirect effect of the use of ICT on the direct effects of Impact, we will use the descriptive labels ‘Attitude’ and ‘Performance’ instead. 7
Figure 3. General conceptual model applied to e‐learning 1.3 Research approach The overall aim of this study is to describe how ICT drives living standards in the world’s advanced and emerging markets, in terms of economic growth and societal and cultural development. In order to be able to compare the impact of ICT across many countries we have chosen to use a predominantly quantitative approach. This means that our approach was very much driven by the availability of international comparative data sets. Luckily, we do have detailed data available at the level of individual pupils and teachers. This data originates from three worldwide surveys on the education performance of pupils in basic and secondary education: PISA, PIRLS, and TIMSS.4 These studies focus on performance in math, science, and reading, but have recently added specific indicators on the use of ICT on schools. In this study we tried to get the most out of these extensive sources of micro data.5 In section 2, we use the three micro‐data sets to study the impact of ICT on learning in three reference countries: the U.S., Spain, and the Netherlands.8 The micro approach enables us to zoom in on a specific country. This is an essential condition to attribute educational performance towards e‐learning. Another big advantage is that the variables (especially the Readiness indicators) are not “contaminated” by country specific characteristics since all cases can be selected 4
The Program for International Student Assessment (PISA) conducted by the OECD, the Progress in International Reading Literary Study (PIRLS), and the Trends in International Mathematics and Science Study (TIMSS). All three are internationally recognized studies on education performance. The studies are conducted by the OECD (PISA) and the International Association for the Evaluation of Educational Achievement (PIRLS and TIMSS). 5
One of the drawbacks of the heavy reliance on the three aforementioned data sources is that the scope of the study is limited to Kindergarten‐12 and Kindergarten‐15 students. Thus the higher education level is not covered in this study. 8
from the same country. The micro‐data sets from PISA, PIRLS and TIMSS are used in parallel because each of them highlights specific dimensions of the impact of ICT. Another part of section 2 will deal with the impact of ICT, but at the macro level of countries. The analysis is based on the same data as the micro model, but the data of individual students and teachers has been aggregated to the country level and the three data sets have been merged. Furthermore, the resulting data set has been supplemented with other data available at the country level.6 This data is mainly used to describe a general context. The macro approach is intended to gain insight in macro country characteristics of Readiness, Use and Impact of e‐learning. As such it puts the detailed results of the micro approach into a broader perspective. The data from the macro model was also used as the quantitative framework for the more qualitative case studies on the three reference countries, the United States, Spain and the Netherlands in section 3. Hence for each country the various components of the conceptual e‐learning model (General Readiness, e‐learning Readiness, General Use, e‐learning use, etc. ‐‐ see next paragraph) are discussed. The interpretation of the results is based on additional research but is partly driven by the results from the micro model and the macro model (section 2). In the final section, overall conclusions about the impact of ICT on learning are drawn at a more general level. 6
We have tried to trace macro data on the higher education level (e.g., growth and magnitude of online learning at colleges and universities) but were not able to trace suitable (that is, internationally comparative) data. Thus the original focus on late primary (K‐12) and lower secondary (K‐15) education from section 2 remained. 9
2
Micro and Macro model The starting point of our quest to resolve the research questions led us to the depths and delicacies of three international student performance tests: PISA, TIMSS and PIRLS. These studies compare countries on math, science and reading performance and provide a vast number of cases on student, teacher and school level (N > 200.000). There are sufficient cases for each of the reference countries available to derive meaningful statistical conclusions for each country. The three micro models should therefore be interpreted as the building blocks for a generic model and the basis for benchmarking among countries. By using three studies in three separate analyses, some redundancy in indicators is apparent. However, the substantial increase in robustness of the model that is gained legitimates this approach. In this section, these three micro models (one for each study) will be covered in more detail. First (§2.1) the generic approach for developing the micro model will be described, after which the macro model will be introduced in §2.2. The major findings of these two models are dealt with in §2.3. The fine delicacies in country differences will be handled in the country case studies in Section 3. 2.1 Constitution of the micro model The micro‐approach provides a more granular tool, specifically intended to gain insight into the impact of e‐learning on educational performance. The biggest advantage is the large set of available data and the ability to zoom in on each of the three countries. This is an essential condition to relate educational performance to e‐
learning. Another big advantage is that readiness is unbounded by country‐specific characteristics and specific macro‐related country‐specific policies, since all cases are derived from the same country. The drawback, however, is that not all of the available indicators can be used in each of the separate analyses. 2.1.1 Approach A structured approach was followed to develop a procedure to analyse available data of all three studies. The starting point was the conceptual model as described in section 1. In other words, we looked for ways to relate Impact variables to Readiness and Use variables. From a strict statistical point of view there is actually no way to truly control confounding variables and thus isolate the effects of the desired ones. We have used the most optimal feasible method, that is, we used a set of indicators that are (A) mutually exclusive (in one analysis) and are (B) controlled for potential external confounding indicators. In order to achieve this, we performed factor analyses to reduce data and thereby met the requirement of mutual exclusivity. While calculating correlations, we were 10
able to control for potential disturbing influences that were derived from the same source of data. In the following sections of paragraph §2.1 we will described the method that was used in more detail. The method consists of three phases which can in turn be divided into nine steps: Phase 1: Data Collection I.
Find relevant studies with sufficient cases and countries II.
Download data from found studies III.
Merge data from relevant countries Phase 2: Data preparation IV.
Merge data by key variables V.
Recode of variables in the same direction VI.
Combine variables by checking for internal consistency VII.
Conduct factor analyses to “summarize” data and make it mutually exclusive VIII. Interpret factors Phase 3: Data processing IX.
Calculate correlations between readiness, use and impact indicators, controlling for confounders Data collection Searching the Internet for large internationally comparable datasets led to the PISA, PIRLS and TIMSS performance studies [I]. These studies do not have an explicit focus on e‐learning, but rather have a number of ICT indicators as part of a vast number of other background indicators on student and teacher characteristics. Data was collected [II] in November 2009 from the International Association for the Evaluation of Educational Achievement.7The two most recent studies PIRLS (2006), TIMSS (2007_Grade8) with a sufficient reach of countries were selected and downloaded. The third study PISA (2006) was derived from the OECD Programme for International Student Assessment.8 The data that was retrieved had to be merged – that is, data from the relevant countries had to be combined – before it could be used in for the purpose of this study [III]. We use IEA’s Analyser to merge the data. This resulted in ten sizeable micro data sets: 1. PISA (combination of two datasets) The PISA datasets entail questions asked to schools and students. 2. TIMSS (combination of three datasets) 7
8
www.iea.nl http://www.pisa.oecd.org/pages/0,2987,en_32252351_32235731_1_1_1_1_1,00.html 11
The TIMSS datasets contain a student, teacher and school set. Individual students, teachers and school principals were involved in the TIMSS study. 3. PIRLS (combination of four datasets) The PIRLS dataset consists of four underlying datasets. Individual students, teachers, and school principals were involved, as in the TIMSS study. Since the PIRLS study deals with young children, parents of the students were asked for background information as well. Data preparation Data in each of the three studies could be further merged [IV] by key variables like student ID, class ID and country ID or a combination of these. This step resulted in three separate datasets. A selection of relevant variables with regard to general readiness, e‐learning readiness, general use, e‐learning use, impact and control variables was made to come to a first more comprehensible dataset. All ordinal and scale variables were coded in the same direction [V]; that is, in the direction of the question in the survey.9 All individual test performance (e.g. math performance in the PISA test) indicators (impact) were tested for internal consistency [VI] by calculating Cronbach’s Alpha. A value > 0.7 is generally considered to be sufficient to merge a number of variables as one. All performance indicators in each of the three studies proved to have a value > 0.7 and could therefore be averaged to constitute aggregated variables for performance; one for PISA performance (math, reading and science), one for TIMSS performance (science and math) and one for PIRLS performance (reading) A selection of the remaining number of relevant variables including the “performance” variables can be found in the table below. Table 1. Total number of cases in the three micro data sets Study N (valid cases) Number of variables PISA 398.750 127
PIRLS 222.125 148
TIMSS 246.112 94
The next step was to further reduce data to a comprehensible and more manageable number of orthogonal (mutually exclusive) factors [VII] that effectively ‘summarizes’ the data in the main categories of the conceptual model, with an additional category for control variables. The reason for this step was [a] To reduce the number of variables and more importantly [b] to control for the risk of multicollinearity. The 9
The majority of the variables were ordinal using Likert scales ranging from 1 to 4, 5 or 6. 12
constructed factors do not relate to each other (r=0), which enables us to interpret a found correlation between one of the factors and performance indicators directly without having taking into account potential confounders of the specific analysis. However, it should be noted that we did follow the basic research model in which we distinguished between the following “bins” of factors. 




General readiness E‐learning readiness General use E‐learning use Control To be able to do this, principal components factor analyses were performed per study and per bin. Two indicators that were taken in consideration are the Kayser‐
Meyer‐Olkin and Bartlett test of sphericity, which in essence identify if we can meaningfully interpret the factor solution. All factor solutions resulted in satisfying values for KMO (>0.5) and Bartlett’s indicators (p<0.01). Each factor was calculated in such a manner that it entails the underlying concept of similar indicators in the analysis.10 The derived factors were interpreted (and given a clear label) based on the combination of the highest factor loadings of the variables that ‘constitute’ the factor [VIII]. Eventually, the data reduction led to a considerable decrease in the number of variables in the model. For example, in the case of PIRLS the 148 selected original variables were combined into 27 factors. The statistical analyses of the micro data could then be done on the basis of the (workable) number of 27 variables rather than the 148 variables. Table 2 gives an overview of the results of the data reduction. 10
Varimax rotation (50 iterations) was done respectively and residuals were saved as new variables. 13
E‐learning use General use E‐learning readiness General readiness Table 2. Total number of factors (combined original variables) in the three data sets, by bin PISA 127 23
1
2
5
2 2 11
PIRLS 148 27
1
2
1
2 2 19
TIMSS 94 25
1
4
1
3 3 13
Study Control Total number of factors Impact Number of variables Data processing The final step in the analysis is to relate factors from the model to each other, that is, to describe the relations between the Readiness, Use and Impact factors. For this analysis we used (partial) Pearson R correlation rather than linear regression. The decision to use correlation was based on two arguments. First, we do not have a firmly‐based research model in which we distinguish between predictors and dependent variables. The conceptual model is not elaborated enough to do so. It would be rather bold to assume cause‐effect relations. Secondly, the possibility of using partial correlations with which we can control for possible confounding variables is a strong motivation to conduct partial correlations. (Partial) Pearsons R correlations, controlling for the potentially confounding “control” factors were calculated [IX] per study (PISA, PIRLS, TIMSS) for the following pairs of categories: Readiness factors x Readiness factors x Use factors x Use factors Impact Impact[1] (controlling for control variables) As an additional analysis, we added an extra control instrument: Use factors x Impact[2] (controlling for control and readiness variables) In this test, we controlled not only for the potential confounding effects of the “control” variables, but also included the readiness variables in the control group. This enables us to isolate the added value of “use” factors regardless of the availability (readiness) of ICT at home or school. 14
As a cut‐off value, a (high) significance < 0.01 was chosen for relations to be relevant. Another cut‐off value is that of the actual correlation coefficient (r). Given the vast number of cases in the data sets, even a small coefficient (e.g. r=0.04) can already be significant. We used a coefficient r>0.1 as a minimum requirement. Methodological remarks As stated before, the approach described in preceding paragraphs is an attempt to relate effects of indicators to each other. It should be noted, though, that even with a large number of control variables, the potential of confounding effects of variables that were not included in the model (‘factor X’) pose a real threat. However, besides using many control variables (see table 2) the threat is further marginalized by using and interpreting not one but three micro models in parallel. The correlation coefficients in the models are not as high as might be expected. Most likely, this is due to the fact that we used a ‘forced’ factor solution to arrive at mutually exclusive factors. Along the way we lose some expressive relations between individual variables. Factor solutions for the “Control variable” group could not be performed for Spain and the U.S. for the PIRLS study due to missing values in the correlation matrix on which this factor solution is based. We choose to calculate “uncontrolled” correlations here instead between Readiness, Use and Impact indicators. The results from PIRLS for the U.S. and Spain should be interpreted with this notion in mind. 2.2 Constitution of the macro model The original plan for the macro model was to follow the same strategy as was pursued in the micro‐model: calculate Pearson R (zero‐order) correlations between Readiness, Use and Impact variables (the factors in the micro model), while controlling for a number of variables. However, this proved impossible for two reasons: 1. The number of cases was too small to conduct a factor analysis; 2. The alternative – making use of partial correlations to control for the “control” variables—also turned out to be not feasible because the intersection between the Readiness, Use, Impact and control variables contained too few cases (usually N<20) Instead, we resorted to the calculation of direct (and thus uncontrolled) correlations between the variables. Hence an important methodological note to the results of the analysis in §3.4 is that the correlations might have a potential number of confounding variables. Another disclaimer is that the variables in the model are not orthogonal. This was not an issue in the micro model, where we used factor analysis for data reduction. The resulting factors are inherently orthogonal – there is no overlap between the factors. However, variables in the macro model could be mutual dependent. 15
The macro model should be seen as the aggregation of variables to the highest level per country. As is depicted in the figure below, it contains variables of the three studies (PISA, PIRLS and TIMSS), supplemented with country specific variables from ITU11 and UNESCO. Figure 4. Relation between micro and macro model Micro Model
Variable 1 Variable 2
Macro Model
Variable n
Mean standardized (Variable 1 , Variable 2 , Variable n , Variable 2)
Student 1
Country 1
PISA
Student 2
Country 2
Student n
PISA
Country 3
Variable 1 Variable 2
Variable n
PIRLS
Student 1
PIRLS
Student 2
TIMSS
Student n
ITU
Variable 1 Variable 2
Student 1
Variable n
UNESCO
TIMSS
Student 2
Student n
The aggregation of data was performed as follows: 
Data was aggregated to country level [I] 
Highly similar variables (in name) between datasets were standardized and tested on Cronbachs alpha, prior to aggregation by calculating the mean [II]. 
Remaining similar variables within a micro dataset (predominantly based on the constructed factors) were standardized and tested on Cronbachs alpha, prior to aggregation, by calculating the mean [III]. 
Similar constructed variables (after step III) between datasets were tested for Cronbachs alpha (>0.7) and aggregated by calculating the mean. [IV] To give an example of this aggregation chain: [I]: Highest education level of father, highest education level of mother and # of books at home from PISA was averaged per country (country means) [II] Highest education level of father from PISA and Highest education level of father from TIMSS prove to be similar on the country level (Cronbachs alpha>0.7) and were standardized and averaged to form: Educational level of father (PISA+TIMSS) 11
International Telecommunications Union (2008) 16
[III] Educational level of father (PISA+TIMSS) was checked for consistency with Educational level of mother (PISA + TIMSS) and sequentially averaged to constitute: Educational level parents. [IV] Educational level parents and # of books at home (based on one of the factors in PISA) were standardized to a 0‐1 scale, correlated with each other and aggregated (averaged) to a new variable: Intellectual background parents. Table 3 presents the constituted variables that will be used for the country studies and the macro model. These variables are categorized roughly in ”Readiness”, “Use” and ”Impact” indicators. Table 3. Overview of singular and compound variables sub theme abbreviation
indicator (source)
General readiness number of countries
Internet penetration % of homes with Internet. [Compound variable: PISA, ITU, TIMSS] 46
computer penetration % of homes with a personal computer. [Compound variable: PISA, PIRLS, ITU, TIMSS] 46
reading on inet time Student: About how much time do you 27
spend reading stories or articles on the Internet outside of school on a normal school day? [PIRLS] broadband pen
Broadband penetration. [ITU]
46
inet search skills
Student: How well can you do each of these tasks on a computer? Search the Internet for information [PISA] 34
email skills Student: How well can you do each of these tasks on a computer? Write and send E‐mails [PISA] 34
fixed inet pen
Fixed Internet penetration. [ITU] 43
Student: Which of the following are in your home? A computer you can use for school work [PISA] 46
Teacher: Is your school’s capacity to provide instruction hindered by any of 45
comp for class
e‐learning readiness – infrastructure lack inet 17
connection the following?
Lack or inadequacy of Internet connectivity [PISA] comp avail else school Teacher: Are computers available elsewhere in the school? [PIRLS] 27
lack instr comp
Teacher: is your school hindered by a lack of computers for instruction. [Compound variable: PISA, PIRLS, TIMSS (2x)] 46
comp in classroom
Teacher: are computers available in your classroom. [Compound variable: PIRLS, TIMSS (2x)] 34
inet access classroom 34
Teacher: do the computers in the classroom have access to the Internet. [Compound variable: PIRLS, TIMSS (2x)] Student: Which of the following are in your home? Educational software [PISA] 46
School: Does your school provide teachers with a workspace in the classroom? [PIRLS] 26
shared worksp t'ers School: Does your school provide teachers with a workspace shared by several teachers? [PIRLS] 26
lack support staff
26
School: How much is your school’s capacity to provide instruction affected by a shortage or inadequacy of computer support staff? [PIRLS] instr softw shortage 46
Teacher / school: how much is your school's capacity to provide instruction affected by a shortage or inadequacy of computer software for instructional purposes? [Compound variable: PISA, PIRLS, TIMSS (2x)] prof IT dev col
Teacher: In the past two years, what percentage of your <eighth grade> teachers have been involved in professional development opportunities for mathematics and edu softw at home
e‐learning readiness – support infrastructure worksp t'ers in clroom 22
18
science targeted at using information and communication technology for educational. [TIMSS] lack support
Teacher: In your view, to what extent does a shortage of support for using computers limitlimits how you teach the <TIMSS class>? [TIMSS] [Compound variable: TIMSS (2x)] 22
prof devel IT teacher Teacher: In the past two years, have you participated in professional development in integrating information technology into your courses? [TIMSS] [Compound variable: TIMSS (2x)] 22
Student: How often do you use a computer at home? [Compound variable: PISA, PIRLS, TIMSS] 42
General use ‐ computer use at computer home computer use other places 42
Student: how often do you use a computer at other places? [Compound variable: PISA, PIRLS, TIMSS] computer games
Student: on a normal day, how much time before or after school do you spend playing computer games? [Compound variable: PIRLS, TIMSS] 34
hist comp use
Student: How long have you been using computers? [PISA] 34
Student: How often do you use computers for the following reasons? Browse the Internet for information about people, things, or ideas [PISA] 34
browse sports
Student: How often do you use the Internet to look up things about sports? [PIRLS] 27
browse music
Student: How often do you use the Internet to find out about music? [PIRLS] 27
browse other
Student: How often do you use the Internet to find out about other 27
General use ‐ browse people
Internet 19
activities and interests? [PIRLS] e‐learning use – pupil’s perspective e‐learning use – teacher’s perspective chat / email
Student: How often do you use the Internet to chat, e‐mail, or instant message with friends? [PIRLS] 27
time on Internet
Student: On a normal school day, how much time do you spend before or after school using the Internet? [TIMSS] 22
inet groupwork
Student: How often do you use computers for the following reasons? Use the Internet to collaborate with a group or team [PISA] 34
use edu softw
Student: How often do you use computers for the following reasons? Use educational software such as Mathematics programs [PISA] 34
inet for school info
Student: How often do you use the Internet to look up information for school? [PIRLS] 27
comp use school
Student: how often do you use a computer at school? [Compound variable: PISA, PIRLS, TIMSS] 42
softw for read instr 27
Teacher: When you have reading instruction and/or do reading activities with the students, how often do you use computer software for reading instruction? [PIRLS] inet for read instr
27
Teacher: When you have reading instruction and/or do reading activities with the students, how often do you use reading material on the Internet? [PIRLS] inet for info
Teacher: How often do you have students use computers to look up information on the Internet [PIRLS] email / chat school Teacher: How often do you have topics students use computers to email or chat with other students about what 27
27
20
they are learning? [PIRLS]
e‐learning activities maths and science read from comp
Teacher: How often do you have students to read stories or other texts on the computer? [PIRLS] 27
instr softw to dev read skills Teacher: How often do you have students to use instructional software to develop reading skills or strategies [PIRLS] 27
write text on comp
Teacher: How often do you have students use computers to write stories or other texts? [PIRLS] 27
proj other school/countries 27
Teacher: How often do you have students use the Internet to do projects with students in other schools or countries? [PIRLS] sci experiments / sci proc Teacher: In teaching science to the <TIMSS class>, how often do you have students use a computer to do scientific procedures or experiments? [TIMSS] 22
simulations
Teacher: In teaching science to the <TIMSS class>, how often do you have students use [TIMSS] 22
a computer to study natural phenomena through simulations? [TIMSS] practice skills / procedures Teacher: In teaching your lessons to 22
the <TIMSS class>, how often do you have students use a computer to practice skills and procedures? [TIMSS] look up ideas / information Teacher: In teaching your lessons to the <TIMSS class>, how often do you have students use a computer to look up ideas and information? [TIMSS] 22
process / analyze data Teacher: In teaching your lessons to the <TIMSS class>, how often do you have students use a computer to process and analyze data? [TIMSS] 22
21
Impact discover concepts
Teacher: In teaching mathematics to the TIMSS class, how often do you have students use a computer to discover mathematics principles and concepts? [TIMSS] 22
Attitude science
Attitude science aggregated. [Compound variable: PISA, TIMSS (5x)] 46
Attitude maths
Attitude maths aggregated. [TIMSS] 22
Attitude reading
Attitude towards reading. [PIRLS] 28
Reading performance Reading performance. [Compound variable: PISA, PIRLS] 46
Maths performance Maths performance. [Compound variable: PISA, TIMSS] 46
Science performance Science performance. [Compound variable: PISA, TIMSS] 46
2.3 Findings of the three micro models and the macro model Figure 5 below illustrates the most important significant findings from the micro and macro models. A detailed description of findings (per country) can be found in Appendix II to V. Figure 5. Most important findings from micro and macro model ↯
↯
↯
↯
CONTROL
Intellectual background → Attitude towards science & math/reading
Intellectual background → Learning performance
Time in regular lessons vs. activities outside school → Learning performance
Student independence in class/independent reading tasks → Learning performance
(luxury) Home possessions/study facilities at home → Learning performance
READINESS → USE
Computers at home → ICT use at home
Computers at home → self-perceived ICT skills
Computers at home → Learning performance
Availability of computers in class/school → ICT use in class
USE → IMPACT
self-perceived ICT skills → Attitude towards science/reading
self-perceived ICT skills → Learning performance
ICT use at home → Attitude towards science & math
ICT use at home → Attitude towards reading
ICT use at home → Learning performance
Intensity of computer use not at home → Learning performance
Use of ICT for homework → Attitude towards science & math/reading
Use computer at school → Attitude towards science & math/reading
Use computer at school → Learning performance
IMPACT
Learning performance → Attitude towards science & math/reading
Attitude towards science & math → Attitude towards reading
PISA
++
+++
+++
---
TIMSS
0
+
++
+
+
+
+++
+
++
+
+
+
+
++
PIRLS
0
+++
+
+
MACRO
++
++
++
+
+
++
++
0
0
+
0
+
0
++
0
+++
0
++
--
0
+
+
-
-
0
--
++
+
0
+
0
0
0
++
++
++
++
--
22
2.3.1 Similar findings in micro and macro model There is quite a deal of similarity between the results of the micro model and the macro model. This is an indication that the results of the macro model are quite robust: Relations that were found at the level of individual students and teachers still hold on the level of countries. Some of the findings in Figure 5 are rather trivial and do not require much elaboration (e.g. Computer availability at home is positively related to computer use at home). The most important findings of the micro model that we do elaborate upon are: 
The availability of ICT at home is positively related to learning performance. Remarkably, though, the intensity of ICT use at home is negatively related to learning performance. The most likely explanation is that the constituted factors of “ICT use at home” in TIMSS and PIRLS are partly based on ICT use for entertainment purposes, like gaming. This might indicate that the time spent on ICT use for entertainment purposes cannibalizes learning time for schoolwork. 
Self‐perceived performance on ICT skills has a positive relation with attitude towards science. This positive relationship is probably entirely due to self‐
selection – students who consider themselves to have advanced ICT skills generally also like sciences (a.k.a. the ‘Nerd’‐argument). 
Computer use at home has a positive relation with attitude towards science. The rationale behind this relation is similar to that of the previous point. This line of reasoning (the ‘Nerd’‐argument) is strengthened due to the fact that computer use at home does not have a positive relation with attitude towards reading. 
Use of ICT for educational purposes (at home or in school) has a positive relation with the attitude towards science and math. In general, it appears that the use of ICT for/in education boosts the image of science and math. 
Attitude towards science and math has a moderate positive relation with learning performance in all three studies. The control variables proved to have a significant impact on learning performance as well, as will be described below: 
Intellectual background (of the parents) has a profound positive relation with both performance and attitude towards science. 
Time for subjects (that is, actual instruction time) in regular lessons has a positive impact on performance. This is a replication of the findings in the micro studies that traditional class teaching has a better impact on learning performance than newly‐fashioned non‐in class teaching methods. 
(Physical) teaching limitations have a negative relation with the attitude towards science. The teaching limitations as experienced by the teachers 23
seem to have a negative effect on their teaching, and consequently on the motivation of their students. 2.3.2 New findings in the macro model The analysis of the macro model yielded a number of results that were not already found in the micro model. These new findings are predominantly derived from the combination of the three aggregated micro‐models and/or the inclusion of ITU and UNESCO variables. Two of these findings are closely related to the previous findings but are slightly different (they correlate with performance rather than attitude): 
Computer use at home has a positive relation with performance which contradicts with the findings in the micro model. Presumably, the reason is that we did not control in the macro model for GDP. The datasets span a lot of countries including some that are not so well developed. Computer use at home in these countries is (more than in Western countries) an indicator of household wealth or an important predictor of the educational level of the parents. This can, in effect, easily yield a confounding relation. 
(Physical) teaching limitations have a negative relation with performance. This message conveys the same “risk” of a confounding relation as the previous bullet. The less‐developed countries are more likely to be exposed to teaching limitations. However, the frame of reference for the perception of teaching limitations is also different for less‐developed countries compared to developed countries, which indicates that the (physical) teaching limitations might still be an important condition for learning performance. Three other new findings can partly be explained by the welfare of a country (the control variable GDP): 
Public expenditure per pupil as % of GDP has a positive relation with attitude towards science and math 
Gross enrolment ratio in education has a positive relation with attitude towards science and math 
Gross enrolment ratio in education has a positive relation with performance Finally we found one genuinely new relationship, which has no apparent link to the underlying micro data. We also have no immediate logical explanation for the correlation between these two factors. It is, however, a robust significant relationship across all (30) countries: 
Attitude towards science and math has a negative relation with attitude towards reading. Simply stated, students who like science and math dislike reading and the other way around. This is a confirmation of the cliché separation between Science (Dutch: Beta) and Arts (Dutch: Alpha) students. 24
25
3
Country studies 3.1
Introduction Three country studies have been conducted: U.S., Spain, and the Netherlands. The studies are structured along the lines of the general model (Readiness, Use and Impact). For each of the countries, all components of the model are discussed. The individual scores for a country on each of the components are shown in radar charts,12 which have a uniform structure (see Box 2). Box 2. Explanation of radar charts in country studies Individual indicator
Score of the reference country
Missing value
Average score of all countries
(average = 0, standard deviation = 1)
The average score for a particular indicator is always calculated on the basis of the data of 49 countries in the dataset (Appendix I). The data is normalized so that the variables in one chart have the same scale.13 The average of a normalized data set is always 0, and the standard deviation 1. This means that 68% of all cases fall within the first rings (‐1 to +1) around the average (the red circle in the centre of the radar), 95% within the first two rings (‐2 to +2) and 99,7% within the first three rings (‐3 to +3), assuming that the variables are distributed normally. Hence small deviations from the average already indicate significant differences in practice. The radar charts provide a general overview of the situation with regard to e‐learning in a particular country. The readiness, use and impact of e‐learning in the specific countries can be derived from the data. Having said this, impact is also influenced by external factors like the structure and functioning of the school system. Thus it should be kept in mind that the country study is valuable to characterize general relations, but more limited with respect to cause‐and‐effect conclusions. The indicators in the radar charts are either singular variables directly based on the (aggregated) data from one of the variables in the three micro data sets (see section 2) or compound variables. The number of variables is much higher than the number of variables from the macro model because we are not restricted to the variables 12
For the sake of comparison, the same charts are also included in Appendix 3, but now ordered by component rather than by country. 13
The original data is included in appendix II 26
that occur in all three micro sets. Instead we could select all variables from any of the three micro sets that were relevant to e‐learning. The compound data are constructed by indicators from different micro data sets that are supposed to measure the same topic (e.g. presence of computers at home in TIMSS, PIRLS and PISA questionnaire).14 3.2 United States 3.2.1 Context The United States is by far the largest of the three reference countries. Its 306 million inhabitants and 25 million secondary school students (UNESCO, 2007) are spread across a vast area of 9.8 million km2 (CIA, 2010). As a result, school density is relatively small; one elementary school per 152 km2 and one junior high school per 372 km2 (Public school review 2009).15 In the two other reference countries, density is much higher: one elementary school per 37 km2 in Spain (and one junior high school per 60km2). The Netherlands has one elementary school per 6 km2 and one high school per 64 km2. GDP per capita of the United States is $47,400 of which 5.7 % is spent on education. Public expenditure per pupil as % of GDP per capita for all school levels is 23.9% (UNESCO,2006). Compared with Spain and the Netherlands, GDP per capita level is high in the U.S. and so is the percentage spent on education. However, public expenditure per pupil as % GDP per capita for all school levels is comparable in Spain (22.3%) and considerably higher in the Netherlands (25.6%). The United States has a long history in home schooling (Lewis and Setzer, 2005) and virtual high schools (Virtual high school, 2009). The most relevant federal act with regard to e‐learning is the No Child Left Behind act. The federal Department of Education has further elaborated on the act in its educational technology plan (US DoE, 2004). In the plan, the department has formulated seven guidelines for states and districts: strengthen leadership, consider innovative budgeting, improve teacher training, support e‐learning and virtual schools, encourage broadband access, move towards digital content and integrate data systems. The states and districts are responsible for the implementation of these recommendations. In short, the federal e‐learning policy pays attention to a broad range of dimensions, thus not only focusing on infrastructure and hardware, but also on organizational dimensions (leadership, budgeting, and skills of teachers). 14
To test the internal consistency of the compound variables, Cronbach’s alpha was calculated. Only those variables where α>0.7 were included. 15
The United States educational system before university consists of four phases: kindergarten for children of the age 4‐5, elementary school for the age of 6‐11, junior high school for the age of 11‐14 and finally high school for children of the age 14‐18. This structure is the same in all states. 27
3.2.2 Readiness General readiness Figure 6. General readiness in the United States reading on inet time
2
fixed inet pen
broadband pen
1
0
United States of America
‐1
‐2
internet penetration
email skills
inet search skills
average
computer penetration
The United States is well positioned to benefit from the use of e‐learning. It scores very high on broadband, computer and Internet penetration. Some 94% of the Internet users have broadband access (OECD, 2009). The use of broadband Internet is among the highest of the world.16 These figures are in line with the high expenditure on ICT.17 Although the actual implementation of ICT stimulation plans is left to the states, and consequently significant differences might occur between states, the national average is still far above the average score of the other countries. On the side of soft infrastructure, we only have figures on the average time spent by pupils on reading articles on the Internet outside of school time. Again, the U.S. scores much higher than the average. Based on the results from the micro model we might argue that the U.S. also scores high on the two missing variables (email skills and Internet search skills). Both variables are core components of the factor ‘self perceived performance basic skills’ and the number of computers at home have a significant positive effect on this factor. Thus the excellent position on IT penetration probably translates into above average scores on basic IT skills. 16
With a monthly use of 14.24 Gbyte, the United States is positioned third in the list of broadband use per Internet user (Website optimization 2009). 17
Expenditure on ICT as a percentage of GDP is 7.5% in the U.S., 6.6 % in the Netherlands and 5.5% in Spain (Worldbank 2007). 28
E‐learning readiness Figure 7. e‐learning readiness infrastructure in the United States inet access classroom
comp for class
2
1
0
‐1
internet connection
‐2
comp in classroom
United States of America
average
comp avail else school
instr comp
The United States has initially implemented e‐learning as part of distance learning (Concotta, 2008) (Watson, 2008). Initial research indicated that the effort of e‐
learning on learning performance is neither positive nor negative (Means et al. 2009). In the early days of e‐learning, the target was mainly to create cost savings and to expand course catalogues (Collins, 2004). The particular focus on distance learning is obviously directly related to the extensiveness of the country, with many sparsely populated rural regions. This is also reflected in the school density. Home‐schooling and distance learning are possible solutions to reach children in regions that are too far away from any type of school. Various states recognized the possible benefits of e‐learning as a potential cost saving. The recommendations from the No Child Left Behind program have also been implemented from that particular point of view. One of the aspects was to encourage broadband access on the schools. Figure 7 indicates that presence of Internet connection is above average, as is computer availability. However, recent studies from the department of education focus on the impact of e‐
learning on performance (Concotta, 2008), and so it is to be expected that performance will play a larger role in the attitude towards e‐learning. In general, the U.S. has a long history in e‐learning, recognizes benefits, and has heavily invested in computers and Internet connections in school. With the initial focus on cost savings, investments in e‐learning were justified as long as they had no negative impact on learning performance. However, with the current focus on improving performance per se, the initial enthusiasm to invest in e‐learning might be tempered. One of the reasons that the investments in ICT infrastructure have not (yet) translated into improved learning performance is that the investments in hard infrastructure should always be accompanied by investments in soft infrastructure (skills and organizational changes) (again see Brynjolfsson & Hitt, 2000). This argument has been acknowledged in the federal education technology plan (see before, also figure 8 below). The amount of investment needed in the soft infrastructure is at least as large as the previous investments in hard infrastructure. 29
Figure 8. e‐learning readiness – support infrastructure of the United States prof devel IT teacher
support
edu softw at home
3
2
1
0
‐1
‐2
prof IT dev col
worksp t'ers in clroom
shared worksp t'ers
United States of America
average
support staff
instr software
The U.S. policy on e‐learning differs from the Spanish and Dutch policy in the way that it not only focuses on use and applications, but also on the skills of the teacher as well. Figure 8 shows very clearly that the U.S. pays far more attention to improving IT skills of teachers [prof IT dev col] than all other countries (+3.00). The investments seem to pay off: 63% of the teachers consider their computer and classroom technology skills as at least somewhat advanced (CDW‐G, 2006). Other aspects, like support from school, support staff and (absence of) software shortage, are still above average, but not as high as professional development opportunities for using ICT for educational purposes. This can partly be explained by the original application of e‐learning as part of distance learning; in recent years, much attention has been paid to distance learning programs. All good practice examples of e‐learning in the United States (U.S. Department of Education, 2009) are, for instance, examples of distance learning. It is to be expected that most effort to develop digital content focuses on distance learning. Although the teachers in our datasets are based in conventional schools, where face‐to‐face teaching is common practice and distance learning seems to be of limited use, the attention to the specific topic of distance learning seems to have a much broader impact on the IT skills of teachers. The presence of educational software at home could indicate that youth is already accustomed to learning with aid of computers. In the previous section it has been argued that pupils in the U.S. are likely to have above‐average ICT skills. Together with the above‐average use of educational software at home, it seems that the American pupils are in a good position to benefit from e‐learning. 30
3.2.3 Use General Use Figure 9: general use of computer and Internet for the United States hist comp use
2
1
time on internet
0
computer games
‐1
computer use home
‐2
browse people
2
1
0
‐1
‐2
chat / email
browse sports
United States of America
average
browse music
browse other
computer use other
The previous figures showed that the school children in the United States who have computers available at school and at home are not limited by a lack of Internet connection, score above average in the availability of educational software at home and can use computers for school work. Thus the conditions seem to be optimal to benefit from e‐learning. Nevertheless, from figure 9 we can draw the disenchanted conclusions that these children mainly use computers to play computer games. One of the interesting results of the micro model is that the factor ‘self perceived advanced ICT skills’ is correlated with the presence of computers at home but not with the use of computers. Given the omnipresence of computers and Internet in the U.S., the previous harsh conclusion should be nuanced. Nevertheless, the pattern in figure 9 remains remarkable. In contrast with computer use, Internet use is above average. An explanation could be the popularity of online gaming. However, this argument does not fully explain the Internet surfing behaviour. E‐Learning use Figure 10: e‐learning use pupil’s and teacher’s perspective inet groupwork
2
1
proj other schools/coun
tries
0
‐1
Comp use school
‐2
use edu softw
write text on comp
softw for read instr
2
1
0
‐1
‐2
instr softw to dev read skills
inet for school info
inet for read instr
inet for info
United States of America
average
email/chat school topics
read from comp
31
Despite the limited computer use at home, American children use the computer and Internet extensively at school. It seems like the high degree of e‐learning Readiness provides strong incentives for intensive computer use at schools. The teacher’s perspective confirms the picture drawn by the pupils. Teachers apply ICT as a tool for different forms of learning. The only exception is the very low use of email and chat for school activities. In the Dutch case, one finding is mentioned which might also apply to the U.S. case: although ICT fosters new ways of communication (such as online chatting) teachers have many difficulties giving up control and leaving initiative to the students (Balanskat, 2006). Figure 10 also shows that e‐learning has really found its way into the American classrooms. More than half of all teachers (54%) indicate that computers have changed teaching a great deal (CDW‐G 2006). Communication with pupils, parents and colleagues also has changed dramatically. The vast majority of teachers in the U.S. use ICT for administrative functions and communication (90%), for lesson preparation and as a teaching tool (80%). Moreover these numbers have increased during the last years (CDW‐G, 2006). ICT not only changed the way of teaching in regular primary and secondary schools; the impact on distance learning and home schooling has been even more pronounced. Nearly all states (44) now have facilities for online learning for K‐12 students (Watson, 2008) and half of the states have implemented statewide virtual high schools (The center for digital education 2009). The adoption of online learning has been especially high in higher education. By the end of 2007 almost 4 million students in the US (20% of the total population) took at least one online course – a 12% increase compared with 2006 (Allen, 2008). One of the effects has been the rise of ‘online mastodons’ such as Phoenix University and MIT Sloan School of Management. When we observe Figure 11, we see that the image that e‐learning has already become part of regular teaching methods can be confirmed. Figure 11: e‐learning activities math and science 32
sci experiments/ sci proc
2
maths process/ analyze sci simulations
1
data
0
maths look up ideas/ sci practice skills/ ‐1
information
procedures
‐2
maths practice skills/ procedures
sci look up ideas/ information
sci process/ analyze data
United States of America
maths discover concepts
average
Especially for science, the use of advanced ICT applications, such as experiments and simulations, is above average. This also confirms the findings from the micro model. In the particular case of the United States, a significant correlation was found between the factors ‘recently followed IT‐integration course for math and science’ and ‘ICT use in math class’. Again, the U.S. has the highest score on the first factor of all countries. In a similar vein, the presence of computers at schools (which is also very high in the U.S.) also has a positive correlation with ICT use in math class. An obvious reason for the advanced application of e‐learning activities is the professional development of the teachers. Not only ICT skills are necessary to benefit from ICT, but also a change in teaching style. Use of ICT requires student activity and involvement. As a consequence, traditional direct instruction is not the most appropriate teaching technique to make the most of e‐learning. Instead, a more student‐ oriented style is needed for activities like looking up ideas and discovering concepts. Training could ease a change in style for the teachers. 3.2.4 Impact Figure 12: impact attitude maths
2
attitude science
1
0
attitude reading
‐1
United States of America
‐2
reading performance
science performance
maths performance
average
33
Figure 12 shows that the attitude towards science, math and reading is far below average. Given the fact that the U.S. is one of the leading countries in the implementation of e‐learning, a quick and dirty conclusion might be that the use of ICT has a negative impact on attitude towards learning, or at least not a positive one. However, the findings for the U.S. are in striking contrast with the results from the micro and macro models, which showed that the most important positive contribution of the use of ICT is precisely to attitude (and much less so to performance). The results from the micro and macro model do not explain the negative attitude towards math. On the contrary, one of the findings from the micro model is that the use of ICT (which is very high in the U.S.) and the attitude towards math are positively correlated. We only find this pattern for the U.S.. This might be explained by the fact that in the U.S. much attention is being brought to bear on improving the ICT skills of teachers, hence the impact of ICT in math class on attitude towards math is greater in the U.S. than in countries where teachers have less ICT skills. Results from the macro model also do not show a particularly bad position for the U.S. There are few limitations to teaching, the educational level of the parents is relatively high, and other relevant variables are not too far from average. The relative favourable position with specific reference to ICT is, however, entirely explained away by the general attitude towards math, which is very low in the U.S. Thus the starting point for the U.S. is highly unfavourable compared to the Netherlands and Spain. One of the underlying reasons is that the income disparity in the U.S. is very wide.18 Another specific finding for the U.S. from the micro model was that home possessions (books, dictionaries etc.) are positively correlated with the attitude towards math. It could be that the difference in possessions is larger as well, and so the disadvantage of lacking some possessions (books, dictionaries, etc.) has a stronger impact on attitude than in other countries. As for performance, the U.S. scores above average on reading performance (slightly below the Netherlands) but has just average scores on science performance and math performance (on par with Spain but clearly below the Netherlands). Given the strong position of the U.S. on e‐learning Readiness and e‐learning Use, one would have expected higher performance scores. However, just as in the case of attitude, there are various country‐specific factors which hamper the performance of pupils in the U.S. First, one of the most important findings of this study is that attitude is one of the key factors that contributes positively to performance. However, we have seen that the attitude towards math and reading is rather negative in the U.S., thus having a negative influence on performance (especially in the case of math). Furthermore, one of the characteristics of the U.S. high school curriculum is that it is general in nature. Compare, for 18
The GINI coefficient for the USA is 45 (on par with Uruguay and Cameroon), 32 for Spain and 31 for the Netherlands. Sweden (23) and Denmark (24) are the countries with the lowest income disparity. 34
instance, the Netherlands, where several specific (that is, less or more math‐
intensive) curricula are being offered to targeted subgroups of students. The one‐
size‐fits‐all approach in the U.S. might have a negative impact on the general attitude towards math. To sum up, among the three reference countries (and probably among all other countries) the United States has the most advanced use of e‐learning. The findings from the micro model do support the assumption that a high Readiness and Use has some effect on Impact, that is, positively contributes to the educational performance of pupils and students. However, the poor general attitude towards science, math and reading, which in turn have to do with specific contextual circumstances, makes it very hard to draw any conclusions on the impact of e‐learning on performance. 3.3 Spain 3.3.1 Context In absolute terms, public expenditure on education is relatively low in Spain (4.3% of GDP) and substantially below expenditure in the other two reference countries, the U.S. (5.7%) and the Netherlands (5.5%). However, because the number of secondary school students is relatively low in Spain – less than 7% of the total population19 – public expenditure per pupil is almost on par with the U.S. and just slightly below the Netherlands (UNESCO, 2007). As for the ratio between teachers and students, one of the variables directly related to the expenditure per pupil, Spain is exactly on the trend line. Figure 13: Public expenditure per pupil as % of GDP per capita (secondary education) and teacher/student ratio (lower secondary education), selected OECD countries, 2007 40
28
35
23
30
18
25
20
13
15
8
10
3
5
0
-2
MEX
SVK
AUS
teacher/student ratio, lower secondary education
NZL
GER
GRE
KOR
JAP
ICE
IRE
CZE
HUN
ESP
USA
POL
NED
AUT
SWI
FRA
UK
ITA
NOR
FIN
SWE
BEL
PRT
DEN
Public expenditure %GDP, secondary education
trendline (R2=0,32)
The relatively small share of students can be explained by the rapidly aging population. This is a general trend across Europe but much more pronounced in 19
Percentages for the USA are 8.1% and 8.5%.for the Netherlands 35
Spain. It also has direct consequences for e‐Learning because an aging demography is considered one of the inhibiting factors for the application of training and e‐learning (FeConE, 2006). This might be a temporary bottleneck; with regard to the use of mobile phones for instance, the generation gap no longer exits. But with regard to the use of Internet, the differences between the various age cohorts are still quite large in Spain (see table 4). Table 4. Internet use by age group, Spain versus EU, 2009 (Eurostat) Age group ESP
EU 16‐24 90%
91%
25‐34 77% 82% 35‐44 66% 75% 45‐54 53% 63% 55‐74 21%
36%
average 16‐74 57% 69% There are several national and regional policy initiatives to support the use of computers and Internet in education. An example of a very large governmental program is Avanza2 with a budget that exceeds €9 billion. The overall objective is to develop the information society; “education” is part of the line “contents and public services”. Initially, the focus in Spain was on stimulation of hardware and infrastructure. One of the most recent initiatives, for example, is to provide every 11‐year‐old student with a laptop (Brown, 2009). In the recent past similar national and regional initiatives include a plan to supply each classroom with at least one computer, and to stimulate the use of Internet in the classroom (FeConE, 2006). By 2009, according to the Ministry of Education, 99.5 percent of schools were connected to the Internet (87 percent through broadband). Spain has made large steps forward since 2005 in line with the Avanza2 program. The “Internet in School” program was succeeded by “Internet in the Classroom” in which ICT is further integrated in the teaching‐learning process. Another initiative is the open‐source educational platform Agrega, a facility for the Spanish educational community for placing, searching and accessing educational content. It seems that Spain in general has caught up from its lag compared to other countries with regard to different aspects of e‐learning in the last couple of years. Unfortunately, we do not have recent comparable data. We had to fall back on slightly dated information. The reader should keep this in mind while reading this section, since it might give a biased image of Spain. 36
3.3.2 Readiness General readiness Figure 14: general readiness in Spain reading on inet time
2
fixed inet pen
1
broadband pen
0
‐1
‐2
internet penetration
email skills
Spain
average
computer penetration
inet search skills
The total IT expenditure as % of GDP in Spain is below the average. This is also one of the primary reasons that (fixed) Internet penetration is relatively low in Spain, compared to the U.S. and the Netherlands. For Internet penetration, the big difference between the rural and urban areas is an additional bottleneck. As a consequence, overall Internet penetration in Spain is below average. Another bottleneck which has already been mentioned is the relatively aged population in Spain. In general, elderly people are less involved with ICT and as a consequence overall use of the Internet is also relatively low, at least compared to the U.S. and especially the Netherlands. Initiatives to stimulate the use of ICT focus merely on hardware, so the computer penetration is above average (compared to the other countries in analysis)20. Because the presence of computers at home enhance the self‐perceived performance of basic skills like Internet search skills and email skills, the Spanish children indicate that their ICT skills are slightly above average. Again we find a substantial generation gap – average Internet skills are low in Spain but the skills of the youth are well above the European average. 20
Note that this average is not equal to the OECD or Eurostat average; only countries in our macro model were included in the computation. 37
E‐learning readiness Figure 15: e‐learning readiness – infrastructure of Spain comp for class
2
inet access classroom
1
0
internet connection
‐1
‐2
comp in classroom
Spain
comp avail else school
instr comp
average
The Spanish e‐learning policy has traditionally been strongly supply‐driven. One of the early targets was to have a computer in every classroom by the year 2000, and a more recent initiative was deployed to stimulate the use of the Internet in the classroom. The Ariño project is one of the best‐known examples of the use of high tech ICT in a rural setting. It is one of the showpieces of Microsoft’s Partners in Learning strategy and as such drawn worldwide attention (Microsoft, 2008). Box 3. The Ariño project: Tablet PC’s go to Aragon21 Ariño is a village in Aragon in the northeast of Spain with a population of just 900 people. Its head teacher wanted to raise educational standards and motivate classes by giving them access to online research, and interactive learning tools. Rather than simply increase the number of computers connected to the Internet, the school decided to provide pupils with portable Tablet PCs linked to a wireless network. Now, final‐year pupils use Tablet PCs in most of their classes. Lessons are much more interactive and exciting, helping pupils to learn at their own pace. It has also transformed the role of teachers in the school. Pupils are more motivated, using Tablet PCs and the Internet to research subject matter and follow interactive lessons themselves. Thus, teachers spend less time instructing at the front of the classroom, and more time coaching and directing individuals and smaller groups of children. The teaching day is also radically different. Because the regional government of Aragon has set up a wireless network for the whole town, pupils can go online in the afternoons and evenings after school. They can spend this time collaborating online with their fellow pupils, or send questions to their teachers who make themselves available to answer messages as quickly as possible. This means that the normal school day can be set aside for more social activities. There is more time for discussions, social education, sports, and arts teaching, where there is greater value for pupils learning in groups with a teacher present. 21
paraphrased from Microsoft (2005). 38
The project has been so successful that the Ministry of Education in Aragon has announced that it will provide a further 14,000 Tablet PCs to schools in the region over the next three years. Six other regional governments in Spain have also started to introduce Tablet PCs and wireless networks in their local schools. Overall, the results of these IT infrastructure initiatives seem to be somewhat ambiguous. Teachers indicate that the presence of a computer in the classroom is below average, but Internet access in classroom is above average. In other words, there are relatively few computers available but they are at least connected to the Internet. However, in another survey less than 40% of the students thought that the computers in their school actually worked well (CNICE, 2007). As for the Ariño case, it has certainly been a success in terms of Readiness. However, based on the findings of section 2, the overall effect on Impact – that is, learning performance – remains to be seen. On the one hand, independent (reading) tasks and interactive use of ICT have a positive impact on attitude and thus indirectly on performance. On the other hand, time spent on out‐of‐school activities has a detrimental effect on performance. A remarkable fact in the Spanish case is that teachers indicate they are not hindered by a lack of computers for instruction although the presence of computers in classrooms is below average. This means that either the available computers are used very efficiently, or, more likely, are simply not used for educational purposes. The perception of the pupils shows the other side of the story. In the same CNICE survey, 47% of the pupils feel that the lack of teacher training hinders the use of e‐
learning. Moreover, 78% of the teachers think their ICT skills are insufficient to use e‐
learning. This supports the argument that ICT is currently not yet well‐integrated in teaching. A recent report from the Spanish Ministry of Education (Ministerio de educación, 2010) confirms this hypothesis. Although this research showed that practically all schools (99,5%) have access to the Internet, of which 88.3% ADSL, it also revealed that teachers use ICT primarily for support in presentations. Only 25% of the teachers use ICT more than once a week. The availability of a computer for school work is slightly above average. It is to be expected that if every 11‐year old child will receive a laptop computer in the foreseeable future that this number will increase, just as the self‐perceived ICT skills. If these trends continue, the gap between the teachers and the pupils – which is in fact a reflection of the wider generation gap – will widen further. Spain will have a teacher workforce unable to use the computer and students who are well inclined to benefit from the use of e‐learning. 39
Figure 16: e‐learning readiness – support infrastructure of Spain edu softw at home
2
prof devel IT teacher
worksp t'ers in clroom
1
0
‐1
support
shared worksp t'ers
‐2
Spain
average
prof IT dev col
support staff
instr software
Figure 16 shows a very contrasting picture. On the one hand, the support from school leadership is high and real. Support from school (read: management) is far above average and the schools actually provide workspace where teachers can prepare their e‐learning lessons. In addition, teachers themselves also have a positive attitude towards e‐learning. They think students are more motivated by the use of ICT and see potential performance‐related benefits, especially for students with special needs (CNICE 2007). At the same time, both teachers and students feel that the lack of ICT competency and support staff hinders the use of ICT in classroom (CNICE 2007). It is likely that this is the result of a lack of ICT training for teachers and lack of supporting staff – the soft side of infrastructure. (Since 2006, however, the ENSENA program has provided onsite courses and support services for teachers in the use of ICT. By 2009, nearly 90 percent of teachers have undertaken ICT‐related training courses.) The fact that there is no perceived lack of instructional software – it scores slightly above average – should also be interpreted in the light of a general lack of ICT skills of teachers. Although supply is probably low there is no shortage because the effective demand is also low. In other words, Spanish teachers do not (yet) perceive the lack of instructional software as a bottleneck to their teaching because they hardly use ICT anyway. This is in sharp contrast to the Netherlands, where teachers think the potential of e‐Learning is greatly underutilized. 40
3.3.3 Use General use Figure 17: General use computer and Internet in Spain hist comp use
2
1
0
‐1
computer games
‐2
computer use home
time on internet
browse people
2
1
0
‐1
‐2
Spain
browse music
chat / email
computer use other
browse sports
average
browse other
Although computer penetration in Spain is slightly above the sample average, the use of computers at home is far below average, as is the use of computer games, and the time spent on the Internet. The use of the Internet is still mainly focused on information retrieval (browsing) and use for communication purposes (chat & email) is still limited. Yet, Spanish are very eager users of social networking sites, which might be regarded as the next‐generation electronic communication. In this respect, the relatively low score on chat and email could actually be regarded as an indicator for the high maturity of use of Internet in Spain, at least when it comes to social communications purposes. Spain ranks second highest in Europe (much higher than the Netherlands) but has by far the highest growth rate among the leading countries. Table 5. Social Networking Reach by Country in various selected countries, age 15+, home and work locations, December 2007‐December 200822 Country Dec.2007 Dec.2008 Growth
Canada* 83,9 86,5
3,1%
Brazil* 76,0 85,3 12,2% United Kingdom 78,4 79,8
1,8%
Spain 62,8 73,7 17,4% Mexico* 67,3 73,0
8,5%
Portugal 65,7 72,9 11,0% USA* 65,8 70,7 7,4% Denmark 54,4 69,7
28,1%
Italy 56,5 69,3 22,7% Belgium 59,0 68,2
15,6%
Australia* 56,5 67,5 19,5% Germany 48,4 67,3
39,0%
Ireland 59,9 66,9 11,7% Finland 61,2 66,2
8,2%
Sweden 61,3 65,4 6,7% 22
Source: ComScore World Metrix (last visited March 4 2010). 41
Switzerland 53,7 64,7 France 52,2 63,9
Russian Federation 46,1 63,5
Netherlands 57,2 63 Norway 54,2 58,9 Austria 38,8 49,7 Average 60,0 68,9
Europe 61,0 74,6
* September 2007 ‐ September 2008
20,5% 22,4%
37,7%
10,1% 8,7% 28,1% 14,8%
22,3%
E‐learning use Figure 18: General use computer and Internet in Spain inet groupwork
2
1
proj other schools/cou
ntries
0
‐1
Comp use school
‐2
use edu softw
write text on comp
softw for read instr
2
1
0
‐1
‐2
instr softw to dev read skills
inet for school info
inet for read instr
inet for info
Spain
average
email/chat school topics
read from comp
The situation at school does not differ very much from the unfavorable situation at home. Use of computers at school is well below average and the Internet is mainly used to look up information for school, not to do group work. The same applies ceteris paribus to teachers. The score on the use of (stand‐alone) instructional software for developing reading skills is well above average in Spain but so is the score in the U.S. and the Netherlands. The same applies to reading from computers. In all the other uses Spanish teachers have relatively poor scores, with the possible exception of the use of the Internet to do projects with other schools. This is due to the fact that the (perceived) ICT skills of teachers are not very strong. As a consequence only few e‐learning activities are implemented in the teaching program. Apart from the curriculum related activities, Spanish teachers also are not frequent users of other possibilities that ICT offers. Only 28% of the schools use ICT to communicate with families and 28% of the teachers use ICT to communicate with colleagues (CNICE 2007). From an international perspective these are low penetration figures. The use of ICT in the secondary (administrative) process is very limited in Spain. Although 77% of the schools use ICT for text processing and 69% to search information on the Internet, only 44% of the schools say to benefit from the administrative possibilities of ICT (CNICE 2007). ICT is more considered as a technological issue than a training tool (Santillana Formacion 2004). 42
The use of e‐learning is more common in tertiary education than in secondary and primary education. This is a similar pattern we found in the U.S., but compared to the U.S. absolute numbers are much lower. Relatively few universities have adopted the potential of e‐learning for their curricula (Michels 2006). Lifelong learning is another strong growth market for e‐learning. As recently as 2006, the general uptake of lifelong learning was low in Spain, in all age cohorts. Figure 19: Lifelong learning in Spain (UNESCO, 2006) Lifelong learning in Spain
60%
50%
training participation
40%
European Union
30%
Spain
20%
10%
0%
25-34
35-44
45-54
age
55-64,
However, a more recent source from Eurostat shows the big strides that Spain has made with regard to lifelong learning in the last few years. Figure 20 below illustrates the percentage of individuals that used the Internet for training and education. Figure 20: Percentage of individuals that used the Internet for training and education by age group in the last three months in 2007 and 2009 43
80
70
60
50
Spain 2007
40
Spain 2009
EU(27) 2007
30
EU(27) 2009
20
10
0
16‐24
25‐34
35‐44
45‐54
55‐64
What can be seen is an above‐average growth compared to the EU average. The level of Internet usage for training and education has in general almost achieved the EU averages. Especially the younger generation (16‐44) has shown a tremendous upward movement in the last couple of years. Figure 21: e‐learning activities math and science maths process/ analyze data
sci experiments/ sci proc
2
1
sci simulations
0
maths look up ideas/ information
maths practice skills/ procedures
maths discover concepts
sci practice skills/ procedures
‐1
‐2
Spain
average
sci look up ideas/ information
sci process/ analyze data
Figure 21 shows clearly, though, that the use of e‐learning in classrooms is still in its infant stage in Spain (at least in 2006). In the particular domain of math and science, e‐learning activities for math are almost absent and activities for science are almost exclusively limited to basic activities such as looking up information and practicing 44
skills and procedures. In contrast to, for instance, the U.S. (figures for the Netherlands are lacking), more advanced uses such as simulation, experimentation, and concept discovery, score far below average. The limited use of e‐learning does not come as a surprise. As already mentioned, ICT skills of teachers are very limited, there is a shortage of support staff, and use of computers at home by pupils is relatively low. 3.3.4 Impact Figure 22: impact of e‐learning in Spain attitude maths
2
1
attitude science
0
attitude reading
‐1
‐2
science performance
Spain
reading performance
maths performance
average
The attitudes towards science and reading are relatively high in Spain, and only the attitude towards math is slightly below average. Since the use of e‐learning in Spain is considerably lower than in the United States and the Netherlands, this case shows factors that influence attitude with limited interference of e‐learning. Two factors that, according to the macro model, correlate positively with attitude are the intellectual background of the parents and the computer use at home. However, for both factors Spain scores below average. The influence of intellectual background on attitude in the micro model is significant for Spain, but not for the complete dataset. This could be the result of the typical distribution of education levels of the Spanish population (CBS, 2009). Compared with other European Union member countries, there is a large group of low‐
educated workers and a large group of highly‐educated people, and only a small group of medium‐educated people. Other macro factors that have a positive correlation with attitude are time for subjects in self‐study, and lack of (physical) teaching limitations. Especially time for self‐study could stimulate attitude in Spain. The value is almost one standard deviation above average and the influence of time for self study in the micro model is significant for Spain and not for the complete dataset. It seems the Spanish children prefer time for self‐study with a positive attitude towards science as a result. Apart from self‐study, Spanish children also value contextual topic description. The effort of contextual topic description, compared with the traditional conceptual topic 45
description, is a very interesting one for educational purposes, but is beyond the scope of this study. Anyway, in this particular case it seems that the attitude towards science benefits from the typical Spanish teaching method. Despite the positive attitude towards science, the performance is below average. This could be influenced by the low score on intellectual background (one standard deviation below average). Also computer use at home – another factor in the macro model that correlated positively with performance—is relatively low. The same holds for presence of computers at home and performance in the micro model. However, in the particular case of computer use and presence, the causal direction is not clear: it could be that low computer use is caused by low performance rather than the other way around (see section 2). In general, the implementation of e‐learning in Spain is moderate at best. Despite the relatively low use of e‐learning, attitudes towards science and reading are above average. The positive attitude towards science could be explained by the characteristics of the Spanish teaching system, where regular teaching demands are met, students have time for self‐study and contextual topic descriptions are valued. However, the actual school performance of the Spanish youth is still below the all‐
country average. The influence of attitude is seemingly not strong enough to offset the hindering factors from the general context. One of the conclusions we can already draw based on the U.S. and Spanish cases alone is that country specific non‐
ICT‐related factors have a much bigger impact on educational performance. However, we should keep in mind that Spain has made big investments in stimulation of ICT in the field of education, the first signs of which could be seen in Figure 20. 3.4 The Netherlands 3.4.1 Context The Netherlands is a small country with one of the highest population densities in the world. There is one elementary school in every 6 km2 and one secondary school in every 64km2. Consequently, distance learning is of little relevance and e‐learning is targeted towards improving the quality of education rather than lower the costs of providing education.23 The ambition for high quality education is one of the pillars of the official mission of the Dutch government to create a knowledge‐based economy. The ambitious mission has not really been substantiated by financial means. Public expenditure on education as a percentage of GDP is just moderate (5.5%) compared with other European Countries, the U.S. and Canada (UNESCO, 2006). Nevertheless, public expenditure as a percentage of GDP per pupil is considerably higher than in the U.S. and Spain, despite relatively big student populations. Use of computers in classrooms in the Netherlands dates from 1985. In the early days, the policy was merely focused on the centralized provision of hardware 23
One obvious exception is the use of distance learning to provide education to children that are physically unable to attend regular face to face courses (e.g., due to sickness). There are several pilots running to connect hospitals with schools. 46
(Ministerie van OCW, 1997). This is somewhat similar to the situation in Spain as of 2006. In the second phase (1998 to 2002) the focus shifted towards the local level and many school projects were supported (Ministerie van OCW, 1999). In the current third phase, the government has centralized the implementation of the e‐learning policy again, but this time with a focus on the development of educational software and the improvement of support staff (Ministerie van OCW, 2006). In addition, there is large financial support for ICT projects (De Leeuwe, 2006). 3.4.2 Readiness General readiness Figure 23: general readiness in the Netherlands reading on inet time
2
fixed inet pen
1
broadband pen
0
‐1
‐2
email skills
inet search skills
Netherlands
internet penetration
computer penetration
average
General Readiness is very high in the Netherlands. The combination of a strong international orientation (early adoption of foreign technology, widespread use of English) and abundant supply of cheap bandwidth (due to the fierce competition between DSL and cable) has led to very high average use of Internet in the Netherlands.24 With specific reference to e‐learning, the ICT skills of children are also very high. The early introduction of computers at school might also have been beneficial here. Given the abundant ICT infrastructure and the relatively high ICT skills it is remarkable that Dutch pupils spend little time on reading articles on the Internet. This low score might be explained by a general negative attitude towards reading (see also further below). The average time spent on reading has dramatically declined during the last 25 years.25 24
This has boosted the technological upgrading of cable to such an extent that it is currently even competing with glass fiber in the SME‐market (DOCSIS 3.0). 25
From 4.6 hours in 1975 to 1.5 hours in 2000 (De Vries 2007) 47
E‐learning readiness Figure 24: e‐learning readiness – infrastructure of the Netherlands comp for class
2
1
inet access classroom
0
internet connection
‐1
‐2
comp in classroom
Netherlands
comp avail else school
average
instr comp
As could be expected from the very strong position on General Readiness, the availability of computers in classrooms is also high. Internet access is just average, but it should be noted that the average for this particular indicator is very high (90%) and the standard deviation very low. Thus, around 90% of the computers at school are connected to the Internet. Yet, teachers indicate that the capacity of their school to provide instruction is hindered by the lack or inadequacy of Internet connectivity. With the steadily improving ICT skills of teachers (De Leeuwe 2006), the demand for Internet connections will only increase. Figure 25: e‐learning readiness – support infrastructure in the Netherlands edu softw at home
2
prof devel IT teacher
1
worksp t'ers in clroom
0
‐1
support
‐2
prof IT dev col
shared worksp t'ers
Netherlands
average
support staff
instr software
The policy shift during the third phase towards the ‘soft side’ of ICT infrastructure has clearly paid off. Instruction is not hindered by a lack of software and the availability of support staff is above average. The presence of workspace remains a weak spot. This means that teachers who want to use e‐learning tools in their lessons have to prepare the ICT‐activities at home. With the widespread use of ICT and Internet at home, this is not a big hurdle. On the other hand it does show that relatively little attention is being paid to the role of the teacher. This is supported by another study (Kennisnet 2009) which concludes that every Dutch teacher has some basic ICT skills but that only half of the teachers have sufficient skills to benefit from ICT as an educational tool. 48
At the student side, the situation is more favourable. The widespread presence of educational software at the student’s home in combination with the presence of fast Internet connection indicates that at least for the Dutch children the statement that “e‐learning is considered as just a normal accepted way of learning within all kinds of educational situations and institutions” (De Leeuwe 2006) is true. 3.4.3 Use General use Figure 26: general use computer and Internet of The Netherlands hist comp use
2
1
0
time on internet
‐1
computer games
‐2
computer use home
browse people
2
1
0
‐1
‐2
chat / email
browse sports
Netherlands
average
browse music
browse other
computer use other
Given the high scores on General Readiness and e‐learning Readiness it could be expected that the position on General Use is also strong. This certainly holds for the use of computers – especially the use of computers at home – but much less so for the use of Internet. The low scores on browsing (with the exception of browsing for people) are remarkable. The lack of browsing the Internet for articles on sports might be explained by a general lack of interest in sports among the Dutch youth. However, there is no apparent explanation for the very low score on browsing for music, all the more so in the light of the rather progressive policy towards intellectual property rights. The extremely high score on the use of chat and email by pupils is in striking contrast to the rather low average score of the overall Dutch population on the use of online social networking. The phenomenal success of the Dutch online social network Hyves, which is especially popular among children and adolescents, is entirely accountable for these figures.26 The relatively high score on browsing people is probably also related to this phenomenon. 26
Hyves boosts 9.5 million users – in a population of 16.4 million people. In 2007, it was voted most popular website in the Netherlands (Wikipedia.nl 2009). 49
E‐learning use Figure 27: e‐learning use pupil’s and teacher’s perspective inet groupwork
2,00
1,00
0,00
Comp use school
‐1,00
‐2,00
proj other schools/co
untries
use edu softw write text on comp
instr softw to dev read skills
inet for school info
softw for read instr
2
1
0
‐1
‐2
read from comp
inet for read instr
inet for info
email/chat school topics
Netherlands
average
In the Netherlands, the use of e‐learning is especially geared toward improving the quality of education (hence cost‐efficiency is much less important than in the U.S.). So far, one of the main purposes of e‐learning is to stimulate curiosity (Martens, 2004). This is at the fringe of the core teaching activities since e‐learning is mainly used for ‘fun’ (e.g., as a reward after traditional lessons have been completed). More closely connected to the core activities are the possibilities of e‐learning to offer material in various ways to meet different learning strategies among students or offer the material in smaller steps so that weaker students can benefit (Kennisnet, 2009). From the perspective of the students, it seems that e‐learning is still mainly used to complement traditional teaching methods. Thus although they think computers are used widely in schools, the use of specific educational software is clearly below average. Computers are used a lot in school but not necessarily for educational purposes. One explanation comes from the side of the teacher. The use of educational software demands advanced ICT skills from the teachers. With only 50% of the teachers having sufficient ICT skills for educational purposes, the number of pupils who are exposed to subject related ICT activities is limited. On the positive side: in the Netherlands a positive relation exists between ICT use and teaching limitations due to a lack of ICT. In general, this relation is negative. The reverse relation in the Netherlands could be an indication that improved ICT use stimulates the desire to use ICT for educational purposes, and that the demand for educational ICT applications is not met. 50
At first glance, the perception of the teachers seems to be more positive than the perception of their students. However, the U.S. and Spain have similar high scores in the use of instructional software. The two indicators in which Dutch teachers rank higher than their U.S. colleagues (writing texts on computers and using the Internet to look for information) are rather basic activities. The use of Internet for reading instructions and for projects with other schools is much lower than in the U.S.. The latter could probably be explained by the small size of the Netherlands – compare the low use of distance learning – but the international orientation of the Dutch would on the other hand be very conducive for online collaboration with schools abroad. The use of email and chat is very unpopular among teachers, albeit not much lower than in the U.S. In both countries the reservations of teachers to partly transferring control to students might explain the conventional attitude towards communication. Although Dutch students are avid users of email and chat, only 12% of them use the Internet to communicate with teachers (Kennisnet, 2009). In a similar vein, the use of virtual learning environments for parents to monitor their children’s performance is not uncommon but teachers use these environments predominantly as a student tracking system, not as a communication platform (Van Essen, 2006). More common is the use of ICT for lesson preparation and administration. Roughly half of the teachers use ICT for more than 5 hours per week for preparation and administration (Kennisnet, 2009). With regard to lifelong learning, the situation as of 2006 was more favorable than in Spain. However, the countries’ relative position had changed by 2009. Figure 28: training participation per age group (UNESCO, 2006) lifelong learning in the Netherlands
60%
50%
training participation
40%
European Union
30%
The Netherlands
20%
10%
0%
25-34
35-44
45-54
age
55-64,
Within the particular domain, there is indeed a strong focus in the use of e‐learning (Schreurs and Wijnhoven, 2005). Lifelong learning participation in the Netherlands is above average in the age group that grew up with computers. The Dutch market for e‐learning has tripled in size since 2003 (NIDAP, 2004). As figure 29 illustrates, it 51
appears that the Netherlands is comparable to the EU average with regard to lifelong learning. Figure 29: Percentage of individuals that used the Internet for training and education by age group in the last three months in 2007 and 2009 (Eurostat, 2010) 80
70
60
50
Netherlands 2007
40
Netherlands 2009
EU(27) 2007
30
EU(27) 2009
20
10
0
16‐24
25‐34
35‐44
45‐54
55‐64
The Netherlands was on par with the EU average in 2007, but has shown little growth in education and training online. 3.4.4 Impact Figure 30: impact of e‐learning in the Netherlands attitude maths
2
attitude science
1
0
attitude reading
‐1
Netherlands
‐2
science performance
average
reading performance
maths performance
52
The impact of e‐learning on learning attitude in the Netherlands is somewhat difficult to interpret. The general attitude towards science is very positive whereas the general attitude towards reading is very negative. One of the remarkable findings from the macro model is that attitudes towards science and reading are consistently negatively correlated across the entire set of countries. We have no immediate explanation for this pattern. In the Netherlands, the opposition is even more pronounced. As has been mentioned before, average time spent on reading has declined dramatically in the Netherlands during the last 25 years (De Vries, 2007). This trend could partly explain the negative attitude towards reading. One could argue that the rise of the Internet has also contributed to the decreased interest in reading traditional media. We find a similar negative attitude towards reading in the other leading Internet country, the United States. However at least in the Netherlands the decline already started at least a decade before computers were introduced to schools and roughly fifteen years before the use of the Internet really took off. In the same study, the diminished quality of reading education was mentioned as one of the reasons for the deterioration of the attitude towards reading.27 This study indicates some other possible causes. Even though the intensity of reading for information outside of school is above average, it seems that Dutch children are overloaded: intensity of reading for homework is well above average, but has a negative correlation with attitude in the Netherlands. This line of reasoning can be expanded in the light of the Dutch school system: The higher the level of the student, the higher the intensity of reading for homework and thus the stronger the negative influence of this factor. With a high reading intensity for homework, students at all levels have to read a lot. This results in a decreased influence of student background on reading performance. Another factor that negatively contributes to the attitude towards reading is the attitude of the parents towards reading. This attitude is relatively low in the Netherlands, and is thought to cause a downward spiral (negative attitude from parents is reinforced in the attitude from their children). The link between performance and e‐learning is not straightforward. One of the main findings of the micro model is that the use of ICT has a positive impact on attitude towards learning, and a key finding from both the micro and macro model is that this in turn has a positive impact on performance. However this line of reasoning only seems to apply to science and math, not to reading. Spain has a very positive attitude towards reading but the country clearly lags behind the U.S. and the Netherlands with regard to e‐learning Readiness and Use. 27
Other possible reasons that were mentioned were: an increase in leisure time activities, increased choice of books (which causes a choice problem), increased pressure in life, decrease in opportunities for transfer of culture (an increase in time spent with the peer group from school and less with parents), and the cumulative effort of decreasing attitude (parents who read less will have children that read even more less). 53
If we turn to science and math, the contribution of ICT to both attitude and performance is much clearer. First of all, the Netherlands has a strong position on most of the factors that have a positive correlation with attitude towards science. These factors include public expenditure per pupil, gross enrolment ratio in education, the absence of teaching limitations, the self‐perceived performance on ICT skills, and computer use at home. Moreover, there are several additional factors that were only found for the Netherlands: self‐perceived performance on advanced ICT skills, ICT program software use, and computer use at school. The performance of Dutch children is above average in any field of study. This might be an outcome of the general quality of the educational system. The positive attitude might also be an important driver. Attitude, in turn, is strongly correlated to the use of ICT. There are also some factors that have a direct correlation with performance. The gross enrolment ratio is linked to both attitude and performance, while computer use at home and presence of computers at home are only linked towards performance. The Netherlands scores high in these fields. On the other hand, the Netherlands scores far below average on other factors which are directly positively correlated to performance ‐‐ time for subjects in regular lessons and time for subjects in self‐study. Spain, where traditional class teaching is still the dominant model, scores very high on this particular factor. The relatively strong relation between time for subjects in self‐study and performance is remarkable.28 This could be a consequence of the Dutch school system, where the selection of students already occurs at an early stage (at the age of 12 pupils are sent to schools that match their level).29 The self‐study time is higher at higher levels of secondary education. It could be that self‐study enhances performance, but it is more likely that the best students use more time for self‐study. To sum up, the implementation of e‐learning in school programs has not progressed as far as in the U.S. but is still above average. The Dutch situation is characterized by intense computer use at home and a very positive attitude towards science. Both factors contribute to performance. The direct impact of the use of ICT on schools on performance seems to be limited. Hence the fact that ICT is not (yet) really interwoven with regular teaching – which is mainly due to a lack of advanced ICT skills from the teachers – does not seem to be a major hindrance to the Netherlands. The significant correlation between contextual topic description and education performance is also typical for the Netherlands (and Spanish) situation. This correlation indicates that the way materials are presented (contextual versus conceptual) also influences the performance. The introduction of contextual topic description is closely aligned with new, more practice oriented, didactic and pedagogical methods (e.g., the ‘New Learning’ paradigm in the Netherlands). These 28
R = 0.20 for the Netherlands while the average is below 0.10. Generally speaking there are three levels of secondary education in the Netherlands: lower (VMBO), medium (HAVO) and higher (VWO). 29
54
methods also require a much more active role for the pupil. Advanced e‐learning activities such as the use of online simulations and experiments fit very well with the emphasis on independence (active role) and practice (contextual topic description). Hence the introduction of new teaching methods provides an excellent opportunity for the further diffusion of e‐learning, as is witnessed by the example of the Amadeus case (see section 1, Box 1). 55
4
Overall conclusions and implications for further research In this concluding section, we summarize the most important findings and indicate some implications for further research. What should be stressed first is that the field of e‐Learning is developing rapidly. Readers should bear this in mind when interpreting the concluding remarks below. 4.1 Conclusions 1. Influence of e‐learning Readiness and Use on Impact is limited The analysis does not reveal a strong direct relationship30, between the presence and use of ICT and educational performance as traditionally measured. We are not able to provide a definitive explanation for this result. Thus, it may be the case that effective use of ICT is able to make a contribution to performance, but that in many of the cases where ICT was being used at that time (2006, 2007) it was not being used effectively. Another possible explanation is that it might take some incubation time in order to find an effect of ICT on educational performance. The recent OECD report (OECD 2010) in which PISA 2006 results were analyzed revealed a moderate relation between ICT familiarity / use (at home) and educational performance (after controlling for socio‐economic status). We found a similar effect from computer use at home in PISA, but this was not confirmed in PIRLS and TIMSS. In fact, it is more likely that intelligence implies a broad interest in general (including ICT). Computer use at school was neither in our study nor in the OECD study related to educational performance, which emphasizes the bottom line of conclusion 1. 2. The impact of the Use of ICT on Impact is mainly indirect, via Attitude Whereas the direct influence of the use of ICT on performance is limited, it is considerably stronger on attitude. Attitude, in return, has a positive correlation with performance. We found the latter relation in all three studies, for all three subject matters (math, science, and reading) and in all reference countries. Thus indirectly, the use of ICT (both at home and at school) contributes to education performance. In one particular case we found a direct relation between a Readiness indicator (presence of computers at home) and performance. However in this particular case the causal direction is the other way around: students who perform well at school come from a background that is both conducive to school performance as well as the propensity to own computers.31 This argument is supported by the fact that whereas the availability of ICT at home is positively correlated to performance, the use of ICT at home is negatively related. The latter can be explained by the fact that time spent on computers at home (for social or entertainment purposes) cannot be spent on school work. 30
One important disclaimer is that the data from PISA, TIMSS and PIRLS is from respectively 2006, 2006 and 2007. At least in the Netherlands, the use of ICT in secondary and especially primary education has grown rapidly in since then. 31
But note that we did control for the intellectual background of the parents. 56
Note that the same line of reasoning might also apply to Attitude. However we found a lot of ICT‐related indicators that were only linked to Attitude and not to performance. 3. Readiness contributes to Use It will come as no surprise that the availability of ICT is conducive to the use of ICT. The relationship should rather be stated in the negative: without availability there is obviously no use. However the strength of the relationships we found was weaker than we expected, and even weaker with regard to education (thus the correlations between e‐learning Readiness and e‐learning Use). The difference can probably be explained by the importance of preconditions for use. The most important Readiness factor here is ICT skills of teachers. One of the reasons investments in ICT have not yet translated into improved learning performance is that the investments in hard infrastructure (computers and Internet connections) always have to be accompanied by investments in soft infrastructure (organizational changes and skills). Teachers should play a critical role not just by becoming proficient themselves, but also in fully integrating technology into their teaching methods. The United States has made the biggest investments in ICT skills of teachers. This certainly seems to pay off. For instance, the investments in IT‐integration courses for math and science have a strong positive influence on the actual use of ICT in math class. It is also only in the U.S. that we found a direct positive correlation between the use of ICT at school and performance. Note that in general the use of ICT at home is negatively correlated to performance. The favorable situation at schools does not apply to the situation at home. Especially in the U.S., children seem to use computers at home mainly for entertainment purposes. In the particular case of Spain, we found a negative correlation between e‐
learning Readiness (availability of ICT at school) and e‐learning Use (teaching limitation due to lack of ICT). Computers are hardly being used in teaching yet (which is due to a lack of ICT skills of teachers, see above) even though computers are available. It should be repeated, however, that Spain seems to have made much progress in the last couple of years with regard to implementation of ICT in the field of education. 4. Difference between attitude toward science, math and reading The thesis that ICT has an indirect impact on performance via attitude is only valid at the level of individual subjects and cannot be made in general. The patterns we found for science and math are distinctively different than the patterns we found for reading. For all three subjects we found a robust correlation between attitude and performance. Yet when it comes to the impact of the use of ICT it is only positive for science and math but negative for reading. The contrast between science and math, on the one hand, and reading on the other hand is in fact so strong that we found a strong negative correlation in the macro model (thus across all countries) between attitude towards science and math and attitude towards reading. In a similar vein, whereas the use of 57
computers at home has a positive relation with the attitude towards science and math (and the latter thus on performance in science and math), it has a negative relation to reading performance. With regard to the latter factor it should be noted that the impact of ICT is notoriously hard to isolate from overall organizational changes. In the Netherlands, for instance, the use of ICT in education is closely related to the introduction of new ways of teaching with a much stronger focus on the autonomy and independence of pupils (see Box 1). 5.
Mixed results of the organizational changes accompanying e‐learning obscure the actual impact of the use of ICT on education performance In general, the benefits of ICT can only be fully exploited when the implementation is accompanied by the necessary organizational changes (see 4). It makes little sense to put new technology into old organizations. In the domain of education, the use of ICT is closely related to the introduction of new forms of learning and teaching. Traditional class teaching is increasingly under pressure. ICT (and in particular the rise of the Internet) is regarded as a cause and an enabler. The current young generation has grown up with the Internet. The traditional class teaching does not seem to fit very well with the learning style of these digital natives. In the Netherlands, the introduction of ‘new learning’ has been officially adopted by the Ministry of Education. The aim of the new methods is to decrease the gap between the learning environment and the environment in which the pupils will later have to live and work. The policies seek to shift more responsibility to pupils and to translate teaching material into practical, real‐time assignments (e.g., arithmetical problems are translated into narratives). The first change renders much more independence to students; the second change involves a shift from conceptual topic description (the traditional method) into contextual topic description. So far the results of the introduction of the ‘new learning’ have been mixed. This is also explained by the control variables in our model. ‘Independence of students’ is negatively correlated to educational performance. One of the reasons is that time spent on self‐study during school hours is at the cost of traditional class teaching hours (that is, ‘instructional time at school’) and the latter variable is strongly positively correlated to performance. On the other hand, ‘topic description’ is also positively correlated to performance, albeit less strongly. Proponents of the ‘new teaching’ have argued that the negative impact on performance is partly due to the fact that the judgment of the new system is still based on the old system (e.g., traditional grading) and that it should be judged by its own standards instead (e.g., increased independence of pupils). Nevertheless, many schools in the Netherlands have already backtracked on the more radical forms of ‘new learning’. That is, they are increasing the number of hours in class again vis‐a‐vis the hours for self‐study. The successful further introduction of e‐
58
learning will very much depend on the degree to which it is able to bridge the gap between traditional class teaching and the digital natives. The goal for the educational field should be to position E‐learning in an effective manner: to find the right balance between traditional and innovative (ICT‐driven) teaching methods. 4.2 Implications for further research To conclude this chapter, we address some implications and possibilities for further research: 




The field of E‐Learning is not new, but indicators to monitor its development (and impact) in an international setting are scarce. The foundation of this study was formed by three international performance assessments that “as a by‐product” collected some ICT‐related indicators. However, traditional impact and achievement indicators cannot measure all of the skills imparted by E‐Learning. There is a need to develop more in depth E‐learning related indicators (e.g. audiovisual and presentation skills, the electronic learning environment, serious gaming) from different perspectives (e.g. student, school, teacher, parent) is evident. This would boost comparative research within this field. Besides the necessary “hard” indicators, there should also be sufficient attention for the contextual learning environment. This can manifest itself at the level of the school, the region or the country. This study revealed some large differences between the U.S., Spain and the Netherlands. In order to understand and explain the development of ICT in education, it is crucial to understand regional and country differences, policies and organizational changes. Against the background of available indicators and contextual differences, another important research issue is effective use of ICT. Countries have different approaches with regard to adaptation of teaching methods with ICT. Again, the vision of the position of ICT in education plays a crucial role here. Nonetheless, researchers should monitor different visions and approaches of implementing ICT in education closely in order to distinguish successful from unsuccessful cases. Another interesting conceptual point of view, which might deserve attention in a research context, is that of cause and effect; to resolve, for example, the question: Does ICT lead to better educational performance or is better educational performance a predictor for (more?) ICT in a school? Arguments for both sides can be found and it would be interesting to resolve this matter. To conclude, to better grasp the effect of ICT in an educational setting, it would be very interesting to make a pairwise comparison based on future 59
international performance assessments. In other words: What is the difference in educational performance and/or ICT use within one specific school in one specific country? This would better illustrate the potential role of ICT with regard to educational performance. 60
5
Literature Allen, E., Seaman, J. (2008). Staying the Course: Online Education in the United States. [online] http://www.sloan‐
c.org/publications/survey/pdf/staying_the_course.pdf [accessed 12‐12‐2009] Available at: www.govhs.org [accessed 7‐12‐2009] Balanskat, A., Blamire, R., Kefala, S. (2006) The ICT Impact Report: A Review of Studies of ICT Impact on Schools in Europe. European Schoolnet(EUN). [online] http://insight.eun.org/shared/data/pdf/impact_study.pdf [accessed 12‐12‐2009] Business Monitor International (2009). Spain Telecommunications Report Q1 2009. London: BMI. Brennenraedts, R. (2007). The Internet as Data Source. Music: Lega, illegal, digital and analogue. Dialogic: Utrecht. Available at: http://www.ez.nl/dsresource?objectid=157710&type=PDF [accessed 26‐01‐2010] Brown, P. (2009). Microsoft encroaching on Spanish schools… again. Linuxpromagazine. 13‐21‐2009. Available at http://www.linuxpromagazine.com/Online/News/Microsoft‐Encroaching‐on‐
Spanish‐Schools‐Again [accessed 10‐12‐2009] Brynjolfsson, Erik and Hitt, Lorin (2000) "Beyond Computation: Information Technology, Organizational Transformation and Business Performance," Journal of Economic Perspectives, Vol. 14, No. 4, pp. 23‐48. Brynjolfsson, Erik and Hitt, Lorin (2003) "Computing Productivity: Firm‐level Evidence,Review of Economics and Statistics. CBS (2009). Webmagazine: Uitgaven aan onderwijs Nederland net boven EU‐
gemiddelde. woensdag 17 juni 2009. [online] http://www.cbs.nl/nl‐
NL/menu/themas/onderwijs/publicaties/artikelen/archief/2009/2009‐2806‐wm.htm [accessed 8‐12‐2009] CDW‐G (2006) Teachers talk tech 2006. [online] http://www.edweek.org/media/viewpoints_cdw‐g_031507.pdf [accessed 16‐12‐
2009] Centro National de Información y Communicación Educativa (2007). Informe sobre la implantación y el uso de las TIC en los centros docentes de educación primaria y secundaria. Ministerio de Educación y Ciencia. CIA (2010). The World Factbook. [online] (updated 8‐1‐2010) https://www.cia.gov/library/publications/the‐world‐factbook [accessed 6‐1‐2009] Colecchia, A. Towards the development of an OECD methodology, Conference on the measurement of electronic commerce, Singapore, 6‐8 dec. 1999. Collins, S.R. (2004) E‐learning framework for NCLB. [online] http://www.ed.gov/about/offices/list/os/technology/plan/2004/site/documents/S.C
ollins‐e‐LearningFramework.pdf [accessed 3‐12‐2009] Concotta, H. (2008) Online learning changes the face of American Education: Internet‐based courses opening doors to knowledge and training for millions. 61
America.gov 1 January 2008 Available at http://www.america.gov/st/educ‐
english/2008/January/20080108101202attocnich0.4027674.html [accessed 3‐12‐
2009] De Leeuwe (2006). National report the Netherlands. FeConE. http://promitheas.iacm.forth.gr/fe‐
cone/docs/national%20reports/The%20Netherlands.pdf [accessed 3‐12‐2009] De Vries, N. (2007). Lezen we nog? Een inventarisatie van onderzoek op het gebied van lezen en leesbevordering. [online] http://www.lezen.nl. [accessed 6‐1‐2010] EDDY (2009). Electronic Distance‐learning for Disabled Youngsters. http://eddycollege.nl [accessed 12‐12‐2009] Eurostat (2010). Online Database, May, 2010. FeConE (2006). National report Spain. [online] http://promitheas.iacm.forth.gr/fe‐
cone/docs/national%20reports/Spain.pdf. [accessed 3‐12‐2009] Holland, C., F. Bongers, R. Vandeberg, W. Keller, R.A. te Velde (2004), “Measuring and evaluating e‐Government. Building blocks and recommendations for a standardised measuring tool”, in: M. Khosrow‐Pour (Ed.), Practicing E‐Government: A Global Perspective, Idea Group: Hershey, PA. Kennisnet (2009). Four in balance monitor 2009. [online] http://onderzoek.kennisnet.nl/onderzoeken/monitoring/fourinbalance2009 [accessed 11‐12‐2009] Lara, A.J. (2008). Elderly and use of ICT in Spain. International Cross‐Disciplinary Conference on Web Accessibility. Madrid 2009. Lewis and Setzer (2005). Distance education for public elementary and secondary school students: 2002‐03. [online] Washington: National center for education statistics http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2005010 [accessed 15‐
12‐2009] Martens, R., Gulikers, J., Bastiaens, T. (2004) The impact of intrinsic motivation on e‐
learning in authentic computer tasks. Journal of computer assisted learning, v20 n5 p368‐376 Means, B., Toyama, Y., Murphy, R., Bakia, M., Jones, K. (2009) Evaluation of Evidence‐
Based Practices in Online Learning: A Meta‐Analysis and Review of Online Learning Studies [online] Washington D.C. : Center for technology in learning. http://www.ed.gov/rschstat/eval/tech/evidence‐based‐practices/finalreport.pdf [accessed 4‐12‐2009] Microsoft (2005). Rural Primary School Uses Wireless Internet to Transform Teaching and Learning. http://download.microsoft.com/download/8/2/b/82b2555c‐b21b‐
4e91‐bdd0‐c5dbade46573/155_Arino.pdf. [accessed 4‐3‐2010] Microsoft (2008), Microsoft Aims to Create New Opportunity for Everyone Through Education. Press realease. Berlin, January 22 [accessed 4‐3‐2010] Ministerie van OCW (1997) Investeren in voorsprong 1997‐1998. Ministerie van OCW (1999) Onderwijs online 1998‐2002. Ministerie van OCW (2006) Actieplan verbonden met ICT. 62
Ministerio de educación (2010) INFORMACIÓN Y COMUNICACIÓN EN LA EDUCACIÓN EN ESPAÑA Y EUROPA, INSTITUTO DE TECNOLOGÍAS EDUCATIVAS DEPARTAMENTO DE PROYECTOS EUROPEOS. Nidap (2004). Het NIDAP bedrijfsopleidingen en e‐learning rapport OECD (2001). E‐learning: The Partnership Challenge. Paris: OECD. OECD(2010). Educational Research and Innovation: Are the New Millennium Learners Making the Grade?: Technology Use and Educational Performance in PISA 2006. Paris: OECD OECD (2009). Statistics. [online] http://titania.sourceoecd.org/vl=2216854/cl=39/nw=1/rpsv/dotstat.htm [accessed 10‐12‐2009] Public school review (2009). Find public schools by state. [online] http://www.publicschoolreview.com [accessed 15‐12‐2009] Santillana Formacion (2004), Estudio de la demanda y expectativas del Mercado E‐
Learning en Espana 2004 y 2005 Schreurs, A., and Wijnhoven, L. (2005). Leren en werken versterken: plan van aanpak 2005. ministerie van OCW and ministerie van SZ [online] http://www.leren‐
werken.nl/front/docs/leren_en_werken_pva.pdf [accessed 24‐12‐2009] Smith, R., Clark, T., Blomeyer, R.L. (2005). A Synthesis of New Research on K‐12 Online Learning. Naperville, IL: Learning Point Associates. The center for digital education ( 2009). Online learning policy: A survey of the states. [online] http://media.convergemag.com/documents/CDE09+REPORT+Nacol_Short_V.pdf [accessed 4‐12‐2009] Unesco (2006). Data centre. [online] (updated 10‐7‐2009)http://www.uis.unesco.org [accessed 15‐12‐2009] Unesco (2007). Data centre. [online] (updated 10‐7‐2009) http://www.uis.unesco.org [accessed 15‐12‐2009] US department of education (2001). PL 107‐110, the No Child Left Behind Act of 2001. [online] http://www.ed.gov/policy/elsec/leg/esea02/index.html [accessed 8‐12‐
2009] US department of education (2004). Toward A New Golden Age In American Education: how the Internet, the law and today’s students are revolutionizing expectations. [online] http://www.ed.gov/about/offices/list/os/technology/plan/2004 [accessed 4‐12‐
2009] US department of education (2009). Success stories: e‐learning and virtual schools. [online] http://www.ed.gov/about/offices/list/os/technology/plan/2004/site/stories/edlite‐
E‐Learning.html [accessed 16‐12‐2009] Van Essen, A. (2006). Gebruik van de Elektronische Leeromgeving in het Voortgezet Onderwijs voor communicatie in leersituaties. [online] 63
http://web.kennisnet2.nl/portal/onderzoek/onderzoeken/monitoring/elektronischel
eeromgeving [accessed 11‐12‐2009] Virtual High School (2009). The virtual high school. [online] (updated 2010) Watson, J., Gemin, B., Ryan, J. (2008). Keeping pace with K‐12 online learning: a review of state policy and practice). [online] http://kpk12.com/downloads/KeepingPace_2008.pdf [accessed 4‐12‐2009] websiteoptimization (2009). Study: American Leads world in broadband use – US broadband penetration jumps to 94.5% ‐ December 2009 bandwidth report. 31 December 2009. available at http://www.websiteoptimization.com/bw/0912/ [accessed 4‐1‐2010] Wikipedia (2009) Hyves. [online] http://nl.wikipedia.org/wiki/Hyves [accessed 5‐1‐
2010] Worldbank (2007). Development data & statistics. [online] http://worldbank.org [accessed 12‐12‐2009] 64
Appendix I: List of developed countries that contribute to the average in the country studies Argentina Japan Australia Korea Austria Latvia Belgium Liechtenstein Brazil Lithuania Bulgaria Luxembourg Canada Mexico Chile Netherlands Chinese Taipei New Zealand Croatia Norway Czech Republic Poland Denmark Portugal Estonia Romania Finland Russia France Serbia Germany Slovak Republic Greece Slovenia Hong Kong Spain Hungary Sweden Iceland Switzerland Ireland Turkey Israel United Kingdom Italy United States of America 65
Appendix II: PISA Results The Program for International Student Assessment (PISA) is an international study to compare the schools and students of different countries (OECD PISA 2005). The assessment consists of the following items: -
student questionnaire school questionnaire ICT questionnaire The target population for the student questionnaire is the 15‐year‐old student. The questionnaire does not pay extra attention towards a specific field of study (e.g., math, biology or language). Description of the PISA model Based on the original set of variables, factor analyses resulted in the following control, readiness, use and impact factors; see figure 31 below. A line in this figure indicates a significant relation. The numbers on or near the line indicate Pearson r coefficients. Green numbers indicate a positive relation, red ones a negative. The dotted lines represent the additional analysis of controlling for “readiness” variables on top of control for the control variables. Figure 31. General factor correlation model PISA Readiness
Actual use
Impact
(Indirect effects)
Use_Gen1
0.10
0.31
0.16
0.10
R_Gen1
0.15
Use_Gen2
Use_Gen3
0.11
-0,19
Use_Gen4
Use_Gen5
Imp_direct
-0.18
0.22
0.22
Imp_indirect
-0.19
-0.11
0.18
Use_Elear1
R_Elear1
R_Elear2
Use_Elear2
Control1
Control1
Frequency
of
instructional
activities in class
Frequency of Teacher-Student
interaction in class
Student independence in class
R_gen_1
Number of Computers at home
R_Elear1
Time for subjects in out of
school lessons
Luxury possessions at home
Use_Gen1
Use_Gen3
Control7
Frequency of contextual topic
description in class
Time for subjects in self-study
Proportion of computers for
instruction
Proportion
of
computers
connected to web at school
Self-perceived
Performance
basic ICT-skills
Self-perceived
performance
advanced ICT-skills
ICT Program/software use
Use_Gen4
ICT entertainment use
Control8
Intellectual background
Use_Gen5
Control9
Use_Elear1
Intensity of computer use not
at home
Use computer at school
Control10
Time for subjects in regular
lessons
Proportion of certified teachers
Use_Elear2
Use educational software
Control11
GDP per capita/education
Imp_direct
Performance reading, science,
math aggregated
Attitude towards science
Control2
Control3
Control4
Control5
Control6
R_Elear2
Use_Gen2
Imp_indirect
-0,11
-0,29
-0,22
Control2
Control3
Control4
Control5
Control6
Control7
0,39
0,30
Control8
Control9
Control10
0,15
Control11
66
Results of the PISA model The table below shows all statistically significant relations from the general model and the two reference countries (Spain and the Netherlands) in which the PISA study was conducted. Table 6. Correlation between indicators in PISA model Factor 1
Factor 2
Overall
Netherlands
Spain
C ONTROL
Frequency of instructional activities in class
Attitude towards science
Frequency of Teacher-Student interaction in class
Performance reading, science, math aggregated
-0.11
-0. 25
Performance reading, science, math aggregated
-0.29
-0.23
Student independence in class
0.19
Attitude towards science
Time for subjects in out of school lessons
Performance reading, science, math aggregated
Luxury possessions at home
Performance reading, science, math aggregated
-0.13
Performance reading, science, math aggregated
0.23
Frequency of contextual topic description in class
-0.22
-0.36
Attitude towards science
Time for subjects in self-study
Performance reading, science, math aggregated
Performance reading, science, math aggregated
0.2
0.39
0.38
0.38
0.3
0.31
0.32
Performance reading, science, math aggregated
0.15
N/A
Self-perceived Performance basic IC T-skills
0.31
Performance reading, science, math aggregated
0.2
Attitude towards science
GDP per capita/education
0.11
0.17
Attitude towards science
Time for subjects in regular lessons
0.16
0.28
Attitude towards science
Intellectual background
-0.27
-0.12
0.16
N/A
READINESS -> USE
Number of computers at home
Self-perceived performance advanced IC T-skills
0.1
0.12
IC T entertainment use
0.11
0.12
Intensity of computer use not at home
-0.19
Performance reading, science, math aggregated
0.1
Self-perceived Performance basic IC T-skills
0.18
Self-perceived Performance basic IC T-skills
Performance reading, science, math aggregated
0.16
Self-perceived performance advanced IC T-skills
Attitude towards science
IC T Program/software use
Attitude towards science
Intensity of computer use not at home
Performance reading, science, math aggregated
-0.19
Use computer at school
Performance reading, science, math aggregated
-0.11
Proportion of computers connected to web at school
-0.15
0.1
USE -> IMPAC T - C ontrolling for C ontrol
0.18
0.12
0.12
0.12
Attitude towards science
-0.23
-0.18
0.11
USE -> IMPAC T - C ontrolling for C ontrol+ Readiness
Self-perceived Performance basic IC T-skills
Performance reading, science, math aggregated
Self-perceived performance advanced IC T-skills
Attitude towards science
IC T Program/software use
Attitude towards science
Intensity of computer use not at home
Performance reading, science, math aggregated
Use educational software
Attitude towards science
0.15
0.17
0.12
0.11
0.13
-0.18
-0.23
-0.18
0.11
IMPAC T - C ontrolling for C ontrol
Performance reading, science, math aggregated
Attitude towards science
0.22
0.23
0.21
Attitude towards science
0.22
0.23
0.21
IMPAC T - C ontrolling for C ontrol + Readiness
Performance reading, science, math aggregated
Control variables  Impact variables Overall, the control variables have a much stronger impact on learning performance than any of the other ICT‐related variables. This suggests that the relative contribution of ICT to learning so far is very limited. From the control (background) variables, the intellectual background of 67
parents [Control8] is the strongest determinant for good learning results. The overall welfare effect – pupils in richer countries tend to perform better than pupils in less rich countries – is also present albeit much weaker. Another interesting picture that emerges from the group of control variables is that new‐
fashioned, that is, non‐class teaching methods [Control 2, 3 and 4], have a negative impact on learning performance. The lower the frequency of teacher‐student interaction in class [Control2], the higher the student independence in class [Control3], and the more time spent on subjects in out of school lessons [Control4], the lower the learning results. The other way around, the control variable Time for subjects in regular lessons [Control9] has a pretty strong positive relation with learning performance. The fact that the negative relations occur stronger in the Netherlands than in Spain can be explained by the fact that newly‐fashioned teaching methods have already made more inroads in the Dutch than in the Spanish education system. Readiness variables Use variables The number of computers at home [R_gen_1] is a good predictor for ICT use and even for performance. With regard to the first relationship, the link is obviously less strong for advanced than for basic ICT‐skills because in this case other factors besides computer ownership play a strong role. The last relationship is probably entirely spurious: both the number of computers and learning performance are influenced by the same shared cause (e.g., income of parents). Use variables  Impact variables The negative relationship between intensity of computer use not at home [Use_Gen5) and learning performance is probably due to the fact that this type of use often involves social use of computers (e.g., gaming, social networking) that is detrimental to learning performance. The relationship is entirely absent in the Netherlands because of the very high penetration rates of computers and Internet connections at home (one of the highest in the world) – there is hardly any computer use left out of home. The positive relationship between self‐perceived advanced ICT skills [Use_Gen2] and ICT program/software use [Use_Gen3] and attitude towards science is probably entirely due to self‐selection – students who use software and who think they have advanced ICT skills generally also like sciences (a.k.a. the ‘Nerd’‐argument). The negative relationship between use of computers at school and learning performance disappears when we control for Readiness. Use of computers at school does have a positive indirect effect on learning performance (that is, on the attitude towards science, see further), although this effect only occurs in the Netherlands. It might again be due to self‐selection (see before). On the other hand, in the same country we find a similar link between use of educational systems and attitude towards science. It might thus also be that science is especially suitable for the application of computer‐based education, and that the use of computers especially increases the motivation to learn science. 68
Impact variables  Impact variables There is a moderately strong positive correlation between attitude and performance. This is an important finding for E‐Learning since most of the impact of ICT is on attitude, and not on performance. Hence the contribution from E‐Learning to learning performance is mainly indirect. This does assume that the causal direction goes from attitude towards performance. It might also go the other way around: students that perform better are also better motivated. In practice, the two variables will mutually reinforce themselves. PISA results specific for The Netherlands Figure 32. PISA results for the Netherlands Readiness
Actual use
Impact
(Indirect effects)
Use_Gen1
0.10
0.18
Use_Gen2
0.12
R_Gen1
Use_Gen3
0.12
0.12
0.12
Use_Gen4
Use_Gen5
Imp_direct
0,23
Imp_indirect
-0,23
0.11
Use_Elear1
R_Elear1
R_Elear2
Use_Elear2
0,19
Control1
Control1
R_gen_1
Number of Computers at home
R_Elear1
Time for subjects in out of
school lessons
Luxury possessions at home
Use_Gen1
Use_Gen3
Control7
Frequency of contextual topic
description in class
Time for subjects in self-study
Proportion of computers for
instruction
Proportion of computers
connected to web at school
Self-perceived Performance
basic ICT-skills
Self-perceived performance
advanced ICT-skills
ICT Program/software use
Use_Gen4
ICT entertainment use
Control8
Intellectual background
Use_Gen5
Control9
Use_Elear1
Intensity of computer use not
at home
Use computer at school
Control10
Time for subjects in regular
lessons
Proportion of certified teachers
Use_Elear2
Use educational software
Control11
GDP per capita/education
Imp_direct
Performance reading, science,
math aggregated
Attitude towards science
Control2
Control3
Control4
Control5
Control6
N/A
Frequency of instructional
activities in class
Frequency of Teacher-Student
interaction in class
Student independence in class
R_Elear2
Use_Gen2
Imp_indirect
-0,25
-0,23
-0,36
-0,13
0,23
0,20
0,38
0,31
Control2
Control3
Control4
Control5
Control6
Control7
Control8
Control9
Control10
Control11
69
PISA results specific for Spain Figure 33. PISA results for Spain Readiness
Actual use
Impact
(Indirect effects)
Use_Gen1
0.12
Use_Gen2
R_Gen1
Use_Gen3
-0,15
Use_Gen4
-0,18
Imp_direct
0,21
Imp_indirect
Use_Gen5
Use_Elear1
R_Elear1
R_Elear2
Use_Elear2
Control1
Control1
Frequency of instructional
activities in class
Frequency of Teacher-Student
interaction in class
Student independence in class
R_gen_1
Number of Computers at home
R_Elear1
Time for subjects in out of
school lessons
Luxury possessions at home
Use_Gen1
Frequency of contextual topic
description in class
Time for subjects in self-study
Use_Gen3
Proportion of computers for
instruction
Proportion of computers
connected to web at school
Self-perceived Performance
basic ICT-skills
Self-perceived performance
advanced ICT-skills
ICT Program/software use
Use_Gen4
ICT entertainment use
Control8
Intellectual background
Use_Gen5
Control9
Use_Elear1
Intensity of computer use not
at home
Use computer at school
Control10
Time for subjects in regular
lessons
Proportion of certified teachers
Use_Elear2
Use educational software
Control11
GDP per capita/education
Imp_direct
Performance reading, science,
math aggregated
Attitude towards science
Control2
Control3
Control4
Control5
Control6
Control7
N/A
R_Elear2
Use_Gen2
Imp_indirect
Control2
-0,27
-0,26
Control3
-0,12
Control4
Control5
0,16
0,11
0,38
0,32
Control6
Control7
Control8
Control9
0,28
0,17
0,20
0,16
Control10
Control11
70
Appendix III: TIMSS results The Trends in International Mathematics and Science Study (TIMSS) is a study on mathematics and science education to both fourth‐grade students and eighth‐grade students (IAE 2009).32 We merely used the eighth‐grade survey for this research. The survey is put out among three different groups of respondents: students, teachers and school managers. Again, the overall model and summarizing table reveal all factors and significant relations we encountered. This will be dealt with in the following paragraph. Description of the TIMSS model Based on the original set of variables, factor analyses of the TIMSS dataset resulted in the following control, readiness, use and impact factors; see figure 34. Figure 34. General factor correlation model TIMSS Readiness
Actual use
Impact
0.18
R_Gen1
0.38
Use_Gen1
-0.10
Imp_indirect1
0.10
R_Elear1
R_Elear2
0.14
R_Elear3
0.13
R_Elear4
Use_Elear1
0.12
0.17
Use_Elear2
Imp_indirect2
0.21
0.16
0.24
0.16
Use_Elear3
-0.15
Class size
R_Gen1
ICT at home
Control2
R_Elear1
Math teaching limitations due to
lack of ICT
Science teaching limitations due
to lack of ICT
Availability of computers in class
Control6
Inadequate Science teaching
equipment
Inadequate
Mathematics
teaching equipment
Background discrepancy of
Math students
Background discrepancy of
Science students
Troublesome students
Control7
Intellectual background
Control8
Attitude towards learning
Control9
Intensity of homework
Control10
Frequency of homework
Control11
Activities outside of school
Control 12
Instruction time in class
Control 13
Home possessions
Control4
Control5
0.15
Imp_direct
Control1
Control3
(Indirect effects)
R_Elear2
R_Elear3
R_Elear4
Use_Gen1
Recently followed IT-integration
courses for math/science
ICT use at Home
Use_Elear1
ICT use in Science class
Use_Elear2
ICT use in Math class
Use_Elear3
Use of ICT for homework
Imp_direct
Imp_indirect1
Performance aggregated
(science, math)
Attitude towards math
Imp_indirect2
Attitude towards science
Control1
Control2
-0.15
Control3
Control4
Control5
Control6
Control7
Control8
0.33
0.33
0.10
0.12
Control9
Control10
-0.22
Control11
Control12
0.22
Control13
Results of the TIMSS model The table below shows all statistically relations from the general model and the United States in which the TIMSS study was conducted. TIMSS was also held in parts of Spain (Basque country), 32
The typical age of the students is 14. 71
but these findings are probably not representative for Spain as a whole and are therefore left out of analysis. Table 7. Correlation between indicators in TIMSS model Factor 1
Factor 2
Overall
USA
C ONTROL
C lass size
Performance aggregated (science, math)
-0.15
Inadequate Mathematics teaching equipment
Performance aggregated (science, math)
-0.15
Background discrepancy of Math students
Performance aggregated (science, math)
-0.29
Background discrepancy of Science students
Performance aggregated (science, math)
-0.10
Intellectual background
Performance aggregated (science, math)
Attitude towards learning
Attitude towards math
0.33
0.35
0.27
0.13
Attitude towards science
0.33
Frequency of homework
Attitude towards math
0.12
Activities outside of school
Performance aggregated (science, math)
-0.22
Home possessions
Performance aggregated (science, math)
0.22
Attitude towards science
0.1
Attitude towards math
-0.28
0.20
0.16
READINESS -> USE
IC T at home
Availability of computers in class
Recently followed IT-integration courses for math/science
IC T use at Home
0.38
0.25
Performance aggregated (science, math)
0.18
0.14
IC T use in Science class
0.14
IC T use in Math class
0.13
IC T use in Math class
0.10
0.28
USE -> IMPAC T - C ontrolling for C ontrol
IC T use in Science class
Attitude towards math
IC T use in Math class
Attitude towards math
0.1
Use of IC T for homework
Attitude towards math
0.17
0.19
Attitude towards science
0.21
0.17
Performance aggregated (science, math)
-0.1
0.10
USE -> IMPAC T - C ontrolling for C ontrol+ Readiness
IC T use at home
IC T use in Science class
Attitude towards science
0.10
IC T use in Math class
Attitude towards math
0.12
Use of IC T for homework
Attitude towards math
0.12
0.14
Attitude towards science
0.16
0.11
Attitude towards math
0.16
0.21
IMPAC T - C ontrolling for C ontrol
Performance aggregated (science, math)
Attitude towards science
0.16
IMPAC T - C ontrolling for C ontrol + Readiness
Performance aggregated (science, math)
Attitude towards math
0.24
0.23
Attitude towards science
0.15
0.15
Control variables  Impact variables TIMSS has a number of additional background variables that are correlated with learning performance. Bigger [Control1], heterogeneous [Control4] classes and lack of teaching 72
materials [Control3] obviously have a detrimental effect on learning performance. The high degree of heterogeneity in U.S. classes – students have very different backgrounds – is a handicap compared to countries with much more homogenous students, such as the Netherlands. The findings from PISA on the negative impact of newly‐fashioned, non‐class teaching [Control 11] and the positive impact of traditional class teaching [Control10] are replicated in TIMSS. We also find the welfare effect again – students from homes with more home possessions [Control13] perform better than students from homes with less home possessions. It should however be noted that the intellectual background of parents – regardless of the income – has more impact than the income level of the parents per se. Readiness variables  Use variables There is a strong and direct link between ICT Readiness [R_Gen1] and ICT Use [Use_Gen1]. However this link is considerably weaker in the U.S.. We have no immediate explanation for this difference. The link between ICT at home and performance might again be due to self‐selection (see before, PISA). The link between Readiness and Use also holds for eHealth (but only for science and math, not for reading) but is much weaker than the general case. The relatively strong correlation between use and the ICT‐skills of teachers [R_Elear3] is an important finding for the U.S. as this country has invested heavily in courses for the integration of ICT into science and math teaching. Use variables  Impact variables The positive correlation between ICT use in science and math and attitude (albeit only towards math) sheds light on the hitherto unknown causal direction between use of ICT and attitude. Since there is no self‐selection in use of ICT in class, the use of ICT explains the improved attitude, and not the other way around. In the case of use of ICT for homework we do not know whether self‐selection occurs but in the line of the previous finding we might cautiously conclude that here the use of ICT also improves the attitude (and thus indirectly the learning performance). When we correct also for Readiness, a negative correlation appears between ICT use at home and performance. This is another indication that the self‐selection (‘Nerd’) argument does not fully explain learning performance. It is not the use of ICT per se, but the way ICT is being used that matters. If ICT as home is mainly used for social uses (gaming, entertainment) the effects on learning performance might be negative (cf. PISA: intensity of computer use not at home). Impact variables  Impact variables The important correlation between attitude and performance that was found in PISA was also found in TIMSS. 73
Figure 35. TIMSS results for the United States Readiness
Actual use
Impact
0.14
R_Gen1
0.25
Use_Gen1
Imp_indirect1
Imp_indirect2
R_Elear1
R_Elear2
R_Elear3
R_Elear4
(Indirect effects)
Use_Elear1
0.10
Use_Elear2
0.21
0.19
0.16
0.17
0.28
Imp_direct
Use_Elear3
Control1
Control1
Class size
R_Gen1
ICT at home
Control2
R_Elear1
Math teaching limitations due to
lack of ICT
Science teaching limitations due
to lack of ICT
Availability of computers in class
Control6
Inadequate Science teaching
equipment
Inadequate
Mathematics
teaching equipment
Background discrepancy of
Math students
Background discrepancy of
Science students
Troublesome students
Control7
Intellectual background
Control8
Attitude towards learning
Control9
Intensity of homework
Control10
Frequency of homework
Control3
Control4
Control5
R_Elear2
R_Elear3
R_Elear4
Use_Gen1
Recently followed IT-integration
courses for math/science
ICT use at Home
Use_Elear1
ICT use in Science class
Use_Elear2
ICT use in Math class
Use_Elear3
Use of ICT for homework
Imp_direct
Control11
Activities outside of school
Imp_indirect1
Performance aggregated
(science, math)
Attitude towards math
Control 12
Instruction time in class
Imp_indirect2
Attitude towards science
Control 13
Home possessions
Control2
Control3
-0.29
-0.10
Control4
Control5
Control6
0.13
Control7
Control8
0.27
0.35
Control9
Control10
-0.28
Control11
Control12
0.20
Control13
0.16
74
Appendix IV: PIRLS results The PIRLS dataset is collected through the Progress in International Reading Literature Study Questionnaire (PIRLS) 2006 (IEA 2008). The questionnaire is divided into five main sections: 1. Student questionnaire 2. Home questionnaire (learning to read survey) 3. Teacher questionnaire 4. School questionnaire 5. Curriculum questionnaire The student questionnaire is designed for fourth‐grade students (typically age 10) and focuses on reading. Results for PIRLS are presented in a similar way as the previous studies. Description of the PIRLS model The overall model with correlations over the entire dataset is illustrated in figure 36 below. The interpretation of the lines and colors is the same as described appendix II and III Figure 36. General factor correlation model PIRLS Readiness
R_Gen1
R_Elear1
Actual use
0.19
-0.12
R_Elear2
0.15
0.10
Use_Gen1
Impact
-0.19
-0.20
Imp_direct
0.19
0.19
(Indirect effects)
Imp_indirect
Use_Elear1
Use_Elear2
Control1
0.10
0.20
Control1
Availability of teaching resources
R_Gen1
Computers at home
Control2
Intensity of reflection in reading tasks
R_Elear1
Control3
Early childhood home literacy activities
Control4
Intensity of reading for info outside of school
R_Elear2
Teaching limitations due to
lack of ICT
ICT availability in school
Control5
Intellectual background student
Use_Gen1
ICT use at home
Control6
Teacher satisfaction
Use_Elear1
ICT Software use within class
Control7
Reading aloud in class tasks
Use_Elear2
Interactive ICT use
Control8
Parent extra attention for child’s reading
Control9
Intensity reading for homework
Control10
Variety of teaching materials
Control11
Independent reading tasks
Control 12
Use of standard teaching materials
Control 13
Level of basic literacy skills
Control 14
School library diversity
Control 15
Parent attitude towards reading
Control 16
Social environment of school
Control 17
Frequency of library visits
Control 18
Study facilities at home
Control 19
Cooperation between teachers
Imp_direct
Reading performance
Imp_indirect
Attitude towards reading
Control2
Control3
Control4
0.32
0.13
0.28
Control5
Control6
Control7
-0.22
0.10
Control8
Control9
Control10
0.10
Control11
0.23
Control12
0.14
Control13
Control14
0.13
Control15
Control16
Control17
0.13
Control18
Control19
Results of the PIRLS model 75
The table below sums up the results over the three reference countries. Unfortunately, the control factors were unavailable for Spain and the United States. Table 8. Correlation between indicators in PIRLS model Factor 1
Factor 2
Overall
Netherlands
Spain
USA
C ONTROL
Intensity of reflection in reading tasks
Reading performance
0.10
Early childhood home literacy activities
Reading performance
0.20
Attitude towards reading
0.13
0.15
Intensity of reading for info outside of school
Attitude towards reading
0.28
0.30
Intellectual background student
Reading performance
0.32
0.21
Reading aloud in class tasks
Attitude towards reading
0.10
Parent extra attention for child’s reading
Reading performance
-0.22
Intensity reading for homework
Attitude towards reading
Independent reading tasks
Reading performance
0.10
Attitude towards reading
0.23
Reading performance
0.14
Level of basic literacy skills
School library diversity
Reading performance
Parent attitude towards reading
Reading performance
Frequency of library visits
Reading performance
Study facilities at home
Reading performance
-0.28
-0.17
0.38
-0.18
0.13
0.24
0.26
0.13
Attitude towards reading
0.16
0.14
READINESS -> USE
C omputers at home
IC T use at home
0.19
Teaching limitations due to lack of IC T
IC T Software use within class
-0.12
IC T availability in school
0.19
0.42
-0.11
IC T Software use within class
0.15
0.24
0.31
Interactive IC T use
0.12
0.45
0.13
IC T use at home
Reading performance
-0.19
Interactive IC T use
Reading performance
USE -> IMPAC T - C ontrolling for C ontrol
-0.23
-0.30
0.10
Attitude towards reading
0.16
USE -> IMPAC T - C ontrolling for C ontrol+ Readiness
IC T use at home
Reading performance
-0.20
Attitude towards reading
0.19
Attitude towards reading
0.19
-0.22
-0.32
0.30
0.20
0.22
0.23
0.21
0.24
IMPAC T - C ontrolling for C ontrol
Reading performance
IMPAC T - C ontrolling for C ontrol + Readiness
Reading performance
Control variables  Impact variables Similar to PISA and TIMSS we find a very strong correlation between the intellectual background of students [Control4] and learning (here: reading) performance. The relationship is less strong in the Netherlands because there are other societal factors at stake as well – there is a general deteriorating trend in reading and reading performance of children. This is confirmed by the relative importance in the Netherlands of the parents’ attitude towards reading [Control15] and the frequency of library visits [Control17]. The negative correlation between parents paying attention for child’s reading [Control8] and reading performance obviously is due to self‐selection: children who are less good in reading get more attention from their parents. In a similar vein, the negative correlation between 76
reading attitude and intensity reading for homework [Control9] and the positive correlation with independent reading tasks [Control11] could be explained: children who are pushed to read for homework generally dislike reading. The other way around: children who do a lot of independent reading tasks generally have a positive attitude towards reading. The negative correlation between school library diversity [Control14] and reading performance, which was only found in the Netherlands, is an odd case and cannot be readily explained. Readiness variables  Use variables Similar to TIMSS, we find a direct positive correlation between both general Readiness [R_Gen1] and Use [Use_gen1], and E‐Learning Readiness [R_Elear2] and Use [Use_Elear1]. In the latter case, the link with interactive ICT use [Use_Elear2] is in interesting one, as it has an indirect positive effect on the attitude towards reading [Imp_indirect] as well. The effect only occurs in the Netherlands. Not surprisingly then, this country also has a very high score on interactive ICT use. The positive relationship between interactive ICT use and attitude towards reading is probably akin to the correlation between independent reading tasks and attitude towards reading (see before, Control). The negative correlation between perceived teaching limitations due to lack of ICT [R_Elear1] and use of ICT software within class [Use_Elear1] is an obvious one. However, the extremely high positive correlation in the Netherlands is a very odd anomaly which requires careful interpretation. In the particular Dutch context, it seems that the teachers who perceive the greatest limitations due to lack of ICT nevertheless are the most avid users of ICT software in the classroom. This is because the limitations arise from the perceived potential of E‐Learning – which is seemingly very high amongst this group of teachers. In other words, they are already keen users of ICT in the classroom but they feel that they could still make much more of the use of ICT. What is probably at stake here is not so much a lack of ICT skills per se, but the inability to integrate ICT into teaching. This is in sharp contrast to the United States, where policies have paid specific attention to this issue (see before, TIMSS). Use variables  Impact variables The negative relation that we found in TIMSS between ICT use at home and learning performance is also found in PIRLS. In the latter case this does not come as a surprise since ICT for entertainment is one of the most prominent variables in the PIRLS factor [Use_Gen1]. Note, however, that the relationship is absent in the Netherlands. Again we have no obvious explanation for this deviation from the general pattern. Impact variables  Impact variables The important correlation between attitude and performance that was found in PISA and TIMSS is also found in PIRLS. Thus it also holds for reading, not just for science and math. This suggests that the relation is a general one, regardless of the specific subject. 77
Figure 37. PIRLS results for The Netherlands Readiness
Actual use
R_Gen1
R_Elear1
R_Elear2
Impact
Imp_direct
Use_Gen1
0.42
0.24
0.45
0.30
(Indirect effects)
Imp_indirect
Use_Elear1
Use_Elear2
0.16
Control1
Control2
0.15
Control1
Availability of teaching resources
R_Gen1
Computers at home
Control2
Intensity of reflection in reading tasks
R_Elear1
R_Elear2
Teaching limitations due to
lack of ICT
ICT availability in school
Use_Gen1
ICT use at home
Use_Elear1
ICT Software use within class
Use_Elear2
Interactive ICT use
Imp_direct
Reading performance
Imp_indirect
Attitude towards reading
Control3
Early childhood home literacy activities
Control4
Intensity of reading for info outside of school
Control5
Intellectual background student
Control6
Teacher satisfaction
Control7
Reading aloud in class tasks
Control8
Parent extra attention for child’s reading
Control9
Intensity reading for homework
Control10
Variety of teaching materials
Control11
Independent reading tasks
Control 12
Use of standard teaching materials
Control 13
Level of basic literacy skills
Control 14
School library diversity
Control 15
Parent attitude towards reading
Control 16
Social environment of school
Control 17
Frequency of library visits
Control 18
Study facilities at home
Control 19
Cooperation between teachers
Control3
Control4
0.21
Control5
Control6
Control7
-0.28
-0.17
Control8
Control9
Control10
Control11
0.38
Control12
Control13
-0.18
0.24
Control14
Control15
Control16
0.26
0.16
Control17
Control18
Control19
0.30
0.14
78
Figure 38. PIRLS results for Spain Readiness
Actual use
0.14
R_Gen1
R_Elear1
R_Elear2
0.19
Use_Gen1
-0.11
Impact
-0.23
Imp_direct
0.20
(Indirect effects)
Imp_indirect
Use_Elear1
0.31
Use_Elear2
0.13
Control1
Control2
Control3
Control1
Availability of teaching resources
R_Gen1
Computers at home
Control2
Intensity of reflection in reading tasks
R_Elear1
Control3
Early childhood home literacy activities
Control4
Intensity of reading for info outside of school
R_Elear2
Teaching limitations due to
lack of ICT
ICT availability in school
Control5
Intellectual background student
Use_Gen1
ICT use at home
Control6
Teacher satisfaction
Use_Elear1
ICT Software use within class
Control7
Reading aloud in class tasks
Control8
Parent extra attention for child’s reading
Control9
Intensity reading for homework
Control10
Variety of teaching materials
Control11
Independent reading tasks
Control 12
Use of standard teaching materials
Control 13
Level of basic literacy skills
Control 14
School library diversity
Control 15
Parent attitude towards reading
Control 16
Social environment of school
Control 17
Frequency of library visits
Control 18
Study facilities at home
Control 19
Cooperation between teachers
Control4
Control5
Control6
Control7
Use_Elear2
Interactive ICT use
Imp_direct
Reading performance
Control8
Imp_indirect
Attitude towards reading
Control9
Control10
Control11
Control12
Control13
Control14
Control15
Control16
Control17
Control18
Control19
79
Figure 39. PIRLS results for the United States Readiness
Actual use
Impact
0.16
R_Gen1
Use_Gen1
R_Elear1
Use_Elear1
R_Elear2
-0.30
Imp_direct
0.22
(Indirect effects)
Imp_indirect
0.10
Use_Elear2
Control1
0.12
Control2
Control3
Control1
Availability of teaching resources
R_Gen1
Computers at home
Control2
Intensity of reflection in reading tasks
R_Elear1
Control3
Early childhood home literacy activities
Control4
Intensity of reading for info outside of school
R_Elear2
Teaching limitations due to
lack of ICT
ICT availability in school
Use_Gen1
ICT use at home
Use_Elear1
ICT Software use within class
Use_Elear2
Interactive ICT use
Control5
Intellectual background student
Control6
Teacher satisfaction
Control7
Reading aloud in class tasks
Control8
Parent extra attention for child’s reading
Control9
Intensity reading for homework
Control10
Variety of teaching materials
Control11
Independent reading tasks
Control 12
Use of standard teaching materials
Control 13
Level of basic literacy skills
Control 14
School library diversity
Control 15
Parent attitude towards reading
Control 16
Social environment of school
Control 17
Frequency of library visits
Control 18
Study facilities at home
Control 19
Cooperation between teachers
Imp_direct
Reading performance
Imp_indirect
Attitude towards reading
Control4
Control5
Control6
Control7
Control8
Control9
Control10
Control11
Control12
Control13
Control14
Control15
Control16
Control17
Control18
Control19
80
Appendix V: Macro Model results Figure 40 summarizes the findings from the macro model. The lines indicate a significant relationship between variables. The red numbers near the lines indicate a significant negative correlation coefficient, the green numbers a positive one. The light blue represent a variable with insufficient cases (<=30) to interpret the correlations. Attitude towards reading (with exactly 30 cases) is a boundary case and is still included in the macro model. Figure 40. Results of the macro model 81