Identifying the factors influencing mathematical literacy in several Spanish regions

 

 

Ana María Lara-Porras^I; María del Mar Rueda-Garcia^I; David Molina-Munoz^II

^IDepartment of Statistics and Operational Research, Faculty of Sciences, University of Granada, Granada, Spain. mrueda@ugr.es
^IIDepartment of Didactics of Mathematics, Faculty of Educational Sciences, University of Granada, Granada, Spain

 

 

---

ABSTRACT

The aim of this paper is to analyse the students' performance in the mathematical competency aspect of the Programme for International Student Assessment (PISA) 2015 tests and to compare 5 Spanish regions in this respect - Navarra, Castile-León and Catalonia, with results above the national average; and Extremadura and Andalusia, with results below the national average - in order to identify the factors causing these differences. To do so, we computed the degree of association between variables related to the students and to the schools with the scores obtained in the mathematical tests. The purpose of this analysis was to better understand the meaning of these scores and their causes and, above all, to propose educational policy actions for improving the students' mathematical performance. In this study, a 2-level regression model was applied to the data collected in the PISA 2015 tests. The first level included the factors related to the students and the second was composed of the variables related to the schools. Our results highlight the significant influence of factors such as immigrant status, grade repetition, location of the school (rural or urban) and economic and sociocultural status. The relevance of these factors to students' academic performance has been observed in previous editions of the PISA tests. We emphasise the need for action to improve students' mathematical performance and, therefore, their educational success.

Keywords: educational assessment; mathematical comprehension; multilevel analysis; performance factors; PISA; regions

---

 

 

Introduction

The PISA is a macro study that has been conducted by the Organisation for Economic Co-operation and Development (OECD) every three years since 2000. The results of the PISA tests are used to evaluate the effectiveness of educational systems and to compare countries in terms of educational success. Three subject areas - science, reading, and mathematics - are evaluated in each edition.

Attention is usually devoted to the countries where the highest scores are recorded, in order to identify and incorporate the factors producing these good results. In international comparisons, Spain has experienced significant stagnation in the three competencies considered. Thus, according to the 2009 and 2012 results, the scientific, reading and mathematical performance of Spanish students was below the OECD average in every case. In 2015, only reading skills were (slightly) above the OECD average. The PISA data is disaggregated by region in some respects, in some countries, but only those for Spain are complete. Within Spain, important differences between the regions have been detected. For example, the mathematics scores obtained in the 2015 PISA tests by students from Navarra, Castile-León and Catalonia (518, 506, and 500 points, respectively) are well above those of students from Extremadura and Andalusia (473 and 466 points, respectively). In fact, the mathematical performance of the top Spanish students is similar to that of students from Canada, Finland, and Germany (516, 511, and 506 points). At the other end of the scale, the mathematical performance of the students from Extremadura and Andalusia might be compared with that of students from countries such as Israel, Croatia and Kazakhstan (470, 464, and 460 points, respectively). This finding suggests that the international differences highlighted by PISA are reproduced on a small scale between the regions of each country. Therefore, the analysis of between-country differences in mathematical scores can be reduced to the analysis of regional differences, which in most cases is a much simpler task. In this regard, Carabana (2008) argued that the region would be a more appropriate unit of analysis than the country for the PISA tests, due to the wide regional diversity present in many countries.

Challenging socio-economic realities in the communities in which learners reside pose a significant challenge to learning and teaching (Setlhare, Wood & Meyer, 2017). On the equality dimension, stark inequalities exist in Spain between autonomous communities with respect to population groups, gender and socio-economic descent, and these inequalities can affect educational outcomes as it happens in other countries like South Africa (Wolhuter, 2014).

The above considerations motivated the present study, which compares the mathematical scores obtained in the PISA 2015 tests between various Spanish regions. This study had two main purposes: first, to identify the factors underlying the differences between regions with scores above the average, such as Navarra, Castile-León and Catalonia, and regions with scores below the average, such as Extremadura and Andalusia. According to the OECD, this difference is significant, and in the case of Navarra and Andalusia it is nearly equivalent to two academic years. On the basis of the findings obtained, we propose actions to boost the performance of low-scoring regions.

Accordingly, the following research questions guided this research:

• What proportion of the variance in mathematics performance in each of the analysed regions is situated at student and school level?

• What is the relationship between variables related to the students (gender, repeater, pre-primary schooling, immigrant status, mother and father schooling, internet connection and books at home, and socioeconomic status) and the students' mathematical performance in each of the analysed regions?

• What is the relationship between variables related to the schools (type of school, school location, student-teacher ratio, number of students enrolled, and school responsibility for curriculum, assessment, and resource allocation) and the students' mathematical performance in each of the analysed regions?

• Which actions might be considered to reduce the differences arising between regions?

Literature Review

PISA reports often originate major controversy and intense discussions in the educational environment. Some critics of the PISA approach argue that it is not suitable for measuring what it is intended to measure and criticise the strong efficiency-based conception of the study (Carabana, 2008, 2015). Other detractors highlight important methodological issues in the design of PISA (Araujo, Saltelli & Schnepf, 2017; Fernandez-Cano, 2016). On the other hand, supporters of the PISA tests cite them as a benchmark for improving student performance (Maná Lloria, 2014; Núnez, 2015; Zayas, 2012). Discussion is especially heated when a new wave of results is released, which is when the media, educational experts, and politicians perform detailed analyses of the data, interpret the results, and propose improvements.

International research has highlighted a number of factors that influence mathematics performance among learners, fundamentally related to school resources and those arising from home resources (Fardin, Alamolhodaei & Radmehr, 2011). Gender has been pointed out as one of the main factors causing the differences in the performance of the students, with boys outperforming girls in most studies. Attitudinal and sociocultural issues were highlighted as some of the reasons of this gap (Fennema, 2000; Niederle & Vesterlund, 2010; Zhu, 2018). Furthermore, students' socioeconomic status has been shown to be positively correlated with their academic performance in countries as diverse as China (Wang, Li & Li, 2014), Germany (Kriegbaum & Spinath, 2016), or Uganda (Kiwanuka, Van Damme, Van Den Noortgate, Anumendem & Namusisi, 2015). Another factor that has been traditionally identified as relevant when analysing the disparities in mathematical literacy is immigrant status. Significant differences in the mathematical performance of immigrant students in comparison with native students have been detected in many countries (OECD, 2006), although the magnitude of these differences vary depending on contextual attributes of host countries, origin countries, and communities (Levels, Dronkers & Kraaykamp, 2008).

In this study, the factors analysed with respect to the PISA 2015 test results are those related to the school, the home and the socioeconomic context of the students' families, restricted to the variables considered relevant by PISA reports and specialised papers in this field (Jamet, 2006; Molina Marfil, Marcenaro Gutierrez & Martín Marcos, 2015; Pérez & Soto, 2011; Sanchez Miguel, Garcia Pérez & Rosales Pardo, 2010). The study also considers additional variables that were incorporated in the 2009 PISA edition, including whether the student had attended pre-school. Cordero Ferrera, Crespo Cebada, Pedraja Chaparro and Santin Gonzales (2011) and Cordero Ferrera, Manchón López and Garcia Valinas (2011) report the noticeably beneficent effect of pre-primary schooling on students' subsequent academic performance. By means of multilevel models, these variables can be combined to reveal new relationships among the factors related to students and schools.

Regression models are commonly used to explain the relationship between academic performance and two or more independent variables. Visser, Juan and Feza (2015) use multiple regression analyses Trends in International Mathematics and Science Study (TIMSS) data to determine the resources factors that influence South African learners' performance in Mathematics.

Multilevel regression models are a useful means of modelling the hierarchical organisation of the PISA data, because they take into account the covariance structure. According to Blanco-Blanco, López Martín and Ruiz de Miguel (2014:142), the main contribution of multilevel regression models is "the possibility of studying the correlation between the dependent variables of the different levels considered." These models are built from the "empty" or "null" model (the one that does not include any independent variable), which is used as a reference to test the significance of alternative models. In this paper, we consider an additive approach according to which the complexity of the null model is progressively increased by adding independent variables from the two levels (Dronkers & Robert, 2008).

 

Method

The study sample was obtained from the 2015 edition of the PISA study. Multilevel regression models with two levels (student and school) were applied to conduct an in-depth analysis of the students' mathematical performance in various Spanish regions. Independent variables from the two levels were considered in each model to detect the variables related to the students and those related to the schools, to determine their influence on mathematical literacy. The results compared were obtained from students in five Spanish regions: Navarra, Castile-León, Catalonia, Extremadura, and Andalusia.

Values were missing from some of the independent variables. Accordingly, the missing data pattern was analysed for each one, and imputation techniques were used to complete the values of the predictors.

The students' mathematical performance was estimated using their plausible values in the mathematics test. The 2015 edition of PISA contains 10 plausible values for each student. In our study, these plausible values were taken as the dependent variables. Multilevel regression models were then applied, as indicated by the OECD (2017).

Variables

The variables addressed concern the environment of the students and of the school, and correspond to those considered significant in previous research into mathematical literacy (see Table 1). Three types of variables are distinguished: dichotomous (dummy), discrete, and continuous.

Multilevel regression models were applied to the data from five of the seventeen Spanish regions participating in the 2015 edition of PISA: Navarra, Castile-León, Catalonia, Extremadura, and Andalusia. These regions were selected taking into account the average score obtained in Mathematics by the Spanish students in this edition of the study (486 points). Regions with results above and below this average were analysed. Thus, Navarra, Castile-León, and Catalonia reported above-average scores, with 518, 506, and 500 points, respectively. Results for Extremadura and Andalusia were below the national average, at 473 and 466 points, respectively.

RESPCUR measures the relative level of responsibility of school staff in issues relating to curriculum and assessment. It is computed taking into account who has responsibility for the following four aspects of curriculum and assessment: establishing student assessment policies, choosing which textbooks are used, determining course content, and deciding which courses are offered. Similarly, RESPRES is an index that indicates the relative level of responsibility of school staff in allocating resources. To compute this, six items regarding who has responsibility for tasks related to resource allocation (namely, selecting teachers for hire, firing teachers, establishing teachers' starting salaries, determining teachers' salary increases, formulating the school budget, and deciding on budget allocations within the school) are considered. Both indices are calculated on the basis of the ratio of "yes" responses for school governing board, principal, or teachers in each item, to "yes" responses for regional/local education authority or national educational authority. Higher values of each index indicate relatively higher levels of school responsibility in the corresponding area (OECD, 2017).

Sample

The 2015 edition of the PISA study examined the performance of 565,000 students in 72 countries. In Spain, this involved over 40,000 students from 1,000 schools located in 17 regions. The sample size in each region was larger than in previous editions of PISA to ensure that regional results were representative.

Our study sample is composed of 9,123 students from the following regions: 1,874 from Navarra, 1,858 from Castile-León, 1,769 from Catalonia, 1,809 from Extremadura, and 1,813 from Andalusia. In these regions, the PISA tests were applied in 268 schools (of which 176 were public schools, with 6,036 students evaluated and 92 were private schools, with 3,087 students evaluated). These schools were distributed by region as follows: 52 in Navarra (32 public, 20 private), 57 in Castile-León (35 public, 22 private), 52 in Catalonia (31 public, 21 private), 53 in Extremadura (39 public, 14 private) and 54 in Andalusia (39 public, 15 private).

Data Analysis

The PISA data was separated into two levels, student and school, to facilitate the use of two-level regression models. Some authors have even considered the region or the country as a third level for modelling PISA data (Blanco-Blanco et al., 2014; Ruiz de Miguel, 2009).

Multilevel models are used to explain the maximum variance in each of the levels considered. In this study we first analysed the "empty" or "null" model (with no independent variables). The complexity of this null model was then progressively increased by adding variables from the two levels, considering separately the independent variables from the student level and those from the school level. However, a significant number of the between-school and within-school variances remained unexplained, and so the variables from both levels were then included in the model in order to analyse the relationships among all the predictors, as a whole, with the students' performance.

 

Results

The model parameters were analysed and the main results obtained are shown in Table 2. The effect of the school factor was evaluated by comparing the model that includes this effect (Model A) with the one that does not (Model B). Table 2 shows the estimation of the parameters associated with the fixed and the random parts of these models (standard errors in brackets). The -2 log likelihood (-2LL) statistic associated with each model, which is very useful for such comparisons, is also shown.

The difference between the values of the -2LL statistic was significantly distinct from zero in the five regions; therefore, we reject the hypothesis that the effect of the school factor is null. The value of the intercept of each model indicates that the average score of the students was 518.259 points in Navarra, 505.926 in Castile and León, 500.328 in Catalonia, 473.485 in Extremadura, and 469.587 in Andalusia. The corresponding standard errors show that the differences are statistically significant.

In each model, the variance of the school factor indicates the differences in the average performance of the students. Moreover, in each case the variance of the residuals reflects the differences in the performance of students who attend the same school. These two variances are significant in all five models, and so we conclude that there is an unexplained relationship between the average performance of the students and that of the schools.

The percentage of the total variance attributable to the school is obtained from the intra-class correlation coefficient, p, which explains the degree of variability between the schools in comparison with the variability between students at the same school. The values of the intra-class correlation coefficients show that 8.34%, 6.34%, 12.38%, 8.85%, and 7.88% of the difference in the students' performance is explained by the school effect in Navarra, Castile-León, Catalonia, Extremadura, and Andalusia, respectively. These are fairly low percentages, from which we deduce that the inequalities in the students' performance arise more from the students' own characteristics than from those of the schools.

As observed above, there are statistically significant differences in the performance of the students within each school that are not attributable to the school itself. Therefore, the variables related to the students that influence their mathematical performance need to be identified. This justifies our decision to expand the null model by including independent variables from the student level. Accordingly, all the variables shown in Table 1 were analysed, together with the interactions between them, but focusing on the variables with a significant influence on the students' mathematical performance. The parameters associated with these variables are shown in Table 3.

These results show that the average mathematical performance of girls was lower than that of boys in the five regions analysed. The difference, in favour of the boys, was -10.417 points in Navarra, -11.187 points in Castile-León, -18.856 points in Catalonia, -9.049 points in Extremadura, and -12.309 points in Andalusia.

In recent years, reducing the rate of grade repetition has been one of the main goals addressed in the Spanish education system. The challenge to be faced is not only the unacceptably high percentage of grade repetition but also the important differences in the results obtained by repeater students. When the variable Repeat is included in our model, the corresponding coefficient is negative and significant for all five regions. These coefficients indicate that, after controlling for gender, students who had not repeated any grade achieved an average performance that was -91.919, -89.975, -74.774, -96.688, or -98.820 points better in Navarra, Castile-León, Catalonia, Extremadura, and Andalusia, respectively, than that of students who had repeated at least one grade.

The parameter associated with the variable Pre-primary schooling was positive and significant for Navarra, Castile-León, Catalonia, and Andalusia. Students in these regions who had attended pre-primary schooling achieved average scores that were 22.591, 27.912, 30.727, and 47.041 points higher, respectively, than those who had not.

Students' immigrant status had a significant effect in Navarra (parameter value: -37.041), in Castile-León (parameter value: -14.830), and in Catalonia (parameter value: -36.847).

Regarding the parents' educational background, the mother' s education had a significant influence on the students' performance in all regions except Castile-León. The parameter associated with the father' s education was significant in all regions except Navarra. The highest value of the coefficient associated with the variable Mother schooling was recorded for Catalonia (20.070) and that for the variable Father schooling (19.274) was recorded in Andalusia.

The variable Internet at home was only significant for Castile-León. Possessing books at home had a significant effect on the students from Navarra and Andalusia.

To measure different aspects of the students' social and family environment, an index of ESCS was computed, including the parents' occupational status and education level, and the household resources available. It is assumed that the less this index influences the students' academic performance, the fairer the educational system. In our study, the variable ESCS had a positive effect on the students' performance in the five regions studied. Thus, for each point of increase in this index, the students' average mathematical performance increased, by 14.169, 5.121, 12.998, 14.038, and 13.715 points in Navarra, Castile-León, Catalonia, Extremadura, and Andalusia, respectively.

When all the independent variables of the student level are included in the model, both the within-school and the between-school variances significantly decrease with respect to the null model. Thus, the predictors of the student level contribute to explain the unexplained variance. The intra-class correlation coefficient in Navarra, Castile-León, Catalonia, Extremadura, and Andalusia decreased from 0.0834, 0.0634, 0.1238, 0.0885, and 0.0788 to 0.019, 0.039, 0.029, 0.031, and 0.032 with respect to the null model. This implies that 1.9%, 3.9%, 2.9%, 3.1%, and 3.2% of the total variance of the mathematical performance is due to the variability at student level. Moreover, the school effect decreased after including the predictors of the student level.

Alternative multilevel models were constructed, incorporating predictors from the school level into the null model. As was the case with the student level, we focused on the predictors that significantly influenced performance. These results are shown in Table 4.

The differences observed in the mathematical performance of the students can be explained, in part, by the type of school. Therefore, we first introduce the variable Stratum to the model. Table 4 shows this parameter had a significant negative value in all regions except Castile-León. In Navarra, Catalonia, Extremadura, and Andalusia the average performance of the students attending private schools was -33.444, -38.101, -38.355, and -19.009 points higher than that of the students at public schools. The school location (rural or urban area) influenced the performance of the students in Castile-León: the performance of the students attending urban schools in these regions was 25.542 points higher than that of those in rural areas.

In the next models, we progressively introduced variables describing characteristics of the schools, such as School location, Student-teacher ratio and Total school enrolment. The variable Student-teacher ratio was only significant in Extremadura (with a coefficient of 1.971) and that of School enrolment had a relevant effect only in Navarra (with a coefficient of 0.036).

The variable RESPCUR had a significant negative impact on the performance of the students in Castile-León (with a coefficient of -12.426). The increase of the school self-government in issues relating to curriculum and assessment in this region had a detrimental effect on the mathematical performance of the students and, therefore, more global policies would be preferable.

The following tables present the results obtained from models that include variables both from the student level and from the school level. Tables 5, 6 and 7 show the estimated parameters for each variable in each level. For reasons of clarity, standard errors are not shown. Table 8 shows the estimated variances, together with the likelihood ratio coefficient as a goodness-of-fit measure.

 

 

 

 

 

 

In the model incorporating all the significant predictors related to the students, the effects of the variables Gender and Repeat are negative in all five regions. The lowest value of the coefficient associated with gender was obtained in Castile-León (- 44.478) while that of the coefficient regarding student repeater status was recorded in Extremadura (- 89.241). The variable Pre-primary schooling was no longer significant in Navarra or Catalonia, but remained significant in Castile-León and in Andalusia, with similar values of the corresponding coefficients. The immigrant status of the students continued to have a negative impact on mathematical performance in Navarra and in Catalonia. These two regions present the highest percentages of immigrant population (13.33% and 17.24%, respectively). The coefficient associated with the variable Mother schooling was significant only in Andalusia and that related to the variable Father schooling remained significant only in Castile-León. The ESCS index was significant in the five regions, with similar values to those shown in Table 3 (where the predictors were introduced one by one). The interactions between variables listed in Table 3 remained significant and with similar values of their coefficients, with the exception of Repeat * Pre-primary schooling which was no longer significant in Navarra.

In these models, when all the variables of the student level were included, the variable Stratum remained significant, with a negative coefficient. The school location had a positive significant effect only in Castile-León. Furthermore, the student-teacher ratio was significant only in Extremadura. With respect to the variables Total school enrolment and RESPCUR, the former was only significant in Navarra and the latter, in Castile-León.

Finally, the variables from both levels were included in the models in order to analyse the relationship of all the predictors, as a whole, with the students' performance. In the models including individual characteristics of the students and the characteristics of the schools (see Table 7), the variable Gender was significant in the five regions. The maximum values of the coefficients associated with this variable were obtained for Castile-León and Catalonia, where the average performance of boys was 45.389 and 38.626 points higher than that of girls, respectively. The coefficient associated with the variable Gender was initially significant and negative in each of the regions, but the differences between boys and girls increased considerably when the remaining predictors and the significant interactions were introduced in each model.

The parameter corresponding to the predictor Repeat was significant and negative in all regions and its value was almost unchanged from that obtained with the models that only considered variables for the student level. The maximum and minimum values of this parameter were obtained in Extremadura (-88.468) and in Castile-León (-71.649). This means that the students from these regions who had not repeated any grade achieved an average performance that was -88.468 points and -71.649 points higher, respectively, than that of the students who had repeated at least one grade.

The variable Immigrant had a significant effect only in Catalonia (with an associated coefficient of -37.029) and in Navarra (with an associated coefficient of -35.282). In these regions, the significant influence of the immigrant population on the students' performance was due to the large number of immigrants resident, in comparison with the other regions considered.

The variables Mother schooling and Father schooling were significant only in Andalusia and in Castile-León, with corresponding coefficients of 9.470 and 8.146, respectively. It is important to note that the parents' schooling is the parameter that correlates most strongly with the students' results and that the percentage of parents with low or medium-low educational attainment in Spain is three times that of the European Union.

The students' cultural and socioeconomic status was positively related to academic results, in all regions. Thus, when the cultural and socioeconomic status increased by one unit, the students' average mathematical performance increased by 13.351, by 6.117, by 11.153, by 7.898, and by 15.881 points in Navarra, Castile-León, Catalonia, Extremadura and Andalusia, respectively.

With respect to interactions between the predictors, the variable Gender interacted with the variables Pre-primary schooling and ESCS. The former interaction was significant in Castile-León and in Catalonia, with coefficients of 25.687 and 21.280, respectively; the latter had a positive significant effect in Catalonia (9.618). The predictor Repeat interacted with the variables Pre-primary schooling, Immigrant and ESCS. The first of these interactions was significant in Castile-León (-20.742), the second was significant in Catalonia (20.669) and the third was significant in Navarra (-10.149), in Castile-León (-6.984) and in Catalonia (-9.108). The interactions Pre-primary schooling * Books at home and Mother schooling * ESCS were significant in Navarra (33.686) and in Extremadura (8.215), respectively.

The predictor of the school level Stratum was significant only in Extremadura, with a coefficient of -11.908. In the final model considering the predictors of the school level (Table 6) the effect of the variable Stratum was significant and negative in Navarra, Catalonia, Extremadura, and Andalusia. In the final model with the variables of the two levels, this predictor remained significant only in Extremadura. It would be very interesting to perform a specific analysis to determine which factors related to the individual characteristics of the students reverse the sign of this relationship.

In Navarra, in the final model with variables for the two levels, the only significant variable for the school level was Total school enrolment (with a parameter of 0.012). The effect of this variable was lower than in the model including only the variable for the school level (0.027).

The location of the school had a notable effect only in Castile-León, where the average performance of the students attending schools in urban areas was 17.010 points higher than that of those attending schools in rural areas.

Comparison of each of these models (which includes all the predictors of the student and the school levels) with the corresponding null model shows that both the within-school variance and the between-school variance decreased in all regions. For example, in Navarra, the within-school variance decreased from 6,215.451 to 4,580.629 and the between-school variance fell from 565.777 to 42.090. The joint effect of the student-related and the school-related variables very largely explains the within-class and between-class differences observed in the null model, as evidenced by the significant decrease in the residual variance.

When both groups of predictors were included, the ρ coefficient decreased in all regions in comparison with the null model, from 0.0834, 0.0634, 0.1238, 0.0885, and 0.0788 in the null model to 0.0009, 0.0207, 0.0272, 0.0244, and 0.0307 in the final model in Navarra, Castile-León, Catalonia, Extremadura, and Andalusia, respectively.

Finally, we evaluated the goodness of fit of the proposed models, comparing the likelihood ratio coefficient of each final model with its counterpart in the null model/ As an example, and for reasons of space, Table 9 shows the goodness of fit of the models corresponding only to Castile-León and to Extre-madura.

In both of these regions the three models made a significantly greater contribution than the null model. From the results obtained, we conclude that the model including variables relating both to the student and to the school is the most appropriate for explaining the mathematical performance achieved by the students evaluated in PISA 2015.

 

Discussion

Our findings contribute to those reported previously by studies such as Blanco-Blanco et al. (2014) and Topgu, Ankan and Erbigilin (2015), who show that multilevel models are an appropriate means of studying the factors that may be relevant to students' academic performance and of determining the sign of this influence. Moreover, multilevel models allow us to study interactions between the variables, and to clarify the complexity of educational phenomena.

According to the Spanish Ministry of Education, Culture and Sport (2016), the relationship between the socioeconomic and cultural level of the family and students' academic performance is often used as a measure of the fairness of the educational system. A weak relationship between these two variables may mean that the educational system reduces the impact of differences in students' social and family environments and therefore reduces the effect of these differences on academic performance.

In Spain, the average value of the ESCS index is -0.51, which is lower than that of the European Union as a whole (-0.07) and of the OECD countries (-0.04). The correlation between this index and the average score in Mathematics is positive; thus, the higher the ESCS, the higher the score obtained in Mathematics. Figure 1 shows the position of the 17 Spanish regions according to their ESCS and average mathematical score. Our results reflect the existence of a significant association between the students' average performance and the ESCS index in all regions except Castile-León, where the performance is better than expected according to the ESCS for this region (-0.44). In the remaining regions, the ESCS explains more than 50% of the variability in Extremadura (-0.79) and in Andalusia (-0.87); in the regions with higher mathematical scores, such as Navarra and Catalonia, the value of the ESCS is - 0.32 and -0.35, respectively. These data are consistent with the regions' per capita income in 2015, except, again, Castile-León (21,922 euros): the corresponding incomes for the other regions were 28,682 euros in Navarra, 27,663 in Catalonia, 17,263 in Andalusia, and 16,166 in Extremadura.

The results of our analysis indicate that in Mathematics, the performance of boys is better than that of girls. These results are similar to those obtained in previous editions of PISA. The differences between boys and girls increase when the remaining variables of the student levels are included in the models.

The behaviour pattern of the variable Pre-primary schooling was similar to that reported by Cordero Ferrera, Manchón López, et al. (2011), which underscores the importance of early schooling and of early mathematical literacy for successful mathematical performance in the future. In our analysis, the highest value of the coefficient associated with the variable Pre-primary schooling was obtained in Andalusia, the region with the lowest score in Mathematics, which highlights the relevance of this variable.

Grade repetition was found to be significant in all the regions analysed, which is in line with the results of all the editions of the PISA study. However, these results should be considered with caution due to the unequal behaviour of this variable in each region. Moreover, some studies show that grade repetition is often linked to other predictors related to academic failure (available economic and social resources, immigrant origin, the transmission of basic literacy skills before entering the school or the help with homework, among others) that should be investigated in depth (Agasisti & Cordero, 2017). Early intervention in high-risk students seems to be effective (Cordero Ferrera, Crespo Cebada & Pedraja Chaparro, 2013).

In our study, the native-born students recorded an average score in Mathematics of 492 points, which was 43 points higher than that of the immigrant students. This difference is slightly higher than that reported by the OECD and the European Union (EU) (36 and 37 points, respectively). The effect of immigrant status on academic performance is complex and should be viewed in association with other factors such as the possession of textbooks and family interest in the child' s education, as observed by Lorenzo-Moledo, Godás-Otero and Santos-Rego (2017), or the rigidity of the educational system and problems in adapting to a diversity of languages and cultures, as indicated by Fernandez Sierra (2017).

In the regions with lowest scores, private schools achieved better results than state schools, but the type of school ceases to be significant when the student-related and the school-related variables are considered jointly in the model. This is also true for the variable Student-teacher ratio (which was also indicated by Topcu et al., 2015 and by Topcu, Erbilgin & Arikan, 2016) and for the variable Total school enrolment. The effect of the school location was only significant in Castile-León, which may be explained by the dispersion of the rural population in this region and the consequent disappearance of the "competition between schools" factor.

Our results coincide in part with Visser et al. (2015) who suggest that it is not only the socio-economic factors of schools that impact learners' mathematics performance, but also that higher levels of parental education have a significant positive influence.

The reduced relevance of the variables of the school level when the family and socioeconomic characteristics are considered was previously reported by Cordero Ferrera et al. (2013). Therefore, it is important to focus on the individual characteristics of the students, since this area is where the greatest differences in academic performance arise.

This study also has its limitations: the variables related to schools are focused on the issue of resources at schools as well as school management, but other important variables were not taken into account, such as the effect that teacher classroom practice practices have on learner performance (Arends, Winnaar & Mosimege, 2017 show that teacher classroom practices affect learner performance in Mathematics significantly) because these variables are not available in the PISA survey in Spain.

 

Conclusion

From an educational standpoint, it is important to note the potential offered by multilevel regression models for identifying the factors influencing the fact that most of the 15-year-old Spanish students in our study lack the mathematical literacy that is assumed for this age group. This shortcoming may have important consequences for the students beyond academic failure, since substantial mathematical knowledge is required in daily life.

According to our analysis, repeating one or more academic years has proven to be the most relevant variable affecting the students' mathematical performance in all the regions analysed, with a negative effect. Usually, the lack of motivation and the discouragement are the underlying causes of the lower performance of the repeater students. Highlighting the importance of Mathematics and its application to everyday life and using properly contextualized mathematical tasks are means to increase students' motivation. Teachers should also upgrade their training and adapt practices and educational materials in order to incorporate into their teaching certain aspects that could improve students' acquisition of mathematical competence. In that sense, innovative teaching strategies such as the technology-based approach, the gamification or the game-based methodologies are highly recommended due to their positive effect on attitudes towards mathematics and mathematical performance (Divjak & Tomic, 2011; Drijvers, Ball, Barzel, Heid, Cao & Maschietto, 2016; Lister, 2015).

Reducing the gap between girls and boys in terms of mathematical performance is a complex challenge, since it implies to eliminate widely-accepted clichés considering that the mathematical ability is exclusive to men. Schools may contribute to this end with activities that raise the role of women along the history of mathematics and their contribution to the development of this area. In addition to including female mathematicians in part of the classroom learning, having female mentors who have been successful in mathematics is also beneficial. Teachers need to also examine their own classroom practices (for example wait time for girls vs. boys) and practices that promote self-efficacy.

The immigrant status also has a substantial negative effect on the mathematical performance in some of the regions analysed. Immigrant students face personal, social, and cultural changes that may have an adverse impact on their academic development. The most important difficulties usually come from the language barriers they find when they move to a new country. Mathematics especially allows for the use of visual teaching materials that may facilitate the learning of mathematical concepts to the immigrant students without the need to be proficient in the language of the host country.

But such measures are not enough. Taking into account the significant impact of the students' socioeconomic and cultural characteristics, it is not only educational policies that should be considered. To promote mathematical literacy and to improve the results obtained in the PISA tests, there must be greater commitment by society at all levels: political, economic, and cultural.

Apart from the concerns aroused by the stagnation of the scores obtained by Spanish students in the PISA tests, it is important to promote the acquisition of mathematical competency, and not only to improve the test scores obtained. A key point in this endeavour is to promote fairer educational systems across the regions in which the effect of the socioeconomic and cultural factors on the students' academic performance should be negligible. Delineating those "fairer educational systems" is beyond the scope of our paper but an area for further research and action. The unequal behaviour of the variables analysed in the regions with scores above the average and in those with scores below the average suggests a further study of the effect on the students' mathematical literacy of the differentiating characteristics of the regions such as the gross domestic product or the unemployment rate.

 

Acknowledgements

This study was funded by Excellence Project HUM 1413, offered by the Economics, Scientific Innovation and Employment Department of the Regional Government of Andalusia (Spain).

Authors' Contributions

AMLP was responsible for the statistical analyses; MMRG and DMM contributed to writing and editing the manuscript. All authors reviewed the final manuscript.

 

Notes

i. A widely used rule is to examine whether the difference between the likelihood ratios of the models being compared is less than twice the difference between the number of parameters considered in each model (Gaviria Soto & Castro Morera, 2005).

ii. Published under a Creative Commons Attribution Licence.

 

References

Agasisti T & Cordero JM 2017. The determinants of repetition rates in Europe: Early skills or subsequent parents' help? Journal of Policy Modeling, 39(1):129-146. https://doi.org/10.1016/jjpolmod.2016.07.002        [ Links ]

Araujo L, Saltelli A & Schnepf S 2017. "Do PISA data justify PISA-based education policy?" International Journal of Comparative Education and Development, 19(1):20-34. https://doi.org/10.1108/IJCED-12-2016-0023        [ Links ]

Arends F, Winnaar L & Mosimege M 2017. Teacher classroom practices and Mathematics performance in South African schools: A reflection on TIMSS 2011. South African Journal of Education, 37(3):Art. # 1362, 11 pages. https://doi.org/10.15700/saje.v37n3a1362        [ Links ]

Blanco-Blanco Á, López Martín E & Ruiz de Miguel C 2014. Aportaciones de los modelos jerárquico-lineales multivariados a la investigación educativa sobre el rendimiento. Un ejemplo con datos del alumnado espanol en PISA 2009 [Contributions of the multivariate hierarchical-linear models to educational research on academic performance. An example with data from the Spanish students in PISA 2009]. Revista de Educación, 365:122-149.         [ Links ]

Carabana J 2008. Las diferencias entre países regiones y en las pruebas PISA [Differences between regions and countries in the PISA tests]. Madrid, Spain: Universidad Complutense de Madrid. Available at http://www.colegiodeemeritos.es/docs/repositorio//es_ES//documentos/pisa_carabana_(vf).pdf. Accesed 30 January 2018.         [ Links ]

Carabana J 2015. La inutilidad de PISA para las escuelas [The futility of the PISA tests for schools]. Madrid, Spain: La Catarata.         [ Links ]

Cordero Ferrera JM, Crespo Cebada E & Pedraja Chaparro F 2013. Rendimiento educativo y determinantes según PISA: Una revision de la literatura en Espana [Educational achievement and determinants in PISA: A review of literature in Spain]. Revista deEducación, 362:273-297. https://doi.org/10.4438/1988-592X-RE-2011-362-161        [ Links ]

Cordero Ferrera JM, Crespo Cebada E, Pedraja Chaparro F & Santin Gonzales D 2011. Exploring educational efficiency divergences across Spanish regions in PISA 2006. Revista de Economía Aplicada, XIX(57): 117-145. Available at https://www.redalyc.org/pdf/969/96922243005.pdf. Accessed 10 October 2019.         [ Links ]

Cordero Ferrera JM, Manchón López C & Garcia Valinas MA 2011. "Los resultados educativos en PISA 2009 y sus condicionantes" [The educational results in the PISA 2009 tests and their conditiong factors]. XX Jornadas de la Asociación de Economía de la Educación. Available at http://2011.economicsofeducation.com/malaga2011/Segundo_cordero.pdf. Accessed 11 October 2019.         [ Links ]

Divjak B & Tomic D 2011. The impact of game-based learning on the achievement of learning goals and motivation for learning mathematics - Literature review. Journal of Information and Organizational Sciences, 35(1):15-30.         [ Links ]

Drijvers P, Ball L, Barzel B, Heid MK, Cao Y & Maschietto M 2016. Uses of technology in lower secondary mathematics education: A concise topical survey. New York, NY: Springer. https://doi.org/10.1007/978-3-319-33666-4        [ Links ]

Dronkers J & Robert P 2008. Differences in scholastic achievement of public, private government-dependent, and private independent schools: A cross-national analysis. Educational Policy, 22(4):541-577. https://doi.org/10.1177%2F0895904807307065        [ Links ]

Fardin D, Alamolhodaei H & Radmehr F 2011. A meta-analyze on mathematical beliefs and mathematical performance of Iranian students. Educational Research, 2(4):1051-1058. Available at https://profdoc.um.ac.ir/articles/a/1022014.pdf. Accessed 10 October 2019.         [ Links ]

Fennema E 2000. Gender and mathematics: What is known and what do I wish was known? Paper presented at the fifth annual forum of the National Institute for Science Education, Detroit, MI, 22-23 May. Available at http://archive.wceruw.org/nise/News_Activities/Forums/Fennemapaper.htm. Accessed 18 October 2019.         [ Links ]

Fernandez-Cano A 2016. Una critica metodológica de las evaluaciones PISA [A methodological critique of the PISA evaluations]. RELIEVE, 22(1):art. M15. https://doi.org/10.7203/relieve.22.L8806        [ Links ]

Fernandez Sierra J 2017. Alumnado inmigrante en la ESO: Vulnerabilidad pedagógica del sistema educativo [Immigrant students in compulsory secondary education: Pedagogical vulnerability of the educational system]. Educación XX1, 20(1):121-140. https://doi.org/10.5944/educXX1.12855        [ Links ]

Gaviria Soto JL & Castro Morera M 2005. Modelos jerárquicos lineales [Hierarchical-linear models]. Madrid, Spain: La Muralla.         [ Links ]

Jamet É 2006. Lectura y éxito escolar [Reading and school success]. Buenos Aires, Argentina: Fondo de Cultura Económica.         [ Links ]

Kiwanuka HN, Van Damme J, Van den Noortgate W, Anumendem DN & Namusisi S 2015. Factors affecting Mathematics achievement of first-year secondary school students in Central Uganda. South African Journal of Education, 35(3):Art. # 1106, 16 pages. https://doi.org/10.15700/saje.v35n3a1106        [ Links ]

Kriegbaum K & Spinath B 2016. Explaining social disparities in mathematical achievement: The role of motivation. European Journal of Personality, 30(1):45-63. https://doi.org/10.1002/per.2042        [ Links ]

Levels M, Dronkers J & Kraaykamp G 2008. Immigrant children's educational achievement in Western countries: Origin, destination, and community effects on mathematical performance. American Sociological Review, 73(5):835-853. https://doi.org/10.1177%2F000312240807300507        [ Links ]

Lister MC 2015. Gamification: The effect on student motivation and performance at the post-secondary level. Issues and Trends in Educational Technology, 3(2):1-22. Available at https://www.learntechlib.org/p/171075/. Accessed 8 October 2019.         [ Links ]

Lorenzo-Moledo M, Godás-Otero A & Santos-Rego MA 2017. Principales determinantes de la implicación y participación de las familias inmigrantes en la escuela [Main determinants of immigrant families' involvement and participation in school life].Cultura y Educación [Culture and Education], 29(2):213-253. https://doi.org/10.1080/11356405.2017.1305074        [ Links ]

Maná Lloria A 2014. La competencia lectora en la educación secundaria obligatoria: Descripción y dificultades del uso de información para responder preguntas [Reading competence in compulsory secondary education: Description and difficulties of using information to answer questions]. Cultura y Educación [Culture and Education], 26(1):184-202.         [ Links ]

Ministry of Education, Culture and Sport 2016. PISA 2015. Programa para la evaluación internacional de los alumnus: Informe espanol [PISA 2015. Programme for International Student Assessment. Spanish report]. Madrid, Spain: Author. Available at http://iaqse.caib.es/documentos/avaluacions/pisa/pisa_2015/pisa_2015_preliminar_espanya.pdf. Accessed 30 January 2018.         [ Links ]

Molina Marfil JA, Marcenaro Gutierrez OD & Martin Marcos A 2015. Educación financiera y sistemas educativos en la OCDE: Un análisis comparativo con datos PISA 2012 [Financial literacy and educational systems in the OECD: A comparative analysis with the PISA 2012 data]. Revista de Educación, 369:85-108.         [ Links ]

Niederle M & Vesterlund L 2010. Explaining the gender gap in math test scores: The role of competition. Journal of Economic Perspectives, 24(2): 129-144. https://doi.org/10.1257/jep.24.2.129        [ Links ]

Núnez P 2015. La comprensión lectora: Aspectos teóricos y didácticos [Reading comprehension: Theoretical and didactic aspects]. In J Mata, P Núnez & J Rienda (eds). Didáctica de la lengua y la literatura [Didactics of Spanish language and literature]. Madrid, Spain: Pirámide.         [ Links ]

OECD 2006. Where immigrant students succeed: A comparative review of performance and engagement in PISA 2003. Paris, France: OECD Publishing. Available at https://www.phil-fak.uni-duesseldorf.de/ew/bf/bf_veranstaltungen/ss06/HS_Bildungssoziologie/9806021E.pdf. Accessed 26 October 2019.         [ Links ]

OECD 2017. PISA 2015 technical report. Paris, France: OECD Publishing. Available at http://www.oecd.org/pisa/data/2015-technical-report/. Accessed 30 January 2018.         [ Links ]

Pérez AI & Soto E 2011. Luces y sombras de PISA. Sentido educativo de las evaluaciones externas [PISA's lights and shadows. Educational sense of external assessment]. Cultura y Educación [Culture and Educatioon], 23(2): 171-182. https://doi.org/10.1174/113564011795944758        [ Links ]

Ruiz de Miguel C 2009. Las escuelas eficaces: Un estudio multinivel de factores explicativos del rendimiento escolar en el área de matemáticas [The effective schools: A multilevel study of explanatory factors of the school performance in the area of mathematics]. Revista de Educación, 348:355-376.         [ Links ]

Sánchez Miguel E, García Pérez JR & Rosales Pardo J 2010. La lectura en el aula. Qué se hace, qué se debe hacer y qué se puede hacer [Reading in the classroom: What is done, what should be done and what could be done]. Barcelona, Spain: Graó         [ Links ].

Setlhare R, Wood L & Meyer L 2017. Exploring Group Life Design with teachers in the context of poverty related psychosocial challenges. South African Journal of Education, 37(4):Art. # 1493, 8 pages. https://doi.org/10.15700/saje.v37n4a1493        [ Links ]

Topcu MS, Arikan S & Erbilgin E 2015. Turkish students' science performance and related factors in PISA 2006 and 2009. The Australian Educational Researcher, 42(1):117-132. https://doi.org/10.1007/s13384-014-0157-9        [ Links ]

Topcu MS, Erbilgin E & Arikan S 2016. Factors predicting Turkish and Korean students' science and mathematics achievement in TIMSS 2011. Eurasia Journal of Mathematics, Science & Technology Education, 12(7):1711-1737. https://doi.org/10.12973/eurasia.2016.1530a        [ Links ]

Visser M, Juan A & Feza N 2015. Home and school resources as predictors of mathematics performance in South Africa. South African Journal of Education, 35(1):Art. # 1010, 10 pages. https://doi.org/10.15700/201503062354        [ Links ]

Wang L, Li X & Li N 2014. Socio-economic status and mathematics achievement in China: A review. ZDM Mathematics Education, 46(7):1051 -1060. https://doi.org/10.1007/s11858-014-0617-8        [ Links ]

Wolhuter CC 2014. Weaknesses of South African education in the mirror image of international educational development. South African Journal of Education, 34(2):Art. # 868, 25 pages. https://doi.org/10.15700/201412071120        [ Links ]

Zayas F 2012. 10 ideas clave. La competencia lectora según PISA: Reflexiones y orientaciones didácticas [Ten key ideas. The reading comprehension according to the PISA study: Reflections and didactic orientations]. Barcelona, Spain: Graó         [ Links ].

Zhu Y 2018. Equity in Mathematics education: What did TIMSS and PISA tell us in the last two decades? In G Kaiser, H Forgasz, M Graven, A Kuzniak, E Simmt & B Xu (eds). Invited lectures from the 13th International Congress on Mathematical Education. Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-319-72170-5        [ Links ]

 

 

Received: 8 January 2018
Revised: 30 March 2019
Accepted: 27 April 2019
Published: 31 December 2019