Conference Agenda

This session combines studies that examine different forms of missing data in international comparative large-scale studies. The overall aim is to investigate current challenges that have emerged in recent years, including issues around the sample representativeness and the validity of performance measures. Five contributions from international scholars use data from the international studies TIMSS (Trends in International Mathematics and Science Study), PIRLS (Progress in International Reading Literacy Study), and PISA (Programme for International Student Assessment) to investigate missing data related issues including the bias, scaling or reliability of these data. The contributions in this session evaluate issues in achievement tests as well as background surveys.

Missing values are a practical problem in virtually all empirical surveys. In particular, non- intentional missing values that are missing by design are a problem in empirical research because they can compromise the integrity of the data. Typical problems relate to the representativeness of data or the accurate measurement of constructs. In this session, we examine missing values in international comparative assessments, looking at missing values in both achievement tests and background surveys. Furthermore, the individual contributions examine both unit and item non- response. The different contributions study both the reasons for missing data and consequences for the integrity of the data. The papers address non-response and exclusions in background surveys and performance tests in the large-scale assessments.

The first two papers look at missing values in performance data. In the first paper, the authors look at students who were excluded from PISA in Sweden and draw on the results of national tests. The other paper, which uses performance data, compares what happens when missing values on individual test items are judged to be incorrect or not administered. This study is based on PIRLS data as well as a simulation study. The other three papers look at missing values in survey data. One of them investigates how exclusion rates have changed over a 20-year period using data from TIMSS, PIRLS, ICCS and ICILS, and the authors note an increase in some countries. The remaining paper the impact of the administration mode on the survey participation comparing paper- vs online parental surveys in TIMSS.

The session consolidates research on a theme that often receives too little attention, and that is non-response and exclusion in large-scale tests and surveys. The session investigates different methodological issues related to missing data in different international assessments. The session is divided into six parts, four presentations, a discussion by a renowned expert, and an open discussion.

The OECD claims that the PISA study allows for representative statements about student performance. In some countries, however, entire schools and individual students are excluded from taking the tests. In addition, an increasing number of students do not sit the test for various reasons and are therefore missing in the data. In PISA 2018 more than 10 percent of the Swedish students in the PISA sample were excluded from the test or did not participate for other reasons. While this is exclusions and non-response is considered a problem (Anders et al., 2020; Micklewright et al., 2012), it is difficult to quantify the bias because the performance levels of the excluded/non-responding schools or their students are generally unknown. To address this problem, we constructed a unique database combining Swedish PISA data with Swedish register data. This database includes national tests results, subject grades as well as information on parental education and migration background for all Swedish students tested in PISA 2018. Moreover, our database also comprises corresponding information for the full cohort of students that was eligible to sit the PISA test (100 000 students born in 2002). We compare the performances and background data of the PISA sample with the entire population of 15-year- olds to shed light on any bias in PISA. The results of the analyses reveal certain degree of bias and cast doubt on the representativeness of the 2018 PISA results in Sweden. Based on the results of the national tests available for all students from Sweden, we find that students who participated in PISA perform on average more than one standard deviation better than students who were excluded from PISA or did not participate for other reasons. The findings are discussed in relation to the general problem of missingness in survey data as well as in relation to the comparability of results over time in PISA.

Anders, J., Has, S., Jerrim, J., Shure, N., & Zieger, L. (2020). Is Canada really an education superpower? The impact of non-participation on results from PISA 2015. Educational Assessment, Evaluation and Accountability, 33, 1, 229-249. https://doi.org/10.1007/s11092-020- 09329-5 Micklewright, J., Schnepf, S. V., & Skinner, C. J. (2012). Non-response biases in surveys of school children: the case of the English PISA samples. Journal of the Royal Statistical Society. Series A (General), 175, 915–938.

In international large-scale assessments, students may not be compelled to answer every test item; hence, how these missing responses are treated may affect item calibration and ability estimation. Nevertheless, using a tree-based item response model or IRTree, it is possible to disentangle the probability of attempting to answer an item from the probability of a correct response (Debeer, Janssen, & De Boeck, 2017). In an IRTree, intermediate individual decisions are represented as intermediate nodes and observed responses as final nodes. Intermediate and end nodes are connected by branches that depict all possible outcomes of a cognitive subprocess. For each branch, it is possible to estimate a distinct probability (De Boeck, & Partchev, 2012). In the present study, we evaluate the usefulness of an IRTree model for skipped (omitted) responses, first with a simulation study and then on PIRLS data from 2006, 2011, and 2016. In the simulation study, we tested missing at random (MAR) and missing not at random (MNAR) scenarios. Moreover, we tested four missing response treatments, simulating the strategies of different large-scale assessments.The simulation study proved that the IRTree model maintained a higher accuracy than traditional imputation methods within a high proportion of omitted answers. In a second step, the IRTtree model for skipping responses was implemented for data of the last three cycles (2006, 2011, and 2016) of the Progress in International Reading Literacy Study (PIRLS). Correspondence between the official PIRLS results, a Rasch model, and the IRTree model was compared at three levels: items, students, and countries. We found some differences between the PIRLS and the Rasch model estimates at the item level; however, these do not significantly impact either the estimation of student ability or the country means and rankings. Moreover, the correlation between the scores estimated by the Rasch model and the IRTree model at the student level is high; however, it is not linear. In general, the results showed that while a change in the model may impact specific countries, it did not significantly impact the overall results or the country rankings. Nonetheless, when the information is disaggregated to compare a country's results over time, it is possible to observe how the increase or decrease in the proportion of skipped items can affect their overall results.

Debeer, D., Janssen, R., & De Boeck, P. (2017). Modeling Skipped and Not-Reached Items Using IRTrees. Journal of Educational Measurement, 54(3). 333-363. https://doi.org/10.1111/jedm.12147 De Boeck, P., & Partchev, I. (2012). IRTrees: Tree-Based Item Response Models of the GLMM Family. Journal of Statistical Software, 48, 1-28. https://doi.org/10.18637/jss.v048.c01

Cross-national comparisons of educational achievement rely upon each participating country collecting nationally representative data. When it comes to missing data, researchers would usually think of omitted answers, not reached questions, and perhaps more general non- response due to non-participation. However, in ILSAs, in most countries specific parts of the international defined target population are - due to various reasons - not considered as of interest or inaccessible, which results that right from the start a part of the population is disregarded. While obtaining high response rates are a key part of reaching this goal, other potentially important factors may also be at play. As noted by Anders et al (2021) and Jerrim (2021), response rates are only part of the story. Other factors – such as the precise definition of the target population and decision about how many schools/students to exclude – also have an impact as well. When taken together, this can result in the data collected having sub-optimal levels of population coverage, jeopardizing a key assumption underpinning cross-country comparisons - that the data for each nation is nationally representative. The paper focuses upon one such issue – exclusion rates – which has received relatively little attention in the academic literature. We elaborate on the causes of missing out a part of the target population, how these are calculated and reported, and how they changed over time. The data we analyze about such exclusion rates are drawn from all IEA studies conducted between 1999 and 2019. This incorporates six rounds of TIMSS, four rounds of PIRLS, two rounds of ICCS and two rounds of ICILS. All countries that took part in any of these studies/cycles are included. Using descriptive analyses (e.g. benchmarks, correlations, scatterplots) and OLS (ordinary least squares regression model) methods, we find there to be modest variation in exclusion rates across countries, and that there has been a relatively small increase in exclusion rates in some over time. We also demonstrate how exclusion rates tend to be higher in studies of primary school students than for studies of secondary school students. Finally, while there seems to be little relationship between exclusion rates and response rates, there is a modest association between the level of exclusions and test performance. Given the relatively high and rising level of exclusions in some countries, it is important that exclusion rates do not increase any further and – ideally – start to decline.

Anders, J., Has, S., Jerrim, J., Shure, N., & Zieger, L. (2020). Is Canada really an education superpower? The impact of non-participation on results from PISA 2015. Educational Assessment, Evaluation and Accountability, 33, 1, 229-249. https://doi.org/10.1007/s11092-020- 09329-5 Jerrim, J. (2021). PISA 2018 in England, Northern Ireland, Scotland and Wales: Is the data really representative of all four corners of the UK? Review of Education, 9(3). doi:10.1002/rev3.3270

International large-scale assessments (ILSA) administrate home questionnaires to parents or guardians to collect important information regarding students’ home context and early learning experience literacy that cannot be surveyed from students directly. Non-participation rates in these home surveys are often much higher than in student surveys and thus the representativeness of these surveys may be in jeopardy. Non-participation in these questionnaires may result in biased estimates. To make the home surveys more accessible, countries now have the option of administering them either via a paper-and-pencil or a computer-based format (e.g., DeLeeuw, 2018). In this study, we investigate two conditions related to parent’s non-participation. Firstly, we will investigate the non-participation bias accordingly to different levels of non-participation in home questionnaires in both TIMSS and PIRLS studies. Secondly, we investigate whether non-response rates can be attributed to an administration mode effect. To identify this mode effect, we take advantage of the international data from several TIMSS and PIRLS cycles and compare the participation rates on the home questionnaires before and after the switch from a paper-and-pencil format to an online survey system. Based on data from more than 70 countries, we examine the effect of changing the form of administration in fixed-effects analyses for countries and study cycles. In further analysis, we examine the interaction between respondents’ characteristics (e.g., education level) and mode effect on the participation rates. We find lower participation in almost all countries that switch to online surveys, with dramatic differences of up to about 20 percent. Furthermore, we find evidence that especially socially weak families participate less in the surveys.

DeLeeuw, E. D. (2018). Mixed-Mode: Past, Present, and Future. Survey Research Methods, 12(2), 75-89. https://doi.org/10.18148/srm/2018.v12i2.7402