Conference Agenda

This paper discusses a peculiarity in institutionalized educational measurement practices. Namely, an inherent contradiction between guidelines for how scales/tests are developed and how those scales/tests are typically analyzed.

Standard guidelines for developing scales/tests emphasize the need to identify the intended construct and select items to capture the construct’s full breadth, leading items (or subsets of items) to target different aspects of the construct. This occurs in test development through specifying the test’s content domain along with a blueprint allocating items to content domains, item formats, and/or cognitive demand levels (AERA, APA, & NCME, 2014, ch. 4). Similarly, scale development guidelines emphasize identifying sub-facets of constructs, such that items can be targeted to capture each sub-facet, ensuring that the full construct is measured (e.g., Gehlbach & Brinkworth, 2011; Steger et al., 2022). These guidelines intentionally ensure that items (or subsets of items) contain construct-relevant variation that is not contained in every other item (e.g., it is recommended to include geometry-related items when measuring math ability because such items capture construct-relevant variation in math ability that is not present in, e.g., algebra-related items; c.f., Stadler et al., 2021).

At the same time, scales/tests are typically analyzed with reflective measurement models (Fried, 2020). I focus on factor models for simplicity, but the same basic point applies to item-response theory models, as a reparameterization of item-response theory models to non-linear factor models would show (McDonald, 2013). In the unidimensional factor model, the item, Xip, is modelled as Xip=(alpha_i+lambda_i*F_p)+e_p, where i represents items, p is persons, alpha_i is an item intercept, lambda_i is a factor loading, F_p is the latent factor construct, and e_p is the person-specific error. The (alpha_i+lambda_i*F_p) term can be understood as an item-specific linear rescaling of the latent factor (that is on an arbitrary scale) to the item’s scale, just as one might rescale a test to obtain more interpretable scores. The factor model, then, consists of two parts, the rescaled factor and the error term. Since each item is defined as containing a rescaling of the factor and this is the only construct-relevant variation contains in items, each item must contain all construct-related variation (i.e., all changes in the construct are reflected in each item). Note that these points are conceptual, stemming from the mathematics of the factor model, not claims about the results of fitting models to specific data.

There is a contradiction here: Scales/tests are intentionally designed so that each item (or subset of items) captures unique, construct-related variation, but analyses are conducted under the assumption that no item (nor subset of items) contain unique, construct-related variation. To have such a clear contradiction baked into the institutionalized practices of measurement in the educational and social sciences is peculiar indeed.

Reading literacy is considered an essential factor for learning and personal development (Mullis & Martin, 2015). International assessments like PIRLS are tracking trends and shaping literacy policies. They seek to evaluate global student learning, offering crucial insights into educational performance to shape informed policy decisions. Given the ongoing technological expansion and innovation, a shift in delivery mode became an inevitable progression (Jerrim, 2018). PIRLS has adapted to these changes, introducing the digital format in 2016 (ePIRLS) and achieving a significant milestone in 2021 with the partial transition to a digital assessment, through a web-based digital delivery system. Digital PIRLS included a variety of reading texts presented in an engaging and visually attractive format that were designed to motivate students to read and interact with the texts and answer comprehension questions. While considerable effort has been invested to ensure content similarity between the two formats, variations persist due to the distinct modes of administration (Almaskut et al., 2023). This creates the need for further analysis and exploration to better understand the impact of these differences on the overall outcomes and effectiveness of the administered modes.

Previous research has highlighted the presence of a mode effect, varying in magnitude, when comparing paper-based and digital assessments (Jerrim, 2018; Kingston, 2009). Jerrim's (2018) analysis of PISA 2015 field trial data across Germany, Ireland, and Sweden indicates a consistent trend of students scoring lower in digital assessments compared to their counterparts assessed on paper. Furthermore, Kingston's meta-analysis (2009) indicates that, on average, elementary students score higher on paper and exhibit small effect sizes when transitioning from paper-based to digital reading assessments. On the other hand, PIRLS 2016 was administered both in paper and digitally in 14 countries, where students in nine countries performed better in digital assessments, while only in five countries did students perform better in paper (Grammatikopoulou et al., 2024).

Formulärets överkant

Additionally, research underscores the distinct consequences of printed and digital text on memory, concentration, and comprehension (Delgado et al., 2018; Baron, 2021). Furthermore, previous findings support the fact that there is variation when it comes to the factors influencing performance in these two modes. Time spent on internet and computer use for school was found as a significant predictor of digital assessments, but not of paper-based (Gilleece & Eivers, 2018).

The present study

Sweden was among the 26 countries out of 57 that administered the digital format in PIRLS 2021. Another paper-based text -replicated from PIRLS 2016- was also administered to a ‘bridge’ sample. To maintain consistency across formats, both digital PIRLS and paper PIRLS share identical content in terms of reading passages and questions. However, digital PIRLS utilizes certain features and item types that are not accessible in the traditional paper and pencil mode. The digital version showcased advantages such as operational efficiency and enhanced features, while maintaining content consistency with the paper format. The primary aim of the present study is to investigate a potential mode effect between digital and paper formats, if there, and explore any variations in reading achievement between the two formats. Despite advancements in digital assessment, there remains a gap in our understanding of how the shift from traditional paper-based assessments to digital formats may impact reading literacy outcomes. By delving into these potential differences, we aim to contribute valuable insights into the evolving landscape of educational assessments, informing educators, policymakers, and researchers about the effectiveness and potential challenges associated with the integration of digital modes in literacy evaluation.

International Large Scale Assessments (ILSA), such as PISA and ICCS, provide internationally comparative data on students' knowledge and abilities in various subjects. The results across assessments permit countries to make comparisons of their educational systems over time and in a global context. To make this possible, the implementation and the methodology on which the studies are based need to be rigorously standardized and of high quality. But even in a well-designed study, missing data almost always occurs. Missing data can reduce the statistical power of a study and can produce biased estimates, leading to invalid conclusions. The mechanisms by which missing data occurs are many. Such a mechanism emerge, for example, from studies based on low stake tests (which ILSA should be considered as). In low stake tests the students nor their teachers receive any feedback based on the students' results. Besides risking reduced validity of results from comparisons, both over time and between countries, low stake tests run the risk of giving rise to a greater proportion of missing data.

Sweden has a long tradition of high quality population administrative register data and this tradition has led us into having a great deal of data linked to the individuals via so-called social security numbers. It is relatively common for researchers and authorities to employ these high quality data in their analysis entailing more reliable results. The Swedish National Agency for Education regularly use register data when producing the official statistics and to a certain extent also when carrying out evaluation studies.

The ILSA:s, used to evaluate the condition of the Swedish schooling system, both by the Swedish National Educational Agency as well as by decision-makers and other stakeholders. To further the possibilities of secondary analyses and to increase relevance t to the national context, it is therefore pertinent to collate data from registers with data from the ILSA:s.

Historically, the Swedish National Agency for Education has only been able to link register data to ILSA data for participating students. This is because the participating students are considered as having given their consent for such linkages. However, before conducting PISA 2022 and ICCS 2022, the legal requirements (?) changed so that it became possible for the Swedish National Agency for Education to link register data also to nonresponding students, i.e. not only to the participating students.