Conference Agenda

Large Scale Assessment Data (LSA) is a rich source of data on educational achievement, educational inequalities and other socio-psychological indicators surrounding schooling and learning across the world (e.g. Mullis & Martin, 2017; OECD, 2019). Exemplary are the six PISA cycles between 2000 and 2015 which resulted in over 650 peer-reviewed articles in the English language only (Hopfenbeck et al. 2018, p.337). LSA studies have also been increasingly pooled to create quasi-longitudinal data (i.e. repeated cross-sections of representative student populations) or to track a cohort from primary to secondary school (e.g. by pooling PIRLS and PISA studies). These approaches extend the boundaries of cross-sectional data and allow for closer approximations of causal inference. Productive examples of this research include studies on the effect of educational systems and institutions (e.g. Hanushek & Woessmann, 2006; Teltemann & Schunck, 2020). LSA studies like PISA, TIMSS and PIRLS therefore provide increasing opportunities for secondary and quasi-longitudinal data analysis, covering various research areas.

Yet, the richness and complexity of LSA data can also be a challenge for researchers, since over 16 000 items have been collected in 24 studies over the last 25 years. Our research project aims at helping researchers find their way around the dense jungle of observed items and constructs in LSA studies.

Working with LSA is generally not trivial, since it involves complex sampling structures and (often) so-called plausible values as measures of student achievement. However, another factor is the diversity and sheer number of items which have been collected since the first TIMSS study in 1995. LSA studies provide items from the perspective of students, parents, teachers and principals which are collected by different questionnaires that can vary considerably within (e.g. PISA 2012 to 2015) and between studies (e.g. TIMSS-8 2015 and PISA 2015). According to our own estimations, the items collected by TIMSS, PIRLS and PISA between 1995 and 2019 sum up to roughly 16 000.

The amount of items but also the differences in questionnaires over time and across studies pose obstacles for researchers, especially those who are interested in pooling several cycles and/or studies. In our own experience, pooling several LSA studies is quite challenging partly, because it is unclear from the beginning when and how certain constructs were collected and whether they can be compared across studies. As a consequence, the rich potential of LSA data is still not fully exploited. The aim of our project is to provide a comprehensive database of all items covered in regular cycles of PISA, TIMSS and PIRLS since 1995. The resulting database would include the technical name of the items, the question text, answer categories, official scales, countries covered and rates of missingness.

Most importantly, however, our research attempts to connect similar and comparable items within and across studies in order to facilitate harmonization. This step creates an underlying network structure of our database, in which ties between single items indicate their resemblance to each other. By making our data publicly available, we hope to facilitate further research with LSA data in general and research with pooled (e.g. pseudo-longitudinal) LSA studies in particular.

We believe that such a database is an important contribution to educational research. Whereas several valuable contributions exist that facilitate the use of LSA’s complex sampling structure (e.g. Breit & Schreiner, 2016), plausible values (e.g. Breit & Schreiner, 2016) or survey weights (Jerrim et al., 2017), there is so far very limited help to find a way around the dense jungle of 16 000 LSA survey items.