Conference Agenda

This workshop is designed for researchers and practitioners willing to learn about computer assisted qualitative and mixed-methods research. This hands-on session will comprise the presentation of the main options available for coding and extracting meaning from data using MAXQDA. Workshop participants will grant the possibility of testing different options, ranging from the more traditional approaches to automation using Artificial Intelligence (AI) tools.

The workshop will consist of a quick guided practice tour through the opening stages of a research data project. Main features and tasks will be practiced in detail, including importing data, creating and applying codes, performing searches and queries, writing memos, retrieving selected coded data segments, analyzing data, and reporting results using some of the special features available (e.g. summaries, QTT, reports, AI Assist). The main goal of this workshop is to provide an overview of the data analysis process relying on MAXQDA assistance, a state-of-the-art software package for qualitative and mixed-methods research. Upon completing the session, workshop participants will identify important options available for tackling their qualitative research data.

Verbi GmbH (2023), MAXQDA User Manual (https://www.maxqda.com/help-mx22/welcome)

A high number of European students face challenges in reading (Betthäuser et al., 2023; Mullis et al., 2023) and require reading support as the new Progress in International Reading Literacy Study (PIRLS) displayed last year. Consequently, it is crucial to employ formative diagnostics to record the learning levels of students, enabling the design of targeted interventions at an early stage. As a result, the German Kultusminsterkonferenz [Commission of the Conference of Ministers of Education] advocates for the early implementation of nationwide diagnostics (Köller et al., 2022). Furthermore, the PIRLS, is not only conducted through a computer-based assessment since 2016, but also has a focus point in digital forms of reading (Mullis & Martin, 2019). Formative assessment serves not only to monitor the students’ learning but also to provide ongoing feedback, refine teaching approaches, and address the unique needs of individual students (OECD, 2005; OECD, 2008). Formative assessment is essential for both students and teachers. Additionally, in the realm of monitoring and aggregating student data, is learning analytics, which plays a vital role for researchers and developers of educational applications. Researchers seek to understand students' learning behaviors and their utilization of platforms, while developers strive to leverage this information for enhancing learning platforms (SoLAR, 2021).

In the project DaF-L an adaptive, digital, and competence-oriented reading screening with aligned reading packages, consisting of literary texts and reading exercises, was developed, tested, standardized, and subsequently made available as an Open Education Resource (OER) on the online learning platform Levumi (Gebhardt et al., 2016) for primary schools in Germany.

The digital reading packages were developed for three reading ability levels, in which the students were sorted into through the screening. The packages’ reading ability levels consist of the same story line for the literary texts and the same exercise formats with some variation depending on the ability level.

One key importance of the reading packages is the digitalization. The reading packages support different ability levels and an individualized learning where students can work on the exercises at their own speed with integrated tools such as immediate individualized feedback, second try-options, and solutions. Additionally, and essential is that the reading packages can be used as a diagnostic tool, which enables teachers to support students in the best way possible. Teachers are required and encouraged to conduct and to employ diagnostics with the focus to support their students. However, it is very time consuming and difficult for teachers to execute as they often do not know what ability test/s they should convey, how to determine if the application was helpful, if students used their full potential to answer questions, where they might need help, and what tools were helpful or unnecessary in an application. Therefore, the DaF-L project provides diagnostic tools for teachers with everything they need in order to support their students.

Furthermore, through the intervention study, researchers who developed the digital reading packages received essential data. The gathered data offered insights into the reading packages, allowing for an assessment of the reading packages' strengths and weaknesses. This information was utilized to make essential adjustments, ensuring the development of the most optimal application.

Moreover, the research conducted a four-week intervention in a regular classroom setting continuously gathering data to gain a deeper understanding of students' learning behaviors and their utilization of the learning platform.

The presentation will discuss the digitalization of the reading packages with focusing on learning analytics. With the objective of exploring how learning analytics, such as examining the time dedicated to reading literary texts, time spent on answering questions, and assessing the use of the skip-button, can improve learning platforms worldwide.

Recent evidence suggests that multi-informant assessments of children and adolescents with Attention-Deficit/Hyperactivity Disorder (ADHD) and co-occurring problems are more likely to provide sufficient sensitivity and specificity for population screening and clinical use than single measures (De Los Reyes et al., 2015); however, the perspectives of children and adolescents are underrepresented in scientific studies (Caye et al., 2017; for review see Mulraney et al., 2022). The question remains whether children and adolescents are reliable sources of information about their own ADHD symptoms. This may point to the need to investigate the complex interplay between self- and other (i.e. teachers, parents) reported ADHD symptoms (e.g. hyperactivity, inattention) and other externalizing (e.g. aggression) and internalizing (e.g. anxiety) problems. This review aims to systematically analyze and examine existing empirical studies that have focused on the comparison of self-reported and other-reported ADHD symptoms and co-occurring behavior problems in children and adolescents with ADHD. The purpose is to evaluate (1) the overall inclusion of self-reports in the assessment process (2) the agreements between informants (3) which types of informants are frequently used and (4) the instruments utilized.

Student questionnaire data, typically collected via Likert-scale items, is commonly used to compare different groups of students, be it across countries or based on student characteristics such as gender and socioeconomic status (e.g., OECD, 2017). However, such analyses can lead to inaccurate conclusions as the data might be biased due to the differences in reporting behavior between different groups of students (e.g., He & van de Vijver, 2016; Kyllonen & Bertling, 2013). Students can, for example, differ in the amount of effort they put into filling in the questionnaire.

Our theoretical framework relates to reporting behavior in surveys. Terms careless responding or insufficient effort responding have been used to describe responding patterns in which respondents lack motivation to answer accurately and do not pay attention to the content of items and survey instructions (Goldammer et al., 2020). A number of approaches have been suggested to identify such careless responding, the analysis of response time (to the whole survey or parts of it) being one of them (Curran, 2016; Goldammer et al., 2020). It rests on an assumption that there is a minimum time needed to read and answer a questionnaire item (Goldammer et al., 2020). The term “speeding” has been used for responding too fast to questionnaire items to give much thought to answers (Zhang & Conrad, 2014).

The analysis of response times is a promising tool for identification of the differences in the amount of effort put into filling in questionnaire surveys between different groups of students. It could help identify careless responding (a) between different groups of students during a single wave of measurement as well as (b) changes in careless responding of particular groups of students across different waves of measurement. This could be exploited, for example, in longitudinal research studies using questionnaires (e.g., [foreign language] learning motivation studies) as well as international large-scale assessment (ILSA) studies such as, for example, Programme for International Student Assessment (PISA). So far, however, the knowledge concerning the differences in questionnaire item response times between different groups of students in the context of ILSA studies is rather limited.

Previous research has suggested that students’ reporting behavior may differ across different socioeconomic groups (Vonkova et al., 2017), encouraging further exploration of the reporting behavior-socioeconomic status relationship. In this contribution, we address this research area. Our aim is to analyze the relationship between students’ response times to achievement motivation questionnaire items and their socioeconomic status in European PISA 2015 participating countries. Our research question is: How do questionnaire response times to achievement motivation items differ between students with parents achieving different education levels in European PISA 2015 participating countries?

This paper discusses a peculiarity in institutionalized educational measurement practices. Namely, an inherent contradiction between guidelines for how scales/tests are developed and how those scales/tests are typically analyzed.

Standard guidelines for developing scales/tests emphasize the need to identify the intended construct and select items to capture the construct’s full breadth, leading items (or subsets of items) to target different aspects of the construct. This occurs in test development through specifying the test’s content domain along with a blueprint allocating items to content domains, item formats, and/or cognitive demand levels (AERA, APA, & NCME, 2014, ch. 4). Similarly, scale development guidelines emphasize identifying sub-facets of constructs, such that items can be targeted to capture each sub-facet, ensuring that the full construct is measured (e.g., Gehlbach & Brinkworth, 2011; Steger et al., 2022). These guidelines intentionally ensure that items (or subsets of items) contain construct-relevant variation that is not contained in every other item (e.g., it is recommended to include geometry-related items when measuring math ability because such items capture construct-relevant variation in math ability that is not present in, e.g., algebra-related items; c.f., Stadler et al., 2021).

At the same time, scales/tests are typically analyzed with reflective measurement models (Fried, 2020). I focus on factor models for simplicity, but the same basic point applies to item-response theory models, as a reparameterization of item-response theory models to non-linear factor models would show (McDonald, 2013). In the unidimensional factor model, the item, Xip, is modelled as Xip=(alpha_i+lambda_i*F_p)+e_p, where i represents items, p is persons, alpha_i is an item intercept, lambda_i is a factor loading, F_p is the latent factor construct, and e_p is the person-specific error. The (alpha_i+lambda_i*F_p) term can be understood as an item-specific linear rescaling of the latent factor (that is on an arbitrary scale) to the item’s scale, just as one might rescale a test to obtain more interpretable scores. The factor model, then, consists of two parts, the rescaled factor and the error term. Since each item is defined as containing a rescaling of the factor and this is the only construct-relevant variation contains in items, each item must contain all construct-related variation (i.e., all changes in the construct are reflected in each item). Note that these points are conceptual, stemming from the mathematics of the factor model, not claims about the results of fitting models to specific data.

There is a contradiction here: Scales/tests are intentionally designed so that each item (or subset of items) captures unique, construct-related variation, but analyses are conducted under the assumption that no item (nor subset of items) contain unique, construct-related variation. To have such a clear contradiction baked into the institutionalized practices of measurement in the educational and social sciences is peculiar indeed.

Reading literacy is considered an essential factor for learning and personal development (Mullis & Martin, 2015). International assessments like PIRLS are tracking trends and shaping literacy policies. They seek to evaluate global student learning, offering crucial insights into educational performance to shape informed policy decisions. Given the ongoing technological expansion and innovation, a shift in delivery mode became an inevitable progression (Jerrim, 2018). PIRLS has adapted to these changes, introducing the digital format in 2016 (ePIRLS) and achieving a significant milestone in 2021 with the partial transition to a digital assessment, through a web-based digital delivery system. Digital PIRLS included a variety of reading texts presented in an engaging and visually attractive format that were designed to motivate students to read and interact with the texts and answer comprehension questions. While considerable effort has been invested to ensure content similarity between the two formats, variations persist due to the distinct modes of administration (Almaskut et al., 2023). This creates the need for further analysis and exploration to better understand the impact of these differences on the overall outcomes and effectiveness of the administered modes.

Previous research has highlighted the presence of a mode effect, varying in magnitude, when comparing paper-based and digital assessments (Jerrim, 2018; Kingston, 2009). Jerrim's (2018) analysis of PISA 2015 field trial data across Germany, Ireland, and Sweden indicates a consistent trend of students scoring lower in digital assessments compared to their counterparts assessed on paper. Furthermore, Kingston's meta-analysis (2009) indicates that, on average, elementary students score higher on paper and exhibit small effect sizes when transitioning from paper-based to digital reading assessments. On the other hand, PIRLS 2016 was administered both in paper and digitally in 14 countries, where students in nine countries performed better in digital assessments, while only in five countries did students perform better in paper (Grammatikopoulou et al., 2024).

Formulärets överkant

Additionally, research underscores the distinct consequences of printed and digital text on memory, concentration, and comprehension (Delgado et al., 2018; Baron, 2021). Furthermore, previous findings support the fact that there is variation when it comes to the factors influencing performance in these two modes. Time spent on internet and computer use for school was found as a significant predictor of digital assessments, but not of paper-based (Gilleece & Eivers, 2018).

The present study

Sweden was among the 26 countries out of 57 that administered the digital format in PIRLS 2021. Another paper-based text -replicated from PIRLS 2016- was also administered to a ‘bridge’ sample. To maintain consistency across formats, both digital PIRLS and paper PIRLS share identical content in terms of reading passages and questions. However, digital PIRLS utilizes certain features and item types that are not accessible in the traditional paper and pencil mode. The digital version showcased advantages such as operational efficiency and enhanced features, while maintaining content consistency with the paper format. The primary aim of the present study is to investigate a potential mode effect between digital and paper formats, if there, and explore any variations in reading achievement between the two formats. Despite advancements in digital assessment, there remains a gap in our understanding of how the shift from traditional paper-based assessments to digital formats may impact reading literacy outcomes. By delving into these potential differences, we aim to contribute valuable insights into the evolving landscape of educational assessments, informing educators, policymakers, and researchers about the effectiveness and potential challenges associated with the integration of digital modes in literacy evaluation.

International Large Scale Assessments (ILSA), such as PISA and ICCS, provide internationally comparative data on students' knowledge and abilities in various subjects. The results across assessments permit countries to make comparisons of their educational systems over time and in a global context. To make this possible, the implementation and the methodology on which the studies are based need to be rigorously standardized and of high quality. But even in a well-designed study, missing data almost always occurs. Missing data can reduce the statistical power of a study and can produce biased estimates, leading to invalid conclusions. The mechanisms by which missing data occurs are many. Such a mechanism emerge, for example, from studies based on low stake tests (which ILSA should be considered as). In low stake tests the students nor their teachers receive any feedback based on the students' results. Besides risking reduced validity of results from comparisons, both over time and between countries, low stake tests run the risk of giving rise to a greater proportion of missing data.

Sweden has a long tradition of high quality population administrative register data and this tradition has led us into having a great deal of data linked to the individuals via so-called social security numbers. It is relatively common for researchers and authorities to employ these high quality data in their analysis entailing more reliable results. The Swedish National Agency for Education regularly use register data when producing the official statistics and to a certain extent also when carrying out evaluation studies.

The ILSA:s, used to evaluate the condition of the Swedish schooling system, both by the Swedish National Educational Agency as well as by decision-makers and other stakeholders. To further the possibilities of secondary analyses and to increase relevance t to the national context, it is therefore pertinent to collate data from registers with data from the ILSA:s.

Historically, the Swedish National Agency for Education has only been able to link register data to ILSA data for participating students. This is because the participating students are considered as having given their consent for such linkages. However, before conducting PISA 2022 and ICCS 2022, the legal requirements (?) changed so that it became possible for the Swedish National Agency for Education to link register data also to nonresponding students, i.e. not only to the participating students.

Flanders has a history of participating in International Large-Scale Assessment studies (ILSAs) like TIMSS, where it has often ranked highly. However, the last cycles of TIMSS have shown a gradual decline in the academic achievement of Flemish students. This has sparked a debate about the quality of education in Flanders. Between the TIMSS cycles of 2015 and 2019, Flemish students' achievement levels decreased by 14 points for mathematics and 11 points for science (Faddar et al., 2020).

Although ILSAs are crucial tools for policymakers to assess the quality of educational systems, their primary purpose is periodic benchmarking (Addey and Sellar, 2019). However, the decline found among Flemish students has prompted a deeper investigation and monitoring of the evolution of Flemish learning gains throughout the remaining two years of primary schooling, which goes beyond benchmarking. To this end, a longitudinal study based on TIMSS-2019 was set up in Flanders: TIMSS-repeat.

Using a longitudinal design, TIMSS-repeat retested students who participated in the TIMSS 2019 cycle in 2021, when most of the students were in the sixth grade of primary education. In total, 4.301 students, their teachers, and their school principals participated in TIMSS repeat. The main purpose of TIMSS-repeat was to investigate the learning gains of Flemish students during the last two years of primary school, allowing an inquiry into the connection between students' background characteristics and their learning gains for mathematics and science. Moreover, the specific timing of the data collection in May 2021, just after the school closures and quarantines due to COVID-19, allowed for additional information regarding the impact of COVID-19 to be collected. This enabled the investigation of COVID-19's impact on the learning gains in mathematics and science. The following research questions were central:

• What are the achievement gains for Flemish students in the last two years of primary education?
• What are the differential effects of student background characteristics on their achievement gains in the last two years of primary education?
• How did the COVID-19 pandemic and the resulting school closures impact student achievement gains in Flanders?

The first and second questions aim to analyze how students in Flanders progress through the last years of primary school. With these questions, we aim to reveal how students’ learning gains increase and whether specific background characteristics facilitate or hamper student learning gains. In previous TIMSS studies, it was found that home language or students’ socioeconomic status is linked to their achievement (Faddar et al., 2020; IEA, 2020). The third question seeks to provide valuable information to both researchers and policymakers regarding the impact of the COVID-19 pandemic on student learning and achievement. Not only does TIMSS-repeat in Flanders provide answers to these research questions, but it also aligns with the research goals of the TIMSS longitudinal study that is following the TIMSS 2023 cycle (The International Association for the Evaluation of Educational Achievement (IEA), 2022).

TIMSS-repeat in Flanders provided a valuable but tentative insight into Flemish learning gains during the final grades of primary education in Flanders, characterized by one of the most impactful global events of our time. In this presentation, we will discuss the different steps taken to conduct the TIMSS-repeat study in Flanders and present our most important findings.

Metacognition or metacognitive skills refer to students’ “understanding and control of their own cognition” (Sternberg, 2007, p. 18). Metacognition or knowledge about thinking includes declarative, procedural, and conditional knowledge (McCormick, 2003). Students who have well developed metacognitive skills tend to thrive academically. For example, research shows that systematic metacognitive monitoring leads to better understanding and academic performance (Zimmerman & Cleary, 2009). However, many studies in education report on low to medium associations between metacognition and academic achievement (Fleur et al., 2021; Winne & Azevedo, 2022).

Self-efficacy is another construct that relates to academic achievement across educational settings and age groups (DiBenedetto & Schunk, 2022). Self-efficacy refers to students’ beliefs that they can successfully tackle a task (Anderman & Wolters, 2008; Bandura, 2006). Students’ self-efficacy is related to their engagement with a task and the types of strategies they use (Bandura, 1994). Years of research indicate that self-efficacy relates to students’ learning, motivation, achievement, and self-regulated learning (DiBenedetto & Schunk, 2022). High self-efficacy is a strong predictor of students’ achievement and success (DiBenedetto & Schunk, 2022) and strongly relates to academic achievement for middle school students (Carpenter, 2007).

Available research studies suggest positive yet small correlations between metacognition and general and domain-specific self-efficacy (Cera et al., 2013; Ridlo & Lutfia, 2016). In addition, metacognitive scaffolding improved metacognitive awareness, academic self-efficacy, and learning achievement of biology students (Valencia-Valejo et al., 2019). Research evidence from other countries provides support in positive relationships among metacognition, self-efficacy, and academic achievement. However, it is not clear how these constructs relate to each other in other contexts such as Russia. Therefore, the goal of this study is to examine the role of metacognitive skills and self-efficacy in predicting middle school students’ academic results.

Theoretical framework

The role of metacognition and self-efficacy in students’ academic results in this study is examined through a Model of Self- and Socially Regulated Learning (Author). The model is organized around three broad areas: self-regulated learning (SRL; C–I, M–N), socially regulated learning (SoRL; A–B, J–N), and culture (O). Each area has its own set of processes contributing to the development of self-/socially regulated skills. Thus, SoRL includes instructional techniques (A–B) and formative assessment practices, such as feedback, which occur in classrooms (J–N). SRL includes the processes that activate student’s background knowledge and motivational beliefs, which lead to the choice of goals and strategies to do the task (C–I, M–N). Finally, culture (O) situates both types of processes within a socio-cultural context.

This model reflects the complexity of school classrooms and includes a number of variables. In this paper, however, the focus is on such components of SRL as metacognition and self-efficacy. For the purposes of this study, metacognition includes the processes of planning, progress monitoring, and reflection. According to Albert Bandura (2006), self-efficacy is domain-specific, which is why separate self-efficacy scales were developed for each of the domains. The main purpose of this study was to examine the role of metacognition and self-efficacy in predicting middle school students’ academic results. The study addresses the following research questions:

1. Are there differences in middle school students’ metacognitive skills, self-efficacy, and academic results by gender and grade?
2. Do metacognition and self-efficacy predict students’ academic results by subject domains?

The negative grading effect (NGE) is the decrement in grade outcomes associated with the process of being assessed and graded. By exploiting the natural experimental conditions resulting from the introduction or abolition of grades earlier in the school career, researchers have been able to contrast the outcomes of comparable groups of Swedish students with different grading backgrounds, i.e. whether they were previously graded or not. The effect has repeatedly been identified in students’ year 9 (age 15/16) grades, and seems to particularly affect low-ability students and boys (Facchinello, 2014; Klapp, Cliffordson, & Gustafsson, 2016; Clarke, Klapp, & Rosen, under review). Despite substantial reforms to the grading and assessment system, the effect persists and thus seems to have an enduring and robust impact on compulsory school students’ grades.
When investigating the effect, previous research has, however, tended to combine the results from multiple or all school subjects together, thus losing the possibility to identify detail and nuance in how the NGE may differentially impact individual school subjects. There is a strong imperative for investigating the NGE for individual subjects rather than using aggregated grades like SAT- or GPA-scores. As they are the primary means of student sorting to higher education and employment Swedish school grades have become increasingly high-stakes (Lundahl, Hultén, & Tveit, 2017) and may be driving student testing and grading anxiety, a factor known to affect test performance. Previous research indicates that students’ levels of test anxiety varies between subjects with the suggestion that there may even be subject-specific anxieties (e.g. mathematics anxiety (Mammarella, Donolato, Caviolo, & Giofrè, 2018)). Student self-efficacy beliefs also vary between subjects, with high self-efficacy thought to be a protective factor against test anxiety (Marsh, 1990; Mammarella, Donolato, Caviolo, & Giofrè, 2018). Different subjects are also taught by different teachers, have different content requiring different modes of thinking, and are likely to receive very different test feedback, another factor which impacts subjects differently (Azmat & Iriberri, 2010). These few examples suggest that the impact of being graded is dependent on several factors which vary from subject to subject, and supports the need to investigate the NGE for different subjects.
This research intends to replicate and extend previous research to investigate how the NGE manifests in different school subjects. By comparing the year 9 grade outcomes in different subjects of students who were either previously graded in year 6 or not it is hoped to establish how the NGE varies across different subjects. As with previous research, how factors such as gender, parental educational background, immigration status, and student’s cognitive ability affect the expression of the NGE will also be investigated.
The research is relevant also to wider audiences than just the Swedish education system. As the NGE has been shown to be robust across variations in the Swedish system it is reasonable to infer that it might exist in other the results of other countries, especially given that Sweden is not unique in using end of year grades for high-stakes purposes like admission to further education and employment.

Value-added indicators are a more accurate method of assessing school performance since they eliminate more non-school factors (Meyer et al., 2017). Slovenian upper secondary schools in the General education track finishing with General Matura have been able to assess value-added measures and track changes over time since 2014. The two time points in question are achievement at the end of Grade 9, just before entering upper secondary schools, and achievement at General Matura examinations. Lower secondary schools can similarly check value-added between Grade 6 and Grade 9 (finishing grade) in different subjects since 2018. These measures are not part of any accountability scheme and are provided for schools’ self-evaluation purposes along with other achievement results.
Value added measures used by slovenian schools are calculated as average residual between actual and predicted students' achievement (Cankar, 2011). Usual method for calculation of predicted values is 'median method', where population is sorted on scores from time point 1 and divided into equal sized groups. Median scores on scores from time point 2 over all groups constitute a series of predicted values for midpoints of groups on timepoint 1. All other values are interpolated from those.
This method of calculating value-added proved robust and relatively straightforward for explaining to teachers and the general public. However, there are also indications that the calculations aren’t optimal. In this paper, we will address the issue of negative national average value-added measures. As a rule, the average value-added over all schools for a chosen year tends to be negative.

There can be many reasons, and within this presentation, we will explore the following research questions:

Could the observed negative average value be associated with school composition factors (primarily the size of the school)?
Could we associate negative average values with school background characteristics (for example, the average income of the school’s municipality)?
Could we associate negative average values with the implementation of the value-added measure (either the median method or some other step in computation)?
Since value-added models rely on differences between time points, which increases the measurement error (Papay et al., 2011), it is important to reduce additional sources of error and insights from this research might be useful to others in the field of value added models.

Teachers who use assessment for formative rather than summative purposes are more effective in promoting student learning outcomes (Chen et al., 2017; Kyriakides et al., 2020). Teachers appear to acknowledge the benefit of formative assessment. However, their assessment practice remains mainly summative oriented (Suurtamm & Koch, 2014; Wiliam, 2017). This can partly be attributed to the fact that teachers do not receive sufficient training in classroom assessment (DeLuca & Klinger, 2010). Teacher Professional Development (TPD) programs intended to improve assessment practice have so far provided mixed results regarding their impact on teachers’ assessment skills (Chen et al., 2017), whereas many studies do not provide any empirical evidence on the impact of student assessment TPD programs on student learning outcomes (Christoforidou & Kyriakides, 2021). In this context, this study aims to explore the impact of a TPD course in formative assessment on improving teachers’ assessment skills and through that on promoting student learning outcomes in mathematics (cognitive and meta-cognitive).

During the first phase of the study, a framework that enables the determination and measurement of classroom assessment skills was developed. This framework examines assessment looking at three main aspects. First, skills associated with the main phases of the assessment process are considered (Gardner et al., 2010; Wiliam et al., 2004): (i) appropriate assessment instruments are used to collect valid and reliable data; (ii) appropriate procedures in administering these instruments are followed; (iii) data emerging from assessment are recorded in an efficient way and without losing important information; (iv) assessment results are analysed, interpreted, and used in ways that can promote student learning; and (v) assessment results are reported to all intended users to help them take decisions on how to improve student learning outcomes. The second aspect of this framework has to do with the fact that assessment skills are defined and measured in relation to teachers’ ability to use the main assessment techniques. Specifically, the framework looks at assessment techniques by considering two important decisions affecting assessment technique selection: a) the mode of response and b) who performs the assessment. Finally, the third aspect of the framework refers to the five measurement dimensions suggested in the Dynamic Model of Educational Effectiveness (Kyriakides et al., 2020): frequency, focus, stage, quality and differentiation. These dimensions allow us to better describe the functioning of each characteristic of an effective teacher (Scheerens, 2016).

Based on the theoretical framework and its dimensions, a questionnaire measuring teachers’ skills in assessment was developed. A study provided support to the validity of the instrument. It was also found that assessment skills can be grouped into three stages of assessment behaviour. These stages were used to make decisions in relation to the content and design of the TPD course which was based on the main assumptions of the DA. First, the DA considers the importance of identifying specific needs and priorities for improvement of each teacher/group of teachers. Second, it is acknowledged that teachers should be actively involved in their professional development to better understand how and why the factors addressed have an impact on student learning. Third, the DA supports that the Advisory and Research Team has should support teachers in their efforts to develop and implement their action plans. Fourth, monitoring the implementation of teacher action plans in classroom settings is considered essential. This implies that teachers should continuously develop and improve their action plans based on the information collected through formative evaluation.

Teachers’ instruction is at the heart of education, and previous research has shown that teaching quality is important for students’ learning outcomes (e.g. Charalambous & Praetorius, 2020; Seidel & Shavelson, 2007). However, teaching is a two-way process, and less is known about how the composition of the classroom affects teaching quality (TQ). Do for instance high socio-economic (SES) classrooms receive different TQ than low-SES classrooms? To examine this, one would first need to establish whether a so-called compositional effect exists. Compositional effect refers to the effects of, for instance the classroom’s socio-economic status (SES) on student learning outcomes, over and above the effect of students’ individual SES (Van Ewijk & Sleegers, 2010).

Both compositional effects and unfair distribution of high-quality teachers have been found in previous studies in a number of countries (Gustafsson et al., 2018; Luschei & Jeong, 2018; Van Ewijk & Sleegers, 2010) However, in Norway, that for a long time was considered an egalitarian society (Buchholtz et al., 2020), there is a lack of such studies. At the same time, educational inequality has increased in Norway (Sandsør et al., 2021). Hence, the overarching aim of the present study is to examine whether a compositional effect exists, and how the composition of the classrooms affects TQ. We further aim to describe more in depth what characterizes the TQ in classrooms of different compositions in Oslo where the gaps between students are larger and there are more minority students than in the rest of Norway (Fløtten et al., 2023).

The following research questions were asked:

1) What is the effect of the classroom composition (in terms of SES and minority status) on students learning outcomes in science, over and above students’ individual SES and minority status (i.e. the compositional effect)? How does this differ between Oslo and the rest of Norway?

2) What is the effect of the classroom composition on TQ in science, and how does this differ between Oslo and the rest of Norway?

3) What characterizes TQ in science classrooms of different compositions in Oslo?

Theoretical framework for teaching quality.

We chose The Three Basic Dimensions (TBD) framework (Klieme et al., 2009; Praetorius et al., 2018) to conceptualize TQ as this framework is the most commonly used in Europe and by the international large-scale studies (Klieme & Nilsen, 2022). TQ is here defined as the type of instruction that predicts students learning outcomes, and includes the following three dimensions:

1) Classroom management refers to how teachers manage the classroom environment and includes, for instance, preventing undesirable behaviors and setting clear and consistent rules and expectations for student behavior.

2) Supportive teaching focuses on the teacher’s ability to support students both professionally and socio-emotionally, such as providing clear and comprehensive instruction and seeing and listening to every individual student.

3) Cognitive activation includes instruction that enables students to engage in higher-level cognitive thinking that promotes conceptual understanding. Such instruction is characterized by challenging and interactive learning.

The TBD is a generic framework used across subject domains. To address research question 3, and further investigate more in depth the subject-specific aspect of TQ in science, a fourth dimension from the framework Teacher Education and Development Study–Instruct (TEDS-Instruct, e.g. Schlesinger et al., 2018) was included. This framework was adapted to the Norwegian context and to the subject domain of science, and validated. The fourth dimension is called Educational structuring and refers to subject-specific aspects of instruction such as inquiry or dealing with students misconceptions in science.

Our aim in this study was to learn about teachers’ understanding and appreciation of their autonomy in the context of student's assessment. The study’s context was a reform in Israel’s national matriculation exams (declared in 2022), that involved transitioning from external state-governed examinations into school-based assessment. The reform triggered discussions and re-evaluation of teachers’ professional autonomy, and of assessment policy. In this context we explored teachers' perceptions of the effect of assessment on professional autonomy. Furthermore, we broadened the scope of our study beyond the confines of the local reform, utilizing this specific case to draw more general insights regarding how teachers attribute significance to the professional conditions within which they work and how these conditions effect their sense of autonomy. We looked at the relation between autonomy in assessment, and autonomy in other aspects of teachers work. Furthermore, we studied the role of autonomy in the teachers' professional identity.

Our main research questions were: Which factors do the teachers acknowledge as contributing to their sense and preferences of autonomy? What are teachers' perceptions of the effect of assessment on their professional autonomy?

The theoretical framework of the study includes several types of literature.

First, we draw on a philosophical analysis of teachers’ freedom and responsibility, based on Existential philosophy of Jean-Paul Sartre (1946/ 2017). Teachers’ professional identity has been recognized as an extreme case of human destiny portrayed by Sartre (Author 1, 2022). While practicing the art of teaching (Schwab, 1983), teachers have a constant need to make choices in class, interpreting system’s regulations, practicing an inevitable autonomy, and exerting professional responsibility.

Secondly, we looked at current studies, and learned that teacher autonomy research mirrors trends in national and global education. Several studies indicate the favorable effects of teacher autonomy on teachers' perceived self-efficacy, work satisfaction, and empowerment, and on creating a positive work climate. They also show constraints on autonomy correlate with teacher turnover and the risk of emotional exhaustion, and burnout (Skaalvik & Skaalvik,2014).

Despite the recognition of the importance of teacher autonomy for job satisfaction (Juntunen, 2017), successful schools, and professional development (Wermke et al., 2019), there is less consensus on its definition (Pearson & Moomaw 2005). Autonomous teachers have a high control over daily practice issues (Wermke et al., 2019). Friedman’s scale for teacher-work autonomy (TWA 1999) includes four functioning areas pertinent to teachers’ sense of autonomy: class teaching, school operating, staff development, and curriculum development. In a re-evaluation of Friedman's scale (Strong & Yoshida, 2014), the number of autonomy areas grew to six and included assessment. In this paper we adopt Lennert-Da Silva’s (2022) definition which relates to the decision-making scope and control teachers have in relation to the national educational policy.

Thirdly, we read studies that look at autonomy in the context of student assessment and examine it as part of the larger theme of accountability. In the context of marketization, schools’ decentralization places school leaders within a framework including bureaucratic regulations, discourses of competitive enterprise, and external public accountability measures, that are spreading worldwide (Hammersley-Fletcher et al., 2021; Verger et al., 2019).

External assessment is a central factor in accountability (Ben-Peretz, 2012). High-stakes accountability casts a shadow on teachers' professional practice (Clarke, 2012; Mausethagen & Granlund, 2012), and their everyday practice is constrained by external testing.(Ball, 2003, 2008a, 2008b). Focusing on assessment as one expression of accountability, studies discuss the tension between external testing and autonomy. State-controlled assessment is viewed as a shift away from teacher professionalism towards the adoption of teaching methods that erode teacher autonomy in9 curriculum development and instructional decision-making (Day & Smethem, 2009).

We report ongoing research that assesses how well AI can evaluate teaching, which we define as “effective” to the degree it helps students learn. Our current research builds on a body of prior work in which we assessed how well human judges performed the same task. Under varying conditions (length of instructional sample; instruction documented as video, audio, and transcript; and judgments based on intuition alone, high-inference rubrics, and low-inference rubrics) human judges demonstrate significant limitations. Experts and nonexperts did no better than chance when they relied solely on their intuitive judgment. Experts fared no better when using high-inference rubrics. However, experts and nonexperts were more accurate than chance when they used low-inference rubrics, and just as accurate using transcripts of instruction compared to video. Machines are very good at performing low-inference tasks, and AI in particular is very good at “understanding” written text, such as transcripts. Is AI better at judging teaching effectiveness from transcripts than humans? If so, should human judges be replaced by machines? We provide data that may help answer these questions, and engage our audience in a discussion of the moral dilemmas it poses.

Children today are growing up in a digitally connected world which sets them apart from previous generations. For example, 42% of 5- to 7-year-olds have their own tablet and 93% of 8- to 11-year-olds spend an average of 13.5 hours online (Ofcom 2022). Digital media provides opportunities for easier access to information and communication with peers. However, it also presents a range of risks, especially for children who are particularly vulnerable due to their young age. This becomes clear when they are confronted with violent, sexual, advertising, or judgmental content in the digital space (Livingstone et al. 2015). Other challenges in digital communication and information channels include fake news, propaganda and deepfakes. With regard to the aforementioned aspects, it is necessary to possess skills that enable a critical examination of information. For this reason, information evaluation is considered an important subskill for social participation and learning inside and outside of school.

When examining the internet preferences of children and young people, it becomes apparent that they are primarily interested in extracurricular activities rather than child-friendly services commonly discussed in school settings, such as children's search engines. The top four internet activities include WhatsApp, watching films and videos, and using YouTube and search engines (Feierabend et al., 2023). In this respect, WhatsApp, YouTube, and TikTok are the most popular (social media) platforms (Reppert-Bismarck et al. 2019). The evaluation of content is not limited to online research alone. It can also occur in different scenarios, such as browsing the internet for entertainment or out of boredom). In this regard, the strategies for evaluating content vary depending on the purpose of the discussion (Weisberg et al. 2023), allowing the assessment of information, data, and content from different angles. One approach to evaluate content is to verify its credibility. In research literature, credibility encompasses multiple aspects. This includes assessing the trustworthiness of content, such as recognizing intention, or the expertise of the author. However, studies show that young people tend to lack critical evaluation skills when it comes to the credibility of online content (Kiili et al. 2018) and are also insufficiently prepared to verify the truthfulness of information (Hasebrink et al. 2019). In the context of social media in particular, the question of the realism of the shared content (e.g., factuality or plausibility) arises. Recipients are faced with the challenge of multiplicity resulting from the different ‘realities’ on social media. These realities are shaped by different motivations, attitudes, and political or social contexts which can blur boundaries (Cho et al. 2022). Overall, the evaluation process of online content is influenced by various factors. For instance, research suggests that reading competence affects the evaluation process. Furthermore, the socioeconomic status has been found to influence the digitalization-related skills of young people (see ICILS results). Another important aspect to consider is the influence of platform-specific knowledge, such as understanding the YouTube algorithm, and topic-specific knowledge on content evaluation, such as the subject of a news video. In addition, the design of both the platform and the content can also have an impact. This includes factors such as image-to-text ratio, layout, effects, and the focus of the central message.

To which extent the presented assumptions apply to primary school children is unclear, as most empirical results relate to adults or adolescents. Therefore, the overarching goal of the project is to develop a standardized measurement instrument for primary school children in order to assess to which extent they are able to evaluate internet content. The creation of a standardized measurement instrument involves several substeps which are outlined below.

Student Teaching Evaluation (STE) is the procedure by which teaching performance is measured and assessed through questionnaires administered to students. Typically, these questionnaires or scales refer to the teaching practices of academic staff and are conducted in one of the last meetings of the semester. Generally, and from a practical standpoint, the primary purpose of implementing this procedure is the necessity of universities to report STE results to quality assurance agencies. Another main objective of STE procedures, and certainly the most important from a pedagogical perspective, is to provide feedback to teachers about their teaching practices.

Previous studies on the highlighted topic present arguments both for and against the validity and utility of STE. On one hand, there are studies suggesting that STE results are influenced by other external variables, such as the teacher's gender or ethnicity (e.g., Boring, 2017), lenient grading (e.g., Griffin, 2004), or even the teacher's personality (e.g., Clayson & Sheffet, 2006).

On the other hand, there are published works showing that STE scales are valid and useful (e.g., Hammonds et al., 2017; Wright & Jenkins, 2012). Furthermore, when STE scales are rigorously developed and validated, as is the case with SEEQ (Marsh, 1982, 2009), there is a consistent level of agreement and evidence suggesting that STE scale scores are multidimensional, precise, valid, and relatively unaffected by other external variables (Marsh, 2007; Richardson, 2005; Spooren et al., 2013).

Even though this debate was very active in the 1970s and the evidence leaned more in favor of STE validity (Richardson, 2005; Marsh, 2007), a recent meta-analysis (Uttl et al., 2017) presented evidence that seriously threatens the validity of STE results. They suggest that there is no relationship between STE results and student performance levels. The existence of this relationship is vital for the debate on STE validity, starting from the premise that if STE results accurately reflect good or efficient teaching, then teachers identified as more performant should facilitate a higher level of performance among their students.

In light of all the above and referring to the results of the meta-analysis conducted by Uttl et al. (2017), the present study aims to investigate whether the relationship between STE results and student learning/performance is stronger when the STE scale used is more rigorously developed and validated. For this purpose, a multilevel meta-analysis was conducted, allowing us to consider multiple effect sizes for each study included in the analysis.

The results of this study can be useful in nuancing the picture of the validity of STE scales, in the sense that they can show us whether scales developed and validated in accordance with field standards can measure the quality of teaching more correctly and precisely. Additionally, this research can help outline a picture of which psychometric characteristics of STE scales contribute to a better measurement of teaching efficiency/effectiveness.

Therefore, the research questions guiding the present study are as follows:

1. What is the average effect size of the relationship between STE results and student performance, in all STE studies with multiple sections published to date?
2. Does the average effect of the relationship between STE results and student performance differ based on evidence regarding the validity of the STE scales used?
3. Does the average effect of the relationship between STE results and student performance differ based on the content of the dimensions of the STE scales used?
4. Does the average effect of the relationship between STE results and student performance differ based on the level of observability of the teaching behaviors in the items that make up the STE scales used?

In recent years, Kosovo has implemented a number of measures to improve quality assurance mechanisms in Kosovo. Participation in large-scale international comparative assessments is part of a monitoring strategy to compare the educational performance of Kosovo's primary and secondary school students with that of children and young adults in neighboring countries, the region, the European Union and other parts of the world, and as such has served as a key indicator of educational quality (MEST, 2020). International monitoring reports and the few academic studies that have been published reveal substantial differences in educational achievement at both primary and secondary levels in terms of student background and family indicators, place of residence (urban vs. rural areas) and school type (OECD; 2023; Mullis et al., 2017). However, in Kosovo, the rich sources of data obtained through participation in international large-scale assessments remain underutilized for educational research and monitoring, and thus have little to offer for evidence-based policy making in education. To date, inequalities have only been partially documented for achievement, but not for other important outcomes of schooling (Pavesic et al., 2022). Moreover, the causes of inequality have not been systematically investigated.

This symposium aims to bring together different perspectives to provide a more coherent picture of educational equity and quality in Kosovo. It will also broaden the perspective of educational equity, whereas to date, large-scale educational monitoring studies have mainly been used to analyse variations in student performance and equity within and across education systems over time. Other perspectives of educational equity, such as participation and recognition equity, are often neglected. These perspectives do not only take into account achievement levels, variances or minimum competence levels, but also focus on capabilities for social and political participation and on factors such as the quality of social relations, the recognition of individual autonomy and voice, and well-being as an end in itself. On the basis of the four papers, inequality will be examined in terms of a) approaches to teaching, b) the relationship between motivation and achievement, c) explanations of gender differences, and d) comparisons with neighboring countries. The symposium will thus provide a basis for a critical review of the extent to which large-scale studies such as TIMSS and PISA incorporate different foci of educational equity in their concepts, indicators and analytical approaches. The relevance and challenges of including valid and reliable measures of equity concepts in the indicator set of these studies will be illustrated by discussing some secondary analyses of LSA data for Kosovo.

Teacher instructional practices are considered amongst the main determinants of student achievement (Cordero & Gil-Izquierdo, 2018). Moreover, depending on the practices used, teachers can either weaken or promote student achievement (Caro et al., 2016; Hwang et al., 2018). Since student achievement is positively related to teacher practices, the impact of different instructional practices on student achievement represents a topic of great relevance for educational equity (Le Donné et al., 2016). This study explores the prevalence of different instructional practices in classrooms and their association with students’ reading achievements, focusing on the 2018 PISA results in Kosovo. Drawing on a dataset of 3,906 students, the research employs exploratory factor analysis to identify three latent variables representing student-centered instruction: individualized learning instructional practices (ILIP), research-based instructional practices (RBIP), and feedback-oriented institutional practices (FOIP). The study aims to answer two main research questions: (1) Which teacher instructional practices are prevalent among teachers in Kosovo classrooms? (2) How do students of different reading proficiency levels perceive ILIP and FOIP, and is there a significant difference in reading performance among students exposed to different instructional practices? Results indicate that ILIP is the most prevalent instructional practice, followed by FOIP and RBIP. Students predominantly report occasional or minimal exposure to ILIP and FOIP. The study also reveals a negative correlation between ILIP and RBIP with FOIP, suggesting a potential trade-off between student-centered and teacher-oriented practices. Benchmarking analyses demonstrate an equal distribution of students across FOIP and ILIP categories based on reading scores, indicating that exposure alone does not guarantee higher performance. However, RBIP stands out, showing a positive correlation with better reading scores, even at low exposure levels. This research contributes to the ongoing discourse on educational equity by examining the relationship between teaching practices and student outcomes in Kosovo's education system. The findings underscore the importance of considering various instructional practices in the pursuit of equitable education and inform policy discussions around teacher professional development. The study emphasizes the need for continued efforts to align teaching practices with the goals of the education reform implemented in Kosovo, fostering a student-centered approach to enhance reading performance and reduce achievement gaps.

Caro, D. H., Lenkeit, J., & Kyriakides, L. (2016). Teaching strategies and differential effectiveness across learning contexts: Evidence from PISA 2012. Studies in Educational Evaluation, 49, 30–41. Cordero, J. M, Gil-Izquierdo, M. (2018). The effect of teaching strategies on student achievement: An analysis using TALIS-PISA-link. Journal of Policy Modeling 40, 1313–1331. Hwang, J., Choi, K. M., Bae, Y., and Shin, D. H. (2018). Do teachers’ instructional practices moderate equity in mathematical and scientific literacy?: An investigation of the PISA 2012 and 2015. Int J of Sci and Math Educ 16, 25–45. Le Donné, N., Fraser, P., & Bousquet., G. (2016). Teaching strategies for instructional quality: Insights from the TALIS-PISA link data. OECD Education Working Papers, No. 148, OECD Publishing.

Sub-paper had to be withdrawn.

.

It is not only performance and the skills acquired that are of great importance within school and teaching, but also the students' attitudes towards the respective subjects. Learners' self-concepts, motivation and emotions are fundamental factors in learning, this is also evidenced by the performance of the students (OECD, 2023). In terms of gender, international studies report that girls are better readers than boys, partly due to differences in motivation and contextual effects. In addition, girls tend to have more positive attitudes towards reading and consider themselves to be more literate than boys (Mullis et al., 2017; OECD, 2023). Several studies identified the following characteristics as the cause for the better grade point averages of girls. Girls have higher self-discipline, self-control and self-regulation (Weis et al., 2013) and a higher interest in school in general (Houtte, 2004), they exert themselves more and work more, while disrupting lessons less (Downey & Vogue, 2004) are less avoidant of work, show less problem behaviour and better social behaviour (DiPrete, 2008). Although the additional effort of girls is mentioned here as the cause of the gender differences, the costs borne by girls are hardly taken into account, therefore the focus of this article is the elaboration of gender differences in reading-related self-concept and reading competencies, taking into account the family background of 15-year-old students in Kosovo. For this purpose, data from the PISA 2018 study are analyzed using regression analysis with the IEA IDB Analyzer. Rather than testing factual knowledge, PISA tests students' ability to apply and connect this knowledge. The study is conducted every three years and covers three areas, reading, mathematics and science, with reading being the focus of the assessment in 2018 (OECD, 2023). The results show that although girls perform better in reading and have a higher reading-related self-concept, even taking into account their family background, this is associated with higher costs for these girls, as they also report greater fear of failure.

DiPrete, T. A. & Jennings, J. L. (2012). Social and behavioral skills and the gender gap in early educational achievement. Social Science Research, 41 (1), 1–15. Downey, D. B. & Vogt Yuan, A. S. (2005). Sex differences in school performance during high school: Puzzling patterns and possible explanations. The Sociological Quarterly, 46 (2), 299–321 Houtte, M. v. (2004). Why boys achieve less at school than girls: The difference between boys’ and girls’ academic culture. Educational Studies, 30 (2), 159–173. Mullis, I. V. S., Martin, M. O., Foy, P., Olson, J. F. & Preuschoff, C. (2017). PIRLS 2016 International Results in Reading.: Findings form IEA's trend in international mathematics and science study at the fourth and eighth grades. TIMSS & PIRLS International Study Center Lynch School of Education Boston College. https://timss.bc.edu/TIMSS2007/mathreport.html OECD. (2023). Programme for International Student Assessment. OECD. PISA - PISA (oecd.org) Weis, M., Heikamp, T. & Trommsdorff, G. (2013). Gender differences in school achieve[1]ment: The role of self-regulation. Frontiers in Psychology, 4 (442), 1–10

Empirical educational research has shown that social inequalities influence the student´s educational achievement. Different theoretical frameworks, such as Bourdieu's capital theory refer to how different types of capital can be related to achievement. For several countries, the correlation of socio-economic status and student´s educational achievement has been shown (Wendt et al., 2012). About 23% of people in Kosovo live in poverty and the GDP is a quarter of the European average (UNICEF, 2021), which indicates a high level of social inequalities. The fact that post-conflict countries, such as Kosovo, have less beneficial conditions and are therefore also associated with lower levels of educational achievement is e.g. shown by the results of the TIMSS study (Mullis et al., 2020). However, analyses show that social inequality in Kosovo is quite small (Wendt et al., i.p.). Moreover, this raises the question of whether the operationalization of socio-economic status in post-conflict countries can take place in the same way than in other countries and whether constructs developed in one culture are valid for other cultures (Matsumoto, 2003). In this article we therefore analyze the operationalization of the socio-economic status indicators. As a theoretical framework we used Bourdieu´s theory (2003, 2012) of capital. Therefore, we conducted secondary analysis of the TIMSS 2019 data of Kosovo at grade 4 to analyze the extent of differences in mathematics and science performance that can be explained by Bourdieu's theory of capital. We use data from the household survey and the student questionnaire, self-reported by parents and students (nstudents= 4496; mean age 9.9). We conducted multivariate regression analysis using the IEA IDB Analyzer. We found a significant relationship for parental education level and number of books for both math and science The lack of economic resources at home is negatively related to mathematics and science achievement. However, the largest difference in achievement is found among children who come to school hungry, with a difference of 31.5 points in science. The "effect" of language practices on science achievement remains significant when controlling for cultural and economic resources. This can be seen as a first indication that the cultural capital acquired by parents through conflict-related migration makes an independent explanatory contribution to the differences in their children's performance. The overall variability in student performance that this model can explain is limited, explaining only about 13% of the variation in student performance, indicating that other important factors may not be accounted for.

Boudon, R. (1974). Education, Opportunity, and Social Inequality. Changing Prospects in Western Society. Wiley. Bourdieu, P. (2003). Interventionen, 1961-2001: Sozialwissenschaft und politisches Handeln. Raisons d'agir. VSA-Verlag. Bourdieu, P. (2012). Ökonomisches Kapital, kulturelles Kapital, soziales Kapital. In U. Bauer, U. H. Bittlingmayer, & A. Scherr (Eds.), Handbuch Bildungs- und Erziehungssoziologie (pp. 229–242). VS Verlag für Sozialwissenschaften. https://doi.org/10.1007/978-3-531-18944-4_15 Matsumoto, D. (2003). Cross‐cultural Research. In S. F. Davis (Ed.), Handbook of Research Methods in Experimental Psychology (pp. 189–208). Wiley. https://doi.org/10.1002/9780470756973.ch9 UNICEF. (2021). Annual Report. https://www.unicef.org/kosovoprogramme/media/2931/file/English-2022.pdf Wendt, H., Stubbe, T., & Schwippert, K. (2012). Soziale Herkunft und Lesekompetenzen von Schülerinnen und Schülern. In W. Bos, I. Tarelli, A. Bremerich-Vos, & K. Schwippert (Eds.), IGLU 2011Lesekompetenzen von Grundschulkindern in Deutschland im internationalen Vergleich (pp. 175–190). Waxmann Verlag.

Learning to learn skills are fundamental cognitive, metacognitive, motivational, and affective resources to help reach a learning goal (James, 2023). Acquiring these skills and abilities is vital for lifelong learning in the 21st century. The Finnish Learning to Learn (L2L: Hautamäki, 2002; Vainikainen & Hautamäki, 2022) scales have been developed and utilised in national and regional assessments since the late 1990s. They cover general cognitive competences needed in different school subjects, such as reading comprehension, mathematical thinking skills, general thinking and reasoning skills, and problem-solving.

This paper reports on a longitudinal L2L study, in which around 1000 children were followed through the nine-year basic education in Finland. Longitudinal studies can collect a broad range of information and provide unique insight into the importance of cognitive development in the early stages of education, identify connections between student abilities and academic achievement, and allow for adjustments to the pedagogical process throughout schooling. Studying the characteristics of stability and trends in the development of cognitive abilities in different age groups makes it possible to identify the weakest points and direct pedagogical efforts to increase the level of abilities and motivation (Metsämuuronen, J., & Tuohilampi, 2014). The level of development of cognitive abilities largely determines performance in mathematics and other subjects and seems to influence children's goal orientation in learning (Mägi et al., 2010; Williams, T., & Williams, K. 2010). Longitudinal assessments of them also make it possible to identify certain trends in the development of certain skills at different age periods, which must be taken into account in the diagnosis and evaluation of the learning process (Weinstein, 2015).

The present study focuses on the development and changes in the cross-curricular cognitive competences and learning-related beliefs measured by the Finnish L2L scales. We also study how they are reflected on pupils’ school achievement as measured by grade point average (GPA). We aim at analysing how individual and group-level differences develop from when the pupils enter the formal education system until they complete basic education and move to the tracked upper secondary education. We answer the following questions:

1. How are the cognitive L2L competences, learning-related beliefs and school achievement connected and how do they influence each other over the years during basic education?

2. How stable are the individual and group-level trends observed in cognitive L2L competences, learning-related beliefs and school achievement throughout the school years?

Student’s effort and motivational factors behind it have an essential role in determing how students approach new tasks and perform in them (e.g., Kupiainen et al., 2014). Together, they affect the ability to apply the cognitive processes fundamental to identifying problems and designing and applying solutions (Kong & Abelson, 2019; Skinner ym., 1998). These processes have traditionally been measured and evaluated through self-reports and observation. While these methods undoubtedly have an important place in the human sciences, they have challenges regarding validity and large sample sizes. One solution to these challenges is that the technology's vast potential allows seamless data collection from individuals in digital environments without disrupting their natural activities (Wise & Gao, 2017). Hence, this paper focuses on investigating what time on task, number of trials, and use of problem-solving strategies in different tasks tell us about student performance and whether the results in different tasks are consistent with each other. The relations between these task behavior indicators are examined from the perspective of motivational profiles students may hold by examining whether the profiles differ in this matter.

In this study, the focus is on students' control-related beliefs within the framework of Action-Control Theory (Skinner et al., 1988). According to the theory, perceived control encompasses beliefs about the relation of agents, means, and ends, shaping a student's perception of how school outcomes are achieved and the extent to which they are actively involved. These beliefs are found to be related to school achievement in to a varying degree and varying hindering or fostering effects. Accordingly, while some students with beliefs that have shown to positively predict school performance have done well, other students with similarly above average beliefs have done less well, highlighting the existence and importance of different combinations of beliefs when considering their association with motivational orientation and performance (Malmberg & Little, 2007).

Treating time use as a measure of motivational investment in a task is grounded in Carroll's Model of School Learning (Carroll, 1989). According to the model, students vary in the time they need to learn, which in turn depends on students' aptitude for the task, their ability to understand instruction, and the quality of instruction. Higher aptitude corresponds to shorter learning times, while lower aptitude may require more effort. The time students ultimately invest in learning is composed of the time allocated for learning and the time students are willing to dedicate. The required time, the time spent, and the quality of instruction act as the determinants of the level of learning (Kupiainen et al., 2014). Computer-based assessment (CBA) research has confirmed that students' too short time on task indicates a lack of effort and task commitment (e.g., Wise & Gao, 2017). This results from reacting too quickly compared to the time needed for a proper task solution (Schnipke, 1995). This supports findings in problem-solving tasks, indicating that in every ability level longer response times positively correlate with correct answers as task difficulty increases (Goldhammer et al., 2014).

The study delves into the diverse strategies individuals employ during problem-solving that guide the problem-solving process and ultimately influence how effectively they navigate problem-solving situations (Stubbart & Ramaprasad, 1990). Some problems may require multiple trials and inductive reasoning, while in other problems the most appropriate way is to test how individual variables affect the outcome, isolating the effect of other variables. CBA enables the exploration of these strategies by utilizing log data collected during tasks, which have been done in the past, particularly for studying the differentiation of the effect of variables in solving more complex problems (e.g., Greiff et al., 2016).

Previous studies have shown that the use of information and communication technologies (ICT) in leisure time, and also at school, is related to lower level of school performance (Biagi & Loi, 2013; Gubbels, Swart, & Groen, 2020). Furthermore, data from the Programme for International Student Assessment (PISA) studies have indicated that higher levels of ICT use is related to lower scores in reading literacy both internationally and in Finland (OECD, 2011; Saarinen, 2020). Analyses of the PISA data from 2012 have also shown no significant improvements in student achievement in reading, mathematics or science in the countries that had invested heavily in ICT for education (OECD, 2015). These findings have sometimes been interpreted as an indication of the harmful effects of digitalisation of education.

PISA results have shown a declining trend in many countries (OECD, 2023). The most recent decrease in PISA 2022 scores have been explained, at least in Finland, for example, by the excess use of ICT. On the other hand, mixed results have also been reported, and it is difficult to draw clear conclusions about the relationship between the use of digital technologies and learning (Harju, Koskinen, & Pehkonen, 2019). PISA studies have found that students who use computers moderately and for a variety of purposes have the highest levels of literacy (Leino et al. 2019, p. 94; OECD, 2011).​

The use of ICT in schools can be seen as a target of learning but also as a learning tool, which means that ICT can also be used as a mean to support students (Jaakkola, 2022). Based on previous research, there are some indications that the digital technology is used to differentiate teaching (Biagi & Loi 2013; Lintuvuori & Rämä, 2022; OECD 2011, pp. 20-21). This study will test the hypothesis that the use of ICT could be targeted especially to lower performing students.

The research questions investigated in this study are:

1. How the use of ICT at school is related to students’ reading literacy scores in PISA? Do the levels of proficiency in reading literacy explain the relationship between ICT use and reading performance?

2. Does the student’s special educational needs (SEN) status explain the relationship between the ICT use and reading performance scores?

Future of Jobs Report 2023 projected the possible job creation and displacements for the next 4 years, revealing a great increase in all the domains in which AI knowledge and skills will be most wanted and used (ex AI and Machine Learning Specialists, Sustainability Specialists, Business Intelligence Analysts, Information Security Analysts) whereas other domains will decrease in their demand for employees (Administrative and Executive Secretaries, Data Entry Clerks, Bank Tellers and Related Clerks) (World Economic Forum, 2023).

In these times of uncertainty and challenge, the selection of an academic path with the potential to lead to a successful career brings a complex decision-making process for adolescents. The achievement in career choices and job performance is significantly shaped by vocational interests (Rounds & Su, 2014). In this context, obtaining a clear understanding of the vocational interests of high school students belonging to Generation Z (born between 1997 and 2008) would prove particularly valuable. A substantial number of these students make decisions about their college majors during the 10th and 11th grade. If their chosen path aligns with their vocational interests, it is likely to enhance their motivation to complete college (Nye, Prasad, & Rounds, 2021), in a period when Higher education is confronted with serious drop-out rates (Eurostat Statistics, 2022). This latter source reveals that in 2022, the proportion of early leavers from education and training (ages between 18 and 24) in the EU ranged from 2.3% in Croatia to 15.6% in Romania.

According to a recent national survey in Romania focusing on Generation Z, it was found that 76% of respondents identified a passion for their work as the primary motivating factor in their job search (Romanian Business Leaders, 2022). This indicates that, for this demographic, vocational interests take precedence over financial compensation when considering employment opportunities.

As all the previous generations, Gen Z has its distinct futures, being described as more pragmatical and future-oriented compared with the more idealistic Millennials (Twenge, 2020, p. 231). Being born in a digitalized and tech world, vocational interests have also changed, as the current generation is interested in more fields of activity than the previous with an increased interest in information technologies (Roganova & Lanovenko, 2020).

Interests are defined as a cognitive and motivational factor encompassing both engagement and participation in specific content areas. The effectiveness of interest lies in its capacity to generate a rewarding experience through the information search process (Renninger & Hidi,, 2020). Interests have a significant influence on career choices and academic achievement (Hoff, Song, Wee, Phan, & Rounds, 2020), (Stoll, et al., 2020). This is why the present research endeavors to explore the patterns or clusters of interests within the Generation Z adolescent demographic.

A key objective of the study is to ascertain whether distinct patterns of interests emerge among the cohort based on factors such as the year of the examination, age, or gender. This multifaceted approach seeks to provide a nuanced understanding of the intricate interplay between vocational interests and demographic variables, contributing valuable insights to the broader discourse on college domain decisions among adolescents.

Therefore, this research aims to address the following questions:

1. Are there variations in vocational interest preferences among high school students who took the test during the periods 2012-2014, 2015-2019, and 2020-2023?
2. Are there distinctions in vocational interest preferences between females and males across and between the established subcohorts?
3. What are the most prominently scored preferences in the Work Roles scales among the participants across and between subcohorts?
4. What types of Work Styles do Generation Z individuals predominantly favor across and between subcohorts?

Recent empirical research in the social sciences has emphasized the importance of causal inference. However, causal inference is challenging when using observational data, primarily cross-sectional, even if it includes relevant variable information, as in international and large-scale educational surveys. The difficulty is more pronounced when the variable of interest, such as a national-level policy, is systemic. This paper demonstrates that by using pseudo-panel data derived from repeated cross-sectional data, we can obtain findings relevant to policy-making, thereby mitigating some of the challenges in causal inference, particularly biases from unobserved confounding factors.

The specific topic addressed in this paper is the assessment of policy factors related to the youth's choice to teach. In general, improving the availability and quality of teacher personnel is a universal and important issue for public education policy (OECD 2018). These research areas concerning the choice of teaching career and teacher supply have been interdisciplinary in education (educational policy studies, sociology of education, educational psychology, etc.) and economics (economics of education, labor economics). In particular, empirical research on the basic issues of "who chooses to teach" and "what factors increase the number of people who want to teach" has been conducted in many countries. While educational and psychological research have pointed out the importance of psychological factors, work environment factors have not been recognized as the main factors influencing career choice (Watt et al. 2017). On the other hand, empirical studies in the economics of education and labor economics have focused exclusively on the impact of salary levels as a policy variable on entry and exit from the workforce and have partially argued for its contribution (Corcoran et al. 2004; Dolton 1990; Manski 1987).

Moreover, Japan, where the presenter is from, has historically excelled in maintaining high-quality teachers, as evidenced by their high competency (Hanushek et al. 2019) and low turnover rates, compared to other countries. However, recent years have seen a growing trend among young people to avoid teaching careers. Japan now faces challenges similar to many countries experiencing a structural teacher shortage. Public debates often cite the relatively inferior work environment of teaching compared to other white-collar jobs as a factor in this avoidance. Yet, substantial evidence is lacking to inform policy priorities in this area.

In this study, we position and extend the groundbreaking recent studies that have used PISA student-level data to analyze the youth’s choice of teaching profession (Park & Byun 2015; Han 2018) as important prior work. We differ from that study in terms of methodology, using pseudo-panel data composed of subpopulations of countries as units; we apply a cross-classified hierarchical model to ask "Which policy factors" promote "whose" entry into the teaching profession among young people? We specifically focus on policy factors related to the working environment, namely, the relative salary level of teachers compared to other professions and the workload of teachers (working hours, number of students per teacher, and time spent on non-teaching tasks).

Applying a cross-classified hierarchical model to the pseudo-panel data, we respond to the question of "which policy factors" encourage "whom" of young people to enter the teaching profession, addressing both causal inference (controlling for time-invariant confounders) and policy relevance (heterogeneity of policy effects). The cross-classified model, which sets up the random effects/coefficients in two types of units, country, and subpopulation, has a major advantage in that it allows for different policy implications for each country. To further increase the robustness of our model, we are expanding it into a semiparametric model (infinite mixture model) that does not rely on a multivariate normal distribution for random effects and coefficients.