Conference Agenda

Quality monitoring of education is preferably based on different types of information from multiple sources (Baker & Linn, 2002; Schafer, 2003). In Flanders (Northern Belgium), the following sources are used: the school inspections of the Flemish inspectorate of education, the national assessments and the international large-scale assessments in which Flanders participates. All these sources provide school performance feedback (SPF). Therefore, Flemish schools will receive SPF-reports of various sources. These sources can lead to more reliable decision making about the effectiveness of schools, if, the information from different sources is accurate and consistent (Chester, 2005; Porter et al., 2004). However, it remains unclear to what extent the results of different sources align for Flemish schools. After all, those sources have completely different designs and use different techniques for estimation and analysis.

We examine the alignment between SPF-reports in primary and secondary education at the school level. We focus on sources that address results in mathematics and reading:

• National assessment for reading 2018 (primary education);

• National assessment for mathematics 2021 (primary education);

• National assessment for mathematics 2018 for the general track in the second year of secondary education;

• National assessment for mathematics 2019 for the vocational track in the second year of secondary education;

• PIRLS 2016;

• TIMSS 2019;

• PISA 2018;

• Inspections for mathematics and Dutch in school year 2018-2019 and 2019-2020.

For each school, school effects were first estimated for all national and international assessments the school participated in, using type 0, type A and type B contextualised achievement models (Leckie & Prior, 2022). Based on these estimated school effects, every school was categorised multiple times as below average, average or above average. As such, a school could be categorised as below average for PISA, but average for the national assessment. Note that schools are already classified in four categories by the inspectorate: below expectation, approaches expectation, meets expectation and exceeds expectation. Secondly, we linked the results (i.e. the estimated school effects and/or the categorisation by the inspectorate) of every school by considering the administrative code of each school. In this way, we can compare – for instance – the results of a school on TIMSS 2019 with the SPF from the inspection for mathematics.

Our results indicate that the SPF of different sources align for the majority of the Flemish schools, especially when controlling for background characteristics of students and schools by using a type B model. We find the lowest alignment when comparing different sources based on type 0 school effects. However, for a limit number of schools, the results of multiple sources are different and sometimes even contradictory. Notice that these contradictory results can have a substantial impact on schools, for instance when the results on one particular source determine the attribution of specific consequences (e.g. an additional school audit by the inspectorate).

This paper proposal is most closely related to the MoREi network and the subtheme ‘Leveraging research and data for inquiry, insight, innovation and professional learning’.

During the past two decades, extensive research has demonstrated the importance of a rich and comprehensive home literacy environment (HLE) in students’ reading comprehension development (Dong et al., 2020). Because most of these studies solely assumed a direct association, the underlying mechanisms - including potential mediating factors - remain unclear (Wiescholek et al., 2018). The investigation of mediating factors is nevertheless of crucial importance to gain a deeper understanding through which specific pathways HLE affects students’ reading comprehension skills.

We propose a model that includes both a direct path between HLE and students’ reading comprehension skills and multiple paths that run via two mediating factors, being students’ reading enjoyment and motivation as well as their reading frequency. These factors are likely to influence the relationship between HLE and reading comprehension, as previous research has shown that students’ reading enjoyment, motivation and frequency are, on the one hand, affected by HLE (Baker et al., 1997; Arzubiaga et al., 2002; Altun, 2022) and, on the other hand, might contribute to the reading comprehension ability of students (Becker et al., 2010; Soemer & Schiefele, 2018; Stutz et al., 2016; Yang et al., 2018). Additionally, we considered HLE as a complex, comprehensive construct reflecting various components in students’ home environment that might contribute to their language and literacy development. In line with Burgess and colleagues (2002), we distinguished HLE as either active (e.g. literacy-related activities parents do with their child) or passive (e.g. parents’ reading enjoyment, the number of books at home). These specific aspects were separately included in our model, allowing us to examine their individual contribution on students’ reading comprehension. This study aimed to assess whether students’ reading enjoyment, motivation and frequency influenced the association between various aspects of the active as well as passive HLE and Grade 4 students’ reading comprehension.

We reported on Flemish data (Belgium) of the Progress in International Reading Literacy Study (PIRLS) 2021. Participants were 5114 students from 141 schools and their parents. A parallel mediation analysis using structural equation modelling is tested to simultaneously assess relations between the variables under investigation. We demonstrated differential associations between specific aspects of HLE and fourth graders’ reading comprehension. Both an active reading climate (β = .178, p < .001) and the amount of books at home (β = .252, p < .001) were found to be positively associated with students’ reading comprehension. We found no contributions of parents’ reading enjoyment (β = -.006, p > .05), and their reading frequency (β = -.026, p > .05) on students’ reading comprehension. Regarding the investigation of mediating factors in the relation between aspects of HLE and students’ reading comprehension, we only found evidence for a mediating role of students’ reading frequency in the association between the amount of books at home and students’ reading comprehension abilities (β = .006, p < .01).

Our findings confirmed the importance of conceptualizing HLE as a broad, multi-componential construct as well as pointed out the inevitable role of parents in students’ reading comprehension development.

Conceptual framework and context.

School inspections are widespread accountability and educational quality improvement practices implemented worldwide. They involve external evaluation of schools based on standards and site visits, resulting in judgments. Comparative studies have been conducted in Anglo-Saxon and European countries (Ehren et al., 2015; Jones et al., 2017). In general terms, there are different effects found in school inspections: conceptual effect (a significant change in the comprehension about the school), symbolic effect (reinforcement of previous ideas) and instrumental effect (decisions based on inspection’s results) (Penninckx et al., 2016). However, evidence about their impact on school improvement is noticeably mixed. Moreover, educational settings in the global south face different systemic challenges and their school inspections are scarcely researched (Bravo, 2019; Díaz & Rodríguez, 2020).

Aim and research questions.

This study aimed to understand schools’ experiences with inspections in Chile. The issues explored in the research were ‘How do school leaders and teachers experience inspections?’ and ‘What kind of effects are produced by inspections?’.

Methodology.

A mixed-methods approach was developed to answer the research questions. A combination of interviews and non-participant observations were conducted three times during the inspections cycle and 8-12 weeks after. This approach allowed a comprehensive understanding of the phenomenon and experiences of school actors (Cresswell & Poth, 2017). 21 schools (public and private subsidized) participated in the study Interviews were transcribed verbatim and thematically analyzed (Braun & Clarke, 2021).

Results, findings, learning.

School actors perceive school inspections in Chile as valuable. They experience some initial stress and anxiety; however, this emotional effect dissipates quickly during the inspection. Inspectors emphasize the formative dimension and orientation. Compared with findings in high-stakes systems, this study shows less resistance to feedback and more acceptance. Moreover, school actors value the external perspective and the opportunity to examine their practices and receive expert feedback. The inspections’ results are generally consistent with the diagnosis held by school leaders. The challenge is that the feedback provided little new information, and the chance to prompt new insights about the school is somewhat limited.

Consequently, the effect is short-term and adjustment to previously planned actions, and less substantial on the institutional capacities. Since feedback frequently reinforces schools’ analysis, these results suggest a noticeable symbolic effect (McCrone et al., 2007). Only in a few cases was observed a conceptual effect when feedback allowed schools to identify blind spots and recognize overlooked management aspects (Penninckx et al., 2016).

The use of information for decision-making and the articulation among educational public agencies are critical dimensions to transform lessons from the inspections into sustainable changes. The potential effect inspections can have on systemic improvement seems limited by insufficient local administrators' and ministerial advisors' involvement and minimal articulation with improvement policies.

Connection to the conference theme.

These findings offer a salient opportunity to understand diverse forms of evaluation and shed light on the role of inspections in professional learning and capacity building.This research is closely connected with the sub-theme "Leveraging research and data for inquiry, insight, innovation and professional learning".

Framing

In school improvement projects, where schools collaborate with university researchers, it is crucial to establish a solid foundation for the collaboration (Barnett et al., 2010; Frerichs et al., 2017; Solvason et al., 2018). Establishing trusting collaborations is seen as essential for integrating research and practice (Kirschenbaum & Reagan, 2001; OECD, 2022). In my project, the collaboration is beneficial for the school leader teams, as the researchers provides them with support in managing their school improvement projects. For us researchers, the collaboration provides access and insights into the school's development processes. In such collaborations, trust plays an important role in securing the quality of the data collected and the validity of the research findings (Cunliffe & Alcadipani, 2016). The establishment of a trusting and open collaboration is, however, a demanding process. It takes time and energy from both parties.

Research questions

Which aspects are of importance when establishing trusting collaborations in school-university partnerships?

Context

The study builds on experiences from a project in which eight school-university collaborations have been established. The aim of the collaborations is to enhance the sustainability of the school improvement projects led by school- and project leader teams as well as to gain insight into the factors important for sustainable school improvement. The interest in writing this paper is not directly connected to the aim of the collaborations. Instead, the interest stems from the experiences gained through the demanding process of establishing them.

Methods and data material

The study is inspired by action research methods, which involve intervening in the practice that we are simultaneously researching (McIntosh, 2010).

This empirical case study draws on interviews, communication data and meeting observations and analyses how trusting collaborations were built between the research team and the school leader teams within the first year of the project.

Findings

The study finds the following three crucial aspects of building trusting collaborations between school leader teams and university researchers: (1) alignment of expectations, (2) the level of demands from researchers to practitioners, and (3) the choice of communication connected to translating theory into practice.

Educational importance

The findings are applicable to the practice of academic researchers or consultants who are to work with school practitioners. Working with a focus on the three aspects contributes to building trusting collaborations in Research and Development projects. The study may be of interest to other conference participants involved in school-university partnerships, regardless the theme and aim of their study, as this study is primarily a methodological contribution.

Connection to the conference theme

The study is related to the parts of the conference theme concerned with engaged and purposeful dialogue between academic researchers and school communities.