Conference Agenda

International Large-Scale Assessments (ILSAs) receive considerable attention from the media and policymakers globally. Critiques of these studies and their influence on policy have been widely discussed (e.g. Gorur, 2016), though some see potential synergies between ILSAs and other methodological orientations to the study and pursuit of educational effectiveness and improvement (e.g. Kyriakides, Charalambous & Charalambous, 2021). This paper takes a critical perspective and draws on findings from a specific example of a recent ILSA to illustrate two not-uncommon issues arising: Troubling/puzzling findings related to teacher professionalism and professional learning, and how to constructively engage with those findings and avoid their misuse.

The Progress in International Literacy Study (PIRLS), conducted in 5-year cycles since 2001 (Mullis et al., 2023), is an ILSA that compares overall reading achievement at approximately age 10 across about 60 participating education systems and across study cycles. Results from the 2021 cycle of PIRLS, specifically those relevant to teacher characteristics (qualifications, years of experience, job satisfaction) and their relationship to pupils’ reading achievement, are used to problematise:

1) What explains some possibly counter-intuitive or discouraging findings concerning teacher characteristics; and

2) How multiple stakeholders (academic researchers, government funders, policymakers, politicians, school leaders and teachers) can play a role both in making meaningful use of these research findings and avoiding (or even actively combatting) their misuse.

While PIRLS 2021 took place across 57 participating education systems, the context for this paper is predominantly national, given the author’s research role in PIRLS 2021 in England. Data come from a nationally-representative sample of 178 teachers and 4150 pupils in 162 schools. Reading achievement is based on pupil assessments, while teacher characteristics are based on self-report questionnaires.

Initial findings in England (Author et al., 2023) showed no clear relationships between reading achievement and teacher qualifications, years of experience, emphases in formal teacher education (on language, pedagogy, and/or reading theory), participation in reading-related professional development, nor job satisfaction. This seems counter-intuitive, and conflicts with some previous research (e.g. Kini & Podolsky, 2016). Several possible causes for these null relationships include problems of measurement (categories that do not work well in context) and analysis (where alternative statistical approaches might provide different results) as well as genuine underlying issues in the education system. The final paper will discuss insights from an exploration of each of these alternatives, including activities currently in progress to engage directly with headteachers, teachers and policymakers to discuss study results, questions raised and practical implications.

The educational importance of this work lies in its contribution to lessons learned both about problems of ILSAs in relation to teacher professionalism and professional learning and the possibilities afforded by open dialogue about ILSA findings across professional boundaries. Connection to the conference theme and sub-theme of “Leveraging research and data for inquiry, insight, innovation and professional learning” is tightly bound to the active and open engagement with different stakeholders around PIRLS 2021 findings in England in an effort to make meaningful and responsible sense and use of results in policy and practice.

The purpose of this research is to shed light on collaboration and support given by the municipal education authority to schools regarding school improvement in upper secondary schools in Norway, and school owners' follow-up of the schools cf. their overall responsibility for quality work (Education Act § 13-3e). The study will provide an example of how a change in thinking can contribute to better cooperation between local school authorities and upper secondary schools. This presentation fits in with the theme of the conference: continuing professional development for school leaders. Several studies have shown the importance of support between the different levels in the education systems, such as at district and municipal level (Anderson and Young, 2018; Hargreaves and Shirley, 2020; Louis et al., 2010; Moos, Nihlfors et al., 2016), and how support from management at district level is important for schools' improvement efforts (George & Kincaid, 2008; Wilkinson et al., 2019). In the Norwegian context, through the national quality system, the local educational office has a great responsibility for quality in education (Ministry of Education, 2017). Datnow, 2012, points out that the relationships between the actors at the various levels are of decisive importance for cross-level cooperation in the education system. The research project started in 2019. Point of departure was that the leaders and advisors at municipal education office wanted to develop themselves as learning leaders. During the years 2019 to 2023, in collaboration with researchers, and based on a knowledge base of "learning leadership" and "genuine inquiry" (Argyris & Schon,1978; Emstad & Birkeland, 2021; Robinson, 2011,2020), they practiced having a “learning approach” in their in meetings with the schools. This means that they have placed emphasis on being a professional learning community (PLC), and developing their own communication. By recording conversations and meetings, analyzing and disclosing their own theories of action, they aimed to become genuine listener and being respectful towards the school's and their knowledge, but at the same time having respect for their own knowledge and their responsibility as school authorities. All the way researchers observed meetings, gave feedback and new input. The study can be described as intervention research, with continuous feedback loops where data is used both for the participants and the researchers to adjust their own course and assess necessary input in order to reach the desired goal: to be a better support for the county's schools, and to ensure good quality in the teaching. The data material consists of leaders' presentations of their own analyzes and evaluations in the PLC-meeting with the researcher, observations of meetings with the schools, timeline of activities with defined turning points (change in communication or actions), feedback notes from the researchers and school leaders, reflection notes). In 2023 principals and middle leaders were asked to send in reflection notes on their experience of change – if any. What was particularly emphasized was that the meetings between the local authorities and the schools' leadership groups have become more characterized by trust than by supervision and control.

School self-evaluation (SSE) is considered a key lever to promote bottom-up school improvement (Chapman & Sammons, 2013; Jacobsen et al., 2003). In recent years, many countries have promoted centrally mandated SSE aiming at school improvement in official discourse, even though accountability and economic logics in top-down models for SSE are in place (McNamara et al., 2022).

In the Italian context, the National Evaluation System requires schools to draw up SSE reports and school development plans, according to a common theoretical framework. A centralized professional development programme (PDP) with five online courses has been designed by the National Institute for School Evaluation. The program was aimed both at supporting the implementation of the aforementioned system mandated SSE and at providing opportunities to make sense of the SSE process. The main contents were: SSE, school improvement, accountability and reporting, external evaluation, students’ assessment, evaluation theory, social research methodology, data use, communication and collaboration skills. In SY 2021-22 attended the program 320 teachers and 73 school principals.

This explorative study aims at understanding how the PDP intercepts participants’ needs, as well as how it can support them to implement SSE.

Four online focus groups with a merit sample of PDP’s participants were carried out in July 2022. The sample consisted of 25 teachers and 5 principals. According to a common protocol, one moderator, one co-host assistant and one co-host observer were present (Authors, 2023). The audio recordings were transcribed and the transcripts underwent an abductive thematic analysis (Braun & Clarke, 2006; Thompson, 2022). The analysis combined an inductive approach (descriptive codes emerging from the reading of the text) with a deductive approach (interpretative categories grouping codes thematically related) (Authors, 2022).

As for the relationship between PDP and participants’ needs, the contents were widely appreciated for their comprehensiveness, richness, and theoretical soundness. In all focus groups some participants perceived the asynchronous modality as a crucial affordance to enhance their self-paced learning. Furthermore, teachers highlighted the usefulness of the practice-oriented e-tivities. On the other hand, some teachers lamented the scarcity of operational tools and individual feedback to carry out SSE and students’ assessment. Finally, several teachers and principals struggled to cope with the instructional pace, perceived as too stringent.

Moving to the impact of the PDP in real contexts, the reluctance or the unpreparedness of most teachers to work cooperatively on SSE were indicated as the main obstacles to school improvement. The lack of support at national and local level to guide SSE resulted in a further constraint for the implementation of evaluation competences. Nevertheless, the PDP provided participants with valuable opportunities to reflect on their evaluation practices. In this respect, two significant insights correspond to the rethinking of working practices to involve new colleagues in SSE and the inclusive re-design of the assessment and evaluation strategies.

Such preliminary results suggest that the PDP was functional to consolidate individual knowledge rather than to implement a cooperative SSE. Indeed, this latter appears as a system constraint that needs further attention both at policy and research level.

This paper critically examines the complex interplay between centralised/de-centralised governance structures, mechanisms and infrastructures to diffuse educational improvements across a system, and partnership configurations that facilitate or challenge system improvement, intentional or otherwise. Drawing on Mary Douglas’s Cultural Theory (1992,1996) and recent Learning Sciences’ theories on infrastructuring (Penuel,2019;Bielaczyc,2006), we focus on Singapore’s translational and mobilisation efforts for research innovations that occur at both system and school levels to highlight the relationships between power dynamics and diffusion processes in an education system driven by strong-state principles (Gopinathan,1994).

Research translation and mobilisation have become a global educational imperative for education research (Coope,2014), with research funding creating new system structures to scale up school innovations or university-school partnerships to drive improvements. Singapore’s two decades of governmental research funding has evolved to a stage where translation and mobilisation has become a central pillar of accountability, creating opportunities and challenges for school- and system-based solutions to problems of practice and policy.

Set in this context, three case studies of government-funded research studies are presented along with analysis of the governance structures that aim to exploit and explore system resources for translation/mobilisation in Singapore. We argue that these case studies signify different ways of organizing and perceiving social relations distinguished by Cultural Theory: hierarchy, egalitarianism, individualism, fatalism. Cultural Theory argues that these four ‘ways of life’ are underpinned by two dimensions of sociality – grid (status differentiation) and group (collectivity) (eg,Cornford, Baines & Wilson used cultural theory for network governance). Concomitantly, recent work into Research-Practice Partnerships have begun to shift attention away from educational infrastructures that are developed to support partnership processes, towards infrastructuring (i.e.,how practices shape infrastructures). Infrastructuring takes on a relational and praxeological perspective on infrastructures as cooperative activities and socio-cultural-technical arrangements that create “conditions that support educators in making innovations into working infrastructures for organizing learning activities” (Penuel,2019,p.660). Seen this way, both infrastructuring and Cultural Theory can be mobilised to better understand how social-cultural-political configurations, such as partnerships and governance structures, may generate different nuanced translational pathways and solutions that disrupts the long-standing linear perception of research translation drawn from the health sciences.

The three case studies are unique in scale and partnership models, lending themselves to analysis using our theoretical framing. Case study methodology along with interviews with key actors in the cases and document analysis are used to critically examine and map them to Cultural Theory/infrastructuring. The first is a large-scale longitudinal classroom-based study into teaching and learning in Singapore schools, with a hierarchical partnership model driven by top-down needs from the centralised government. The second is a self-sustaining Research-Practice Partnership that thrives through bottom-up school needs. The third draws on different partnership processes to spread school innovations. Each maps onto 3 of Cultural Theory’s dimensions, with implications for how they engage in infrastructuring processes. In examining these cases, we will discuss how systems can encourage broader partnership approaches that can lead to wider translations and meaningful impact across the system.