Conference Agenda

Remark: An earlier version of the project has been presented two years ago. We previously presented preliminary results of this study.

Our rapidly changing society influences the work of schools. Today, public schools in different countries are under enormous pressure to adapt and change as they need to to deal with technological improvements and cultural changes that leads to the increase heterogenization of societies (cf. e.g. OECD 2019, 2016; Imlig, Lehmann & Manz 2018; European Union 2016; Altrichter & Maag-Merki 2016).

In spite of the abundant literature about the conceptual and practical notions and processes of educational change, there is still a need to learn more about the ways to conceptualize and put into practice transitional change processes in individual schools and how to conceptualize and put into practice transitional change in individual schools and school district (Steffens, Heinrich, Dobbelstein, 2016; Reinbach 2016). Transitional change processes refer to the steps and strategies that a district takes to implement significant changes in its operations and policies such as curriculum reform, or the implementation of new technologies. Transitional change processes can be complex and multi-faceted, involving a range of different stakeholders, such as students, teachers, administrators, and community members.

For principals, teacher leaders and project managers in charge of implementation processes at different levels, these accelerated changes raise the question about how to bring successful and sustainable implementation that is accepted and carried through by all those involved?

Many leaders have assumed and taken on willingly the challenge of implementing numerous, simultaneous and, in part, highly complex transitional processes. They are often confronted with scepticism, reserve, doubt and even outright boycott, often by politicians, staff and parents (Rasmussen 2017; Landert 2014).

This is where the project Shaping Transitional Processes in Dynamic Societies comes in. This project was conducted by a research group of the Zurich University of Teacher Education and the education authority of the Principality of Liechtenstein. The project aims at developing relevant and helpful support services and practices for school principals and project leaders with which current and future reforms and improvements can be mastered alongside the demanding daily school routine.

In order to answer the questions and to advance the research process of the project as transparently and comprehensibly as possible, the approach of Design Based Research (DBR) was used (Euler and Sloane 2014; Euler 2017). Due to its methodological design, the chosen approach opens up the possibility to think and design practical school improvement processes in complementarity with school improvement research (Rau, Gerber u. Grell 2022, 353) and thus to consider both the relevance for practice and the further development of theoretical findings.

School performance feedback (SPF) systems present educational professionals with student achievement data in order to support self-evaluation and data-based decision making (Schildkamp & Teddlie, 2008; Visscher & Coe, 2003). However, in order to arrive at (formative) conclusions based on SPF, recipients need to make sense of the (summative) data they are presented with (Schildkamp, 2019; van der Kleij et al., 2015). Attribution, or reflecting on the causes of (learning) outcomes, is an integral part of this sensemaking process (Coburn & Turner, 2011). In line with the basic propositions of attribution theory (Weiner, 1985, 2010), the nature of educators’ causal explanations for student outcomes has been found to affect their emotions and subsequent (instructional) behavior (Wang & Hall, 2018).

Research finds that educational professionals often struggle to pinpoint factors that (may) have led to certain outcomes, particularly when those outcomes are unfavorable (Verhaeghe et al., 2010). Moreover, in defiance of ideals relating to data-based decision making, student performance is often attributed to external causes such as student characteristics, rather than matters internal to educational professionals, such as instruction (Evans et al., 2019; Schildkamp et al., 2016). This is especially apparent in cases of student failure (Wang & Hall, 2018). Consequently, it can be difficult to formulate productive decisions and constructive actions based on SPF (Schildkamp et al., 2016).

In the present study, we examine educational professionals’ causal explanations for results presented in a SPF report from a low-stakes national assessment (NA) in Flanders, Belgium. Like typical external standardized assessments, the Flemish NA relate school performance to standards and to the performance of reference groups (AERA et al., 2014; Visscher & Coe, 2003). We investigate educational professionals’ attributions of these data, with a particular interest in the locus of causality of the attributions they make. To what extent is the SPF interpreted introspectively (i.e., with regard to aspects of school policy and instructional practice that can be improved or sustained) and to what extent is school performance ascribed to external factors (such as aspects of the test itself, or input from students)?

Our review of the literature suggests that perceptions of school leaders remain somewhat underexposed in studies on attribution in educational data use. However, SPF intends to inform both school policy and instructional practice. Consequently, we will not only focus on teachers’ attributions in the present study, but also on causal explanations made by school leaders. Furthermore, we will examine causal explanations for both outcomes perceived as favorable, and those perceived as unfavorable. Perhaps in line with the very term “diagnosis”, we find that the attributions and attributional processes discussed in empirical literature are predominantly focused on explanations for student failure (van Gasse & Mol, 2021; Verhaeghe et al., 2010) and not so much for student success. However, school improvement is not only a narrative of identifying problems, but also of fostering what works.

In summary, the three research questions we address, are:

RQ1 To which internal and external factors do teachers and school leaders attribute their school’s performance on a national assessment?

RQ2 Do attributions differ according to the attributor’s work role?

RQ3 Do attributions differ according to the perceived favorability of the result?

We adopt a qualitative approach and make use of authentic educational data, because our aim is to illuminate how individuals and groups make meaning of something they experience (here, their schools’ SPF) from their own perspectives (Savin-Baden & Major, 2013).

This three-year study aims to design and validate the Kazakhstani Teacher Observational Protocol (KTOP) aimed at gauging the degree to which teachers/ instructors in Kazakhstan successfully implement reform-based practices in high school and undergraduate STEM lessons.

The 2011-2020 Educational Strategy for Kazakhstan outlined the goal of further development of the ‘training system and professional development’ with the primary objective being to improve teachers' learning and professional mastery in the Kazakhstani school system. Education reform has been considered necessary as the secondary curriculum was perceived as overly dense, with a focus on rote learning that resulted in only providing superficial knowledge rather than deep mastery of topics (Fimyar, Yakavets, & Bridges, 2014).

Education reform across contexts has focused on teachers' critical role in improving education quality (Schleicher, 2016). The question remains as to how initiated changes are validated or measured. Teacher observation instruments have been developed to measure effective teaching (Mantzicopoulos, et al., 2018). In addition to being used to provide standardized data to monitor education, these instruments can help to determine if professional development programs or the use of reformed curricula are producing changes in teaching practices (MacIsaac et al., 2001). Classroom observation instruments can be constructive tools for aiding in the evaluation of teachers and designing professional development (Evenhouse et al., 2018). Classroom observation allows observers to gather information on student and teacher behaviours and the classroom environment within an authentic setting, which is one way that links theory and practice in order to better understand the classroom environment (Snyder, 2012). Observations have been used to gather data on teachers’ integration of technology (Helmer et al., 2018); student-teacher interactions (Darling-Hammond, 2006); and explicit subject-area learning (Waxman et al., 2009). In Kazakhstan, the fast pace and wide range of reforms raises questions about teachers’ readiness to implement reforms in the classroom. Moreover, historically “open lessons” where teachers and administrators observe lessons may be rehearsed without systematic criteria for evaluation.

To address these issues, this paper reports on the pilot study where researchers collected data to determine how well the KTOP is suited to the reform-based practice in Kazakhstan. The reform-based practice was conceptualized by way of the following six subscales: lesson design and implementation, methods/teaching strategies, communicative interactions, student-teacher relations, assessment interactions, and integration of content and language in teaching.

. The specific questions addressed:

RQ1: What is the overall level of inter-rater reliability of the Kazakhstan Teaching Observational Protocol (KTOP) instrument?

RQ2: What is the level of inter-rater reliability of the KTOP instrument’s (a) lesson design and implementation (5 items), (b) methods/teaching strategies (5 items), (c) classroom culture (communicative interactions; 6 items), (d) classroom culture (student-teacher relationships; 5 items), (e) assessment interactions (3 items), and (f) integrating content and language in teaching (2 items) subscales?

RQ3: How well targeted is the KTOP instrument for identifying the higher- and lower-reform-based teaching practice in the Kazakhstani high school and higher educational contexts?