Conference Agenda

Objectives

This paper reports the development work in cooperation between school and university at an introductory programme in Sweden. The introductory programme is an alternative educational path to support students who do not qualify for the national upper-secondary programmes. These students, for instance, migrant students or students with intellectual disabilities, belong to groups that are at-risk and have been marginalised in different ways during their school paths. An important challenge that we focus upon is that a majority of the students do not reach the goals set in the national curriculum for eligibility for a national programme or establishment in the labor market. More specifically, a narrow focus on eligibility for the national programmes implies that eligibility is measured in grades. Many students attend the programme for several years without achieving grades, even though they develop competences that are relevant and may facilitate transitions to future educational and professional paths.

This paper aims to investigate and shed light on alternative methods to map and assess students’ progress in ways that are not constrained by the formal curriculum.

The research questions are:

1. From a practice perspective, is it possible to develop alternative assessment categories?

2. If so, what categories were put into practice?

3. Why were these categories selected?

Perspective

Grades can constitute external motivation for students to engage in short time learning (Stan, 2012). However, when students run the risk of not reaching grades they often stop trying. Therefore, there is a need to develop methods both to motivate, and to show progress for students who experience failure to reach grades. Beyond assessing student achievement, data can support the development of teaching practices (Datnow & Park, 2018). This study investigates what such data could be and how the awareness and use of it is developed by teachers.

Methods

This paper draws from engaged scholarship (Van de Ven, 2007) and collaboration between practitioners in schools and researchers. Different techniques have been tested in the study to create matrixes and templates that support teachers and students in their reflective work on their learning and development. These tools were created after discussions and testing and an iterative process. Interviews were conducted with teachers and students to document this process. The data is compiled from continuous notes and documents generated during the process.

Findings

Preliminary findings show that the process of designing, testing, and implementing templates for assessment is long and complex. It implies that teachers need to come to a shared understanding of relevant data that reflect students’ success and not only their failures. An important consequence is that the process facilitated teachers’ and students’ engagement in deeper conversations about their learning, and the factors that may bring successful transitions for this student group. One challenge is that the process is time-demanding and requires planning and needs a focused investment by the school leadership in order to be implemented. Hopefully, the results of this study can provide an example for others who struggle to promote equity, inclusion and citizenship in their education.

Formal (i.e. systematically collected) school performance feedback (SPF) (such as test scores from central tests) can be a powerful tool for data-based decision making (Schildkamp, 2019). The assumption here is that school leaders and teachers (can) use SPF as a mirror to identify strengths and weaknesses in order to inform and drive teaching and school improvement (Coburn & Turner, 2011; Hulpia & Valcke, 2004; Schildkamp & Teddlie, 2008). However, research has found that both school leaders and teachers mostly fail to translate this data into meaningful information (Goffin, Janssen, & Vanhoof, 2023; Mandinach & Gummer, 2016; van der Kleij & Eggen, 2013). As a result, critical signals may not be picked up, or worse, incorrect or invalid inferences may lead to wrong decisions.

Sensemaking is a crucial stage in the cycle of data use (Mandinach & Schildkamp, 2021; Schildkamp, 2019), in which school leaders and teachers try to understand what the data means for the school or the classroom (Datnow, Park, & Kennedy-Lewis, 2012; Spillane, 2012). In this, comparing school’s performance against a well-chosen comparator, such as other (comparable) school’s performance (i.e. norm comparisons), can support users in understanding what the SPF means (Neumann, Trautwein, & Nagy, 2011; Schildkamp, Rekers-Mombarg, & Harms, 2012; Vanhoof, Mahieu, & Van Petegem, 2009). After all, only after SPF is interpreted (and analysed) it is transformed into information that can be used as a basis for data-based decision making (Mandinach, Honey, Light, & Brunner, 2008; Schildkamp & Poortman, 2015). Although education professionals commonly compare their school’s performance to that of a reference group (Goffin et al., 2023), literature shows that interpreting such norm-oriented information in SPF presents challenges (Hellrung & Hartig, 2013). In addition, it appears that data use processes are hampered when the norm-oriented comparator for SPF is not perceived as 'fair' (Vanhoof, Verhaeghe, & Van Petegem, 2009).

Given these considerations, in this study we explore the importance of comparing SPF in sensemaking (RQ1) and describe - from a user perspective - appropriate indicators and operationalisations for valid norm comparisons (RQ2). To this end, we conducted a Delphi study with two systematic rounds of consultations with both (future) feedback users and educational scientists. 18 informants participated in the study. Their responses were initially (round 1) analysed inductively according to the principles of Grounded Theory (Glaser & Strauss, 1967) and subsequently (round 2) deductively according to principles of framework analysis (Miles & Huberman, 1984). Preliminary results suggest that norm comparisons mainly help to understand the broader context of a school’s performance. In addition, the importance of comparing both 'gross' and 'net' school test scores appears to be endorsed. Finally, informants underline - in pursuit of valid comparisons of SPF - the importance of taking into account (mainly) pupil intake characteristics. As school feedback from central tests only proves relevant when the SPF is interpreted (and used) meaningfully and validly, these findings give rise to a discussion of theoretical lessons and policy implications.

Introduction

Over time it has become evident that peers and networks of students play a key role in their social and academic development (Bhargava et al. [2022], Calvo-Armengol et al. [2009]). Hence, analyzing social networks in classrooms is important to understand and leverage the benefits of these connections (Zarate [2023]). However gauging and collating this information can be daunting for teachers, especially when they have to cater to large classroom sizes and multiple administrative responsibilities in schools (King and Nomikou, 2018). Hence we ask a dual question; What are the social networks of students in Indian classrooms? Second, How do teachers engage with the network data of their classrooms?

Context

This study involved 24 classrooms and teachers from three schools with a total of 584 students in total. Participating classrooms ranged from grades 1 - 10. The schools from the states of Rajasthan and Uttar Pradesh were selected to represent three different economic brackets (low, middle, and high income).

Methods and Analysis

The study utilized a mixed-method approach to answer the research questions. Quantitatively, we used the method of social network analysis (Jackson et al. [2023]) to unpack and visualize these networks.

Additionally, we conducted semi-structured interviews with teachers as they were making sense of the visuals of their classroom network data. The qualitative data were analyzed using a constant comparative method (Glasser, 1965) to surface the different themes in the ways that teachers engage with their classroom data and connect it to daily practices.

Findings

Through our network analysis, we found that there was considerable dispersion in degrees across several dimensions. At least 20% of students in each classroom get nominated by only one other individual as a very good friend. On the other end of the spectrum, 20% of the students within each classroom get nominated by more than 6 individuals as very good friends. Similar levels of dispersion are seen in academic help and recess networks.

In our interviews, we saw teachers being surprised about the students who were isolated in their classrooms. While they had an idea of the “popular” students in their classrooms, they were less aware of the students who were at the margins. Second, teachers were making tangible connections between the data and their future practices. These prospective practices ranged from making seating plans according to the data to nominating isolated students to take a more active part in school events.

Implications and Connection to ICSEI Theme:

Through our network analysis, it is evident that there are students in classrooms who do feel isolated and need more support. Additionally, our engagements with teachers speak to the fact that they value data that is specific to their classrooms. We see teachers making nuanced inferences from the data and relating it back to tangible practices. This study connects to the overarching theme of the conference to create professional support and training for teachers utilizing data. Additionally, it also directly connects to the sub-theme of leveraging research and data for inquiry, insight, innovation, and professional learning.

To improve successfully, schools should engage in data-informed school improvement, and research shows that principals are key players when it comes to such data use in schools (e. g. Datnow & Hubbard, 2016; Demski & Racherbäumer, 2015). Only few schools engage in data-informed improvement systematically (e. g. Demski, 2019), and the implementation of such practices seems to be contingent on several factors (e. g. Altrichter & Maag Merki, 2016; Bremm et al., 2017). The question therefore is how principals (and their schools) can be supported and trained so that their practice is more data-informed. So far, the discourse regarding school improvement in Germany has focused on the responsibility that individual schools have for their own improvement, whereas the tasks and responsibilities of superordinate actors in the local and regional school administration (LRSA) have hardly been addressed (Authors 1). According to German law, LRSA have supervisory responsibility for the schools (Ackeren et al., 2015, p. 97; Authors 2). However, there is no coherent strategy or a common understanding of the role and tasks that LRSA have with regard to school improvement (e. g. Brüsemeister & Gromala, 2020; Authors 1). Instead, there is a lack of concrete guidance at the policy and legal level, on how to shape the role of LRSA, e.g., with regard to their support services for schools (Authors 1). Their self-perceptions primarily involves a controlling, possibly advisory function, but no responsibility for enabling schools to improve (Authors 2). And although “‘data-based’ governance has become a large part of contemporary school supervision” (Dabisch, 2023, p. 65), the support practices of LRSA with regard to data use are highly heterogeneous (Dabisch, 2023; Huber et al., 2020). Moreover, there is little theoretical input or research on how LRSA can and should be trained if they should function as supporting agency for school improvement (Authors 1). Consequently, most schools currently are on their own when it comes to school improvement capacities, such as data-informed school improvement (ibid). This paper therefore aims to explore what LRSA in Germany can or should do to support schools in developing a data-informed approach to improvement. To do so, we used guided interviews (Mayring, 2015) with principals and teachers of successful schools to understand the prerequisites of data-informed practices in schools. We moreover used observation protocols of expert workshops with experts from administration, practice, and research, in which they discussed what structures, resources, and professional support schools need to develop data-informed practices based on the interview data. Initial findings point to the importance of school-wide – and possibly cross-organizational – structures, networking between schools as well as between different actors in the field, and collaboration as relevant organizational structures that need to be supported by LRSA. Moreover, LRSA must cultivate positive attitudes toward data and improvement in principals and teachers, which necessitates that LRSA actors model that attitude. Since LRSA often are not specifically trained for their task, it is also vital that they acquire knowledge about data (use) as well as helpful structures for effective data use first.