Conference Agenda

Quality professional education, issues of fairness and equity, questions of improvement and effectiveness highly depend on decisions teachers and school leaders make daily. How can educators make professional decisions that reduce decision bias? An evidence-informed approach provides the opportunity to improve education that serves the needs of pupils and the community within a context and set of values (Brown et. al., 2017). In this symposium authors provide a broad view on evidence-informed practice from different perspectives. The first paper argues that, under the right conditions, intuition counts for as evidence. The author presents an integrated framework for professional decision making in education. The second paper focuses on the sense-making process in evidence use, discussing insights on the levels of the data, the data use process, the individual user, the social context of the user, users’ interactions, and the broader system level. Paper 3 investigates teacher leaders’ perception of their role in evidence-informed school decision-making processes. The discussant will provide overall insights in an interactive discussion with the audience.

Objectives

The urgency for continuous improvement and innovation in education is high, but the struggles are bigger than the success. The bad reputation of educational change is harmful for students who do not get what they need and for educators who feel tired and frustrated. Educational change is dependent on the professional capital in schools, in which decisional capital is of pivotal importance (Hargreaves & Fullan, 2015). The objective of this conceptual paper is to understand and explain how professional decision-making, as an integration of data, intuition and research can strengthen the effectiveness and fairness of educational decisions.

Theoretical Framework

Decisional capital refers to the capacity to make sound judgements (Hargreaves & Fullan, 2015), which can be defined as one’s ability to assess situations using relevant information and apply experience in order to come to a conclusion (Standing, 2010). Thus, professional decision-making builds on information and experience.

Educators have the tendency to decide and act fast, driven by the need to solve problems quickly to reach immediate results. This has led to an educational routine of fast, often intuitive decision-making with little use of information. Intuitive decision-making does not sufficiently tackle deeper problems or fails to address students’ diverse needs (Vanlommel et. al., 2017). Fair and effective judgement also requires data and research to challenge and complement intuition. Attention for evidence-informed practice in education is not new (e.g. Hargreaves 1996), but it still remains largely unclear how educators can use and integrate evidence into their professional decisions, and what counts for as evidence. This conceptual paper integrates insights from data use, evidence use, intuition and decision making theory with the aim of understanding and enhancing professional decision-making in education.

Research Question

How can educators make professional decisions integrating data, intuition and research?

Results

The use of data and research has been combined, and there is growing insight that personal judgement should play a part (Brown, Schildkamp, & Hubers, 2017). Up till now, intuition has not been explained and integrated in evidence-informed practice. In our results we present insight, strengths and weakness from the different views on educational decision-making. We argue that, under certain conditions, intuition can serve as source of evidence. In a framework for professional decision-making in education we integrate data, research and intuition, describing how intuition can be subjected to scrutiny in interactions with colleagues.

Importance

Educational decisions influence the fairness of education, as well as school improvement and effectiveness. It is important that educators make professional decisions, try to reduce decision bias that might lead to the continuation of classical or self-fulfilling procedures instead of innovation. Our framework explains how decisions can be based on a combination of intuition, data and evidence, grounded in context and validated through professional collaboration.

Connection to conference theme

Decisions of teachers and school leaders highly influence the realization of quality professional education. Decision theory is an important aspect of high quality teaching and should be part of both initial teacher education and continuing professional development.

Objectives and focus of inquiry

School performance feedback (SPF) consists of formal data about a school’s functioning – often in terms of student achievement – collected by an external party and confidentially fed back to the school for self-evaluation (Visscher & Coe, 2003). SPF-like data can be a highly informative ‘piece of the puzzle’ for educational professionals to use in data-based decision making, but the complexity of factors that influence the actual use of these data is not yet fully understood. In the present project, we examined how teachers and school leaders make sense and make use of SPF from external standardized assessments, and explored factors that promote or hinder these processes.

Framework, approach and context

We departed from the notion that sensemaking is a central phase in data use for school improvement (Schildkamp, 2019). Raw data acquire meaning, validity, and utility through interpretation and contextualization, which is why data use is not linear or straightforward (Bertrand & Marsh, 2015; Datnow et al., 2012; Ikemoto & Marsh, 2007; Mandinach & Schildkamp, 2021). Based on a conceptual exploration of the sensemaking perspective, as applied to teachers’ and school leaders’ engagement with formal achievement data such as SPF, we constructed a framework to integrate insights on the levels of the data, the data use process, the individual user, the social context of the user, users’ interactions, and the broader system level (Authors, 2022a). These theoretical insights were supplemented with original empirical research conducted within the context of the Flemish national assessments.

Findings

In this contribution, we give an overview of the different levels of the framework and present illustrations based on our own qualitative and quantitative inquiries into (aspects of) educational professionals’ use of SPF. Based on data from 22 semi-structured interviews, we propose that issues regarding the user validity of SPF reports can be explained in part by a disconnect between data providers’ and data users’ frames of reference (Authors, 2023). Additionally, we discuss the complexity of a phase that follows data analysis, i.e. formulating a diagnosis. Teachers and school leaders make wide ranges of causal attributions when interpreting SPF, and tend to address different factors according to their profile (Authors, submitted). This attests to the importance of collective sensemaking. That ‘power of the collective’ is underscored by results from a quantitative study based on survey data from 470 educational professionals. Here, a (perceived) shared vision on SPF use in school teams emerges as a significant driver (Authors, 2022b).

Importance and connection to conference theme

In order for data such as SPF to live up to its potential to effectively inform educational decisions and truly contribute to school improvement, we need to further scrutinize the pitfalls and opportunities that a sensemaking perspective can expose. By shedding more light on what actually happens when a SPF report comes through a school’s proverbial letterbox, we can find ways to make both data users and data providers more conscious of the frames they employ, and provide input for hands-on support, training and professional development.

Objectives and problem of practice

Since the Austrian PISA shock in the 2000s, the awareness for “evidence-informed teaching and leadership” has grown (Jesacher-Rössler & Kemethofer, 2022), associated with the notion that decisions in education should be “based on a combination of personal judgement, research evidence and local school data” (Brown et al. 2017, p. 154). Accordingly, the White Paper of the Austrian Federal Ministry of Education, Science and Research (BMBWF, 2019) mandated that decisions, particularly those related to quality assurance processes, should be based on evidence. This contribution provides a policy-oriented perspective on the newly introduced Austrian Quality Management System (QMS), focusing on school quality coordinators (SQCs). These teachers are assigned by school principals to ensure the implementation of reforms. Collaborating closely with the principal, they are responsible for the interpretation of data feedback and assisting school management processes to derive and implement development measures. Overall, Austrian SQCs are in the position to “form a powerful link between school leader and teaching staff” (Rössler & Schratz, 2018, p. 287), and there is evidence available that SQCs’ impact can be strong (Rössler & Schratz, 2018). Still, they face a multitude of challenges as they are encountering difficulties in their endeavors to establish an evidence-orientation at their schools. They are neither top-down, nor bottom-up change agents (Brown et al., 2021), but stand somewhat in-between.

Research questions

This contribution aims to investigate:

- SQCs’ own perception of their role in the given context;

- how much usefulness SQCs attach to particular types of data and evidence;

- what effect SQCs’ specific professionalization has on their practice;

- how their stance towards data-based or evidence-informed decision-making is reflected in use of data and evidence at site.

Theoretical framework

Frameworks for teacher leadership (e.g., Tomal et al., 2014; Lohman, 2013) and teacher competency models (e.g. Baumert & Kunter, 2011; Kunter et al., 2013) will be juxtaposed in order to develop an understanding of what SQCs might need to professionalize in to be successful in their specific roles, which have formerly been reserved for principals only. Related to latter is the unspoken requirement that SQCs are not only data literate, but also evidence and assessment literate (for the different concepts, cf. Groß Ophoff & Cramer, 2022). Beck and Nunnalley (2021) have described this as the profile of data leaders on the level of expert users, who are able to support other data users on individual, program or institutional level. Accordingly, Austrian university colleges of teacher education offer post-qualification programs and are of “great responsibility to create awareness and sensitization for evidence-informed practice” (Jesacher-Roessler & Kemethofer, 2022, p. 328).

Methods and prospective results

The sample for this study is gathered among teacher leaders (approximately 5 persons) who voluntarily participate in a professional development course for SQCs at the University College of Teacher Education Lower Austria. They will be interviewed using a theory-based guideline. First results (based on qualitative content analysis) will be presented.

Connection to the conference theme

Findings will inform continuing professional development for SQCs regarding evidence-informed decision-making.