Conference Agenda

Great advances have been made in how we conceptualize, operationalize and measure aspects of teaching quality (Charalambous et al., 2021). However, this field of research is fragmented. Scholars work in silos, drawing on their own specific framework despite what are often strong commonalities in ambition, terminology, and structural features across frameworks. We argue that classroom video provides an avenue to work across these silos, allowing multiple frameworks to be applied to the same videos. This provides a common ground for discussions across frameworks, facilitating communication and potentially the integration of different frameworks for understanding teaching.

This symposium uses classroom videos as a common ground to break out of our silos through analyzing the same videos with a broad range of frameworks. This symposium consists of 3 papers (with three additional papers in a linked symposium) that use unique frameworks to investigate teaching quality. The frameworks in this symposia stem from different traditions and are at different stages of development. The three quantitatively-oriented frameworks are International Comparative Analysis of Learning and Teaching (ICALT; van de Grift et al., 2007), Protocol for Language Arts Teaching Observation (PLATO; Grossman, 2015), and the Teacher Education and Development Study-Instruct framework (TEDS-Instruct; Schlesinger & Jentsch, 2016). The two qualitatively driven frameworks are the Model for analysing Teaching Quality derived from the Joint Action framework in Didactics (JAD-MTQ; Sensevy, 2014; Ligozat & Buyck, accepted), the praxeological documentary video analysis (DVA; Martens & Asbrand, 2022). Part 2 of the double symposium (this part) focuses on ICALT, TEDS-Instruct, and DVA. The quantitatively oriented frameworks pre-determine definitions of teaching quality based on rubric dimensions and performance categories. The qualitative approaches understand teaching as a situated practice developing within a dynamic system of social, material, and semiotic interactions.

This symposium’s ambition is to have participants reflect on how one’s framework shapes how one constructs an understanding of teaching and the limitations and benefits of each framework through comparing the decompositions of the focal lessons across frameworks. Through this, we hope to build common understandings across frameworks and break out of our silos. To this end, we have asked each individual paper to attend to three research questions:

1. What are key patterns of teaching quality (in the focal videos) according to this framework?
2. How are aspects of the frameworks shaping constructing decompositions of the focal videos?
3. What are benefits and challenges with using this framework in analyzing aspects of teaching quality?

The contributors provide an overview of their respective frameworks based on the following categories: purpose and the theoretical grounding of the observation framework, facets of teaching captured, specific focus, grain size (e.g., unit of analysis on time scales), and empirical evidence and use. Then, contributors analyze the same four videos of lower secondary mathematics and language arts lessons from Nordic classrooms. Each contributor presents patterns of findings derived and afforded by their respective framework. To that end, we especially discuss patterns of teaching quality and how differences in the above-mentioned categories might shape the construction of findings as well as limitations and affordances across frameworks.

The inclusion of both mathematics and language arts, as well as both quantitatively and qualitatively oriented frameworks, sets this work apart from past important efforts in this area (e.g., Charalambous & Praetorius, 2018).

In the educational effectiveness research tradition, classroom observation has been recognized as a key instrument for uncovering variations in teaching quality in terms of student achievement (Muijs et al., 2018). In general, all existing observation instruments have the common goal for unravelling variations in teaching effectiveness to support teachers with valuable information that can help them develop their teaching skills. Several classroom factors matter for student attainment, including curriculum quality, the amount of learning time, various teaching skills including the creation of a safe and stimulating learning environment, efficient classroom management, the quality of instruction, teaching students how to learn, monitoring student progress, adapting teaching to student differences, and attention for students at risk of falling behind (eg., Creemers, 1994; Hattie, 2012; Scheerens & Bosker, 1997) Notably, not all behaviours synthesized from the literature are easily observable in classrooms. These factors are best revealed by means of teacher interviews and -surveys, student surveys, and value-added measures (Coe et al., 2014; van de Grift et al., 2014). The International Comparative Analysis of Learning and Teaching (ICALT, van de Grift, 2007) is a generic, non-subject specific teaching observation instrument, originally developed by the Dutch Inspectorates in cooperation with the Central Inspectorates in several European countries. This generic observation instrument focusses on capturing observable teaching behaviours of the whole lesson using high- (32 item) and low-inference (120 items) indicators. The indicators provided in the instrument are commonly observed in typical classroom practices, but are not all-inclusive. The observer can add good practices to justify his/her feedback. Although the ICALT framework is used as a formative feedback tool in teacher education and induction of early career teachers in the Netherlands, there is no research illustrating this formative potential. Our results reveal how low inference feedback shapes and provides justification for the overall teaching quality feedback provided by trained observers, which contributes to increasing the objectiveness of ratings. This is revealed by providing a) time indicators alongside illustrative quotes from the lesson, b) examples of (lacking) good practices, c) interaction symbols to increase the clarity of the feedback, following the increasing skill complexity levels inherent in the structure of the instrument.

Coe, R., Aloisi, C., Higgins, S., & Elliott Major, L. (2014). What makes great teaching? A review of the underpinning research. London: The Sutton Trust. Creemers, B. P. M. (1994). The effective classroom. London: Cassell. Hattie, J. (2012). Visible learning for teachers: Maximizing the impact on learning. London: Routledge. Muijs, D., Reynolds, D., Sammons, P., Kyriakides, L., Creemers, B. P. M., & Teddlie, C. (2018). Assessing individual lessons using a generic teacher observation instrument: how useful is the International System for Teacher Observation and Feedback (ISTOF)? ZDM, 50, 395–406. https://doi.org/10.1007/s11858-018-0921-9. Scheerens, J., & Bosker, R. (1997). The foundations of educational effectiveness. Oxford: Pergamon. van de Grift, W. (2007). Quality of teaching in four European countries: A review of the literature and application of an assessment instrument. Educational Research, 49(2), 127–152. https://doi.org/10.1080/00131880701369651. van de Grift, W., Helms-Lorenz, M., & Maulana, R. (2014). Teaching skills of student teachers: Calibration of an evaluation instrument and its value in predicting student academic engagement. Studies in Educational Evaluation, 43, 150-159.

Effective teaching has been extensively researched for decades. Studies have demonstrated the significant influence of teachers' performance on student achievement in various school subjects (Grossman et al., 2014; Hill et al., 2005). Theoretical frameworks and observational systems have been developed to conceptualize and measure teaching quality (e.g., Danielson, 2007; Klieme et al., 2009), often taking either generic or subject-specific perspectives. In this study, we discuss a hybrid observation system that draws on the established generic framework of Three Basic Dimensions but aims to address also subject-specific aspects of teaching quality to better explain student achievement in mathematics classrooms. The observation system was developed in the context of the Teacher Education and Development Study-Instruct (TEDS-Instruct). It captures four dimensions of teaching quality, two of which are considered generic (classroom management, student support), and two of which are considered subject-specific (cognitive activation, and educational structuring). This means that their operationalization is informed by the norms and concepts of the subject (Charalambous & Praetorius, 2018), and teachers need substantial (pedagogical) content knowledge to perform teaching behavior that reflects high levels of cognitive activation or educational structuring (e.g., posing challenging mathematical problems, changes of representations, being precise regarding mathematical language, providing adequate explanations). In this paper, we analyze generic and subject-specific dimensions across two Norwegian double lessons employing high-inference observer ratings. This means that videotaped lessons are presented to trained observers, and after a certain amount of time (i.e., a segment of a lesson), they provide an informed judgement on teaching behaviors and teacher-student interactions on 4-6 items per dimension (Jentsch et al., 2022). The results show that important aspects of teaching quality are captured by the observation system. For example, in the mathematics lesson, the items “teachers’ correctness” and “dealing with error” within educational structuring were assigned substantively higher scores than any other items in that dimension. Moreover, we argue that there was much variability in scores across the items assessing educational structuring. In contrast, for classroom management, all the items were assigned high scores. In the language arts lesson, different patterns emerge. There is more variability across items measuring cognitive activation. In addition, the teacher provides a lot of individual support to students but does little to support collaborative learning. A goal for future research on our observation system (and potentially others) could be to explore for which contexts and purposes valid conclusions can be drawn from classroom observation.

Charalambous, C., & Praetorius, A.-K. (2018). Studying instructional quality in mathematics through different lenses: In Search of Common Ground. ZDM Mathematics Education, 50, 535-553. Danielson, C. (2007). Enhancing professional practice: A framework for teaching. Association for Supervision and Curriculum Development. Grossman, P., Cohen, J., Ronfeldt, M., & Brown, L. (2014). The test matters: The relationship between classroom observation scores and teacher value added on multiple types of assessment. Educational Researcher, 43(6), 293-303. Hill, H. C., Rowan, B., & Ball, D. L. (2005). Effects of teachers’ mathematical knowledge for teaching on student achievement. American Educational Research Journal, 42(2), 371–406. Jentsch, A., Heinrichs, H., Schlesinger, L., Kaiser, G., König, J. & Blömeke, S. (2022). Multi-group measurement invariance and generalizability analyses for an instructional quality observation instrument. In M. Blikstad-Balas, K. Klette & M. Tengberg (Hrsg.), Ways of analyzing teaching quality. Potentials and pitfalls (pp. 121-139). Scandinavian University Press. Klieme, E., Pauli, C., & Reusser, K. (2009). The Pythagoras study. In T. Janik, & T. Seidel (eds.), The power of video studies in investigating teaching and learning in the classroom (pp. 137–160). Waxmann.

This paper examines the role of Documentary Video Analysis (DVA; Martens & Asbrand, 2022) in research on teaching quality. It applies DVA to two different lessons from the Nordic LISA study – one in mathematics and the other in Norwegian language arts (L1). By integrating these cases into an existing typology that emphasizes cognitive activation in classroom interactions (Schreyer, 2024), the study utilizes the qualitative reconstructive capabilities of DVA to examine the intricate dynamics of subject-specific teaching and learning processes. DVA is characterized by its ability to capture the complexity of classroom interactions and allows for uncovering the multifaceted relationships between the development of knowledge, embodied practices and the deeply rooted habitus of both teachers and students (Bohnsack, 2021; Martens & Asbrand, 2022). This methodological approach highlights the interconnected relationships between different aspects of teaching and contrasts with the more deductive methods used in previous research on teaching quality, which relied heavily on standardized observation manuals (e.g. Bell et al., 2019). This descriptive method aims to assess the quality of teaching following the empirical analysis. For this purpose, opportunity-use models (Vieluf & Klieme, 2023) are used to assess whether and how teaching stimuli are understood and used in a subject-specific context. Analyzing classroom situations in mathematics and Norwegian language arts through the lens of DVA reveals contrasting aspects of cognitive activation. In mathematics classrooms, the focus is on the teacher's central role in creating an environment that fosters cooperative learning, metacognition, and problem solving through the presentation of challenging tasks. This practice fits seamlessly with the theoretical constructs of cognitive activation (Praetorius et al., 2018). In contrast, the language arts classroom shows a notable divergence from the teacher's pedagogical standards and objectives, especially in student presentations where important literary devices are insufficiently identified and discussed, underscoring a discrepancy between the targeted instructional goals and actual knowledge development. The study discusses the potential of DVA as a tool for assessing teaching quality and critically examines its limitations in evaluating this. It emphasizes the strength of DVA in providing a descriptive rather than an evaluative analysis and questions the usefulness of the method for a comprehensive understanding of teaching quality through the comparative analysis of two subject-specific lessons.

Bell, C. A., Dobbelaer, M. J., Klette, K., & Visscher, A. (2019). Qualities of classroom observation systems. School effectiveness and school improvement, 30(1), 3-29. https://doi.org/https://doi.org/10.1080/09243453.2018.1539014 Bohnsack, R. (2021). Rekonstruktive Sozialforschung: Einführung in qualitative Methoden. (10th ed.). Barbara Budrich. Martens, M., & Asbrand, B. (2022). Documentary Classroom Research. Theory and Methodology. In M. Martens, B. Asbrand, T. Buchborn, & J. Menthe (Eds.), Dokumentarische Unterrichtsforschung in den Fachdidaktiken: Theoretische Grundlagen und Forschungspraxis (pp. 19-38). Springer VS. Praetorius, A.-K., Klieme, E., Herbert, B., & Pinger, P. (2018). Generic dimensions of teaching quality: the German framework of Three Basic Dimensions. ZDM: mathematics education, 50(3), 407-426. https://doi.org/https://doi.org/10.1007/s11858-018-0918-4 Schreyer, P. (2024). Kognitive Aktivierung in der Unterrichtsinteraktion: Eine qualitativ-rekonstruktive Analyse zu Passungsverhältnissen im Mathematikunterricht. Waxmann. Vieluf, S., & Klieme, E. (2023). Teaching effectiveness revisited through the lens of practice theories. In A.-K. Praetorius & C. Charalambous (Eds.), Theorizing Teaching: Current Status and Open Issues (pp. 57-95). Spinger Nature. https://doi.org/https://doi.org/10.1007/978-3-031-25613-4_3