09. Assessment, Evaluation, Testing and Measurement
Paper
Comparing the Predictive Validity of Ratings on Opportunity and Use of Cognitive Activation: Does the Source of Information Matter?
Charalambos Charalambous, Sergios Sergiou, Maria Perikli
University of Cyprus, Cyprus
Presenting Author: Charalambous, Charalambos
Teaching quality has been empirically shown to be a key predictor of student learning (Stronge, 2013). In studying teaching quality, teaching effectiveness researchers have for years focused on the opportunities provided to students for learning, as these are crafted through teacher-student and student-student interactions with the content. Yet, following Fend’s (1981) distinction between opportunity and use and more recent work in the German-speaking countries on this issue (cf. Vieluf et al., 2020), teaching effectiveness researchers worldwide have started to more increasingly attend to not only the opportunities created for student learning, but also to how students make use of these opportunities, on the grounds that the former without the latter can only partially explain student learning.
Of particular interest in this line of research are the opportunities provided for student cognitive activation, often identified as the potential for cognitive activation, and students’ use of these opportunities, often identified as cognitive activity (Groß-Mlynek et al., 2022; Rieser & Decristan, 2023). This heightened interest in cognitive activation is justified both because of empirical findings corroborating its role for students’ cognitive and affective learning (e.g., Lazarides & Buchholz, 2019), but also due to studies showing cognitively activating teaching to be highly needed worldwide (cf. OECD, 2020).
Despite this increased interest, our review of the literature showed that in most extant studies scholarly attention has mostly been directed to the potential for cognitive activation without also exploring students’ cognitive activity. Only three studies concurrently attended to and measured both the opportunity and use for cognitive activation in relation to student learning (Lipowsky et al., 2009; Merk et al., 2021; Rieser & Decristan, 2023). These studies, however, differ not only in reporting mixed findings, but also in their methodological design: whereas the former two employed expert classroom observers’ ratings to capture the potential for cognitive activation and student ratings to capture cognitive activity, the latter utilized student ratings to measure both. Concurrently attending to different sources of information (e.g., expert classroom observers and students) is, however, critical, given scholarly calls (e.g., Fauth et al., 2020) to more systematically examine how the source of information contributes to the predictive validity of the teaching quality measures employed.
The scarcity of studies that concurrently attend to the predictive validity of opportunity and use (in cognitive activation); the mixed findings of these studies; and the fact that none of them concurrently used different sources of information to capture the opportunity for cognitive activation—note that the use of opportunities is typically captured only through student ratings—raise two questions:
- How does the predictive validity of ratings on the opportunity for cognitive activation compare with that of ratings on the use of cognitive activation?
- Does this differ when different sources of information (expert classroom observers vs. students) are employed to capture the opportunity for cognitive activation?
Addressing these questions can have important methodological implications for measuring aspects of teaching quality in more optimal ways, but also practical implications for teachers’ formative evaluation. However, to more adequately answer these questions, and especially the second one, attention needs to be paid to ensuring that the measures of the different sources obtained are aligned in the sense of tapping into similar—and if possible identical—aspects of teaching quality. Doing so becomes particularly important, given that our review of the literature showed only a few studies comparing aligned measures of teaching quality from different sources (e.g., van der Scheer et al., 2018)—and even in those cases, not with respect to the issue of their predictive validity.
Methodology, Methods, Research Instruments or Sources UsedSample and measures. A sample of 31 elementary school teachers and their sixth-grade students (n=542) participated in the study. For comparability purposes, all participating teachers were observed teaching the same three algebra lessons. Students’ algebra performance before and after these lessons was measured through a validated mathematics test (Authors, 2019).
We measured the potential for cognitive activation in two ways:
(a) Expert observer ratings: The 93 lessons were coded by three expert raters trained and certified for this purpose; the raters first rated these lessons individually and then met in pairs to discuss and reconcile their scores. For this study, we utilized the raters’ reconciled scores on the Common Core-Aligned Student Practices of the Mathematical Quality of Instruction (cf. Charalambous & Litke, 2018) framework, which capture the opportunities provided to students for cognitive activation through working on challenging tasks, providing explanations, and engaging in reasoning.
(b) Student ratings: Drawing on prior work (e.g., Fauth et al., 2014), we used 8 survey items capturing students’ perceptions of how frequently their teacher gave them opportunities to engage in cognitively activating teaching (e.g., through handling different solutions, providing explanations, or working on complex tasks/new content). Student ratings were aggregated to the classroom level to reflect the class’ overall perception of the opportunities provided.
Four items were utilized to measure student cognitive activity, drawing on existing scales (e.g., Merk et al., 2021). Unlike for the potential of cognitive activation, we used student ratings at the individual rather than the classroom level, given that they were taken to reflect students’ individual self-perceptions of how they themselves experienced to be cognitively challenged.
We also administered a validated survey (Kyriakides et al., 2019) measuring students’ SES, gender, and ethnicity. Finally, we collected information on teachers’ gender, years or experience, and education credentials.
Analyses. Two-level (students nested within teachers) multilevel modeling analysis was utilized with students’ performance at the culmination of algebra teaching as the dependent variable. After controlling for student and teacher background characteristics as well as students initial algebra performance, we introduced observer and student ratings on cognitive activation (first in isolation and then in combinations). We ran these analyses twice, first for the ratings as composites, and then for individual items (those that were aligned in content). In comparing the predictive validity of the examined predictors, we considered both their statistical significance and the percentage of the unexplained variance explained.
Conclusions, Expected Outcomes or FindingsFor the composites, both the potential for cognitive activity (opportunity) and cognitive activity (use) were predictive of student learning, regardless of how they were measured. When introduced in isolation to the model, each significantly contributed to student learning. For opportunity, classroom expert ratings explained a much higher percentage of the unexplained variance (4.20% total, all at the teacher level, explaining about 70% of the unexplained variance at that level) compared to that explained by student ratings (1% total, all at the teacher level, explaining 16% of the unexplained variance at that level). Compared to student opportunity ratings, student use ratings explained a slightly higher percentage of the total variance (1.5% total, corresponding to about 7% and 5% of the unexplained variance at the teacher and student level, correspondingly). When all three ratings were introduced, student opportunity ratings were no longer significant. Interestingly, the combination of expert ratings on opportunity and student ratings on use explained the highest percentage of the unexplained variance of all the models considered (5.30% total, explaining 70% and 3% of the unexplained variance at teacher and student level correspondingly).
When comparing the aligned survey and MQI items (e.g., providing explanations; working on challenging tasks/new content), we noticed that whereas in all cases, the expert observer ratings had a significant contribution to student learning, student ratings did have such a consistent contribution (and also explained a smaller percentage of the unexplained variance).
Collectively, these findings underline the value of concurrently attending to both opportunity and use. They also suggest that classroom observer ratings might have more predictive validity than student ratings when it comes to the opportunities provided to students for cognitive activation. Future replication studies with a different student population on a different subject are, however, needed to test the veracity of these arguments.
ReferencesAuthors (2019). [Blinded for peer-review purposes].
Charalambous, C. Y., & Litke, E. (2018). Studying instructional quality by using a content-specific lens: The case of the Mathematical Quality of Instruction framework. ZDM, 50(3), 445–460. https://doi.org/10.1007/s11858-018-0913-9
Fauth, B., Decristan, J., Rieser, S., Klieme, E., & Büttner, G. (2014). Student ratings of teaching quality in primary school: Dimensions and prediction of student outcomes. Learning and Instruction, 29, 1–9. https://doi.org/10.1016/j.learninstruc.2013.07.001
Fauth, B., Göllner, R., Lenske, G., Praetorius, A.-K. & Wagner, W. (2020). Who sees what? Conceptual considerations on the measurement of teaching quality from different perspectives. Zeitschrift für Pädagogik, 66, 63–80. https://doi.org/10.15496/pub likation-41013
Fend, H. (1981). Theorie der schule. Urban & Schwarzenberg.
Groß-Mlynek, L., Graf, T., Harring, M., Gabriel-Busse, K., & Feldhoff, T. (2022). Cognitive activation in a close-up view: Triggers of high cognitive activity in students during group work phases. Frontiers in Education, 7. https://doi.org/10.3389/feduc.2022.873340
Kyriakides, L., Charalambous, E., Creemers, H. P. M. B., & Dimosthenous, A. (2019). Improving quality and equity in schools in socially disadvantaged areas. Educational Research, 61(3), 274–301. https://doi.org/10.1080/00131881.2019.1642121
Lazarides, R., & Buchholz, J. (2019). Student-perceived teaching quality: How is it related to different achievement emotions in mathematics classrooms? Learning and Instruction, 61, 45–59. https://doi.org/10.1016/j.learninstruc.2019.01.001
Lipowsky, F., Rakoczy, K., Pauli, C., Drollinger-Vetter, B., Klieme, E., & Reusser, K. (2009). Quality of geometry instruction and its short-term impact on students’ understanding of the pythagorean theorem. Learning and Instruction, 19(6), 527–537. https://doi.org/10.1016/j.learninstruc.2008.11.001
Merk, S., Batzel-Kremer, A., Bohl, T., Kleinknecht, M., & Leuders, T. (2021). Nutzung und wirkung eines kognitiv aktivierenden unterrichts bei nicht-gymnasialen schülerinnen und schülern. Unterrichtswissenschaft, 49(3), 467–487. https://doi.org/10.1007/s42010-021-00101-2
OECD. (2020). Global teaching in sights: A video study of teaching. OECD Publishing. https://doi.org/10.1787/20d6f36b-en
Rieser, S., & Decristan, J. (2023). Kognitive aktivierung in befragungen von schülerinnen und schülern. Zeitschrift Für Pädagogische Psychologie, 1-15. https://doi.org/10.1024/1010-0652/a000359
Stronge, J. (2013). Effective teachers = student achievement: What the research says. Routledge.
van der Scheer, E. A., Bijlsma, H. J. E., & Glas, C. A. W. (2018). Validity and reliability of student perceptions of teaching quality in primary education. School Effectiveness and School Improvement, 30(1), 30–50. https://doi.org/10.1080/09243453.2018.1539015
Vieluf, S., Praetorius, A., Rakoczy, K., Kleinknecht, M., & Pietsch, M. (2020). Angebots-nutzungs-modelle der wirkweise des unterrichts: Ein kritischer vergleich verschiedener modellvarianten. Z. Pädagog. 66, 63–80. https://doi.org/10.25656/01:25864
09. Assessment, Evaluation, Testing and Measurement
Paper
Effects of Presentation Order on the Reliability of Classroom Observations of Teaching Quality in Norwegian Mathematics and Science Lessons
Armin Jentsch, Bas Senden, Nani Teig, Trude Nilsen
University of Oslo, Norway
Presenting Author: Jentsch, Armin
Teaching quality has been researched extensively in the past years with a high number of empirical studies in educational sciences and psychology. To better understand how learning develops in the classroom, scholars are concerned with the reliable and valid measurement of teaching quality. In doing so, Helmke (2012) considers classroom observation as the “gold standard” amongst other ways of capturing teaching quality (e.g., student ratings in large-scale assessment) because of its direct assessment of teaching practices. However, classroom observation also draws on resources and can be prone to many sources of measurement error. Therefore, when performing classroom observation for any purpose (i.e., research, practical, policy) it is important to consider how to allocate (limited) resources such that high score reliability and valid conclusions about teaching quality are ensured.
Studies suggests that changing the presentation order of lesson segments could particularly affect score reliability (e.g., Mashburn et al., 2014). For instance, using the generic CLASS-Secondary observation system (Pianta et al., 2008), Mashburn et al. (2014) found that 20-minute lesson segments presented in a random order to raters achieved the best combination of reliability and predictive validity. In the present study, we used a different, hybrid observation system (i.e., comprising both generic and subject-specific aspects of teaching quality, Charalambous & Praetorius, 2018) that was first developed to capture teaching quality in German secondary mathematics classrooms, and that draws on the Three Basic Dimensions of teaching quality (classroom management, student support, and (potential for) cognitive activation, e.g., Klieme et al., 2009). The three basic dimensions have been shown to positively relate to students’ achievement in mathematics classrooms across several studies and various operationalizations (e.g., Baumert et al., 2010; for an overview see Praetorius et al., 2018).
Classroom management refers to teachers’ procedures and strategies that enable efficient use of time (time on task), as well as behavioral management (Kounin, 1970). Student support draws on self-determination theory (Deci & Ryan, 1985) and aims at both motivational and emotional support, as well as individualization and differentiation. Cognitive activation, finally, addresses opportunities for "high-order thinking" from a socio-constructivist perspective on teaching and learning (e.g., problem solving, Mayer, 2004).
Empirical evidence suggests that generic and subject-specific measures of teaching quality generate moderately correlated, but still unique information about classrooms (Kane & Staiger, 2012). Evaluating this finding, Charalambous and Praetorius (2018) conclude that subject-specific and generic measures together could explain more variance in student learning in mathematics than generic measures alone. Since subject-specificity might be considered a continuum rather than a binary characteristic, they argue that it could be meaningful for scholars to develop hybrid frameworks of teaching quality, which take both perspectives into account (i.e., generic and subject-specific, see also Charalambous & Praetorius, 2018).
The purpose of the present study is twofold: First, we aim at investigating the effect of presentation order on score reliability in two subjects. Second, we explore an optimal design for the implementation of our observation system in terms of score reliability. Towards this end, we assigned four trained raters to rate videotaped Norwegian mathematics and science lessons either in sequential 20-minute segments, or two nonsequential 20-minute segments.
Methodology, Methods, Research Instruments or Sources UsedData was obtained from schools from the Oslo metropolitan area in Norway, with teachers conveniently participating in the study. In total, 15 classrooms were sampled, and from each classroom one through six lessons are available that were videotaped over the course of several weeks. The length of the lessons varied between 24 and 106 minutes, and they were cut into 20-minute segments for analysis. For the purpose of this study, two segments from every mathematics classroom and two segments from every science classroom were analyzed, and the segments were scored under both study conditions (i.e., sequential and nonsequential).
We applied the observation system from the Teacher Education and Development Study–Instruct (TEDS-Instruct, e.g., Schlesinger et al., 2018). Consequently, the framework and corresponding instrument involved four teaching quality dimensions with four to six items each that also used different indicators for mathematics and science classrooms. Raters were trained extensively over the course of one week by studying the rating manual, conducting video observations, and discussing the results with master raters. However, no benchmarks were applied. All raters were student teachers in mathematics and science programs, and they were at least in their fourth year.
To analyze the effect of presentation order on score reliability, we designed our study as follows. For each lesson, we randomly assigned one rater to the sequential condition. The rater would then score both segments of this lesson. This condition is referred to as the static condition. At the same time, two different raters were assigned to the nonsequential condition. We had these raters randomly score either the first or the second segment of a lesson. This we refer to as the switching condition. Using this experimental design, raters were balanced across subjects and conditions. Since in this study we only analyzed one lesson for each teacher-subject combination, raters would not score the same teacher or classroom twice within the same condition or subject. However, there was a chance that raters could encounter the same teacher in a different subject.
We applied Generalizability theory (GT, Cronbach et al., 1972) to estimate measurement error and reliability in our study. GT was developed specifically for complex measurement situations with many potential sources of error, such as classrooms, lessons, or raters. GT makes use of the linear mixed model to estimate variance components for each measurement facet of interest (G Study).
Conclusions, Expected Outcomes or FindingsOur results show that, overall, presentation order had little impact on score reliability. In more detail, score reliability was high for science lessons in both conditions, and acceptable for two out of four teaching quality dimensions in mathematics with slightly better results for the static condition. A low share of lesson variance and a relatively high share of within-lesson variation was found for cognitive activation. Correlation analysis and mean comparisons revealed no meaningful differences between conditions. Our results could be depended on the fact that we only sampled one lesson per classroom. Other studies show that particularly subject-specific aspects of teaching quality vary severely over time (e.g., Praetorius et al., 2014). However, we did not encounter similar issues in science classrooms, which suggests that (1) teaching quality in science and mathematics lessons varies on different time scales, (2) the observation system functions differently in mathematics and science lessons, or (3) raters have applied the measure differently between subjects.
ReferencesBaumert, J., Kunter, M., Blum, W., Brunner, M., Voss, T., Jordan, A., . . . Tsai, Y.-M. (2010). Teachers' mathematical knowledge, cognitive activation in the classroom, and sudent progress. American Educational Research Journal, 47(1), 133–180.
Charalambous, C., & Praetorius, A.-K. (2018). Studying Instructional Quality in Mathematics through Different Lenses: In Search of Common Ground. ZDM Mathematics Education, 50, 535-553.
Cronbach, L. J., Glaser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. John Wiley.
Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in human behavior. Perspectives in social psychology. Plenum.
Helmke, A. (2012). Unterrichtsqualität und Lehrerprofessionalität: Diagnose, Evaluation und Verbesserung des Unterrichts. Klett-Kallmeyer.
Kane, T. J., & Staiger, D. O. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Bill & Melinda Gates Foundation.
Klieme, E., Lipowsky, F., Rakoczy, K., & Ratzka, N. (2006). Qualitätsdimensionen und Wirksamkeit von Mathematikunterricht: Theoretische Grundlagen und ausgewählte Ergebnisse des Projekts "Pythagoras". In M. Prenzel & L. Allolio-Näcke (Eds.), Untersuchungen zur Bildungsqualität von Schule. Abschlussbericht des DFG-Schwerpunktprogramms (pp. 127-146). Waxmann.
Kounin, J. S. (1970). Discipline and group management in classrooms. Holt, Rinehart & Winston.
Mashburn, A. J., Meyer, J. P., Allen, J. P., & Pianta, R. C. (2014). The effect of observation length and presentation order on the reliability and validity of an observational measure of teaching quality. Educational and Psychological Measurement, 74(3), 400-422.
Mayer, R. E. (2004). Should there be a three-strikes rule against pure discovery learning? The case for guided methods of instruction. American Psychologist, 59(1), 14–19.
Pianta, R. C., La Paro, K. M., & Hamre, B. K. (2008). Classroom Assessment Scoring System™: Manual K-3. Paul H. Brookes Publishing Co.
Praetorius, A.-K., Klieme, E., Herbert, B., & Pinger, P. (2018). Generic dimensions of teaching quality: The German framework of Three Basic Dimensions. ZDM Mathematics Education, 50, 407-426.
Schlesinger, L., Jentsch, A., Kaiser, G., König, J., & Blömeke, S. (2018). Subject-specific characteristics of instructional quality in mathematics education. ZDM Mathematics Education, 50, 475-491.
Shavelson, R. J., & Webb, N. M. (1991). Generalizability Theory: A Primer. SAGE Publications.
09. Assessment, Evaluation, Testing and Measurement
Paper
Supporting Teachers to Participate in Lesson Study with Advisors or Facilitators: Searching for Differential Effects on Student Learning Outcomes
Leonidas Kyriakides1, Maria Vrikki1, Panayiotis Antoniou1, Efi Paparistodemou2
1University of Cyprus; 2Cyprus Pedagogical Institute
Presenting Author: Kyriakides, Leonidas
Lesson Study (LS), a collaborative and inquiry-based model of teacher professional development, has received increased attention in recent years. It involves teachers in small groups identifying an issue in their teaching practice and organising an inquiry to learn more about it. Specifically, teachers jointly plan, teach and reflect on lessons. A rich body of mostly descriptive and qualitative studies suggests that with LS experience teachers may develop pedagogical knowledge, and may be able to identify students’ misconceptions (e.g. Cheung & Wong, 2014; Vrikki, Warwick, Vermunt, Mercer & van Halem, 2017). However, more large-scale controlled studies are needed in order to systematically evaluate the effect of LS (Benedict et al., 2023). In addition, even less evidence exists of the impact of teachers’ participation in LS on their students’ achievements (e.g. Cheung & Wong, 2014; Kager, Mynnott & Vock, 2023).
In addition, variations of the LS model include the presence of an external expert (i.e., LS facilitator, knowledgeable other, moderator). The literature identifies many ways that this external expert can support teachers, including enhancing in-depth discussions about students’ thinking, shaping the quality of the teachers’ inquiry, fostering teachers’ discussions by posing questions, encouraging teachers to share their experiences and managing the LS process (e.g. Akiba et al., 2019; Schipper et al., 2017; Bjuland & Helgevold, 2018; De Vrie, Verhoed & Goei, 2016). These responsibilities are not only vaguely described, but their effects have not been studied either. At the same time, research on teacher and school improvement argues for the important role of an advisory and research team that can work closely with teachers and support their attempt to design, implement and evaluate their action plans (e.g., Creemers & Kyriakides, 2012; Scheerens, 2013).
This paper addresses both limitations in the literature described above. First, it aims to examine how secondary mathematics teachers’ participation in LS affects their students’ achievement in reasoning. Second, it aims to examine how different types of support offered to LS teacher groups can further enhance students’ achievement. Specifically, it examines the impact of the support of a LS facilitator, who guides teachers through the LS process, fosters their discussions and promotes teacher learning which is expected to affect student learning outcomes. It compares this to the impact on student learning outcomes of the support of an LS advisor who in addition to guiding teachers through the LS process, offers subject advice and his/her own ideas to the teachers. Although having these kinds of support is not uncommon, little is known about the effect of different types of support that teachers may have in implementing LS.
Methodology, Methods, Research Instruments or Sources UsedA group randomisation study took place in Cyprus during the school year 2022-2023. A total of 42 lower secondary mathematics teachers, who taught Grades 7 to 10 (students aged 11-14), in 13 secondary schools, were randomly allocated to three groups: two experimental and one control group. Teachers in two experimental groups formed small LS teams (2-3 teachers) and implement a specific variation of the LS model, namely Dudley’s (2019) “Research Lesson Study”. This is a three-cycle model, meaning that to complete one LS teachers had to plan three “research” lessons, one teacher teaching them while the others observed, and then to jointly reflect on the lessons. Each LS team completed two LSs during the school year, that is six research lessons. The difference between the two experimental groups was that teachers in one experimental group were supported by a “LS Facilitator”, who coordinated the discussions and helped teachers through the LS process, while teachers in the second experimental group were supported by a “LS Advisor”, who also provided advice on mathematics pedagogy. Teachers of the control group did not participate in any LS.
Two classes per teacher were randomly selected to participate in the study, giving a total of 966 student participants. The students completed mathematical reasoning tests at the beginning and at the end of the school year. Specifically, a total of five tests were developed by a group of mathematics educators and expert teachers to assess students’ cognitive learning outcomes in relation to mathematical reasoning. These tests were used as pre-tests and post-tests across the four grades. Prior to the intervention, the construct validity of the five tests was examined. Data were analysed by using the Rasch model and support to the validity of the tests was provided. Student background data (i.e., students’ socioeconomic background and gender) were also collected via a student questionnaire.
Conclusions, Expected Outcomes or FindingsUsing one-way ANOVA, it was found that there was no statistically significant differences on student prior achievement among the three groups at the beginning of the intervention. Inferential analysis revealed no statistically significant differences at .05 level in terms of student background characteristics (i.e., SES and gender). To search for the impact of the intervention on student learning outcomes, multilevel analysis of student achievement in mathematical reasoning was conducted for the data collected at the end of the intervention. The empty model revealed that the teacher level rather than the class level should be considered for this analysis. In Model 1 prior achievement in mathematical reasoning was added as an explanatory variable. Prior achievement was found to have a statistically significant effect on final achievement. In Model 2, student background variables including grade were added as explanatory variable. Finally, two dummy variables (with the control group treated as a reference group) were added to model 2. Only the dummy variable concerned with supporting teachers with an advisor to implement LS was found to have a statistically significant effect at .05 level. Thus, the multilevel analysis revealed that students whose teachers participated in the Advisor group had better results in mathematical reasoning than students whose teachers participated in the Facilitator and the Control groups.
Implications of findings for research, policy and practice are discussed. The paper argues about the role of advisor which seems to be crucial for promoting student learning outcomes. Policy makers and school leaders, therefore, should consider options for creating the conditions for in-school models of professional development. Further research is needed to test the generalisability of the findings.
ReferencesBenedict, A. E., Williams, J., Brownell, M.T., Chapman, L. Sweers, A. & Sohn, H. (2023). Using lesson study to change teacher knowledge and practice: The role of knowledge sources in teacher change. Teaching and Teacher Education, 122.
Bjuland, R. & Helgevold, N. (2018). Dialogic processes that enable student teachers’ learning about pupil learning in mentoring conversations in a Lesson Study field practice. Teaching and Teacher Education, 70, 246-254.
Creemers, B.P.M. & Kyriakides, L. (2012). Improving Quality in Education: Dynamic Approaches to School Improvement. Routledge.
Cheung, W. M., & Wong,W. Y. (2014). Does lesson study work?: A systematic review on the effects of lesson study and learning study on teachers and students. International Journal for Lesson and Learning Studies, 3(2), 137e149. https://doi.org/10.1108/IJLLS-05-2013-0024
De Vries, S., Verhoef, N. & Goei, S. L. (2016). Lesson Study: a practical guide for education.
Dudley, P. (2019). Research lesson study: A handbook. https://lessonstudy. co.uk/2015/11/download-a-free-copy-of-the-lesson-study-handbook.
Kager, K., Mynott, J. P. & Vock, M. (2023). A conceptual model for teachers’ continuous professional development through lesson study: Capturing inputs, processes, and outcome. International Journal of Educational Research Open. https://doi.org/10.1016/j.ijedro.2023.100272
Scheerens, J. (2013). The use of theory in school effectiveness research revisited. School, Effectiveness and School Improvement, 24, 1–38.
Schipper, Τ., Goei, S. L., de Vries, S., & van Veen, K. (2017). Professional growth in adaptive teaching competence as a result of Lesson Study. Teaching and Teacher Education, 68, 289-303.
Vrikki, M., Warwick, P., Vermunt, J.D., Mercer, N. & Van Halem, N. (2017). Teacher learning in the context of Lesson Study: A video-based analysis of teacher discussions. Teaching and Teacher Education, 61, 211-224.
09. Assessment, Evaluation, Testing and Measurement
Paper
Are Teaching Actions as Observed and Experienced by Students Predicting Romanian Students’ Achievement in TIMSS 2019?
Daniela Avarvare, Daniel E. Iancu, Lucian Ciolan
University of Bucharest, Romania
Presenting Author: Avarvare, Daniela;
Iancu, Daniel E.
In the contemporary era, the important advancements in technology are closely connected with the paramount importance of achievements in mathematics and sciences disciplines. The current societal landscape, characterized by technological progress and the prevalence of a data-driven environment, underscores the increasing importance of mathematical and scientific knowledge.
Mathematics is acknowledged as the foundational language supporting all STEM (Science, Technology, Engineering, and Mathematics) disciplines. Numerous stakeholders have underscored the imperative for a nation to enhance the mathematical skills and proficiency of its students (Mujtaba et al., 2014).
In a study analyzing TIMSS 2019 data for Turkey, teacher practices such as relating to daily life and prior knowledge, responding to student needs and encouraging students to participate in the discussion predicted mathematics achievement and explained the one-fifth of the between-schools variance (Sezer & Cakan, 2022).
Also, activities such as asking students to complete challenging exercises, which required them to go beyond the instruction, was an important predictor of mathematics achievement and had a positive relationship in 8th-grade (Sezer & Cakan, 2022). In Sweden, an analysis of TIMSS 2019 data showed that teaching activities such as asking to memorize formulas and listening to the teacher were positive predictors of TIMSS 8th-grade mathematics achievement, whereas relating information to daily life was a negative predictor (Eriksson et al., 2019).
The present study aims to investigate the extent to which specific teacher actions rated by students are predicting the Romanian students’ achievement in TIMSS 2019. Results could identify specific actions that could influence student achievement and propose those actions for further research and improvement.
Therefore, the research questions guiding the present study are as follows:
RQ.1 – To what extent do teachers’ actions, as observed and experienced by students, predict 8th-grade Romanian students’ mathematics achievement in TIMSS 2019 after controlling for socio-economic status?
RQ.2 – To what extent do teachers’ actions, as observed and experienced by students, predict 8th-grade Romanian students’ physics achievement in TIMSS 2019 after controlling for socio-economic status?
RQ.3 – To what extent do teachers’ actions, as observed and experienced by students, predict 8th-grade Romanian students’ chemistry achievement in TIMSS 2019 after controlling for socio-economic status?
RQ.4 – To what extent do teachers’ actions, as observed and experienced by students, predict 8th-grade Romanian students’ biology achievement in TIMSS 2019 after controlling for socio-economic status?
RQ.5 – To what extent do teachers’ actions, as observed and experienced by students, predict 8th-grade Romanian students’ earth sciences achievement in TIMSS 2019 after controlling for socio-economic status?
Methodology, Methods, Research Instruments or Sources UsedIn this transversal study we investigated to what extent Romanian students’ achievement in TIMSS 2019 could be predicted by context factors related to teaching actions as perceived by the students. Socio-economic status was also included in the regression model, because it’s a variable known to have a strong positive relationship with students’ mathematics and science achievement in previous TIMSS cycles.
The study sample was established following a random probability sampling process. All the schools in Romania that had the eighth grade in their composition were taken into consideration, each school having an equal chance of being chosen. Following this sampling process, a sample consisting of 199 public schools resulted. From these schools, 4,485 students (14-15 years) participated in the study. Most of the schools participating in the study are located in small towns or villages (40.7%), followed by those in the urban area (26.3%), the suburban area (9.8%), respectively the rural area, with difficult access (7.2%).
Data collection was carried out through two methods: administering tests to students in mathematics and sciences and the administration of context questionnaires to students. All test booklets and context questionnaires were applied on the same day. Firstly, the test booklets were applied and then the context questionnaires. During the test period, the students were supervised by a teacher who didn’t have classes with the tested students.
The study was performed using TIMSS 2019 data from the official website of TIMSS (International Association for the Evaluation of Educational Achievement, 2021). Student achievement test results for Romanian students and the 8th-grade student questionnaire were used as data sources.
From the student questionnaire we extracted the following variables to be investigated as predictors:
Working on problems on their own (only in math);
Conducting experiments (only in sciences);
Teaching actions as observed by students - each item from the composition of the Instructional Clarity scale.
Frequency of homework.
Socio-economic status, which is a composite measure of number of books in the home, number of home study supports and education level of parents was used as a control variable in the regression analyses.
The statistical procedures conducted were descriptive analysis (frequencies and percentages) and multiple simple regression for identifying the predictors of Romanian students’ achievement in TIMSS 2019.
Conclusions, Expected Outcomes or FindingsThe extent in which students work on their own during mathematics classes moderately predicts student achievement in mathematics. Romanian students who work more on their own have on average higher mathematics achievement in TIMSS 2019. At the same time, conducting experiments during science classes is not predicting achievement in any of the science disciplines (i.e., biology, physics, chemistry, earth sciences).
From the teaching actions that were rated by students, the level of teachers being supportive in learning is a significant and moderate negative predictor of the students’ achievement in mathematics and biology. Another predictor is the level of teachers linking new lessons to previous acquisitions, predicting student achievement in mathematics, physics and chemistry.
The last predictor related to teaching observed by students is the level of teachers being easily understood, which has a significant but relatively low prediction effect on achievement in mathematics, chemistry and biology.
The frequency of homework received negatively predicted students’ achievement in biology, chemistry, physics and earth sciences, and did not predict achievement in mathematics at all. For sciences, the more homework they receive for a respective discipline, the lower student achievement in that discipline.
TIMSS 2019 results offer a strong basis for decision-making based on scientific evidence to improve educational policies and practices related to teaching and learning mathematics and sciences. Through the proposed research, we hope to come to the aid of teachers with results that will help them to make their teaching methods more efficient in the classroom in order to improve the results of students in mathematics and science, thus making it possible to increase the advanced benchmark.
ReferencesEriksson, K., Helenius, O., & Ryve, A. (2019). Using TIMSS items to evaluate the effectiveness of different instructional practices. Instructional Science, 47, 1-18. https://doi.org/10.1007/s11251-018-9473-1
Fitzmaurice, O., O’meara, N., & Johnson, P. (2021). Highlighting the Relevance of Mathematics to Secondary School Students – Why and How. European Journal of STEM Education, 6(1). https://doi.org/10.20897/ejsteme/10895
Griffin, P., & Care, E. (2015). Assessment and teaching of 21st century skills: Methods and approach. Springer.
Maass, K., Geiger, V., Ariza, M.R. & Goos, M. (2019). The Role of Mathematics in interdisciplinary STEM education. ZDM Mathematics Education, 51, 869–884. https://doi-org.am.e-nformation.ro/10.1007/s11858-019-01100-5
Mujtaba, T., Sheldrake, R., Reiss, M. J., & Simon, S. (2018). Students’ science attitudes, beliefs, and context: associations with science and chemistry aspirations. International Journal of Science Education, 40(6), 644–667. https://doi.org/10.1080/09500693.2018.1433896
Sezer, E., & Cakan, M. (2022). Role of Teacher Quality and Working Conditions in TIMSS 2019 Mathematics Achievement. Journal of Theoretical Educational Science, 15(2), 395–419. https://doi.org/10.30831/akukeg.971286
TIMSS. (2019). Encyclopedia: Education Policy and Curriculum in Mathematics and Science, Romania. https://timssandpirls.bc.edu/timss2019/encyclopedia/romania.html
TIMSS. (2019). Assessment Frameworks. https://timssandpirls.bc.edu/timss2019/frameworks/
|
