27. Didactics - Learning and Teaching
Paper
Are ”Optimal Learning Moments” Optimal for Learning? – Combining Experience Sampling and Pre-Post Test Design to Study Students’ Situational Engagement
Elisa Vilhunen1, Veli-Matti Vesterinen2, Miikka Turkkila1, Katariina Salmela-Aro1, Kalle Juuti1, Jari Lavonen1
1University of Helsinki, Finland; 2University of Turku, Finland
Presenting Author: Vilhunen, Elisa
The aim of this study is to investigate the hypothesized relation of students’ situational engagement, conceptualized as optimal learning moments (Schneider et al., 2016), and science learning. Engagement in educational contexts has received increasing interest (Fredricks et al., 2019; Pöysä et al., 2020; Sinatra et al., 2015), and its role in learning and socio-emotional development seems evident (Finn & Zimmer, 2012). Enhancing students' engagement in sciences is crucial, as society will persistently require individuals capable of sustaining, advancing, and innovating key functions, including healthcare and technical infrastructure, in the future. However, motivation and interest in the study of natural sciences and technology among students have recently declined globally, and especially in Europe (OECD, 2016; Osborne & Dillon, 2008; Potvin & Hasni, 2014).
Engagement can be understood and defined in several ways, and the definition may depend, for example, on whether engagement is examined at the micro or macro-level. Micro-level engagement refers to a student's engagement to a specific situation, task, or activity, while macro-level engagement may refer to a student's engagement to a class, school, or society (Sinatra et al., 2015). In this study, engagement is examined at a micro-level, referring to a situationally varying construct. Furthermore, we employ a concept of optimal learning moment as a construct of situational engagement. Optimal learning moments are ought to occur when students experience interest and challenge in their task, and concurrently feel sufficiently skilled to perform the task (Schneider et al., 2016, 2020; Shernoff et al., 2003). Interest plays an important role in the manifestation of situational engagement, as it facilitates concentration on the present task and motivates the learner to engage cognitively, even in the face of challenging tasks (Hidi & Renninger, 2006; Schraw & Lehman, 2001). However, situational interest may not necessarily persist for long if the student perceives a lack of competence and necessary skills for completing the task. Therefore, it is important for situational engagement that the student perceives themselves as capable of effectively managing the assigned task, leveraging their knowledge, and applying their skills (Csikszentmihalyi, 1990). On the other hand, for the preservation of students’ interest and learning, it is crucial that the task also presents appropriate levels of challenge (Csikszentmihalyi, 1990; Shernoff et al., 2003). Such situations, characterized by high levels of interest, skill, and challenge, have been referred to as optimal learning moments because they are hypothesized to positively impact learning (Schneider et al., 2016). In this study, we define learning as a process in which a person acquires new skills, knowledge or understanding; whereas performance or achievement are considered as more stationary constructs, reflecting merely the state of a learning process (Gross, 2015).
In the present study, we combine students’ self-reported, real-time experience sampling method (ESM) data about situational engagement to pre and posttest scores measuring students’ academic performance and science learning. We conceptualize pretest performance as prior knowledge, posttest performance as learning outcome, and the change in performance as learning or learning progress. We investigate the relations between optimal learning moments, their components, and learning using mediation analyses. This approach allows us to examine the impact of optimal learning moments on learning outcomes while accounting for prior knowledge, and the mediating role of optimal learning moments and their components in the learning process. The research questions are:
RQ1: How the components of optimal learning moments (interest, skill, and challenge) relate to students’ science learning?
RQ2: How the optimal learning moments relate to students’ science learning?
Methodology, Methods, Research Instruments or Sources UsedThis study comprises two sub-studies. The data for sub-study 1 was collected during autumn 2019 and the data collection for the sub-study 2 is ongoing. The data collection will be finished in March 2024 and the analyses will be conducted during the spring 2024.
In sub-study 1, the data was collected in Finnish upper secondary school physics classes. The participants (n = 148) were first year upper secondary school students from six classes. In each of the classes, the data collection was conducted during a study period of six or seven consecutive lessons (á 75 min). The study period focused on Newtonian mechanics.
Students’ prior knowledge and learning outcomes were evaluated using a pre-posttest design. The exact same summative test served as both a pretest and a posttest, and it covered the topics of the study period. Data on students’ situational engagement was gathered using ESM (Zirkel et al., 2015). Students filled out an ESM questionnaire using their smartphones, three times during each science lesson in the study period. Thus, each student received 18 or 21 opportunities to answer the questionnaire, resulting in altogether 1800 ESM observations. In the questionnaire, situational engagement was measured as components of optimal learning moments, using the following questions: “Were you interested in what you were doing?”, “Did you feel skilled at what you were doing?”, and “Did you feel challenged by what you were doing?”. A four-point Likert scale with the response categories from 1 = ‘not at all’ to 4 = ‘very much’ was used. A situation was considered as an optimal learning moment if a student responded the option 3 or 4 to all the three questions.
To answer RQ1, we first tested for a parallel linear mediation model, in which the effect of prior knowledge to learning outcomes is mediated by interest, skill, and challenge separately. And second, to answer RQ2, we tested a logistic mediation model, in which the effect of prior knowledge to learning outcomes is mediated by optimal learning moments. The ESM data of this study is hierarchical, meaning the situational observations are nested within students, thus a multilevel structural equation modelling (MSEM) framework was applied (Preacher et al., 2010).
In sub-study 2, the study design itself is similar to sub-study 1, only having a slightly bigger sample (about 200 participants). The data is collected in upper secondary physics classes, during study periods focusing on climate change.
Conclusions, Expected Outcomes or FindingsAgainst our expectations, the results of sub-study 1 do not support the idea of optimal learning moments being optimal for learning. According to the parallel linear mediation analysis (RQ1; examining the mediating role of skill, interest, and challenge), prior knowledge was the strongest predictor of the learning outcome (β = .549, p < .001). Prior knowledge also predicted significantly all the components of the optimal learning moments: Students with high scores in the pretest experienced higher levels of interest (β = .230, p < .001) and skill (β = .239, p < .001), and lower levels of challenge (β = -.116, p < .001) during the study period, compared to the students with lower scores from the pretest. However, after accounting for the effects of prior knowledge in the model, none of these components appeared as a significant predictor of learning outcomes.
According to logistic mediation analysis (RQ2; exploring the mediating role of optimal learning moments), prior knowledge was again the strongest predictor of the learning outcome (β = .589, p < .001), as expected. However, prior knowledge had no effect on the occurrence of optimal learning moments (β = .045, p = .282), nor had optimal learning moments an effect on learning outcomes after accounting for prior knowledge (β = -.004, p = .980), which was contradictory to the hypothesis (Schneider et al., 2016). Altogether, the results of sub-study 1 raise questions about the conceptualization and measurement of both situational engagement and learning.
We expect the results from sub-study 2 to further clarify the relation between optimal learning moments and science learning. Based on the results we have gained so far, we see the need for further studies to examine the situational factors influencing learning, and to clarify the dynamic relations between situational affective factors and academic performance.
ReferencesCsikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. Harper Perennial.
Finn, J. D., & Zimmer, K. S. (2012). Student Engagement: What Is It? Why Does It Matter? In S. L.
Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of Research on Student Engagement (pp. 97–131). Springer. https://doi.org/10.1007/978-1-4614-2018-7_5
Fredricks, J. A., Reschly, A. L., & Christenson, S. L. (2019). Handbook of Student Engagement Interventions. Elsevier.
Gross, R. D. (2015). Psychology: The science of mind and behaviour (7th ed.). Hodder Education.
Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational Psychologist, 41(2), 111–127. https://doi.org/10.1207/s15326985ep4102_4
OECD. (2016). PISA 2015 results (Volume I): Excellence and equity in education. OECD Publishing. https://doi.org/10.1787/9789264266490-en
Osborne, J., & Dillon, J. (2008). Science education in Europe: Critical reflections (Vol. 13). The Nuffield Foundation.
Potvin, P., & Hasni, A. (2014). Interest, motivation and attitude towards science and technology at K-12 levels: a systematic review of 12 years of educational research. Studies in Science Education, 50(1), 85–129. https://doi.org/10.1080/03057267.2014.881626
Pöysä, S., Poikkeus, A.-M., Muotka, J., Vasalampi, K., & Lerkkanen, M.-K. (2020). Adolescents’ engagement profiles and their association with academic performance and situational engagement. Learning and Individual Differences, 82, 101922.
Preacher, K. J., Zyphur, M. J., & Zhang, Z. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15(3), 209–233. https://doi.org/10.1037/a0020141
Schneider, B., Krajcik, J., Lavonen, J., & Salmela-Aro, K. (2020). Learning Science: The Value of Crafting Engagement in Science Environments. Yale University Press.
Schneider, B., Krajcik, J., Lavonen, J., Salmela-Aro, K., Broda, M., Spicer, J., Bruner, J., Moeller, J., Linnansaari, J., Juuti, K., & Viljaranta, J. (2016). Investigating optimal learning moments in U.S. and finnish science classes. Journal of Research in Science Teaching, 53(3), 400–421. https://doi.org/10.1002/tea.21306
Schraw, G., & Lehman, S. (2001). Situational interest: A review of the literature and directions for future research. Educational Psychology Review, 13(1), 23–52.
Shernoff, D. J., Csikszentmihalyi, M., Schneider, B., & Shernoff, E. S. (2003). Student engagement in high school classrooms from the perspective of flow theory. School Psychology Quarterly, 18(2), 158–176. https://doi.org/10.1521/scpq.18.2.158.21860
Sinatra, G. M., Heddy, B. C., & Lombardi, D. (2015). The Challenges of Defining and Measuring Student Engagement in Science. Educational Psychologist, 50(1), 1–13. https://doi.org/10.1080/00461520.2014.1002924
Zirkel, S., Garcia, J. A., & Murphy, M. C. (2015). Experience-sampling research methods and their potential for education research. Educational Researcher, 44(1), 7–16. https://doi.org/10.3102/0013189X14566879
27. Didactics - Learning and Teaching
Paper
Enhancing Design and Research Skills in Students: An Academic Inquiry of Integrating Project-Based Learning (PBL) Approach in History Lessons
Dulat Turarbekov1, Dina Ashimova2, Salima Shayanbayeva3, Aiymgul Ungarbayeva4, Kuat Tleuberdinov5, Arailym Shilikbayeva6
1Nazarbayev Intellectual School in Astana, Kazakhstan; 2Nazarbayev Intellectual School of Physics and Mathematics in Uralsk, Kazakhstan; 3Nazarbayev Intellectual School of Chemistry and Biology in Almaty; 4Center for Educational Programmes, AEO “Nazarbayev Intellectual Schools”; 5Center for Pedagogical Measurements, AEO “Nazarbayev Intellectual Schools”; 6Nazarbayev Intellectual School of Physics and Mathematics in Almaty
Presenting Author: Turarbekov, Dulat;
Ashimova, Dina
In the evolving landscape of education, there is a notable shift towards more engaging pedagogical methods to meet diverse student needs. Project-Based Learning (PBL) stands out as an exemplary model, embodying an experiential, collaborative, and interdisciplinary paradigm in education (Thomas, J.W., 2000). PBL not only instills problem-solving skills but also nurtures critical thinking, creativity, and research abilities through the formulation of research questions, case study methodologies, and small-scale studies within lessons (Tretten, R. and Zachariou, P., 1995).
Project-based approaches play a pivotal role in cultivating profound understanding and meaningful learning experiences. They hold immense potential for developing higher-order thinking skills, fostering collaboration, and facilitating the application of knowledge in real-world contexts (Barron B., Schwartz D., Wai N., 1998). These methods exert a substantial impact on student motivation, engagement, and the cultivation of metacognitive skills.
Nazarbayev Intellectual Schools (NIS) explores the concept of PBL in the history of Kazakhstan and its role in developing research skills for high school students (grades 11-12). Despite the implementation of innovative teaching methods at NIS, challenges persist in the effective integration of PBL in the classroom, coupled with difficulties faced by students in completing design and research tasks during lessons. The study is valuable as it identifies challenges and offers insights into how PBL-based research can benefit both teachers and students, proposing strategies for the systematic integration of PBL into history lessons in Kazakhstan.
The research aims to enhance the methods of project-research teaching for NIS history teachers using the PBL method and to develop students' research skills. Specifically, it investigates how PBL affects students' development of various research skills, including media and information literacy, critical thinking, and design thinking, through the creation of research projects of different durations (short, medium, long). The research questions considered are:
-
How does PBL teaching affect students' knowledge and skills?
-
What are the opportunities and challenges of PBL teaching?
-
How is PBL implemented according to the tasks in the textbooks "History of Kazakhstan (Kazakhstan in the Modern World)"?
Project-Based Learning, as a pedagogical methodology, steers students towards addressing complex, regionally specific problems or projects, contrasting with traditional memorization-based methods (Johnson, M., Smith, L., 2017). It champions active exploration, analysis, and construction, promoting individual-centered approaches that stimulate critical thinking, problem-solving, and creativity.
The NIS program's curriculum aligns with the GCE AS Level 2021-23 programs, particularly in the subject of "History of Kazakhstan (Kazakhstan in the modern world)," with tasks for teaching based on PBL integrated into each chapter. The organizational and implementation structure of PBL in history lessons involves short-term, mid-term, and long-term research projects, each contributing distinctively to students' research skills and knowledge acquisition.
Short-term research activities occur during lessons, addressing specific research questions, while mid-term projects, spanning 4-6 classes, allow students to delve into small-scale research endeavors. Long-term projects extend over 2 years, enabling students to engage in extensive project and research work outside the classroom under teacher supervision. This progression underscores the systematic development of research skills through varying project lengths.
In conclusion, the strategic deployment of Project-Based Learning in history education, as evidenced in the NIS context, emerges as an invaluable method for cultivating multifaceted research skills and knowledge acquisition. This study contributes to the ongoing discourse on innovative pedagogical approaches, shedding light on the challenges and opportunities associated with PBL implementation in the history classroom, and offering practical insights for educators and curriculum developers. The focus on research questions and the systematic examination of the impact of PBL on various facets of student learning provides a robust foundation for future educational research endeavors.
Methodology, Methods, Research Instruments or Sources UsedResearch Methodology: The research design employed in this study is a mixed-methods approach, combining qualitative and quantitative data collection and analysis. The primary objective was to assess the impact of project-based learning (PBL) on the development of students' project and research skills within history classes.
Participant Selection: To enhance the reliability of participant selection, students and teachers from the Nazarbayev Intellectual Schools (NIS) in Astana, Almaty, Oral, Semey, and Shymkent were included in the study. The selection also considered the experience levels of teachers in using PBL for history education. This diverse participant pool aimed to provide a comprehensive perspective on the influence of PBL on media literacy, information literacy, and technology literacy skills in history education.
Data Collection: Quantitative data were obtained through structured student surveys, ensuring reliability through mean scores and standard deviations. Qualitative data were gathered through interviews with students and teachers, employing open-ended questions that underwent coding and categorization for thematic analysis. This methodological combination sought to offer a thorough understanding of the research problem.
Data Analysis: Quantitative analysis involved statistical methods, mean scores, and standard deviations, with ANOVA analysis to assess group differences. Survey results indicated that students perceive PBL as effective for understanding societal issues, enhancing media literacy skills, and integrating technology into history education. Qualitative analysis involved examining reports from the Centers for Pedagogical Measurements and Educational Programs and thematic analysis of interview responses. This qualitative approach provided deeper insights into the impact of PBL on design and research skill development.
Comparison: Cross-comparison of qualitative responses from students and teachers identified areas of agreement and disagreement, enriching the understanding of the study. Triangulation of data collected through various instruments further bolstered the study's reliability.
Ethical Considerations: Maintaining confidentiality, informed consent, and adherence to ethical principles were crucial aspects of the research process, ensuring the study's reliability and ethical integrity. These considerations protected participant integrity and contributed to the overall validity of the study.
Conclusions, Expected Outcomes or FindingsThe research, encompassing both quantitative and qualitative data, scrutinized Project-Based Learning's (PBL) positive impact on students' historical understanding and the development of investigative, critical, and creative thinking skills. Integrated into academic programs, PBL fosters an active learning environment, transcending knowledge acquisition into everyday life. Small-scale studies within PBL enable students to analyze societal issues, enhancing problem-solving abilities. To improve coursework quality, we recommend systematically incorporating PBL in Kazakhstan's 11th and 12th-grade history lessons.
PBL Development Aligned with Textbook Objectives: The study highlights PBL's prevalence aligned with "History of Kazakhstan (Kazakhstan in the modern world)" textbooks' objectives. Emphasizing interdisciplinary research projects by integrating PBL into subjects like history, geography, economics, global perspectives, and project-based coursework promises to enhance students' research capabilities.
Skill Development: Qualitative insights underscore PBL's pivotal role in nurturing critical thinking, problem-solving, inquiry, and collaborative skills, applicable across academic and professional domains. Small-scale research projects and in-class coursework significantly contribute to enhancing students' information and media literacy, critical, and creative thinking skills. Future emphasis on Approaches to Learning (ATL) skills in more extended projects and coursework is warranted.
Information Literacy Skills: Emphasis should be placed on crafting references, citations, footnotes/endnotes, and bibliographies according to recognized conventions. Additionally, adept data processing and results reporting are crucial.
Media Literacy Skills: Recommended is the cultivation of effective communication of information and ideas across diverse audiences through various media and formats. Students should refine abilities to locate, organize, analyze, evaluate, synthesize, and ethically use information from various sources, including digital platforms.
Critical Thinking Skills: Imperative is the encouragement of drawing reasonable conclusions, considering ideas from multiple perspectives, and fostering flexible thinking through the development of opposing, contradictory, and complementary arguments.
ReferencesBarron, B., Swartz, D., Vye, N., Moore, A., Petrosino, A., Zech, L., Bransford, J., & Cognition and Technology Group at Vanderbilt. (1998). Doing with understanding: Lessons from research on problems and project-based learning. The Journal of the Learning Sciences, 7(3&4), 271-311.
Brown, P. (2012). Enhancing Historical Understanding through PBL. History Education Quarterly, 45(4), 501-522.
Buzina, I. (2005). Using the project method in history lessons. History of Kazakhstan: teaching at school, 1, 23-25.
Clark, J., & Turner, S. (2018). Media Literacy and Historical Inquiry: The Role of PBL. The History Teacher, 51(2), 255-274.
Gupta, R., & Patel, A. (2019). Integrating Technology Literacy in Historical PBL. Journal of History and Technology, 36(3), 305-323.
Hmelo-Silver, C. E. (2004). Problem-Based Learning: What and How Do Students Learn? Educational Psychology Review, 16(3), 235-266.
Johnson, M., & Smith, L. (2017). Fostering Critical Thinking in History Through PBL. The History Educator, 10(1), 45-64.
Kolmos, A., et al. (2016). Fostering Critical Thinking in History Through PBL. Journal of Engineering Education, 105(3), 442-472.
Peters, E., & Turner, R. (2016). Design Thinking and PBL in History Education. History Education International, 45(3), 333-351.
Strobel, J., & van Barneveld, A. (2009). When Is PBL More Effective? A Meta-synthesis of Meta-analyses Comparing PBL to Conventional Classrooms. Interdisciplinary Journal of Problem-Based Learning, 3(1), 44-58.
Thomas, J. W. (2000). A Review of Research on Project-Based Learning. San Rafael, CA: Autodesk Foundation.
Tretten, R. & Zachariou, P. (1995). Learning about project-based learning: Assessment of project-based learning in Tinkertech schools. San Rafael, CA: The Autodesk Foundation.
Documents
"Nazarbayev Intellectual Schools" Educational Program of the AEO - NIS-Programme "History of Kazakhstan" subject. - Nur-Sultan, 2019
Methodological guide for compiling exam materials for External Summative Assessment on the subject "Kazakhstan in the Modern World" (grade 12). "Nazarbayev Intellectual Schools" was approved and submitted for publication by the decision of the Methodological Council of the AEO ‘NIS’ on February 21, 2019, protocol No. 46
Analytical reports on the results of External Summative Assessment of 12th grade students of Nazarbayev Intellectual Schools in 2018-2019, 2020-2021, 2021-2022, 2022-2023 academic years. - Astana, CPM AEO NIS, 2023
Instruction on organization and conduct of External Summative Assessment of Academic Achievements of Students of Nazarbayev Intellectual Schools, approved by the decision of AEO NIS dated December 14, 2015 (protocol No. 62) with amendments dated November 10, 2016. No. 53; 14.12.2017 No. 65; 22.08.2018 No. 48)
27. Didactics - Learning and Teaching
Paper
Teachers Dealing with Formative Assessment Reforms: an Interplay Between Persons and Contexts
Maurizio Gentile1, Tania Cerni2, Enrico Perinelli3, Francesco Pisanu4
1LUMSA University of Rome; 2University of Ferrara; 3University of Trento; 4Autonomous Province of Trento
Presenting Author: Gentile, Maurizio
Introduction
Assessment dramatically impacts students' learning outcomes and the quality of their involvement in school activities (Black & Wiliam, 1998; Schellekens et al., 2021). In the last 30 years, the concept of assessment has changed, and the discussion has focused on the distinction between "assessment for learning," "assessment of learning," and "assessment as learning" (Dann, 2014). In this proposal, we argue the three components of the assessment process should act according to an interdependent pattern. Each assessment approach connects to the other to maximize students' educational outcomes in both cognitive and non-cognitive domains (Van Der Vleuten et al., 2017). Combining different approaches or using one process instead of another may depend on different educational purposes.
With the O.M. 172.04-12-2020 reform - "Periodic and final evaluation of students' learning in primary school" - the Italian primary school has adopted a new evaluation system. The new rules provide for the replacement of votes - expressed on an evaluation scale ranging from "0 to 10" - with descriptive judgments that indicate four levels of learning accomplishment: "in the initial phase," "basic," "intermediate," and "advanced." The O.M. 172.04-12-2020 formally assumed the perspective of assessment for learning. The reform places the pupils' learning process and outcomes at the center of evaluation and the design of teaching strategies to enhance them (Cerini, 2021). The assessment must improve learning and promote the construction of personal resources and skills (Clark, 2012), which go beyond the specific domains of curricular knowledge (Black et al., 2016). In more strictly cognitive and psychological terms, formative assessment, in addition to impacting learning outcomes, can positively color the school experience of pupils (Black et al., 2016) and could contribute, for example, to activate psychological resources (non-cognitive skills) such as resilience, hope, optimism and a self-efficacy; conscientiousness and open-mindedness; a motivation-oriented learning goal; internal and controllable causal attributions, autonomous motivation and positive academic self-concept (Gentile & Pisanu, 2023).
However, once reform is approved, it is not fully obvious to expect a consistent change in teaching practices and teachers' conceptions. It is suitable for a realistic vision of the reform processes to consider a gap between the intentions of the legislator and the actual application problems that teachers face in implementing reforms (Gouëdard et al., 2020; Wiliam, 2018). Personal (Pan & Wiens, 2023) and contextual factors (Gouëdard et al., 2020) can hinder or facilitate these changes. According to Gentile et al. (2023), the following dimensions can play a crucial role: teachers' self-efficacy, beliefs about the general aims of teaching, openness to innovation, and the perception of the organizational climate of the school workplace. Regarding the evaluation reforms, the following factors could affect the teachers' receptivity: the assessment approaches (assessment for/of/as learning) and teachers' assessment literacy (Coombs et al., 2020).
The study analyzes the following research questions:
- RQ 1: What kind of relationship do teachers perceive between cognitive and non-cognitive learning in the context of the four levels of learning provided by the reform?
- RQ 2: Is there a distinct degree of separation between the assessment approaches, or to some extent, can reciprocal relationships depend on specific evaluative purposes?
- RQ 3: Are there differentiated profiles in primary teachers' approaches to evaluation reform?
Methodology, Methods, Research Instruments or Sources UsedMethods
Procedure
A questionnaire was administered online through Google Forms at the end of the 2020-21 academic year (June 21, 2021 – July 11, 2021). The questionnaire administration was part of a larger project funded by the Italian Ministry of Education. The Ministry sent the link to the questionnaire to all potential respondents. Teachers participated according to their willingness. Before completing the questionnaire, participants were advised to read the informed consent carefully and give their authorization for data collection. Data were collected per the privacy law in force (D. Lgs 196/2003 and UE GDPR 679/2016).
Participants
Participants were 700 primary school teachers (female = 681, 97.29%). They were employed in 155 primary schools located in Tuscany (Italy). 28.29% of the sample was 56 years old or older, 43.29% was between 46 and 55, 24.57% was between 36 and 45, and the remaining 3.86% was equal to or lower than 35. Most participants were tenured teachers (n = 661), while 38 were substitute teachers. 51.29% possessed a high school diploma, 41% had a postgraduate degree, and 7.71% had higher degrees (i.e., Ph.D. or supplementary master's degrees). Participants declared themselves teachers for a mean of 21.34 years (SD = 9.97) and employed in their current school for 12.63 years (SD = 9.68).
Measures
We designed a self-report questionnaire comprising 101 items in which each respondent was encouraged to express their opinion on a series of statements. The statements were selected or adapted from validated international and national literature scales. We analyzed six dimensions: a) assessment practices, b) pupils' non-cognitive skills, c) levels of learning (e.g., initial, base, intermediate, advanced), d) teacher self-efficacy, e) assessment literacy, f) organizational climate, g) openness to innovation, and f) perception of teaching.
Analysis strategy
All analyses were conducted with R (Version 4.3.0; R Core Team, 2023). The data analytic strategy was conducted according to the following steps. First, we calculated one composite score for each construct by averaging their corresponding items. Descriptive statistics and reliability (Cronbach's alpha) were calculated for all the variables. Second, we analyzed the reciprocal relationship system among the factors associated with each dimension (RQ1). Third, we conducted a Principal Component Analysis (PCA) on assessment practices (RQ2). Finally, we examined the teachers' profiles connected to factors under study by the k-means calculation procedure (RQ3).
Conclusions, Expected Outcomes or FindingsRQ 1: What kind of relationship do teachers perceive between cognitive and non-cognitive learning in the context of the four levels of learning provided by the reform?
Teachers associate a high perception of non-cognitive competence (social-emotional resources) with an Advanced Level of Learning (r = 0,33). The result is consistent with the literature: social-emotional competence predicts academic achievement. In contrast, the less intense the perception of pupils' social-emotional resources, the lower the attributed levels of learning (r = -0.13. This result raises an issue of equity.
RQ 2: Is there a distinct degree of separation between the assessment approaches, or to some extent, can reciprocal relationships depend on specific evaluative purposes?
Throughout the PCA procedure, we found and labeled three assessment general approaches: "student-centered assessment" (alpha = 0.88), "brief and continuous monitoring" (alpha = 0.79), and "summative assessment of learning" (alpha = 0.70). We observed an absence of clear boundaries between summative assessment, monitoring of/for learning, and student-centered assessment. For example, teachers use summative tools to gather information and communicate feedback (r = 0.53) or monitor pupils' learning for summative purposes (r = 0.48). The tool is important, but the purpose is more so.
RQ 3: Are there differentiated profiles in primary teachers’ approaches to evaluation reform?
Two profiles of teachers emerged from the research. In the first profile, we identified teachers who are more open to innovation and perceive their pupils to be more capable on a cognitive and socio-emotional level. More positive beliefs prevail in this cluster concerning the view of assessment, self-efficacy in teaching, and perceptions of teaching (N = 334). In the second profile, however, we found teachers less open to innovation, with pupils perceived to be less cognitively and socio-emotionally competent. Low perceptions of self-efficacy prevail in this cluster, and less positive meanings are attributed to assessment and teaching (N = 336).
ReferencesBlack, P., & Wiliam, D. (1998). Assessment and Classroom Learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7-74. https://doi.org/10.1080/0969595980050102
Black, P., Harrison, C., Lee, C., Marshall, B., & Wiliam, D. (2016). Working inside the Black Box: Assessment for Learning in the Classroom. Phi Delta Kappan, 86(1), 8-21. https://doi.org/10.1177/003172170408600105
Cerini, G. (2021). Atlante delle riforme (im)possibili. [Atlas of (im)possible educational reforms]. Tecnodid.
Clark, I. (2010). Formative Assessment: 'There is nothing so practical as a good theory'. Australian Journal of Education, 54(3), 341–352.
Clark, I. (2012). Formative Assessment: Assessment Is for Self-regulated Learning. Educational Psychology Review, 24(2), 205-249. https://doi.org/10.1007/s10648-011-9191-6
Coombs, A., DeLuca, C., & MacGregor, S. (2020). A person-centered analysis of teacher candidates’ approaches to assessment. Teaching and Teacher Education, 87, 102952. https://doi.org/https://doi.org/10.1016/j.tate.2019.102952
Dann, R. (2014). Assessment as learning: Blurring the boundaries of Assessment and learning for theory, policy and practice. Assessment in Education: Principles, Policy and Practice, 21(2), 149–166. https://doi.org/10.1080/0969594X.2014.898128.
Gentile, M., & Pisanu, F. (2023). Insegnare Educando. Promuovere a scuola le risorse psicosociali di chi apprendere: modelli, strategie, attività. UTET Università.
Gentile, M., Cerni, T., Perinelli, E. & Pisanu, F. (2023). Analisi delle pratiche valutative e attuazione della riforma della valutazione nella scuola primaria: un’interazione tra persone e contesti. QTimes, 15(4), 258-274. DOI: 10.14668/QTimes_15420.
Gouëdard, P., Pont, B., Hyttinen, S., & Huang, P. (2020). Curriculum reform: A literature review to support effective implementation (OECD Education Working Papers, Issue N. 329). OECD Publishing. https://www.oecd-ilibrary.org/content/paper/efe8a48c-en
Pan, H.-L. W., & Wiens, P. D. (2023). An Investigation of Receptivity to Curriculum Reform: Individual and Contextual Factors. The Asia-Pacific Education Researcher. https://doi.org/10.1007/s40299-023-00712-6
R Core Team (2023). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/.
Schellekens, L. H., Bok, H. G. J., de Jong, L. H., van der Schaaf, M. F., Kremer, W. D. J., & van der Vleuten, C. P. M. (2021). A scoping review on the notions of Assessment as Learning (AaL), Assessment for Learning (AfL), and Assessment of Learning (AoL). Studies in Educational Evaluation, 71, Article 101094. https://doi.org/10.1016/j.stueduc.2021.101094.
Van Der Vleuten, C., Sluijsmans, D., & Joosten-Ten Brinke, D. (2017). Competence Assessment as Learner Support in Education. In (pp. 607-630). Springer International Publishing. https://doi.org/10.1007/978-3-319-41713-4_28
Wiliam, D. (2018). Assessment for learning: meeting the challenge of implementation. Assessment in Education: Principles, Policy & Practice, 25(6), 678-682. https://doi.org/10.1080/0969594X.2017.1401526
|
