Conference Agenda

This symposium presents the process and results of measurement instrument development of the IMP>ACT project. IMP>ACT is a Horizon Europe project that runs from the start of 2024 until the end of 2027, involving five universities and seven diverse societal partners who co-design an impact assessment framework for sustainability and climate change education (SCCE). The project aims to go beyond traditional output-based evaluations such as counting the number of trained educators, strategic documents produced or activities implemented and instead evaluate actual impact of SCCE initiatives in terms of learners’ action competence and educators’ professional competence to implement action-oriented SCCE in their own context. The overall objective of the project is to develop the IMP>ACT assessment framework, by considering the needs of actors in SCCE policy and practice. Drawing on transdisciplinary expertise from universities, policymakers, NGOs, and other SCCE actors, we adopted a user-centered design approach and a user-centered interpretation of validity (Kane, 2016). The framework focuses on two main concepts; action competence as the key learning outcome of SCCE (Olsson et al., 2025; Sass et al., 2020) and action-orientation as a pedagogical approach in SCCE (Sinakou et al., 2019; Weinberg et al. 2024). The framework further incorporates measurement instruments and indicators that enable diverse users to collect valid and reliable data on learners’ action competence and educators’ competences to implement action-oriented SCCE. Over the past two years of the project, we have worked on conceptualization and operationalization of the measurement instruments both for learners and educators and we have collected both qualitative and quantitative data to test the instruments for providing validity and reliability. In this symposium, we will present results related to the instrument development and validation process. In the next phases in the project, the framework, including updated measurement instruments will be tested in six diverse case studies in the Netherlands, Belgium, Germany, Serbia, the Czech Republic and Sweden. The results of these case studies will show how the framework can support the needs of users in the context of formal and non-formal SCCE context to develop their policies and/or evidence-informed practices.

The current symposium presents cognitive testing and validation results that come from the work of several partners in different European countries. The first paper presents the results of the instrument development for assessing educators’ competences in SCCE context, specifically professional action competence regarding action-oriented SCCE (PACscce) as well as findings from qualitative and quantitative data analyses results conducted across five countries. The second paper shares insights from the instrument development process of four different versions of self-perceived action competence (SPACs) and qualitative and quantitative data results derived from 26 interventions studies applied in the six European partner countries. The third paper provides example applications of action competence instruments for learners in different educational levels (primary, secondary and tertiary) and different contexts (formal and non-formal) as well as the development of action competence through diverse interventions. The fourth paper introduces how these instruments are part of the bigger IMP>ACT assessment framework, which provides a conceptual foundation for the assessment methodology by situating the instruments developed for learners and educators to inform SCCE policy and practices and by outlining its implications for the future SCCE assessment and monitoring systems. We hope to stimulate discussion within the environmental and sustainability education research community by presenting these findings and outlining the broader ambitions of the IMP>ACT project.

Educators require specific competences to develop effective learning environments for implementing sustainability and climate change education (SCCE) and enhancing students’ learning outcomes. While several professional competence frameworks addressing educators’ competences in SCCE context were proposed, there is still a lack of operationalization for assessing these competences (Rieckmann & Barth, 2022). Within the IMP>ACT project, we aim to develop a valid and reliable instrument for measuring educators’ competences in SCCE in the formal education context. We developed a conceptual model of professional action competence in SCCE (PACscce) through integrating several professional competence models including PACesd framework (Sass et al., 2020), teacher competences in ESD (Bertschy et al., 2013) and general professional competence model (Baumert & Kunter, 2013). The PACscce conceptual model consists of three main constructs and relevant sub-constructs: a) professional knowledge for SCCE (content knowledge, pedagogical content knowledge and institutional context knowledge) b) self-efficacy (SE) regarding SCCE (teaching SE and SE for implementing SCCE) c) and willingness to implement SCCE (intrinsic motivation and extrinsic motivation) By integrating the action-oriented SCCE framework developed within the project through a systematic literature review with the PACscce conceptual model, we developed the initial version of the educators’ SCCE competence survey. The survey consisted of 79 items, using a seven-point Likert scale and applicable to primary and secondary school teachers as well as educators in higher education. In spring 2025, to examine the validity and reliability of the instrument, the survey was translated into national languages and applied to 457 educators across five countries (Belgium, Germany, the Netherlands, Serbia, and Sweden). We also conducted cognitive interviews with 17 educators in five countries. By using a decision-making script, we revised items in terms of the percentage of “I don't understand” option, skewness values and factor loadings. Qualitative and quantitative data analyses results showed us which survey items worked well across countries and which items required revision or removal. The updated version of the survey contains 52 items. During the revision process, 22 items were retained, 27 items were removed and 30 items were revised. We deleted some items based on the criteria defined in the script. This process involved integrating statistical data (e.g., descriptive statistics and confirmatory factor analysis) with the insights from the cognitive interviews at item level and construct level. During the symposium, we will present this instrument development process, cross-country piloting results, and the updated version of the survey.

Baumert, J., & Kunter, M. (2013). The COACTIV model of teachers' Professional competence. In M. Kunter, J. Baumert, W. Blum, U. Klusmann, S. Krauss, & M. Neubrand (Eds.), Cognitive activation in the mathematics classroom and Professional competence of teachers: Results from the COACTIV project (pp. 25-48). Springer. Bertschy, F., Künzli, C. & Lehmann, M. (2013). Teachers’ competencies for the implementation of educational offers in the field of education for sustainable development. Sustainability, 5 (12), 5067–5080. https://doi.org/10.3390/su5125067 Rieckmann, M., & Barth, M. (2022). Educators’ competence frameworks in education for sustainable development. In P. Vare, N. Lausselet, & M. Rieckmann (Eds.), Competences in education for sustainable development: Critical perspectives (pp. 19–26). Springer International Publishing Sass, W., Claes, E., Boeve-de Pauw, J., de Maeyer, S., Schelfhout, W., van Petegem, P., Isac, M. M. (2022). Measuring professional action competence in education for sustainable development (PACesd). Environmental Education Research, 28 (2), 260–275. doi:10.1080/13504622.2021.1976731

Action competence is increasingly recognised as a central learning outcome in sustainability and climate change education (SCCE) (Olsson et al., 2020, 2022; Sass, et al., 2020). Within the IMP>ACT, this paper outlines the systematic development of SCCE measurement instruments, informed by a systematic literature review (Olsson et al., 2025) and quantitative and qualitative data from 26 intervention studies in six European countries. Based on the literature, we developed target group specific versions of the Self-Perceived Action Competence scale (SPACs): a generic SCCE version and a specific version (adaptable to themes such as biodiversity or littering), each designed for learners aged 10–14 and above 14 respectively. Using quantitative data obtained from 2,485 and cognitive interview data from 58 participants, we validated and updated all four instruments. Guided by principles for cross-national cognitive testing (Miller et al., 2011), we identified interpretive challenges, linguistic ambiguities, and contextual misunderstandings. These insights were integrated with quantitative analyses, including descriptive statistics and confirmatory factor analysis (CFA), following recommended validation procedures (Hair et al., 2018). A structured decision-making script supported systematic evaluation of each item of each version of the surveys, ensuring consistent decisions regarding retention, revision, or removal of items. Three main areas guided revisions. First, language simplification was essential, particularly for younger learners. Second, stronger connections to learners’ life worlds were established by shifting item contexts from global to school based or local community settings. Third, conceptual clarity was enhanced by replacing problematic wording and refining the constructs underpinning action competence. For learners aged 10–14, the theoretical model was slightly adjusted, with eight items removed and 24 revised to improve accessibility. For learners above 14, no items were removed but 43 of the 48 items were revised to enhance contextual realism and clarify components related to collective action. Preliminary CFA results are not yet fully satisfactory but indicate promising construct validity across age groups. These early findings provide a strong foundation for the next validation phase, during which the updated instruments will be tested in authentic learning environments through several case studies. Overall, this work contributes a theoretically and empirically grounded set of SCCE assessment instruments aligned with contemporary understandings of action competence. It represents an important step toward operationalising assessment and monitoring of sustainability related learning outcomes across diverse educational settings.

Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2018). Multivariate data analysis (8th ed.). Cengage Learning. Miller, K., Fitzgerald, R., Padilla, J. L., Willson, S., Widdop, S., Caspar, R., & Schoua-Glusberg, A. (2011). Design and analysis of cognitive interviews for comparative multinational testing. Field Methods, 23(4), 379–396. https://doi.org/10.1177/1525822X11414802 Olsson, D., Berglund, T., Boeve-de Pauw, J., Gericke, N., & Olsson, D. (2025). A systematic review of self-assessment scales on action competence in environmental and sustainability education: current limitations and development opportunities. Environmental Education Research, 1-28. https://doi.org/10.1080/13504622.2025.2605222 Olsson, D., Gericke, N., & Boeve-de Pauw, J. (2022). The effectiveness of education for sustainable development revisited: A longitudinal study on secondary students’ action competence for sustainability. Environmental Education Research, 28(3), 405–429. https://doi.org/10.1080/13504622.2022.2033170 Olsson, D., Gericke, N., Sass, W., & Boeve-de Pauw, J. (2020). Self-perceived action competence for sustainability: The theoretical grounding and empirical validation of a novel research instrument. Environmental Education Research, 26(5), 742–760. https://doi.org/10.1080/13504622.2020.1736991 Sass, W., Boeve-de Pauw, J., Olsson, D., Gericke, N., De Maeyer, S., & Van Petegem, P. (2020). Redefining action competence: The case of sustainable development. The Journal of Environmental Education, 51(4), 292–305. https://doi.org/10.1080/00958964.2020.1765132.

In this contribution we provide five example cases of application of the instruments for learners discussed in paper 2, which were based on the SPACs (Olsson et al., 2020). 23 intervention activities were implemented and evaluated throughout six European countries in 2025. All interventions were ongoing sustainability and climate change education activities taking place in the regular programmes of institutions that aim to foster action competence as a learning outcome. The aim is exploring the potential of the assessment instruments for learners for assessing action competence at different educational levels (primary, secondary, tertiary), and in different contexts (formal, nonformal). Responsive to the diversity of interventions, we applied three different analytical scenarios during evaluation. The first scenario concerned collecting pre-post survey data with a sample large enough for statistical analyses (e.g., Wilcoxon signed-rank test). The second scenario involved pre-post survey data for a relatively small sample, where descriptive statistics can be used for analysis. Finally, the third analytic scenario concerned interventions where only post-test data were collected, for instance because the intervention lasted only a few hours. The post-test for these interventions included a question where learners reflected on whether they thought the intervention helped them develop action competence. The purposefully selected interventions were spread out across formal (Flanders, Germany, Netherlands, Sweden) and informal (Serbia) learning, at primary (Flanders, Serbia), secondary (Sweden) and higher (Germany, Netherlands) level, and hosted by governmental organisations (Flanders), higher education institutions (Germany, Netherlands), and nongovernmental organisations (Serbia, Sweden). For the paired interventions, we obtained 376 pairs in Flanders, 8 pairs in Germany, 42 pairs in Serbia, and 52 pairs in Sweden. In the Netherlands, we obtained 81 post-tests. Results from our analyses show many significant changes in knowledge of action possibilities, confidence in own influence and in one’s ability to act, and willingness to act for the different interventions. These mainly concern collective rather than individual action competence dimensions. For the activity in scenario 3, the extra question indicated that students generally agreed that the activity benefited their action competence. Together, these five examples show that the assessment surveys for learners can detect developments in action competence in a broad range of educational levels (primary, secondary, higher education), contexts (formal, nonformal), and through different analytical scenarios. In the presentation, we will provide more results and reflect on what these might mean for the programmes under study.

Olsson, D., Gericke, N., Sass, W., & Boeve-de Pauw, J. (2020). Self-perceived action competence for sustainability: the theoretical grounding and empirical validation of a novel research instrument. Environmental Education Research, 26(5), 742-760. DOI:10.1080/13504622.2020.1736991.

We introduce and critically reflect on the IMP>ACT framework, a user-centered assessment framework that connects (survey) data from monitoring and evaluation to quality development and accountability in sustainability and climate change education (SCCE). The first three contributions in this symposium focus on the development, validation, and use of survey instruments to assess learners’ action competence and educators’ competencies for SCCE. This final contribution situates those instruments within our broader impact assessment framework designed to support data-informed decision making in SCCE policy and practice. The IMP>ACT framework moves beyond isolated measurement by embedding validated survey instruments within an assessment cycle that includes goal setting, data collection and interpretation, action, and feedback loops. Central to the framework is the principle of user validity (Kane, 2016): assessment is only meaningful insofar as it aligns with the goals, capacities, and context of its users. Rather than prescribing a single modus operandi, the IMP>ACT project supports differentiated uses of data, ranging from activity-level improvement to system wide monitoring. It recognizes that diverse learners and educators' competence can be relevant for users of the framework, but we do consider action competence (Olsson et al., 2020) and action-oriented pedagogy (eg., Weinberg et al., 2024) key ingredients in this process. The framework is currently being developed and refined through six diverse case studies involving diverse SCCE actors across Europe and lifelong learning contexts. These case studies function as pilot implementations and as codesign spaces: (a) the participating organizations use the framework to articulate assessment goals, collect and interpret survey data, and reflect on how evidence can inform concrete decisions, and (b) their experiences feed back into the iterative refinement of the framework itself. By comparing cases that differ in organizational role and scale, we explore how the same assessment instruments can underpin very different claims and decisions. These range from internal quality development to external accountability reporting. In doing so, we demonstrate how (survey-based) assessments can become part of a wider impact narrative that connects change at the level of learners to organizational learning and system-level insights. We argue that such an integrated framework is essential if SCCE assessments are meant to meaningfully inform policy and practice, rather than remain isolated measurement exercises (Kane, 2016). The contribution will also reflect on the implications of this approach for future SCCE monitoring systems and for strengthening the evidence base underpinning sustainability and climate change education across Europe.

Kane, M. T. (2016). Validation of interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000 Olsson, D., Gericke, N., Sass, W., & Boeve-de Pauw, J. (2020). Self-perceived action competence for sustainability: The theoretical grounding and empirical validation of a novel research instrument. Environmental Education Research, 26(5), 742–760. https://doi.org/10.1080/13504622.2020.1736991 Weinberg, A. E., Jordan, M. E., & Jongewaard, R. (2024). Real work, real consequences: An action-oriented pedagogies (AOP) framework for sustainability education in K–12 classrooms. Sustainability Science. https://doi.org/10.1007/s11625-024-01560