Conference Agenda
| Session | ||
09 SES 06 C: Questionnaire Adaptation, Scale Validation and Technology Acceptance
Paper Session | ||
| Presentations | ||
09. Assessment, Evaluation, Testing and Measurement
Paper Validation of a Dutch STEM Questionnaire on Motivation, Interest, Identity & Need Support Vrije Universiteit Brussel, Belgium Presenting Author:Introduction: Contemporary society is increasingly shaped by digital technologies (OECD, 2024; World Economic Forum, 2025), driving a sustained demand for qualified STEM professionals, as evidenced by labor market indicators across numerous countries (Jones et al., 2017; National Science Board, 2024). However, despite this demand, many countries are reporting declining student interest and retention in STEM fields (National Science Board, 2024; Truchly et al., 2019; World Economic Forum, 2025). Bridging this gap requires not only expanding STEM training capacity but also fostering student engagement and motivation within these pathways. In Belgium, the VUB STEAM Academy, led by Vrije Universiteit Brussel, is developing two educational centers, one in Gooik and the other in Halle, both dedicated to STEM. The Academy offers camps, workshops and clubs for young people and their communities, with an emphasis on 21st century skills, sustainability, accessible hands-on making and technology experiences. As part of its mission, the Academy aims to establish a baseline profile of students’ STEM interest, identity, motivation and satisfaction of the basic psychological needs (autonomy, relatedness, competence) in the Pajottenland and Zennevallei regions, given the documented role of these factors in shaping STEM trajectories and career intentions (e.g., Chiu, 2021; De Loof et al., 2022; Keith, 2018; Kim et al., 2018; Nadelson & Seifert, 2017; Pinxten et al., 2019; Vansteenkiste et al., 2009). This baseline will identify youth subgroups requiring targeted interventions and enable rigorous evaluation of the impact of forthcoming Academy projects on participants’ STEM interest, identity, motivation and career intentions. To conduct this mapping, as well as future impact evaluations, there is a need for a validated, integrated Dutch-language instrument that assesses STEM interest, motivation, identity and need-support (autonomy, relatedness, competence). Existing Dutch measures capture only parts of this construct space. For example, the Dutch version of the Science Motivation Questionnaire II focuses on science motivation rather than the broader STEM construct and is typically used as a subject-specific instrument (Glynn et al., 2011). For STEM interest, a Dutch adaptation of the STEM-LIT instrument exists, but it targets younger learners (Grimmon et al., 2020). Moreover, there are currently no Dutch instruments for assessing students’ STEM identities or their experiences of basic psychological need-support in STEM activities. Although international instruments are available (e.g., Chiu, 2021; Cohen et al., 2021; Ryan & Deci, 2000; Stets et al., 2017; Tyler-Wood et al., 2010), much of their validation has been conducted in United States, raising cross-cultural and linguistic concerns for Flanders (Belgium) (Maric et al., 2023). In this context, the lack of a validated, integrated tool hampers intervention evaluation, the identification of subgroups needing tailored support and the monitoring of trends in STEM engagement. Objectives: The present validation study aims to develop and psychometrically validate such an instrument, enabling multilevel analyses and testing measurement invariance across gender, socioeconomic status and migration background. In doing so, it addresses a gap identified by Moote et al. (2020), namely, the limited insight into why some groups remain excluded from STEM while others participate successfully. Methodology, Methods, Research Instruments or Sources Used This validation study focused on students in the final years of Flemish secondary education, a critical stage for the development of STEM studies and career preferences (Rosenzweig & Chen, 2023). The sample comprised 317 students from two schools. The mean age was 16.9 years (range 15-19); 46.4% identified as male and 53.6% as female; 59.9% were in year 5 and 40.1% were in year 6. The study protocol was reviewed and approved by the Ethics Committee of Human Science, Vrije Universiteit Brussel (approval code: ECHW_586). The survey consisted of four scales assessing STEM interest, STEM identity, STEM motivation and perceived teacher support for basic psychological needs (autonomy, relatedness, competence). STEM interest (Tyler-Wood et al., 2010), STEM identity (Cohen et al., 2021) and basic need satisfaction (Chen et al., 2015) were translated from English to Dutch via a forward backward procedure by two independent language specialists; discrepancies were reconciled to ensure semantic and conceptual equivalence. STEM motivation was adapted from the Academic Self-Regulation Scale (Vansteenkiste et al., 2009) to a STEM specific context, retaining four subdimensions: intrinsic motivation, identified regulation, introjected regulation and external regulation. All the items were rated on a five-point Likert scale (1 = strongly disagree to 5 = strongly agree). The construct validity was examined via confirmatory factor analysis (CFA) in R (lavaan), given the a priori factor structures established in prior research. Models were estimated with full information maximum likelihood (FIML) and evaluated via multiple fit indices: χ², RMSEA, CFI, TLI and SRMR. Considering χ²’s sensitivity at larger samples (n > 250), model adequacy was primarily judged by commonly used thresholds (RMSEA ≤ .05; CFI/TLI ≥ .90; SRMR ≤ .08). Following selection of the best-fitting model, internal consistency was assessed with Cronbach’s α, with α ≥ .70 considered acceptable. This approach enables a rigorous test of the proposed multidimensional structure and the reliability of the Dutch-language scales in the context of Flemish upper-secondary STEM education. Conclusions, Expected Outcomes or Findings Results: This study examined the psychometric properties of a Dutch survey designed to assess STEM related constructs, including interest, identity, motivational regulation and perceived teacher need support. The hierarchical model provided the best representation of the data with good fit (χ² = 1170.01, df = 571, χ²/df = 2.05, CFI = .94, TLI = .93, RMSEA = .058, SRMR = .060) and satisfactory reliability (α = .84-.95; ω = .83-.95). This model comprises nine first-order latent variables (STEM interest, STEM identity, intrinsic motivation, identified regulation, introjected regulation, external regulation, autonomy support, relatedness support and competence support) organized under three second-order factors (STEM, motivation and basic psychological need support by mathematics teacher). In line with self-determination theory (Deci & Ryan, 2000; Ryan & Deci, 2020), the motivation factor was further differentiated into autonomous motivation (intrinsic motivation, identified regulation) and controlled motivation (introjected and external regulation). The convergent and discriminant validity were then evaluated. All first-order constructs demonstrated adequate convergent validity, with average variance extracted values ranging from .59 to .82 and composite reliabilities between .84 and .95, exceeding commonly recommended thresholds (Fornell & Larcker, 1981; Nunnally, 1978). The evidence for discriminant validity was more mixed: while most constructs met the Fornell-Larcker criterion, two notable exceptions emerged. Intrinsic motivation and identified regulation were highly correlated (r = .87) and the three basic psychological need dimensions (autonomy, relatedness, competence) showed strong intercorrelations (r = .81-.90). These overlaps are theoretically plausible: intrinsic and identified regulation are adjacent forms of internalization within self-determination theory, and the basic psychological needs frequently co-occur in practice (Deci & Ryan, 2000; Ryan & Deci, 2020). References Chen, B., Vansteenkiste, M., Beyers, W., Boone, L., Deci, E. L., Van der Kaap-Deeder, J., Duriez, B., Lens, W., Matos, L., Mouratidis, A., Ryan, R. M., Sheldon, K. M., Soenens, B., Van Petegem, S., & Verstuyf, J. (2015). Basic psychological need satisfaction, need frustration, and need strength across four cultures. Motivation and Emotion, 39(2), 216-236. https://doi.org/10.1007/s11031-014-9450-1 Chiu, T. K. F. (2021). Digital support for student engagement in blended learning based on self-determination theory. Computers in Human Behavior, 124, 106909. https://doi.org/10.1016/j.chb.2021.106909 Cohen, S. M., Hazari, Z., Mahadeo, J., Sonnert, G., & Sadler, P. M. (2021). Examining the effect of early STEM experiences as a form of STEM capital and identity capital on STEM identity: A gender study. Science Education, 105(6), 1126-1150. https://doi.org/10.1002/sce.21670 De Loof, H., Boeve-de Pauw, J., & Van Petegem, P. (2022). Engaging Students with Integrated STEM Education: A Happy Marriage or a Failed Engagement? International Journal of Science and Mathematics Education, 20(7), 1291-1313. https://doi.org/10.1007/s10763-021-10159-0 Deci, E. L., & Ryan, R. M. (2000). The ‘What’ and ‘Why’ of Goal Pursuits: Human Needs and the Self-Determination of Behavior. Psychological Inquiry, 11(4), 227-268. https://doi.org/10.1207/S15327965PLI1104_01 Fornell, C., & Larcker, D. F. (1981). Evaluating Structural Equation Models with Unobservable Variables and Measurement Error. Journal of Marketing Research, 18(1), 39. https://doi.org/10.2307/3151312 Maric, D., Fore, G. A., Nyarko, S. C., & Varma-Nelson, P. (2023). Measurement in STEM education research: A systematic literature review of trends in the psychometric evidence of scales. International Journal of STEM Education, 10(1), 39. https://doi.org/10.1186/s40594-023-00430-x National Science Board. (2024). The Skilled Technical Workforce: Crafting America’s Science & Engineering Enterprise (p. 2). National Science Foundation. https://www.nsf.gov/nsb/publications/2024/STW-1-pager-2024.pdf Nunnally, J. C. (1978). Psychometric Theory (2nd edition). McGraw-Hill. OECD. (2024). OECD Digital Economy Outlook 2024 (Volume 2): Strengthening Connectivity, Innovation and Trust. OECD Publishing. https://doi.org/10.1787/3adf705b-en Ryan, R., & Deci, E. (2020). Intrinsic and extrinsic motivation from a self-determination theory perspective: Definitions, theory, practices, and future directions. Contemporary Educational Psychology, 61, 101860. https://doi.org/10.1016/j.cedpsych.2020.101860 Tyler-Wood, T., Knezek, G., & Christensen, R. (2010). Instruments for Assessing Interest in STEM Content and Careers. Journal of Technology and Teacher Education, 18. Vansteenkiste, M., Sierens, E., Soenens, B., Luyckx, K., & Lens, W. (2009). Motivational Profiles From a Self-Determination Perspective: The Quality of Motivation Matters. Journal of Educational Psychology, 101, 671-688. https://doi.org/10.1037/a0015083 World Economic Forum. (2025). The Future of Jobs Report 2025 (p. 290). World Economic Forum. https://www.weforum.org/publications/the-future-of-jobs-report-2025/digest/ 09. Assessment, Evaluation, Testing and Measurement
Paper ***WITHDRAWN*** Psychometric Evaluation of the Turkish Adaptation of the Self-Regulation Strategy Inventory 1: Marmara University (Turkey), Türkiye; 2: Marmara University (Turkey), Türkiye Presenting Author:The purpose of this study was to adapt the Self-Regulation Strategy Inventory into Turkish. Following the examination and confirmation of equivalence between the English and Turkish versions, the scale was administered to 415 students enrolled in an urban high school. To evaluate the validity and reliability of the inventory, exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) were conducted, and internal consistency was examined using Cronbach’s alpha coefficient. The results of the factor analyses indicated that the inventory retained its original three-factor structure. The Cronbach’s alpha coefficient for the adapted inventory was .74, demonstrating acceptable internal consistency. Overall, the findings suggest that the Turkish version of the Self-Regulation Strategy Inventory is a valid and reliable instrument for assessing students’ self-regulated learning strategies. Self-regulated learning refers to the cognitive, metacognitive, behavioral, motivational, and affective actions that individuals employ during learning processes (Alvi & Gillies, 2020). To conceptualize and foster self-regulated learning, several theoretical frameworks have been proposed (Boekaerts, 1992; Zimmerman, 2000). A shared feature of these models is their emphasis on goal setting and the cyclical nature of the self-regulation process (Panadero, 2017). The present study is grounded in Zimmerman’s (2000) theoretical framework and aims to adapt the Self-Regulation Strategy Inventory (SRSI) into Turkish. Accordingly, the research question guiding the study was: Is the SRSI a valid and reliable instrument for measuring high school students’ self-regulated learning strategies? Methodology, Methods, Research Instruments or Sources Used This study employed an adaptation research design. The sample consisted of 415 students enrolled in an urban high school. The Self-Regulation Strategy Inventory (SRSI), developed by Cleary (2006), was used to assess students’ self-regulated learning strategies. The inventory comprises 28 items across three dimensions: (a) Seeking and Learning Information (SLI), (b) Managing Environment/Behavior (MEB), and (c) Maladaptive Regulatory Behaviors (MRB). Responses are recorded on a 5-point Likert scale ranging from 1 (never) to 5 (always), yielding total scores between 28 and 140. Permission to adapt the inventory into Turkish was obtained via email from Timothy J. Cleary. The translation process involved translation, back-translation, and comparison procedures. The Pearson product–moment correlation coefficient between the English and Turkish versions was .92, indicating strong equivalence between the two forms (Puth, Neuhäuser & Ruxton, 2014). Exploratory Factor Analysis (EFA) was conducted using SPSS, and Confirmatory Factor Analysis (CFA) was performed with LISREL. Reliability was assessed using Cronbach’s alpha coefficient. Regarding content validity, the items were derived from the original instrument, and adaptations were limited to contextual adjustments related to learning and instruction. The Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy was .92, indicating that the data were suitable for factor analysis (Shrestha, 2021). Bartlett’s test of sphericity was significant (p < .05), suggesting that the correlation matrix was appropriate for factor extraction (Hadi, Abdullah & Sentosa, 2016). Therefore, maximum likelihood estimation was employed. Items with factor loadings greater than .32 and factors with eigenvalues exceeding one were retained (Tabachnick & Fidell, 2013). Promax rotation was applied. Although the unrotated EFA yielded a structure different from the original form, the rotated solution revealed four factors with eigenvalues greater than one. However, the fourth factor contributed minimally to the total variance; thus, a three-factor solution consistent with the original inventory was retained. Conclusions, Expected Outcomes or Findings The three-factor structure accounted for 47.13% of the total variance, which is considered adequate for multifactor models in social sciences and education (Kim & Ban, 2024). All items loaded onto their respective original factors. Specifically, eight items loaded onto Factor 1 (MRB) with factor loadings ranging from .72 to .84; twelve items loaded onto Factor 2 (SLI) with loadings between .35 and .75; and eight items loaded onto Factor 3 (MEB) with loadings ranging from .32 to .82, consistent with the original inventory structure. Confirmatory Factor Analysis results indicated that the chi-square statistic was significant (χ² = 853.29, p < .001, df = 347), which is expected given the large sample size (χ²/df = 2.46). Fit indices demonstrated an acceptable to good model fit: RMSEA = .059, RFI = .94, GFI = .87, NFI = .94, AGFI = .85, and SRMR = .06 indicated acceptable fit, while NNFI = .96, CFI = .97, and IFI = .97 reflected good fit (Hair et al., 2019). These findings confirm that the Turkish version of the SRSI retains the original three-factor structure. The Cronbach’s alpha coefficient for the overall scale was .74, indicating acceptable internal consistency (Edelsbrunner, Simonsmeier & Schneider, 2025). Variations in adaptation outcomes may arise from translation processes, cultural and linguistic differences, and individual experiences (Chan & Elliott, 2004), which may account for minor discrepancies between the adapted and original versions. Overall, the findings demonstrate that the Turkish adaptation of the SRSI possesses satisfactory psychometric properties and can be used as a valid and reliable instrument for assessing high school students’ self-regulated learning strategies. References Alvi, E., & Gillies, R. M. (2020). Teachers and the teaching of self-regulated learning (SRL): The emergence of an integrative, ecological model of SRL-in-context. Education Sciences, 10(4), 98. https://doi.org/10.3390/educsci10040098. Boekaerts, M. (1992). The adaptable learning process: Initiating and maintaining behavioural change. Applied Psychology: An International Review, 41(4), 377–397. https://doi.org/10.1111/j.1464-0597.1992.tb00713.x Chan, K. W., & Elliott, R. G. (2004). Epistemological beliefs across cultures: Critique and analysis of beliefs structure studies. Educational Psychology, 24 (2), 123-142. https://doi.org/10.1080/0144341032000160100. Cleary, T. J. (2006). The development and validation of the self-regulation strategy inventory—self-report. Journal of School Psychology, 44(4), 307-322. https://doi.org/10.1016/j.jsp.2006.05.002. Edelsbrunner, P. A., Simonsmeier, B. A., & Schneider, M. (2025). The Cronbach’s alpha of domain-specific knowledge tests before and after learning: A meta-analysis of published studies. Educational Psychology Review, 37(1), 4.https://doi.org/10.1007/s10648-024-09982-y. Hadi, N. U., Abdullah, N., & Sentosa, I. (2016). An easy approach to exploratory factor analysis: Marketing perspective. Journal of Educational and social research, 6(1), 215-223. https://doi.org/10.5901/jesr.2016.v6n1p215. Hair, J. F., Black, W. C., Babin, B. J. & Anderson, R. E. (2019). Multivariate data analysis. Hampshire: Cengage Learning EMEA. Kim, M., & Ban, M. (2024). Development of an infertility perception scale for women (IPS-W). BMC Women's Health, 24(1), 513. https://doi.org/10.1186/s12905-024-03336-0. Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, 422. https://doi.org/10.3389/fpsyg.2017.00422. Puth, M. T., Neuhäuser, M., & Ruxton, G. D. (2014). Effective use of Pearson's product–moment correlation coefficient. Animal behaviour, 93, 183-189. https://doi.org/10.1016/j.anbehav.2014.05.003.Lomax, R. G. (2004). A beginner's guide to structural equation modeling. psychology press. Shrestha, N. (2021). Factor analysis as a tool for survey analysis. American Journal of Applied Mathematics and Statistics, 9(1), 4-11. https://doi.org/10.12691/ajams-9-1-2.S. Tabachnick, B. G. & Fidell L. S. (2013). Using multivariate statistics. Boston: Pearson. Zimmerman, B. J. (2000). Self-efficacy: An essential motive to learn. Contemporary educational psychology, 25(1), 82-91. https://doi.org/10.1006/ceps.1999.1016. 09. Assessment, Evaluation, Testing and Measurement
Paper Adaptation SaPS Questionnaire for the European Context: Is It Possible? Masaryk university, Czech Republic (Czechia) Presenting Author:In today’s rapidly changing world, there is an increasing emphasis on transversal competencies—skills that prove invaluable to students regardless of the specific context. A fundamental component of these key competencies is the competence to learn, which fosters the essential skills required for lifelong learning (EU, 2019). Within this framework, student self-assessment plays a pivotal role (Earl, 2012). It serves as a primary driver, encouraging students to gradually assume responsibility for their own educational progress and master the ability to learn throughout their lives. Furthermore, this approach is intrinsically linked to learner-centered education. This paper addresses the level of student self-assessment and the methodologies for its measurement. The instrument selected for this purpose, the SaPS questionnaire (Yan, 2018), was originally developed in an Asian context. This study discusses the process of adapting this tool for the European environment, specifically within the Czech Republic. This adaptation is timely, as the Czech education system is currently undergoing a curricular reform in line with the 2030+ Strategy (Fryč et al., 2020), which shifts the focus toward assessment; notably, formative assessment is being strongly promoted, and traditional grading in the first and second grades of primary school has been abolished. Methodology, Methods, Research Instruments or Sources Used Following Phase 1: Linguistic and Cultural Adaptation The initial stage involved a double-blind forward translation of the original SaPS instrument from its source language into Czech, followed by a back-translation conducted by an independent expert to ensure conceptual equivalence. Beyond literal translation, cultural adaptation was performed to align the items with the specific terminology of the Czech education context. Phase 2: Content Validity and Pre-testing To ensure content validity, the draft version was subjected to a cognitive debriefing process. This involved consultations with a panel of experts in teaching and assessment to assess the context relevance of the items. Simultaneously, semi-structured interviews were conducted with a small sample of pupils (the target population) to evaluate the clarity and comprehensibility of the phrasing, ensuring that the self-assessment items were developmentally appropriate. Phase 3: Pilot Data Collection The adapted instrument was administered to a pilot sample of primary school students. This stage aimed to test the feasibility of the paper-based administration and to gather sufficient data for initial psychometric screening. The sampling focused on wide range of schools. Phase 4: Psychometric Analysis The final stage involves a rigorous statistical evaluation of the tool's internal structure. Exploratory Factor Analysis (EFA) is employed to identify the underlying factor structure within the European context, followed by Confirmatory Factor Analysis (CFA) to test the goodness-of-fit of the model. Furthermore, internal consistency is measured using Cronbach’s alpha to ensure the reliability of the scales. This analysis will determine whether the SaPS instrument remains a valid measure of self-assessment skills and their relation to self-regulated learning when transferred across different cultural and educational frameworks. Conclusions, Expected Outcomes or Findings This paper presents the ongoing process of adapting the SaPS questionnaire for the European educational context, specifically within the Czech Republic. As the research is currently in its developmental and pilot stages, the findings presented here are preliminary. However, the expected findings suggest that the instrument will demonstrate sufficient psychometric characteristics to be utilized within the Czech context. We anticipate that the results from the confirmatory factor analysis will validate the structural integrity of the tool, confirming its effectiveness in measuring students' self-assessment skills. References Earl, L. M. (2012). Assessment as learning: Using classroom assessment to maximize student learning. Corwin press. EU. (2019). KEY COMPETENCES FOR LIFELONG LEARNING. European Union. Fico, M. (2024). Adaptation of selected self-efficacy scales. Ricerche di Pedagogia e Didattica. Journal of Theories and Research in Education, 19(1), 93-107. Fryč, J., Matušková, Z., Katzová, P., Kovář, K., Beran, J., Valachová, I., ... & Štech, S. (2020). Strategy for the education policy of the Czech Republic up to 2030+. Juklová, K., Michek, S., Soukup, P., Vondroušová, J., & Vrabcová, D. (2020). Studentovo hodnocení výuky: validace a adaptace dotazníku CEQ pro podmínky českých vysokých škol. Pedagogická orientace, 30(1), 32-60. Vlčková, K., Mareš, J., Ježek, S., & Šalamounová, Z. (2017). Báze moci učitele: česká adaptace dotazníku Teacher Power Use Scale. Studia paedagogica, 22(1), 87-112. Yan, Z. (2018). The self-assessment practice scale (SaPS) for students: Development and psychometric studies. The Asia-Pacific Education Researcher, 27(2), 123-135. | ||