Conference Agenda

Increasing linguistic diversity in many contexts across the globe and the spread of innovative technologies in teaching and learning pose unique demands on language assessment. In linguistically diverse contexts, individuals' language repertoires may embrace various language skills in multiple languages that necessitate the adjustment of existing instruments to different populations (e.g., ethnic groups) and languages (e.g., heritage and foreign languages; see De Angelis, 2021; Schissel, 2019). Furthermore, the dynamic nature of language skills requires using such tests or other instruments that can capture language development. Tests used for cross-sectional measurements are not necessarily applicable to longitudinal designs, requiring comparability of repeated measurements over time (Barkaoui, 2014). In the digital age, these challenges may also be accompanied by the application of new technologies in administration, scoring, reporting, and interpretation of tests (Neumann et al., 2019). Despite the common nature of challenges around educational assessments in multilingual constellations faced in different national contexts, there has been little consideration of the solutions' generalizability beyond national contexts.

The current symposium addresses such assessment-related challenges of multilingual testing and elaborates on coping strategies. Common perspectives will be presented based on the research conducted in different national contexts. The session strives to raise consciousness about the similarity of challenges across the globe in assessing language skills in linguistically diverse contexts in the digital age. The symposium elaborates on solutions to various assessment-related challenges by bringing together cutting-edge research from different national contexts. The objectives of our symposium are:

1. to enhance the knowledge of the similarities of assessment-related challenges in different national contexts and multilingual constellations;
2. to provide insight into strategies to cope with these challenges;
3. to discuss implications for the international applicability of these coping strategies and their transferability across national contexts.

The first paper investigates a C-test's applicability for foreign language development measures in secondary education in bilingual and monolingual students in Germany. Thereby, different scoring methods are used to compare students' receptive and productive skills. The second paper presents the adaptation of the construct of Core Analytic Language Skills (CALS) to Chinese (CH-CALS). The study advances our scientific understanding of CALS as a theory-driven and robust measure of students' literacy development beyond the western languages. The third paper examines the comparability of the standardized Graz Vocabulary Test (GRAWO: Seifert et al., 2017) with its digital, APP-based version (Paleczek et al., 2021). Further, it addresses the differences in the acceptance of the two modalities by students and teachers and discusses the tests' applicability in a school context. The fourth paper explores the multilingual digital learning platform Binogi used by teachers in Canada to assess the learning gaps of middle-grade students in STEM.

Overall, the projects represent different national contexts (Austria, Canada, China, and Germany), allowing for a reflection of single empirical findings from a global perspective. Discussing the measurement of language skills in linguistically diverse contexts in the digital age and exploring the new modalities and tools will enable us to work jointly and elaborate steps necessary for establishing more efficient and valuable educational assessments.

Structure of the session. The session will begin with a brief introductory talk by the discussant, followed by a presentation of papers. After that, the discussant will launch a discussion on the impact of findings on the assessment of language skills in linguistically diverse settings in the digital age.

This contribution considers the applicability of the C-Test for measuring language skills in multilingual students from a longitudinal perspective. A challenge in this context is finding appropriate tests to tackle development in multilingual repertoires (De Angelis, 2021; Schissel, 2019). Tests need to adhere to the classic standards of test quality – objectivity, reliability, validity – and the modalities of their verification (AERA et al. 2014): (1) If multiple languages are measured simultaneously, tests need to be adjusted to the specifics of particular languages while enabling the cross-language comparison. (2) To capture true development in language proficiency as the trait under study, the measurements must be invariant over time (longitudinal measurement invariance, see Cheug & Rensvold 2002). (3) In the case of multilingual learners, tests must also be sensitive to particular aspects of language proficiency at different stages of language development. In the current study, we explore the potential of C-tests to provide a differentiated approach to the assessment of foreign language development in bilingual and monolingual secondary school students in Germany. C-tests are among the instruments that are widely used as a measure of general language proficiency in foreign language testing (Grotjahn 2002). According to their theoretical construct, they should be (1) appropriate for measuring different stages of language development (Aguado et al. 2007); (2) offer different scoring procedures to distinguish between modes of language skills (receptive/productive) as well as between different aspects of language skills (semantic skills/spelling). However, a C-test's applicability for assessing development in multiple languages has not been explored yet. We draw on data from the German panel study "Multilingual Development: A Longitudinal Perspective" (MEZ) students' language skills in the foreign languages English (n = 1987) and French (n = 662). Our analytic strategy comprises three parts: (1) We investigated longitudinal mean differences in the scoring procedures to investigate the C-test's discriminant validity between semantic and spelling skills. (2) We build a latent measurement model of the C-test using confirmatory factor analysis to test for longitudinal measurement invariance. (3) We explored the applicability of c-tests for a digital testing mode by comparing scores gathered in analogous and digital testing modalities. The results of our study provide evidence that C-tests meet the classic criteria for test quality, such as reliability and validity. Further, a C-test meets the additional requirements of longitudinal testing and allows for capturing both general and specific aspects of language skills.

Aguado, K., Grotjahn, R., & Schlak, T. (2007). Erwerbsalter und Sprachlernfolg: Zeitlimitierte C-Tests als Instrument zur Messung prozeduralen sprachlichen Wissens. In H. J. Vollmer (Ed.), Kolloquium Fremdsprachenunterricht: Vol. 27. Synergieeffekte in der Fremdsprachenforschung: Empirische Zugänge, Probleme, Ergebnisse (pp. 137–149). Lang. American Educational Research Association. (2014). Standards for educational and psycho-logical testing. American Educational Research Association Barkaoui, K. (2014). Quantitative Approaches for Analyzing Longitudinal Data in Second Language Research. Annual Review of Applied Linguistics, 34, 65–101. https://doi.org/10.1017/S0267190514000105 De Angelis, G. (2021). Multilingual Testing and Assessment. Bristol, Blue Ridge Summit: Multilingual Matters. https://doi.org/10.21832/9781800410558 Grotjahn, R. (2002). Konstruktion und Einsatz von C-Tests: Ein Leitfaden für die Praxis. In Rüdiger Grotjahn (ed.), Der C-Test. Theoretische Grundlagen und praktische Anwendungen (vol. 4, pp. 211–225). Bochum: AKS-Verlag. Schissel, J. L., Leung, C., & Chalhoub-Deville, M. (2019). The Construct of Multilingualism in Language Testing. Language Assessment Quarterly, 16(4–5), 373–378. https://doi.org/10.1080/15434303.2019.1680679

Non-fiction reading comprehension is a challenging literacy task in upper-elementary grades. In prior research on English- and Spanish-speaking students, Core Analytic Language Skills (CALS) has been found to significantly predict non-fiction reading comprehension, even controlling for vocabulary knowledge. This construct has yet to be examined in Chinese, a language with distinct orthographic features and discourse structure compared to western languages. CALS refers to a constellation of high-utility language skills needed to unpack sophisticated information in non-fiction texts at the lexical, syntactic, and discourse levels. Guided by the operational definition of CALS and based on a systemtic review of literature and corpora in Chinese, this study has three objectives: 1) to develop and validate a theory-driven and psychometrically-robust assessment of Chinese Core Analytic Language Skills (i.e., the CH-CALS Instrument); 2) to explore the dimensionality of the CH-CALS construct and the instrument; and 3) to examine the contribution of CH-CALS to non-fiction reading comprehension in Chinese, controlling for students’ vocabulary knowledge and demographic characteristics. A cross-sectional sample of 248 Chinese students (Grade 5-6) from diverse socioeconomic backgrounds participated in the study. Using classical test theory and item response theory analysis, we found robust reliability evidence for the CH-CALS instrument (alpha coefficient = 0.88). Consistent with prior research in English and Spanish, most tasks in CH-CALS displayed upward trends in the higher grade yet with considerable within-grade variability. The Rasch modeling results supported the unidimensional structure of the CH-CALS instrument. Principal Component Analysis also revealed that CH-CALS loaded onto a single composite score. The aggregated CH-CALS score significantly predicted non-fiction reading comprehension in Chinese, beyond the contribution of grade, socioeconomic status, and vocabulary. This study advances our scientific understanding of CALS as relevant for students’ literacy development beyond the western languages. The CH-CALS construct and its measurement instrument offer a promising tool to enhance language teaching and assessment practices in Chinese elementary classrooms.

Uccelli, P., Galloway, E.P., & Qin, W. (2020). The language for school literacy: Widening the lens on language and reading relations. In Handbook of Reading Research (Vol. V, pp. 155-180). New York: Routledge. Qin, W., Zheng, Y., & Uccelli, P. (2021). Core Language Skills in EFL Academic Reading Comprehension: A Construct and its Assessment. 6.

Vocabulary is a prerequisite for many school-relevant skills. In particular, reading competence (word reading, reading comprehension) is predicted by vocabulary knowledge in kindergarten (e.g., Ennemoser et al., 2012). For second language (L2) learners, this tight relationship is even closer (e.g., Seifert et al., 2019). L2 learners are outperformed by their L1 learner peers in reading tests (e.g., Melby-Lervåg & Lervåg, 2014). In order to adequately support children's reading skills, valid vocabulary assessments are crucial (Ennemoser et al., 2012). Using digital tools can facilitate assessment and it offers some advantages over traditional print methods, namely in administration, automated scoring, interpretation and feedback, as well as student motivation (e.g., Neumann et al., 2019). Although digitalizing print assessments is a natural step in the transition to technology-based assessments (Neumann et al., 2019), digital test results need to be comparable to those of the print version. However, the growing body of international research on the impact of test modality is inconsistent (Blumenthal & Blumenthal, 2020). The paper compares the print version of the standardized Graz Vocabulary Test (GRAWO: Seifert et al., 2017) with a digital, APP-based version (Paleczek et al., 2021) on a sample of n = 400 first grade children. In addition to the validation (teacher assessment for convergent and discriminant validity), feasibility data (survey of students regarding preference of modality) are presented. The GraWo was conducted at the beginning and at the end of the school year in a classroom setting, while one part of the sample received the classic print version fist and the other part was first presented with the digital version via iPads. The aims of this paper are to examine whether the results of the digital test are comparable to those of the print version, to find out how well the test can be used in a school context, and whether there are differences in the acceptance of the two modalities by students and teachers.

Blumenthal, S., & Blumenthal, Y. (2020). Tablet or Paper and Pen? Examining Mode Effects on German Elementary School Students’ Computation Skills with Curriculum Based Measurements. International Journal of Educational Methodology, 6(4), 669 680. https://doi.org/10.12973/ijem.6.4.669 Ennemoser, M., Marx, P., Weber, J, & Schneider, W. (2012). Spezifische Vorläuferfertigkeiten der Lesegeschwindigkeit, des Leseverständnisses und des Rechtschreibens. Zeitschrift für Entwicklungspsychologie und Pädagogische Psychologie, 44(2), 53-67. Melby-Lervåg, M., & Lervåg, A. (2014). Reading comprehension and its underlying components in second-language learners: A meta-analysis of studies comparing first- and second-language learners. Psychological Bulletin, 140(2), 409–433. https://doi.org/10.1037/a0033890 Neumann, M. M., Anthony, J. L., Erazo, N. A., & Neumann, D. L. (2019). Assessment and Technology: Mapping Future Directions in the Early Childhood Classroom. Frontiers in Education, 4, Article 116, 1–13. https://doi.org/10.3389/feduc.2019.00116 Paleczek, L., Seifert, S., & Schöfl, M. (2021). Comparing digital to print assessment of receptive vocabulary with GraWo-KiGa in Austrian kindergarten. British Journal of Educational Technology. https://doi.org/10.1111/bjet.13163 Seifert, S., Paleczek, L., & Gasteiger-Klicpera, B. (2019). Rezeptive Wortschatzleistungen von Grundschulkindern mit Deutsch als Erst- und Zweitsprache und der Zusammenhang zu den Lesefähigkeiten: Implikationen für einen inklusiven Unterricht. Empirische Sonderpädagogik, 4, 259-278. Seifert, S., Paleczek, L., Schwab, S., & Gasteiger-Klicpera, B. (2017). Grazer Wortschatztest – GraWo. Göttingen: Hogrefe.

Learning the language of the school continues to be considered a key factor in the academic success of language learners. Language learners need at least five years to catch up with their native-speaking peers in school (e.g., Cummins, 1981; Collier, 1987; Levin & Shohamy, 2008). During this time, they invest a substantial proportion of their efforts in learning the school language. As a result, some fall behind academically (Cummins, 1989; Cummins, 2000; Collier & Thomas, 1989; OECD, 2016). One of the crucial advantages of digital technologies is that they can incorporate different languages, which provides a unique opportunity for teachers to assess their students in STEM content in languages they know (Stacy et al. 2017). Such technology can provide a more accurate impression of their knowledge and skills in these domains (Educational Testing Service, 2012). Teachers need to learn, however, how to use and implement novel multilingual digital tools. If applied in the right way, such multilingual digital programs can, not only serve as a learning tool for students, but also provide the teacher with insight into the students' true academic skills. In this study, we explored this issue by examining how 17 teachers used a multilingual digital learning platform called Binogi to assess the learning gaps of middle-grade students in STEM. We implemented and evaluated the impact of this online STEM-focused program that combines a highly engaging interface with a choice of several languages across Ontario in schools where there is a greater than average number of plurilingual students. We sought to elucidate the experiences of teachers with regards to using this technology and various strategies to support them in learning to integrate multilingual assessments across the STEM curriculum. Our study reveals the essential role that multilingual resources like Binogi can play in terms of translanguaging pedagogy. It also allows us to identify a number of preconditions for their effective use. In this presentation, I will emphasize that the condition for a fair assessment for these students is a medium to long term assessment that considers the linguistic and cultural backgrounds of the students in a perspective of reciprocal knowledge. I will show that in order to distinguish between the academic and language profiles of students, it is necessary, but not sufficient, to assess them in a language in which they feel comfortable.

Cummins J. (1981). Empirical and Theoretical Underpinnings of Bilingual Education. Journal of Education, 163(1),16-29. doi:10.1177/002205748116300104 Cummins, J. (2000). Language, power and pedagogy: Bilingual children in the crossfire. Multilingual Matters Ltd. https://doi.org/10.1080/15235882.2001.10162800 Collier, V.P., & Thomas, W.P. (1989). How quickly can immigrants become proficient in school English? Journal of Educational Issues of Language Minority Students, 5, 26-38. Levin, T. & Shohamy, E. (2008). Achievement of immigrant students in mathematics and academic Hebrew in Israeli school: A large-scale evaluation study. Studies in Educational Evaluation, 34(1), 1-14. OECD (2016), Education at a Glance 2016: OECD Indicators, OECD Publishing, Paris. Stacy, S. T., Cartwright, M., Arwood, Z., Canfield, J. P., & Kloos, H. (2017). Addressing the math- practice gap in elementary school: Are tablets a feasible tool for informal math practice? Frontiers in Psychology, 8, 179.