09. Assessment, Evaluation, Testing and Measurement
Paper
Literacy Learning for Students with Intellectual Disability Using Phonic-Based and Comprehension-Based Interventions
Lisa Palmqvist1,2, Mikael Heimann2, Jenny Samuelsson3,4,5, Gunilla Thunberg3,4, Monica Reichenberg1, Emil Holmer2
1Department of Education and Special Education, University of Gothenburg, Sweden; 2Department of Behavioural Sciences and Learning, Linköping University, Linköping, Sweden; 3Institute of Neuroscience and Physiology, Speech and Language Pathology Unit, Sahlgrenska Academy, University of Gothenburg, Sweden; 4Dart - Centre for AAC and Assistive Technology, Sahlgrenska University Hospital, Gothenburg, Sweden; 5Region Västra Götaland, Habilitation & Health, Gothenburg, Sweden
Presenting Author: Palmqvist, Lisa
To participate in today’s society, one has to be able to read and write. For a person with intellectual disability (ID), this requirement may be insurmountable and many never become proficient readers (Cawley & Parmar, 1995; Di Blasi et al., 2019; Lemons et al., 2013; Ratz & Lenhard, 2013a; Wei et al., 2011). Thus, we must provide effective reading instructions early. This study aimed to investigate reading development in individuals with ID enrolled in the Swedish compulsory school system for students with ID using a, for the field, methodologically rigorous design. The study focused on the effects of two reading instruction strategies, phonics-based and comprehension-based, on reading development in beginning readers with ID. The Simple View of Reading (Gough & Tunmer, 1986; Hoover & Gough, 1990; Tunmer & Hoover, 2019) describes reading comprehension as the product of word recognition and language comprehension. The model is supported by meticulous research (e.g., Lervåg et al., 2018; Lervåg & Melby-Lervåg, 2012). Word recognition is the process of correctly identifying meaningful units in text and is associated with several different pre-literacy skills, including phonological awareness. Language comprehension refers to the process of binding together multiple lexical units into coherent semantic representations, using contextual, syntactical, and inferred information. Research has found that individuals with ID have poor pre-literacy skills, word recognition, and reading comprehension. The difficulties for a person with ID also include poor executive functions and working memory, and the difficulties often increase with the severity of the ID. There is a large body of research on reading ability in the population without ID, but the research for persons with ID is lagging (Dessemontet et al, 2019). Recent studies found similar variables associated with word recognition and reading comprehension in individuals with ID as for students with a typical development, suggesting that reading manifests similarly in both groups which in turn would indicate that the methods that are known to support reading development in typically developing children should also produce positive effects on reading development in children with ID. Literature has found that phonic-based interventions are effective for teaching literacy to students with mild ID and students with severe cognitive disabilities (Ainsworth et al., 2016; Dessemontet et al., 2019). Ainsworth et al., (2016) investigated teaching phonics to students with autism, intellectual disabilities, and complex communication needs and found that children increased their letter-sound-knowledge. Dessemontet et al., (2021) performed an RCT phonic-based instruction for students with ID and found positive results. Additionally, studies have shown promising results in combining the instruction of phonic-based and comprehension-based strategies for children with ID (e.g., Browder et al., 2012; Gustafson et al., 2007). However, the samples in the studies are often small without using a control group. The current project builds on these studies using a large-scale controlled study, by implementing a comprehension- and a phonic-based approach (and a combination thereof) using digital media. The aim of the current study is to help children with ID in need of AAC to reach as fluent reading capacity as possible using three different intervention strategies: a phonic-based, a comprehension-based, or a combination of the two. In the present study, we tested three pre-registered hypotheses (Palmqvist, Samuelsson, et al., 2020):
- A phonics-based or a comprehension-based reading strategy improves phonological awareness.
- A phonics-based or a comprehension-based reading strategy improves reading ability (word recognition and reading comprehension).
- The combination of both reading strategies is more effective than either strategy on its own.
The hypotheses will be tested on these outcome variables: phonological awareness (1, 3), word recognition (2, 3), and reading comprehension (2, 3).
Methodology, Methods, Research Instruments or Sources UsedA total of 124 students (ngirls = 54, nboys = 70) were included in the study. They had a mean age of 13.7 years (SD = 3.3) and a mean IQ of 48 (SD = 13). Participants had to attend a Swedish special needs school and be beginning readers. In Sweden, the student receives an ID diagnosis according to the ICD-11 (WHO, 2019) prior to being enrolled in the special needs curriculum. The teachers were instructed to identify students that could not read more than approximately 20 words, which was the operationalization of being a beginning reader. To make the sample representative of the students in special needs schools, no exclusion criteria were set for additional diagnoses or aetiology of the ID. The caregivers of all participating students signed informed consent prior to the testing. The study has been approved by the Swedish Ethical Review Authority (2020-06215).
The study is a longitudinal between-group study, with four time points (t1-t4; before, during, directly after the intervention, and a follow-up), and four groups: phonics-based reading strategy, comprehension-based reading strategy, both phonics-based and comprehension-based strategies (combination group), and a comparison group who received teaching-as-usual.
Before initiating testing, background information about the participants was collected by interviewing parents (e.g., diagnoses). Testing took place in a silent environment at the participants’ school. All children were assessed on general non-verbal cognitive ability Raven’s 2; Raven et al., 2018), phonological awareness (MiniDUVAN; Wolff, 2013), word recognition (OS64 & OLAF; Magnusson & Naucler, 2010), communication skills (BAF; Frylmark, 2015), and reading comprehension (DLS Bas; Järpsten, 2004), and were allocated into one of four intervention groups. The intervention strategies were provided in a digital format (i.e., apps) and the students worked at school together with a teacher or assistant. The intervention was conducted over 12 weeks (3x30 minutes per week).
Linear mixed-effects models were used to evaluate the effects of the intervention. The outcome measures (PA, word recognition, and reading comprehension) were analyzed separately. Days were used as the time variable, starting day 1 at the date of t1. Model fit was assessed using ANOVA. The model with the best fit, indicated by χ2, was chosen. There were three contrasts performed: the comparison group versus the phonics-based intervention (Hypothesis 1), the comparison group versus the comprehension-based intervention (Hypothesis 2), and the combination group versus the phonics-based and the comprehension-based intervention (Hypothesis 3).
Conclusions, Expected Outcomes or FindingsNo initial differences in reading performance between groups were observed at t1. The results showed that reading improved over time, as indicated by a main effect of time across all three outcome measures β ̂=0.09, 95% CI [0.05,0.13], t(336.07)=4.81, p<.001. The combined phonics and comprehension strategy had a positive impact on phonological awareness development β ̂=0.09, 95% CI [0.03,0.15], t(333.59)=2.78, p=.006. However, no significant differences were found in word recognition or reading comprehension based on the reading instruction strategy used.
The results support the idea that systematic instruction that includes explicit teaching of phonics, and comprehension is better than more simple instructional strategies, such as practicing sight-word reading. Phonics-based instruction improves the sensitivity to the sub-lexical structure of spoken words, while comprehension-based instruction leads to richer and more precise lexico-semantic representations. Combining the two strategies may allow students to apply their improved skills in a richer context, making the combination more effective than either strategy alone.
The results add to the previous literature that conventional literacy strategies for persons with typical development also are effective for students with ID when using a combined instructional strategy. Teachers should prioritize intensive and methodologically rich literacy instructions for their students. Furthermore, we provide evidence that digitally-based interventions for reading are more effective than teaching-as-usual for students with ID. One reason may be that the digital format enables the instructions to be provided in an adapted manner for students with ID in terms of adapted speed or response time. Additionally, the focused intervention itself might have contributed to more literacy instruction than their teaching-as-usual which in turn resulted in improved reading.
ReferencesAinsworth, et al., (2016). Teaching phonics to groups of middle school students with autism, intellectual disabilities and complex communication needs. Research in Developmental Disabilities, 56, 165-176.
Browder, et al., (2012). An evaluation of a multicomponent early literacy program for students with severe developmental disabilities. Remedial and Special Education, 33 , 237-246.
Dessemontet, et al., (2021). Effects of a phonics-based intervention on the reading skills of students with intellectual disability. Research in Developmental Disabilities, 111, 103883.
Dessemontet, et al., (2019). A meta-analysis on the effectiveness of phonics instruction for teaching decoding skills to students with intellectual disability. Educational Research Review, 26, 52-70.
Gustafson, et al., (2007). Phonological or orthographic training for children with phonological or orthographic decoding deficits. Dyslexia, 13 , 211–229.
Cawley, J. F., & Parmar, R. S. (1995). Comparisons in reading and reading-related tasks among students with average intellectual ability and students with mild mental retardation. Education and Training in Mental Retardation and Developmental Disabilities, 118-129.
Di Blasi, F. D., Buono, S., Cantagallo, C., Di Filippo, G., & Zoccolotti, P. (2019). Reading skills in children with mild to borderline intellectual disability: a cross‐sectional study on second to eighth graders. Journal of Intellectual Disability Research, 63(8), 1023-1040.
Gough, P. B., & Tunmer, W. E. (1986). Decoding, reading, and reading disability. Remedial and special education, 7(1), 6-10.
Hoover, W. A., & Gough, P. B. (1990). The simple view of reading. Reading and writing, 2(2), 127-160.
Lemons, C., Zigmond, N., Kloo, A., Hill, D., Mrachko, A., Paterra, M., Bost, T., & Davis, S. (2013). Performance of students with significant cognitive disabilities on early-grade curriculum-based measures of word and passage reading fluency. Exceptional Children, 79(4), 408–426.
Lervåg, A., Hulme, C., & Melby‐Lervåg, M. (2018). Unpicking the developmental relationship between oral language skills and reading comprehension: It's simple, but complex. Child development, 89(5), 1821-1838.
Melby-Lervåg, M., Lyster, S. A. H., & Hulme, C. (2012). Phonological skills and their role in learning to read: a meta-analytic review. Psychological bulletin, 138(2), 322.
Ratz, C., & Lenhard, W. (2013). Reading skills among students with intellectual disabilities. Research in developmental disabilities, 34(5), 1740-1748.
Tunmer, W. E., & Hoover, W. A. (2019). The cognitive foundations of learning to read: A framework for preventing and remediating reading difficulties. Australian Journal of Learning Difficulties, 24(1), 75-93.
Wei, X., Blackorby, J., & Schiller, E. (2011). Growth in reading achievement of students with disabilities, ages 7 to 17. Exceptional Children, 78(1), 89–106.
09. Assessment, Evaluation, Testing and Measurement
Paper
The Views of Students with Disabilities on Speech and Reading Compared to Corresponding Test Results
Jenny Samuelsson1,2,3, Emil Holmer4, Jakob Åsberg Johnels2, Lisa Palmqvist4,5, Mikael Heimann4, Gunilla Thunberg2,3
1Region Västra Götaland, Habilitation & Health, Gothenburg, Sweden; 2Institute of Neuroscience and Physiology, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden; 3Dart - Centre for AAC and Assistive Technology, Sahlgrenska University Hospital, Gothenburg, Sweden; 4Department of Behavioral Science and Learning, Linköping University, Linköping, Sweden; 5Department of Education and Special Education, University of Gothenburg, Sweden
Presenting Author: Samuelsson, Jenny
The opinion of people with intellectual disabilities (ID) and communication difficulties can be challenging to capture. Pictorial support to enable communication, such as the methodological framework Talking MatsTM has been successful to solicit the views of both adults and young people with ID (Murphy & Cameron, 2008). We believe that a constructionist way of thinking supports the idea of listening to children and trying to understand their thoughts, likes, fears, hopes, and problems, with the goal of forming a partnership. The process of guided participation, as presented by Rogoff (2003), involves children engaging in communication and acquiring knowledge through close collaboration with their peers and surroundings. This process views children as active and capable agents of change, as described by Rogoff (2003), Tomasello (2013) and Vygotsky et al. (1978).
According to the United Nations Convention on the Rights of the Child (UNCRC) and The Convention on the Rights of Persons with Disabilities, (CRPD), all children have the right to express their views and the views shall be given due weight. The children shall be provided with the opportunity to express their views in accordance with their age and maturity. To provide all children to express their views, also those with ID and communication difficulties, environmental support and adaptations are needed.
It is important to carefully consider the challenges that individuals with ID may face when expressing their views and opinions and to use strategies to support their participation and ensure that their perspectives are included and valued. Many individuals with ID and communication difficulties need Augmentative and Alternative Communication (AAC) to understand and to be understood (Beukelman, 2020). Using interviews and questionnaires developed for the general population is often complicated for individuals with ID and communication difficulties (Santoro et al., 2022). The addition of pictures in conversations, interviews, and questionnaires can support cognitive and communication difficulties as well as executive functions and working memory (Boström et al., 2016). Resources for pictorial support facilitate comprehension and supports expression and Talking MatsTM has been widely used together with people affected by communication difficulties for both research and clinical purposes (i.e. Breeze, 2021; Stans et al., 2019).
Research supporting literacy interventions for individuals with different diagnoses and for all ages has shown promising results indicating that everyone should be given the opportunity to get a literacy education (Yorke et al., 2021). The enjoyment of reading and its positive relationship to reading ability has been well researched among students with typical development (e.g., Rogiers et al., 2020; Smith et al., 2012), but not in children with ID. The conclusion cannot be drawn that the simple fact of enjoying reading leads to increased literacy skills for neither students with typical development nor students with ID. Research in comparing children´s perspective have mostly been focusing on participation and goal setting and in comparison with proxy raters (Stans et al., 2009). The children´s views compared to corresponding test results is sparse.
In the current study, we interviewed students with ID in Swedish special needs schools. They all attended a reading intervention with digital apps and were interviewed before and after the intervention about their own communication and reading. The overall aim was to determine the relationship between students’ own views and their corresponding formal test results on speech sound production and word reading ability.
The study posed two research questions: (1) What are the students' own views of their speech and reading activities? And (2) Is there a positive correlation between the students' views of their speech and their speech sound production, and between their views on reading and their word reading ability?
Methodology, Methods, Research Instruments or Sources UsedA total of 116 students (65 boys and 51 girls) with ages ranging from 7 to 21 years old took part in this study. All the students met the inclusion criteria which were: (1) intellectual disability (2) need for AAC to understand and express themselves, and (3) inability to decode words independently and identify a maximum of 20 sight words.
This study used the pictorial framework Talking MatsTM to enable students with ID and communication difficulties to share their views on speech and reading activities. The process includes identifying a topic, discussing related options, and using a visual scale to indicate views or feelings. A practice mat with simple questions about animals was used to validate students' understanding of the method. The remaining students who completed the practice mat were asked questions about their views on communication and reading activities. The three-point visual scale with pictures of facial expressions representing “like”, “neutral/in between” and “dislike” was transformed into a numbered ordinal scale. In the comparison between students self-rating and the test results on speech sound production and word reading, we focused on four questions related to speech (merged into one variable, Speech) and three questions related to increased difficulty in reading ability (merged into one variable, Reading activities).
For assessing speech ability, a phonological test (Assessment of phonology, Frylmark) was used. The speech was transcribed and calculated as percentage phonemes correct on 138 phonemes. Reliability was excellent, with a high intraclass correlation coefficient of .997 and substantial agreement as indicated by a Cohens Kappa of .78. Two reading tests, OS 64 and OLAF were used to assess reading ability. Both tests were shortened for the present study, with OS 64 reduced to 15 items and OLAF reduced to 13 items. In OS 64, participants were shown written words and asked to match them to pictures, while in OLAF, they were shown pictures and asked to match them to written words. The test procedures for OS 64 were adapted by using enlarged symbols for visibility. The dependent variable was the number of correct answers on both tests, with a good test-retest reliability of .780.
We used descriptive statistics to analyse the interview responses. Additionally, Pearson's correlations were used to examine the relationship between students' perceptions of their speech and reading ability and their actual test results in speech sound production and reading ability.
Conclusions, Expected Outcomes or FindingsThe use of pictorial support was beneficial in enabling students with ID and communication difficulties to express their views and participate in the study. The students had positive views of their speech, with 64% having positive views towards talking to one person at a time, and 46% liking talking in groups. Many students (64%) answered positively about talking on the phone. However, less than half (43%) had positive views towards their speech being intelligible for others. The student’s views on reading activities were more varied, with 47% having positive views toward reading letters, 38% for reading words, and 26% for reading sentences. The results indicate that as the degree of difficulty in the activity increases, the students' ratings were less positive.
The study found that there was a positive association between the students' views on their speech and their actual speech sound production, as well as their views on reading activities and their tested word reading ability. The correlation coefficients were calculated using Pearson's method, and were statistically significant, with small positive values of r(84) = .24, p < .05 for speech and r(104) = .21, p < .05 for reading. This indicates that the student’s views on their speech and reading abilities were in line with their actual abilities as measured by formal tests.
The students had a basic understanding of their own speech sound production and word reading abilities, as reflected in their views on these activities. The study highlights the challenges of including students with intellectual disabilities in research but emphasizes the importance of following the United Nations Convention on the Rights of the Child (UNCRC) and including these students in research. To overcome these challenges, the study suggests using pictorial support such as Talking Mats to facilitate communication and better understand the views of these students.
ReferencesBeukelman, D. R. (2020). Augmentative & alternative communication : supporting children and adults with complex communication needs (Fifth edition ed.). Baltimore, Maryland : Paul H. Brookes Publishing Co.
Boström, P., Johnels, J. Å., Thorson, M., & Broberg, M. (2016). Subjective Mental Health, Peer Relations, Family, and School Environment in Adolescents with Intellectual Developmental Disorder: A First Report of a New Questionnaire Administered on Tablet PCs. Journal of mental health research in intellectual disabilities, 9(4), 207-231. https://doi.org/10.1080/19315864.2016.1186254
Breeze, J. (2021). Including people with intellectual disabilities in the development of their own positive behaviour support plans. Tizard Learning Disability Review.
Murphy, J., & Cameron, L. (2008). The effectiveness of Talking Mats with people with intellectual disability. British journal of learning disabilities, 36(4), 232-241. https://doi.org/10.1111/j.1468-3156.2008.00490.x
Rogiers, A., Van Keer, H., & Merchie, E. (2020, 2020/01/01/). The profile of the skilled reader: An investigation into the role of reading enjoyment and student characteristics. International Journal of Educational Research, 99, 101512. https://doi.org/https://doi.org/10.1016/j.ijer.2019.101512
Rogoff, B. (2003). The cultural nature of human development. Oxford University Press.
Santoro, S. L., Donelan, K., & Constantine, M. L. (2022). Proxy-report in individuals with intellectual disability: A scoping review. Journal of applied research in intellectual disabilities : JARID.
Smith, J. K., Smith, L. F., Gilmore, A., & Jameson, M. (2012). Students' self-perception of reading ability, enjoyment of reading and reading achievement. Learning and Individual Differences, 22(2), 202-206. https://doi.org/10.1016/j.lindif.2011.04.010
Stans, S. E. A., Dalemans, R. J. P., de Witte, L. P., & Beurskens, A. J. H. M. (2019). Using Talking Mats to support conversations with communication vulnerable people: A scoping review. Technology and disability, 30(4), 153-176. https://doi.org/10.3233/TAD-180219
Tomasello, M. (2013). Origins of human communication. MIT Press.
UN Convention on the Rights of the Child, (1989). https://www.ohchr.org/en/instruments-mechanisms/instruments/convention-rights-child
UN Convention on the Rights of Persons with Disabilities (2006). https://www.un.org/development/desa/disabilities/convention-on-the-rights-of-persons-with-disabilities/article-7-children-with-disabilities.html
Vygotsky, L. S., Cole, M., John-Steiner, V., Schribner, S., & Souberman, E. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.
09. Assessment, Evaluation, Testing and Measurement
Paper
The Impact of Multilingualism on Children’s Reading/Writing Skills and Scholastic Performance
Linda Romanovska, Ineke Pit-Ten Cate, Sonja Ugen
University of Luxembourg, Luxembourg
Presenting Author: Romanovska, Linda
While research on multilingualism has shown both, positive (e.g. inhibition; Coderre et al., 2013), and negative (e.g. vocabulary; Bialystok et al., 2008) effects on cognition and language proficiency, its influence on scholastic achievement appears to be largely negative (Hoffmann et al., 2018; Martini et al., 2021). Children in Luxembourg are educated in a multilingual educational system. In Kindergarten, the main teaching language is Luxembourgish. This switches to German for literacy acquisition in elementary school, with French taught as a second language. Despite its small size, Luxembourg is also highly multi-cultural, boasting 170 nationalities (The Government of the Grand Dutchy of Luxembourg, 2023). Thus, many of the children in the school system do not speak the language(s) of instruction at home. Data from the Luxembourgish national school monitoring program reveals significant differences in German reading comprehension in grade 3 depending on the language spoken at home. Because Luxembourgish is linguistically close to German, Luxembourgish-speaking children generally perform better than children who do not speak Luxembourgish at home (Hoffmann et al., 2018; Martini et al., 2021).
Furthermore, the language-based differences in children’s scholastic performance complicate the diagnostic process of children with potential learning disorders, such as dyslexia and/or dyscalculia. In Luxembourg, the language in which children are screened and diagnosed for potential learning disorders is usually identical to the main language of instruction at school, which at time of diagnosis (typically grade 3) is German. It is therefore difficult to distinguish poor performance based on potential difficulties with reading/writing or mathematics from poor performance based on low language proficiency in the test language. Furthermore, the diagnostic tools currently employed in Luxembourg are developed in countries with primarily one language of instruction, challenging the validity of the diagnostic process in a multilingual population (Ugen et al., 2021).
We have thus developed a comprehensive reading/writing test battery adapted to the Luxembourgish educational curriculum and multilingual environment. Children’s potential language proficiency differences in the test language (German) are taken into account using simplified instructions with reduced language load, multiple examples, varying degrees of difficulty of the test materials, as well as the construction of distinct language-group norms, depending on the language(s) spoken at home. This helps avoid over-diagnosis of reading and writing disorders in children who do not speak the language(s) of instruction at home and underdiagnosis of children who do. The developed test battery assesses children’s performance in key domains relevant for reading and writing comprising phonological skills, (non)word and text reading (fluency and accuracy), reading comprehension, writing, and vocabulary. Furthermore, we link children’s performance in the newly developed test battery to their performance in the Luxembourgish national school monitoring program.
Methodology, Methods, Research Instruments or Sources UsedWe have tested 214 children during the pre-test phase of the project (February – June 2022; age 8 – 12; M = 9.59; SD = 0.68; 95 girls) and will test approximately 735 children during the validation and norming phase (February – June 2023). All children attend grade 3 in public primary schools in Luxembourg. The distribution of classes participating in the project covers all 15 regions of the country, resulting in a representative sample of the Luxembourgish school population.
Children complete the 9 sub-tests of the novel reading/writing test battery, which includes precursor skills: Rapid Automatized Naming (RAN), non-word phoneme segmentation, non-word phoneme deletion; reading skills: word and non-word reading, text reading and comprehension; writing skills: gap dictation and text dictation; as well as a receptive vocabulary task. The vocabulary and writing skills are assessed in a group setting (all children complete the tasks together in the classroom), the precursor and reading skills are assessed individually in a quiet room in the school. The total testing time (group test + individual tests) does not exceed 90 minutes per child. All tests are conducted by trained test administrators following a standardised procedure.
The pre-test data were analysed per sub-test using Repeated Measures Analysis of Variance with language group as the between-subject factor and results of the sub-test (per category where applicable) as the within subject factor. Significant main effects of language group were explored using post-hoc pairwise-comparisons (Bonferroni corrected t-tests). Four language groups were created based on the frequencies of the reported language(s) spoken at home: Luxembourgish/German monolingual, Luxembourgish/German bilingual, Romance language (e.g., French, Portuguese, Spanish) mono- and bilingual, Other language (e.g., English, Slavic) mono- and bilingual.
The results of each sub-test of the novel reading/writing test battery were also correlated with children’s performance on German listening and reading comprehension in the Luxembourgish national school monitoring programme (Bonferroni corrected Pearson correlations).
Conclusions, Expected Outcomes or FindingsThe results of the pre-test phase show that children, who speak Luxembourgish or German at home outperform children who speak a Romance or Other language at home. Particularly, significant differences between language groups were observed for: word reading accuracy (F(3,190) = 4.94, p = .003); word reading fluency (F(3,190) = 4.59, p = .004); text reading accuracy (F(3,190) = 8.73, p < .001); text reading fluency (F(3,190) = 11.50, p < .001); text comprehension (F(3,190) = 12.45, p < .001); gap dictation (F(3,180) = 10.52, p < .001); text dictation (F(3,180) = 18.22, p < .001). The significant main effects of language highlight the need for separate language group norms for screening and diagnostic purposes. The lack of main effects of language for non-word phoneme deletion, non-word phoneme segmentation, and non-word reading indicate that the sub-tests using non-words were successfully constructed to account for language proficiency effects.
Significant Pearson correlations between the school monitoring results of German listening (.28 < |𝜌| < .59) and German reading comprehension (.24 < |𝜌| < .65) and the majority of the newly developed sub-tests of the reading/writing test battery were also observed. These correlations provide a measure of construct validity, illustrating the significant link between children’s scholastic performance and performance in the novel reading/writing test battery.
We expect to replicate these initial findings with a larger sample of children during the validation and norming phase of the project and supplement our data analyses with more detailed results highlighting the distribution of scores per sub-test based on language spoken at home and its effect on scholastic performance as assessed by the Luxembourgish national school monitoring program.
ReferencesBialystok, E., Craik, F., & Luk, G. (2008). Cognitive Control and Lexical Access in Younger and Older Bilinguals. Journal of Experimental Psychology: Learning Memory and Cognition, 34(4), 859–873. https://doi.org/10.1037/0278-7393.34.4.859
Coderre, E. L., van Heuven, W. J. B., & Conklin, K. (2013). The timing and magnitude of Stroop interference and facilitation in monolinguals and bilinguals. Bilingualism, 16(2), 420–441. https://doi.org/10.1017/S1366728912000405
Hoffmann, D., Hornung, C., Gamo, S., Esch, P., Keller, U., & Fischbach, A. (2018). Schulische Kompetenzen von Erstklässlern und ihre Entwicklung nach zwei Jahren. In T. Lentz, I. Baumann, & A. Küpper (Eds.), Nationaler Bildungsbericht (pp. 84–96). University of Luxembourg & SCRIPT.
Martini, S., Schiltz, C., Fischbach, A., & Ugen, S. (2021). Identifying Math and Reading Difficulties of multilingual children: Effects of different cut-offs and reference group. In M. Herzog, A. Fritz-Stratmann, & E. Gürsoy (Eds.), Diversity Dimensions in Mathematics and Language Learning (pp. 200–228). De Gruyter Mouton.
The Government of the Grand Dutchy of Luxembourg. (2023, January) Society and culture – Population Demographics. https://luxembourg.public.lu/en/society-and-culture/population/demographics.html
Ugen, S., Schiltz, C., Fischbach, A., & Pit-ten Cate, I. M. (2021). Lernstörungen im multilingualen Kontext. Diagnose und Hilfestellungen. Melusina Press. https://doi.org/10.26298/bg5s-ng46
09. Assessment, Evaluation, Testing and Measurement
Paper
Is Paper-based Reading Achievement a Better Predictor of Later Reading Achievement Than is Digital Reading Achievement?
Elpis Grammatikopoulou, Stefan Johansson, Monica Rosén
Göteborgs Universitet, Sweden
Presenting Author: Grammatikopoulou, Elpis
Both IEA and the Swedish National Agency for Education (Skolverket) stresses the importance of reading literacy as they recognize it as an individual right and a prerequisite to participate in society, to gain knowledge of school subject and being able to argue and participate in decision making situations (Skolverket, 2016; Mullis, Martin & Sainsbury, 2015). The definition of reading literacy has evolved over time from the simple view of reading as a decoding and word recognition to involve further skills as comprehension and meta-comprehension (Gough & Tunmer, 1986; Chall, 1989; Sefton-Green, Marsh, Erstad & Flewitt, 2016) and with the entrance of digital means in everyday and school life has grown further to involve digital and social skills as well (Coiro et al., 2008; Leu et al., 2009). The differences between paper-based and digital reading have been under close scrutiny in recent past indicating that the two modalities differ in several aspects besides the obvious format, such as the skills required, the processes needed and the consequences in comprehension and concentration (Delgado et al., 2018; Baron, 2017). Previous research has explored and confirmed the paper-based reading ability as a predictor of later achievement (Butler et al., 1985; Sparks et al., 2013;). Less is known though about the role digital reading ability as a predictor for later achievement. The present study will shed light to the possible effect of difference between digital reading assessment and paper-based reading assessment as a predictor of later reading achievement and general achievement. So far, most studies have focused on the individual factors that may influence achievement in reading. The present study examines the two modalities in comparison along with individual factors, such as SES and immigrant background.
IEA’s PIRLS and ePIRLS both measures reading literacy in paper-based and digital form respectively. In 2016, Swedish 10-year-olds completed both the paper-based and digital version of the reading assessment PIRLS. The correlation between students’ test scores amounted to 0.79, thus a high, yet not a perfect, relationship. The correlation revealed that there likely are differences between the two formats and that some students perform better in paper-based reading whereas others perform better on the digital test. IEA analyzed the differences that emerged between PIRLS and ePIRLS results for all countries. Swedish 10-year-olds were generally better in digital, but the difference was not statistically significant.
The main purpose of the current study is to investigate the predictive validity of the paper-based and digital reading scores in PIRLS by comparing these scores with the core subject marks and the overall marks in grade 6 for the students that participated in PIRLS 2016. A hypothesis is that the paper-based scores will be more closely related to the subject grade and overall grades, partly because paper-based reading was more prominent in schools at this point in time. Drawing from Clinton’s meta-analysis (2019) suggesting that paper-based reading is associated with deeper comprehension and with metacognitive processes (accuracy in prediction of achievement), we hypothesize that students who are relatively stronger in paper-based reading test to have higher marks in schoolyear 6 than students who scored better in digital reading. Against this background, two specific research questions are posed:
- What is the relationship between paper-based and digital reading achievement in grade 4 and core subject marks and GPA in grade 6?
- Is there a stronger relationship between grades in Grade 6 and the reading results in PIRLS for students who are relatively stronger in paper-based reading?
Methodology, Methods, Research Instruments or Sources UsedThe present study uses Swedish register data together with PIRLS and ePIRLS scores. Sweden participated in both PIRLS and ePIRLS IN 2016 with approximately 4000 students. Through the Swedish unique social security number (personnummer) that was collected from PIRLS students, a number of register variables can be related to their results in PIRLS and ePIRLS. The first subject marks are collected in grade 6, thus for PIRLS students in 2018. Compulsory school in Sweden is 9 years and it is divided into three periods: grades 1-3, 4-6 and 7-9. The evaluating system up to grade 6 is mostly descriptive. By the end of fall semester in grade 6, students are appointed with grades according to their performance in different courses. There are 5 passing grades (A-E), while F indicates that the student has not reached the lowest benchmark for a passing grade. The courses that students attend are 16 (Swedish, mathematics, english, biology, physics, chemistry, technology, geography, history, religion, social studies, visual arts, music, handicrafts, physical education and home economics). Out of them Swedish, mathematics and English are the core subjects. Parental education is used as an indicator of SES and language use at home is used as an indicator of immigrant background.
We will conduct hierarchical regression analysis to investigate if and to what extend can the difference between PIRLS and ePIRLS scores predict the overall marks and the core subject marks.
Conclusions, Expected Outcomes or FindingsSwedish students had an average of 559 points on ePIRLS and 555 points on PIRLS. We calculated correlations which indicated a strong -yet not perfect- relationship between paper-based reading and digital reading, about .80. This suggests that most of those who perform well in one format also perform well in the other. The average score in Swedish language was 14 (0-20), 13.4 (0-20) in maths, 15 (0-20) in English and 205 (0-320) in GPA. In the first step, we calculated the difference between the two test scores (PVepirls –PVpirls). The correlation between the difference of the two tests and Swedish language grade was not significant, while the correlation between the difference and math grades and English language grades was moderate (.52 and .68 respectively) and weak to the GPA (.35).
Preliminary results showed that there is a relationship between grades in mathematics and English language and the difference between PIRLS and ePIRLS scores, indicating that students that scored better in the digital assessment achieved higher marks in mathematics and English language. However, the effect on the Swedish or the overall marks was not statistically significant. Having in mind the considerable predominance of English language online (Meurant, 2009), the present results comes in agreement with previous research indicating the relationship between digital literacy and English language (Alakrash et al., 2021). The present study also confirms previous research that found a pattern of digital literacy related with achievement in mathematics (Hu et al., 2018; Skryabin et al., 2015). Nevertheless, the relationship between digital literacy and Swedish language that was found not significant needs further exploration. A possible explanation can be the fact that Swedish language teachers do not focus on digital literacy enough, as well as the fact that the most common language online is by far English.
ReferencesAlakrash, H., Razak, N. A., & Krish, P. (2021). Social network sites in learning english; an invstigation on attitudes, digital literacy and usage. Linguistica Antverpiensia, 2021(1), 26–43.
Baron, N. S. (2017). Reading in a digital age. Phi Delta Kappan, 99(2), 15–20.
Butler, S. R., Marsh, H. W., Sheppard, M. J., & Sheppard, J. L. (n.d.). Seven-Year Longitudinal Study of the Early Prediction of Reading Achievement.
Chall, J. S. (1989). “Learning to Read: The Great Debate” 20 Years Later: A Response to ‘Debunking
the Great Phonics Myth.’ The Phi Delta Kappan, 70(7), 521–538.
http://www.jstor.org/stable/20403953
Clinton, V. (2019). Reading from paper compared to screens: A systematic review and meta‐analysis. Journal of Research in Reading, 42(2), 288–325. https://doi.org/10.1111/1467-9817.12269
Coiro, J. (Ed.). (2008). Handbook of research on new literacies. Lawrence Erlbaum Associates/Taylor & Francis Group.
Delgado, P., Vargas, C., Ackerman, R., & Salmerón, L. (2018). Don’t throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension. Educational Research Review, 25, 23–38. https://doi.org/10.1016/j.edurev.2018.09.003
Gough, P. B., & Tunmer, W. E. (1986). Decoding, Reading, and Reading Disability. Remedial and Special Education, 7(1), 6–10. https://doi.org/10.1177/074193258600700104
Hu, X., Gong, Y., Lai, C., & Leung, F. K. S. (2018). The relationship between ICT and student literacy in mathematics, reading, and science across 44 countries: A multilevel analysis. Computers and Education, 125, 1–13.
Leu, D. J., O’Byrne, W. I., Zawilinski, L., McVerry, J. G., & Everett-Cacopardo, H. (2009). Comments on Greenhow, Robelia, and Hughes: Expanding the New Literacies Conversation. Educational Researcher, 38(4), 264–269. https://doi.org/10.3102/0013189X09336676
Meurant, R. C. (2009). The Significance of Second Language Digital Literacy Why English-Language Digital Literacy Skills Should be Fostered in Korea. 2009 Fourth International Conference on Computer Sciences and Convergence Information Technology, 369–374. https://doi.org/10.1109/ICCIT.2009.192
Mullis, I. V. S., Martin, M. O., & Sainsbury, M. (2016). PIRLS 2016 Reading Framework.
Sefton-Green, J., Marsh, J., Erstad, O., & Flewitt, R. (n.d.). Establishing a Research Agenda for the Digital Literacy Practices of Young Children. 37.
Skolverket (2016). Att läsa och förstå. Stockholm: Skolverket.
Skryabin, M., Zhang, J., Liu, L., & Zhang, D. (2015). How the ICT development level and usage influence student achievement in reading, mathematics, and science. Computers and Education, 85, 49–58.
Sparks, R. L., Patton, J., & Murdoch, A. (2014). Early reading success and its relationship to reading achievement and reading volume: Replication of ‘10 years later’. Reading and Writing, 27(1), 189–211. https://doi.org/10.1007/s11145-013-9439-2
|
