09. Assessment, Evaluation, Testing and Measurement
Paper
Do COVID-19 Infections Have Effects on Cognitive Abilities of Primary School Students? Results of a Representative Study in Burgenland, Austria
Wolfram Rollett2, Thomas Leitgeb1, Katja Scharenberg3
1University College of Teacher Education Burgenland, Austria; 2University of Oldenburg, Germany; 3Ludwig-Maximilians University München, Germany
Presenting Author: Rollett, Wolfram;
Leitgeb, Thomas
The impact of the COVID-19 pandemic on children's and adolescents' development is a topic that has been intensively studied in recent educational research. However, the focus is often on the consequences that school closures and class cancellations had for students (e.g., Betthäuser, Bach-Mortensen, & Engzell, 2023; Patrinos, Vegas, & Carter-Rau, 2022). The consequences that COVID-19 infections can have on affected children and adolescents have been described primarily in clinical studies. Although younger individuals are less likely to have symptomatic infections or severe infections, they may experience symptomatic consequences that are persistent even after recovery (Behnood et al., 2022; Lopez-Leon et al., 2022). The severity of persistent consequences has been linked to the severity of symptomatology during illness (e.g., Radtke, Ulyte, Puhan, & Kriemle, 2021). For adults, such associations have already been empirically demonstrated regarding cognitive impairments (Hampshire et al., 2021). Although many clinical studies examined the consequences of COVID-19 infections in children, there is a lack of studies presenting results that are representative of specific subpopulations and that allow comparisons of groups of children that have already recovered from COVID-19 with groups that have not yet been infected. The present study addresses these research desiderata and examines the question of whether primary school children who have recovered from COVID-19 show disadvantages in terms of their cognitive abilities.
Methodology, Methods, Research Instruments or Sources Used1,761 second- and third-grade students (49.9% girls, 50.1% boys) in the federal state of Burgenland, Austria, who had parental consent to participate were examined in June 2022 (32.3% of the population of students in the school year 2021/22, from 106 (63.1%) of the 168 elementary schools in Burgenland). In addition, students’ parents or legal guardians were surveyed (n=1,438).
The key independent variable was whether the children had been infected with COVID-19 at the time of the survey. According to parents and students, this was the case for n=1,253 students, whereas n=508 students had not been infected by that time. In addition, we assessed whether the infection was symptomatic or asymptomatic, which symptoms occurred, and whether a physician was consulted due to the COVID-19 infection. In addition, characteristics of students' individual and family background were surveyed (including gender, language spoken at home, native language, parents' country of birth, parental education, etc.). We used standardized instruments of the federal state of Burgenland to weight the gathered data based on state statistics.
As dependent variable, cognitive ability was assessed using the Cognitive Abilities Test (KFT 1-3; Heller & Geisler, 1983) (test duration: 60 minutes), which consists of four subtests: language comprehension, relation recognition, inductive reasoning and numerical thinking.
For analysis, four groups were distinguished: children who had not been infected at the time of the survey (control group, n=502) and three recovery groups (RG): asymptomatically infected children (RG1, n=251), symptomatically infected children (RG2, n=850), and symptomatically infected children who had seen a medical doctor because of the illness (RG3, n=131). The doctor's visit is considered as an indicator of a situation that gave the parents reasons for concern. According to parents, 78 percent of recovered children had been infected with COVID-19 within the five months prior to data collection.
The data of the three recovery groups were compared pairwise with those of the control group. Since small, but significant differences were found between the groups regarding immigrant background, native language and language spoken at home, an analysis of covariance was conducted controlling for these variables. Missing values were treated as Missing at Random and were multiply imputed (MICE, Buuren & Groothuis-Oudshoorn, 2011; CART Breiman, et al, 1984). Data were weighted using iterative proportional fitting (IPF; Deming & Stephan, 1940; Lomax & Norman, 2019) based on representative statistics from the federal state of Burgenland. All statistical tests were conducted with an error probability of p<.05.
Conclusions, Expected Outcomes or FindingsRegarding cognitive abilities, RG1 and RG3 showed a significantly lower test performance in numerical reasoning than the control group (RG1: F(1,748)=7.42**, p=.007, partial Eta²=.010; RG3: F(1,627)=9.18**, p=.003, partial Eta²=.014). Moreover, RG3 also performed significantly lower in language comprehension than the control group (F(1,627)=11.26***, p<.001, partial Eta²=.018). For relation recognition and inductive reasoning, RG3 performed, in tendency, lower than the control group (F(1,627)=3.57, p=.059, partial Eta²=.006; F(1,637)=3.19, p=.075, partial Eta²=.005).
Our findings suggest negative cognitive effects of COVID-19 infections for two of the recovery groups distinguished in the present study. For the recovery group of symptomatically infected children who underwent medical treatment, the findings point more strongly into this direction. The identified effects are of small size. However, given the low prevalence of longer-lasting symptoms after the infection among children (Lopez-Leon et al., 2022), these effects may imply severe consequences for the cognitive functioning of the respective children. Further analyses using propensity score matching are planned to validate our findings obtained by covariance analysis. Beyond this, it has to be considered that the effects reported here emerged at a time when, for most children, the infection happened only a few weeks or months before the survey. The extent to which these effects persist is another important question. Therefore, our sample was re-assessed in June 2023 using the same test instrument to assess students' cognitive abilities. The results of this follow-up study will be available by spring 2024 and will be included in our paper. The findings will be discussed with reference to the medical research literature as a consequence of the impaired central functions (memory, attention) and with regard to consequences for targeted educational support of children after their COVID-19 infections.
ReferencesBehnood, S. A., Shafran, R., Bennett, S. D., Zhang, A. X. D., O’Mahoney, L. L., Stephenson, T. J., . . . Swann, O. V. (2022). Persistent symptoms following SARS-CoV-2 infection amongst children and young people: A meta-analysis of controlled and uncontrolled studies. Journal of Infection, 84(2), 158–170. https://doi.org/10.1016/j.jinf.2021.11.011
Betthäuser, B. A., Bach-Mortensen, A. M., & Engzell, P. (2023). A systematic review and meta-analysis of the evidence on learning during the COVID-19 pandemic. Nature Human Behaviour, 7(3), 375–385. https://doi.org/10.1038/s41562-022-01506-4
Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984). CART: Classification and Regression Trees. Belmont, CA: Wadsworth.
Buuren, S. van, & Groothuis-Oudshoorn, K. (2011). mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45(3), 1–67. https://doi.org/10.18637/jss.v045.i03
Deming, W. E., & Stephan, F. F. (1940). On a least squares adjustment of a sampled frequency table when the expected marginal totals are known. The Annals of Mathematical Statistics, 11(4), 427–444. https://doi.org/10.1214/aoms/1177731829
Hampshire, A., Trender, W., Chamberlain, S. R., Jolly, A. E., Grant, J. E., Patrick, F., . . . Mehta, M. A. (2021). Cognitive deficits in people who have recovered from COVID-19. EClinicalMedicine, 39, 101044. https://doi.org/10.1016/j.eclinm.2021.101044
Heller, K., & Geisler, H. J. (1983). Kognitiver Fähigkeitstest (Grundschulform). KFT 1–3. Weinheim: Beltz.
Lomax, N., & Norman, P. (2016). Estimating population attribute values in a table: “Get me started in” Iterative Proportional Fitting. The Professional Geographer, 68(3), 451–461. https://doi.org/10.1080/00330124.2015.1099449
Lopez-Leon, S., Wegman-Ostrosky, T., Ayuzo del Valle, N. C., Perelman, C., Sepulveda, R., Rebolledo, P. A., . . . Villapol, S. (2022). Long-COVID in children and adolescents: A systematic review and meta-analyses. Scientific Reports, 12(1), 9950. https://doi.org/10.1038/s41598-022-13495-5
Patrinos, H. A., Vegas, E., & Carter-Rau, R. (2022). An analysis of COVID-19 student learning loss. The World Bank. https://doi.org/10.1596/1813-9450-10033
Radtke, T., Ulyte, A., Puhan, M. A., & Kriemler, S. (2021). Long-term symptoms after SARS-CoV-2 infection in children and adolescents. JAMA, 326(9), 869–871. https://doi.org/10.1001/jama.2021.11880
09. Assessment, Evaluation, Testing and Measurement
Paper
School Environments Pre- and Post- Pandemic: Exploring the Irish Context Using TIMSS and PIRLS Data
Sarah McAteer, Brendan O'Neill, Vasiliki Pitsia, Grainne McHugh, Emer Delaney, Aidan Clerkin
Educational Research Centre, Ireland
Presenting Author: McAteer, Sarah;
O'Neill, Brendan
The influence of the school environment on pupils’ educational outcomes has long been established (Kutsyuruba et al., 2015; Mullis et al., 2013). Having a safe, structured and encouraging learning environment is associated with higher achievement and improved wellbeing (Cohen et al., 2009; Mullis et al., 2019; Thapa et al., 2013). Therefore, research on the school environment is important as it can have practical implications for educational policy. International large-scale assessments such as the Trends in International Mathematics and Science Study (TIMSS) and the Progress in International Reading Literacy Study (PIRLS), which are based on nationally representative samples of pupils at the target grade at the time of the assessment, allow researchers to examine aspects of the school environment from different perspectives. Factors such as school climate and school safety and discipline can be examined in both studies. The study cycles that are of particular focus in this paper are TIMSS 2019 and PIRLS 2021. These cycles can be seen as bookending the 2019/20 and 2020/21 academic years, during which extended periods of nationwide school closures occurred in Ireland as a result of the COVID-19 pandemic. These closures resulted in disruption to in-person teaching and learning and a transition to remote learning, which could potentially have impacted the school environment in the longer term.
Due to the unprecedented disruption in education that occurred between the administrations of TIMSS 2019 and PIRLS 2021, these data, stemming from school principals, class teachers, pupils, and parents/guardians, present a key opportunity to examine whether school environments in Ireland differed substantially between these time points. While we cannot infer causation when comparing cross-sectional datasets such as these, the nationally representative findings may help us to better understand the school landscape in the wake of the nationwide closures.
Methodology, Methods, Research Instruments or Sources UsedThis analysis uses data from two studies: TIMSS 2019 and PIRLS 2021. Each study involved a representative sample of pupils in Ireland for the year the study was conducted, with 4,582 pupils in 150 schools taking part in TIMSS 2019 and 4,663 pupils in 148 schools taking part in PIRLS 2021. For TIMSS, pupils in Grade 4 were assessed on mathematics and science, while for PIRLS, pupils at the start of Grade 5 were assessed on reading literacy. In PIRLS 2021, the decision was made in Ireland (and 13 other countries) to move from spring to autumn testing because of the nationwide closures in the academic year 2020/21; therefore, pupils who participated in PIRLS in 2021 were approximately six months older than those who participated in TIMSS 2019. Context questionnaires were completed by participating pupils, their parents/guardians, school principals, and class teachers.
Data on questionnaire items relating to school climate and school safety and discipline that were common to both the TIMSS 2019 and PIRLS 2021 assessments, along with pupils’ home resources for learning (as a proxy for socioeconomic status) and achievement, were analysed. School climate indices included parents’ perceptions of their child’s school, schools’ emphasis on academic success, teacher job satisfaction, and pupils’ sense of belonging at school. School safety and discipline indices included school discipline, school safety and order, and bullying.
The analysis was conducted in three phases using the International Association for the Evaluation of Educational Achievement (IEA) International Database Analyzer (IDB Analyzer) (IEA, 2023). Initially, individual items comprising each index were examined. Secondly, the relationship of the indices with achievement was examined (mathematics and science for TIMSS and reading for PIRLS) with a follow-up analysis that also took pupils’ home resources for learning into account. Finally, hierarchical linear regression models were constructed to examine the extent to which the indices of interest explained achievement in each subject. In each instance, two models were tested: first, a model with only the school environment indices, and second, a model that included both the school environment indices and the home resources for learning index. The use of the IEA IDB Analyzer allowed for the adjustment of regression estimates for sampling error due to the clustered sampling design of TIMSS and PIRLS via the use of the replicate weights.
Conclusions, Expected Outcomes or FindingsResults point to a picture of overall stability in the school environments in Ireland between TIMSS 2019 and PIRLS 2021. In terms of school climate, the proportion of parents who were very satisfied with their child’s school remained high, ranging from 77% in 2019 to 80% in 2021. Fewer pupils in 2021 had teachers who reported that their school placed a very high or high emphasis on academic success, but, these differences were slight. Also, teacher job satisfaction was largely stable between 2019 and 2021. For example, at index level, over half of pupils were taught by teachers who reported being very satisfied in both studies, while the proportion whose teachers were less than satisfied remained small (10% in 2019 and 8% in 2021).There was a small decrease in the proportion of pupils whose teachers reported being often content with their profession as a teacher and those whose teachers very often found their work full of meaning and purpose. School safety and discipline was also relatively unchanged in the bullying and the safe and orderly school indices.
In the regression models, more frequent bullying was associated with lower achievement even after home resources for learning were accounted for. Higher sense of school belonging was associated with higher achievement in all subjects when only school environment indices were included. However, in 2019 it was not significant after home resources for learning were accounted for, whereas it remained significant after they were accounted for in 2021. This may suggest an increased importance of school belonging for other student outcomes post-pandemic, which should be monitored and examined further. Overall, the stability observed in relation to the school environment pre- and post-pandemic may be viewed as positive considering the significant disruption and challenges brought on by the pandemic and associated school closures in Ireland.
ReferencesCohen, J., McCabe, E. M., Michelli, N. M., & Pickeral, T. (2009). School climate: Research, policy, practice, and teacher education. Teachers College Record, 111(1), 180–213. https://doi.org/10.1177/016146810911100108
IEA. (2023). Help manual for the IEA IDB Analyzer (Version 5.0). https://www.iea.nl
Kutsyuruba, B., Klinger, D. A., & Hussain, A. (2015). Relationships among school climate, school safety, and student achievement and well-being: A review of the literature. Review of Education, 3(2), 103–135. https://doi.org/10.1002/rev3.3043
Mullis, I. V. S., Martin, M. O., & Foy, P. (2013). The impact of reading ability on TIMSS mathematics and science achievement at the fourth grade: An analysis by item reading demands. In M. O. Martin & I. V. S. Mullis (Eds.), TIMSS and PIRLS 2011: Relationships among reading, mathematics, and science achievement at the fourth grade—Implications for early learning (pp. 67–108). TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College, and International Association for the Evaluation of Educational Achievement (IEA).
Mullis, I. V. S., & Martin, M. O. (Eds.). (2019). PIRLS 2021 assessment frameworks. TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College, and International Association for the Evaluation of Educational Achievement (IEA).
Thapa, A., Cohen, J., Guffey, S., & Higgins-D’Alessandro, A. (2013). A review of school climate research. Review of Educational Research, 83(3), 357–385. https://doi.org/10.3102/0034654313483907
09. Assessment, Evaluation, Testing and Measurement
Paper
Children At Risk: Association Between PIRLS Reading Achievement and Student Well-Being in Finland
Timo Salminen, Juhani Rautopuro, Mikko Niilo-Rämä
University of Jyväskylä, Finland
Presenting Author: Salminen, Timo;
Rautopuro, Juhani
Being able to read can be seen as the foundation of a functioning democracy enabling learning, equal participation in society, and a condition for a healthy and successful life (European Commission, 2023; EDUFI, 2021). Reading performance is closely linked with other areas of academic performance, and there is a strong association between student well-being in school and reading performance (European Commission, 2023). Moreover, the danger of failing to meet academic or social expectations or to complete school with a basic level of academic proficiency has been termed “at-risk” (e.g. Novosel et al., 2012).
In Finland, the trends in students’ academic well-being (e.g. Helenius & Kivimäki, 2023; Read et al., 2022) and learning performance (e.g. Mullis et al., 2023; OECD, 2023) have been descending in the last decade. For example, Grade 4 students’ performance in reading has decreased from 2011 to 2021 as evidenced by the Progress in International Reading Literacy Study (PIRLS) (Mullis et al., 2023). The performance in reading declined by two points from 2011 to 2016 and by 17 points from 2016 to 2021. When examining the international reading benchmarks, the percentage of advanced achievers has dropped from 18% to 14% during this period. Meanwhile, the percentage of students at the low or below benchmark has doubled from 8% to 16%. We define these students as “students at risk”. They are in danger of not achieving adequate reading proficiency which is crucial for their learning success or failure in subsequent school years.
As for student well-being, the latest School Health Promotion Study (Helenius & Kivimäki, 2023) shows that more than one third of girls and one in five boys felt that their health was average or poor in Finland. The study also reports that experiences of physical threats and bullying have recently increased. Furthermore, school burnout has increased for a long time, especially among girls (Read et al., 2022).
Student well-being in school can be considered as a condition that enables positive learning outcomes but also as an outcome of successful learning and students’ satisfaction with their school experiences (Morinaj & Hascher, 2022). Student well-being in school consists of positive attitudes to school, enjoyment in school, positive academic self-concept, the absence of worries, physical complaints, and social problems in school, which can be used as indicators of well-being (Hascher, 2003).
In PIRLS, student well-being is indirectly measured by several indicators – such as school belonging, academic self-concept, experience of bullying, and absenteeism (Reynolds et al., 2024). PIRLS 2021 data was collected during the COVID-19 pandemic, but then there were no school closures in Finland. However, one year before these students’ schooling was disrupted, and they spent eight weeks in distance learning. Lerkkanen et al. (2022) showed that the Finnish students’ development in reading was slower from Grade 2 to 4 in the COVID sample compared to the pre-COVID sample. Previous research has detected the association between student well-being and learning performance but also the need for further examining this relation and the role of other factors associated with reading achievement, e.g. socioeconomic background (e.g. Bücker et al., 2018; Nilsen et al., 2022). For example, Manu et al. (2023) focused on the role of gender and parental education, and Torppa et al. (2022) the effects of the home literacy environment on the development of Finnish children’s reading comprehension.
In this study, we ask the following research questions, using the PIRLS reading assessment data from 2011 to 2021:
1) How has students’ well-being in school changed, if any, from 2011 to 2021?
2) How do students’ socioeconomic background and well-being factors predict the risk of low academic achievement in reading?
Methodology, Methods, Research Instruments or Sources UsedThe present study is based on the three cycles of curriculum-based PIRLS assessment in Finland. The data includes the 4th graders who participated in PIRLS 2011 (N = 4,640), PIRLS 2016 (N = 4,896), and PIRLS 2021 (N = 7,018). In this study, we use school climate and safety, students’ attitudes, and absenteeism as indicators of well-being. School climate and safety include the scales of Students’ Sense of School Belonging (3 items) and Bullying (6 items). Students’ attitudes include the scales of Students Like Reading (5 items) and Students Confident in Reading (7 items). These four-point scales are from PIRLS student questionnaires. From each scale, we selected those items that were the same in all three cycles of PIRLS assessment. Absenteeism was asked of students (in years 2016 and 2021, not asked in 2011) by a single item reporting how often they are absent from school. As an indicator of student’s socioeconomic background, we used Home Socioeconomic Status and Home Resources for Learning scales, and Parents’ Educational Level separately.
The data was analysed by using various statistical methods. To answer the second research question, binary logistic regression analysis was applied. The low achievement benchmark (cut point 474) was used as a binary response. Students’ socioeconomic background and well-being factors were used as explanatory variables. This analysis was conducted separately for each of the three PIRLS data sets. Five plausible values representing students’ proficiency in reading (see von Davier et al., 2023) were used in the analyses. A two-stage sampling design used in the PIRLS assessment (von Davier et al., 2023) was considered in the analyses.
Conclusions, Expected Outcomes or FindingsOverall, the Finnish 4th grade students’ well-being was relatively good. Examination of the trends of means showed that there are some changes in students’ well-being from 2011 to 2021. After 2011, students’ sense of school belonging increased, and bullying first decreased from 2011 to 2016 but increased again from 2016 to 2021. From 2011 to 2021, both students liking reading and confidence in reading decreased.
The preliminary results of logistic regression showed that there were significant associations between bullying, student confident in reading, student socioeconomic background, parents’ educational level, absenteeism, gender, and low achievement in reading.
In all three cycles of PIRLS (2011, 2016, and 2021), the predictive factors for the risk of low academic achievement in reading were the students’ low degree of confidence in their own reading ability, lower socioeconomic background, parents’ low educational level (in 2021 even below higher education), and gender (boy). In PIRLS 2016 and 2021 datasets, the frequency of absences from school (once a week) also predicts the risk of low academic achievement in reading. Being subjected to bullying about weekly was a risk factor in PIRLS 2021 dataset.
When identifying at-risk students in reading, the results suggest that family background, especially the educational background of parents, has become more important, as has bullying. In Finland, however, about 5% of the students experienced bullying about weekly. In addition, the students’ confidence in their own reading ability seems to be a strong predictor of reading achievement. Furthermore, the gender gap in reading achievement has remained rather large favouring girls for a long time in Finland. It also seems that the factors predicting the risk of low academic achievement in reading are linked to each other. This study supports earlier research on the meaning of students’ well-being and socioeconomic background to learning.
ReferencesBücker, S., et al. (2018). Subjective well-being and academic achievement: A meta-analysis. Journal of Research in Personality, 74, 83–94.
EDUFI. (2023). National Literacy Strategy 2030: Finland - the most multiliterate country in the world in 2030. Finnish National Agency for Education. https://www.oph.fi/sites/default/files/documents/National_literacy_strategy_2030.pdf
European Commission. (2023). Children’s reading competence and well-being in the EU – An EU comparative analysis of the PIRLS results. https://data.europa.eu/doi/10.2766/820665
Hascher, T. (2003). Well-being in school – why students need social support. In P. Mayring & C. von Rhöneck (Eds.), Learning emotions – the influence of affective factors on classroom learning (pp. 127–142). Bern u.a Lang.
Helenius, J., & Kivimäki, H. (2023). Well-being of children and young people – School Health Promotion study 2023. Finnish Institute for Health and Welfare, Statistical Report 50/2023. https://urn.fi/URN:NBN:fi-fe20230913124233
Lerkkanen, M.-K., et al. (2022). Reading and math skills development among Finnish primary school children before and after COVID-19 school closure. Reading and Writing, 36, 263–288.
Manu, M., et al. (2023). Reading development from kindergarten to age 18: The role of gender and parental education. Reading Research Quarterly, 58(4), 505-538.
Morinaj, J., & Hascher, T. (2022). On the relationship between student well-being and academic achievement: A longitudinal study among secondary school students in Switzerland. Zeitschrift für Psychologie, 230(3), 201–214.
Mullis, I. V. S., et al. (2023). PIRLS 2021 International Results in Reading. Boston College, TIMSS & PIRLS International Study Center. https://doi.org/10.6017/lse.tpisc.tr2103.kb5342
Nilsen, T., Kaarstein, H., & Lehre, A. C. (2022). Trend analyses of TIMSS 2015 and 2019: school factors related to declining performance in mathematics. Large-scale Assessments in Education, 10(1), 1–19.
Novosel, L., et al. (2012). At-risk learners. In N. M. Seel (Ed.), Encyclopedia of the science of learning (pp. 348–350). Springer.
OECD. (2023). PISA 2022 Results (Volume I): The State of Learning and Equity in Education, PISA, OECD Publishing, Paris. https://doi.org/10.1787/53f23881-en
Read, S., Hietajärvi, L. & Salmela-Aro, K. (2022). School burnout trends and sociodemographic factors in Finland 2006–2019. Social Psychiatry and Psychiatric Epidemiology, 57, 1659–1669.
Reynolds, K.A., et al. (2024). Aspects of student well-being and reading achievement in PIRLS 2021 (PIRLS Insights). Boston College, TIMSS & PIRLS International Study Center.
Torppa, M., et al. (2022). Long-term effects of the home literacy environment on reading development: Familial risk for dyslexia as a moderator. Journal of Experimental Child Psychology, 215, Article 105314.
von Davier, M., et al. (Eds.). (2023). Methods and Procedures: PIRLS 2021 Technical Report. Boston College, TIMSS & PIRLS International Study Center. https://pirls2021.org/methods
|
