09. Assessment, Evaluation, Testing and Measurement
Paper
EPIC - Education Preparedness Index in COVID-19: Methodology and Research
Arusyak Aleksanyan1, Mariam Muradyan2, Anna Malkhasyan3, Anna Arustamyan4, Narek Yenokyan5, Arayik Tsaturyan6
1YSU, Armenia; 2YSU, Armenia; 3World Bank, Armenia; 4Teach for Armenia; 5Armenian Lawyer's Association, NGO; 6KPMG Armenia
Presenting Author: Aleksanyan, Arusyak
The Education Preparedness Index in Covid-19 (EPIC) is one of the outputs of the project Enabling Learning to Happen for All Children in Emergency Crisis. The project is funded by the Global Campus of Human Rights in partnership with the Right Livelihood Award Foundation. In 2019, people all over the world were faced with new realities and adopted new rules of life. One of those realities also affected the education system. Due to the COVID-19 pandemic, many countries in the world initiated emergency remote education applications and platforms to continue providing education without interruption, aiming for students to continue their learning. Under the crisis of COVID-19 remote/distance learning became a viable alternative to ensure the continuity of the educational process. Under these circumstances, the research aimed at studying the preparedness of the education system to adapt to new realities and to act in crisis conditions. To this end, a group of Armenian experts took the initiative to develop a model for assessing the education system in emergencies - Education Preparedness Index in COVID-19 (EPIC). EPIC assessment of preparedness for education in emergencies is a set of indicators, tools, and methods aimed at measuring education system readiness for emergencies and analyzing the effectiveness of education policy responses in times of crisis. The four main thematic areas that the framework covers are as follows:
• Policy and Legal Framework
• Coordination and Cooperation
• E-readiness
• Capacities and Resources
Each thematic area incorporates a set of indicators and sub-indicators that allow uncovering the level of achievements and the efficiency level of preparedness within each recommended section.
EPIC is applicable in all emergencies entailing physical distancing and education through online means. The emergency context was retrieved from the conditions and limitations appeared through COVID-19 period combined with other crises such as war, internal unrest, context of disability and some other characteristics. The framework is flexible to changes of individual country cases and these are specific characteristics are considered during the assessment.
The research question of this study is as follows:
What is the level of preparedness of the country/countries in provision of education in emergency entailing rapid shift from conventional education to distant/ online education?
EPIC basis on the international principles of child rights. Among them the framework of 4As, Availability, Accessibility, Acceptability and Adaptability of Education of the UN Human Rights Office of the High Commissioner is the corner stone of the study. The methodology is based on the UN CRC Commentary N13 description and statements of the 4As framework. Furthermore, the EPIC grounds its target group that primarily on children of 6-14 age group following the CRC General Comments N13, World Declaration on Education for All, as well as on SDGs, the Minimum Standards for Education in Emergencies, the Global Study of Children Deprived of Liberty (Nowak, 2019), Chronic Crises and Early Reconstruction (2004). the Dakar Education for All (EFA) framework, and the Sphere Project’s Humanitarian Charter.
Methodology, Methods, Research Instruments or Sources UsedTo achieve the research goals, the triangulation strategy was employed. The purpose of the triangulation approach is to use diverse methods to assist each other in explaining and interpreting the data. Thus, the calculation and analysis of the index are based on three data sources: 1. Quantitative survey of teachers and students; 2. Expert interview; 3. Statistical data.
1. The quantitative research includes conducting a representative quantitative survey among students and teachers of secondary schools to assess and calculate the e-Readiness sub-index. The survey covers the following three thematic areas, indicating the readiness of schools for distance education:
• Technological readiness for distance learning
• Social-psychological readiness for e-learning
• Cognitive readiness for online education
Each of the above-mentioned areas has its sub-indicators and a series of relevant questions for teachers and students. We adopted an online standardized questionnaire hosted on Google, the link to which is provided to school administrators who further ensured the dissemination of the data among students and teachers via private messaging systems. The questionnaire includes questions, mainly designed by using a Likert scale. On average, it took the respondents 20 minutes to complete the survey. The responses to these questions were initially collected by the Google drive excel database. After fully completing the survey, all answers were directly exported into an SPSS file and analyzed.
2. As a qualitative method, interviews with experts was distinguished. The experts were involved based on the relevant experience in policy/strategy developing experience, skills in implementation and monitoring. The expert interview questionnaire covers questions related to the existing regulations and policies, capacities, coordination of involved parties, technological availability, and delivered models of e-learning in an emergency. To this end, interviews with representatives of the Government, independent experts, involved CSOs and international organizations were conducted. An important approach is the application of the saturation method when the number of experts is determined by the collected information.
3. Statistical data collection stage of the study involves the collection of statistical data. In case of missing data for the reporting year, the most recent available data can be collected.
In the final stage, all the data are standardized, on the basis of which the INDEX is calculated on a scale from 0 to 100, where 0 is the lowest level of preparedness and 100 is the highest.
Conclusions, Expected Outcomes or FindingsFinally, an emergency situation such as the Covid-19 pandemic has raised awareness of the need for schools and education systems to be prepared for different emergencies. This global crisis made it clear that countries should develop and have a coping strategy for mitigating the adverse impact of the pandemic as well as identify and provide additional support to the most vulnerable groups. This challenge is an opportunity for those schools not having a strategy for emergency situations to develop one and to use it during such eventualities. School closures have shown that online teaching and learning preparedness is not only a trend but also a must to achieve success in the educational process. The effectiveness of distance learning and educational process in general mostly depend on the relevant professional-pedagogical skills of the teaching community, the willingness of teachers to constantly improve and develop, on teachers’ creative thinking, motivation to teach at school, etc. Different international studies have shown that in order to have an effective education system, it is important to have a highly qualified pedagogical community. The effectiveness of the education system is largely measured by the achievements of the students. And the achievements of the students significantly depend on the high professional and pedagogical skills and capacities of teachers. Thus, education systems successfully meet the challenges of emergencies if they regularly evaluate and monitor the system's preparedness for emergencies.
References1.Bensalah, Kacem. 2002. “Guidelines for education in situations of emergency and crisis: EFA strategic planning”. UNESCO.
2.Çağatay, İhsan Ulus. 2020. "Emergency Remote Education vs. Distance Education". European Commission.
3.Chebib, Kinda. 2020. “Education For All in the Time of COVID-19: How EdTech can be Part of the Solution”.
4.Committee on Economic, Social and Cultural Rights. 2020. "Statement on the Coronavirus Disease (COVID-19) Pandemic and Economic, Social and Cultural Rights".
5.Fernando, M. Reimers, Andreas Schleicher. 2020. “A framework to guide an education response to the Covid-19 Pandemic of 2020”. Harvard Graduate school of education.
6.Humanitarian Practice Network. 2006. “Implementing minimum standards for education in emergencies: lessons from Aceh”.
7.INEE (Inter-Agency Network for Education in Emergencies). 2010. “Minimum Standards for Education: Preparedness, Response, Recovery”. Accessed March 8, 2021.
8.INEE. 2004. “Minimum Standards for Education in Emergencies, Chronic Crises and Early Reconstruction”. DS Print.
9.Lasi, Masri bin Abdul. 2021. “Online Distance Learning Perception and Readiness During Covid-19 Outbreak: A Research Review”. International Journal of Academic Research in Progressive Education and Development. 28 February.
10.Nicolai, Susan. 2003. “Education in Emergencies A toolkit for starting and managing education in emergencies”. Save the Children.
11.OECD. 2020. “Education Response to Covid-19: Implementing a Way Forward”. Working Paper No. 224. 9 July.
12.Penna, Maria Pietronilla, Vera Stara. 2007. “The failure of e-learning: why should we use a learner centred design”. Journal of e-Learning and Knowledge Society. January.
13.Phan, Thanh Thi Ngoc, Ly Thi Thao Dang. 2017. “Teacher Readiness for Online Teaching: A Critical Review”. June.
14.UN. 2020. "Policy Brief: Education during COVID-19 and beyond", August.
15.UNESCO. 2016. “Education 2030: Incheon Declaration and Framework for Action for the implementation of Sustainable Development Goal 4: Ensure inclusive and equitable quality education and promote lifelong learning”.
16.UNESCO. 2020b. “COVID-19 Education Response, How Many Students are at Risk of not Returning to School”. Advocacy paper, 30 July.
17.UNESCO. 2020c. “Covid-19 Education Response. Education Sector Issue Notes. Supporting teachers and education personnel during times of crisis”. Issue note no. 2.2, April.
18.UNESCO. 2020а. “COVID-19 Education Response, Distance learning strategies in response to COVID-19 school closures”. Issue note no. 2.1, April.
19. UNICEF. 2020. "Education and COVID-19 report".
20.World Bank. 2021. “Urgent, Effective Action Required to Quell the Impact of COVID-19 on Education Worldwide”.
09. Assessment, Evaluation, Testing and Measurement
Paper
Assessing Distance Learning in Primary Education of Kazakhstan during the COVID-19 Pandemic: Evidence from PIRLS-2021
Nazym Smanova
JSC “Information-Analytical Center”, Kazakhstan
Presenting Author: Smanova, Nazym
The COVID-19 pandemic has led to near-universal closing of schools at all levels worldwide, remaining negative consequences for all participants in the educational process. School support plays a key role in mitigating the negative effect of school-closure during the pandemic period on student learning achievement. The present study assesses the possible mediating role of school support in the effects of COVID-19 disruption on primaty students learning achievement through analysis of data collected through the IEA’s Progress in International Reading Literacy Study (PIRLS 2021).
In Kazakhstan, as in many countries, regular schooling was disrupted since the COVID-19 pandemic broke out in March 2020. In 2021, 77% of all school students started the new academic year via distance learning. In response to the pandemic, the government of Kazakhstan undertook set of systemic measures: distance learning was implemented using online platforms and services, as well as using audio and telework; computer equipment and Internet cards were presented freely to students in need; professional development courses on distance learning were offered for about 347 thousand teachers.
So far, some studies have pointed to significant losses in students' knowledge in Kazakhstan during school-closures led by the pandemic (IAC, 2020; Dzhaksylykov, 2020). Researchers in the USA found that many children in Year 2 and Year 3 in their study lost momentum on fundamental skills such as reading, with the difficulty in creating a language-rich environment on Zoom being one of the primary reasons (Domingue et al., 2021). In another study conducted in the UK, attainment gaps were found for both Year 1 and Year 2 students, with the most profound effects on students from a disadvantaged background (Rose et al., 2021).
The following research questions will guide the study:
- How is the school closure during the pandemic period resulted in reading achievement of Kazakhstani primary grade students?
- Does school support (providing access to digital devices, delivering printed and online learning materials, organizing online activities, providing technical and methodological support for teachers) mediate the relationship between school closure due to COVID-19 and reading achievement?
- Does the mediation effect of school support vary across different socio-demographic groups?
Methodology, Methods, Research Instruments or Sources UsedThe quantitative research method will be employed to evaluate the impact of the selected parameters on student performance by using multilevel modelling techniques. The data for this study will be from the Kazakhstan sample in the IEA PIRLS 2021 database. PIRLS 2021 is an only international large-scale assessment conducted during the COVID-19 school disruption. It provides contextual information about how remote instruction was organized in schools, including information about distance learning resources available for students, methodological and technical support for teachers, etc.
Conclusions, Expected Outcomes or FindingsAs a result, it is expected to find out how school resources could mediate the relationship between school closure and academic performance across SES groups using the national questionnaire data from PIRLS 2021. The study will inform an ongoing process of developing an effective mechanism for designing and implementing the educational recovery program in Kazakhstan.
References1. JSC Information and Analytical Center. (2020). Analytical report on the monitoring of learning using distance technologies in general schools in the framework of emergency distance learning in Kazakhstan [Unpublished research]. https://iac.kz/
2. Bokayev, B., Torebekova, Z., Abdykalikova, M., & Davletbayeva, Z. (2021). Exposing policy gaps: The experience of Kazakhstan in implementing distance learning during the COVID-19 pandemic. Transforming Government: People, Process and Policy, 15(2). Retrieved from https://doi.org/10.1108/TG-07-2020-0147
3. Dzhaksylykov, S. (2020). Distance learning diaries: How was the “distance” school term from students and parents’ point of view. Retrieved from https://drive.google.com/file/d/1a-wo91IsG_puveH2mUVCU9XliZIcC8_2/view
4. Domingue, B. W., Hough, H. J., Lang, D., & Yeatman, J. (2021). Changing Patterns of Growth in Oral Reading Fluency during the COVID-19 Pandemic. Policy Analysis for California Education, Working Paper. Retrieved from https://edpolicyinca.org/sites/default/files/2021-03/wp_domingue_mar21-0.pdf
09. Assessment, Evaluation, Testing and Measurement
Paper
Capturing the Educational and Economic Impacts of School Closures in Poland
Tomasz Gajderowicz1, Maciej Jakubowski1, Sylwia Wrona1, Harry Patrinos2
1University of Warsaw, Poland; 2World Bank
Presenting Author: Wrona, Sylwia
COVID-19 led to strict lockdown measures, which included school closures in most countries. As a result, more than 1.5 billion students were out of school for weeks or months (UNESCO, 2022). The loss of schooling is expected to negatively impact children's cognitive development, even if distance learning modes are enacted. The loss of in-person teaching could also lead to inequality since the only remaining relevant input is parental involvement during school closures (Agostinelli et al., 2022). Most studies document significant learning loss. In Europe, the average learning loss is almost a quarter of a school year, but the estimates are available mainly for Western European countries (Donnelly and Patrinos, 2021). Worldwide the loss is even greater, especially in lower-income countries (Patrinos et al., 2022). Poland is an interesting case because it represents countries in Eastern Europe where school closures lasted longer, and research on learning loss is scarce.
Methodology, Methods, Research Instruments or Sources UsedTo properly estimate the effect of school closures and to distinguish it from the effect of the 2016 structural changes, we compare the expected and actual achievement of three cohorts of students in secondary schools. We assume students should gain a minimum of 0.1 standard deviation (SD) during one year of education. That is a safe assumption but in line with previous studies comparing 15, 16, and 17-year-old student results on the PISA scale in Poland (Jakubowski et al., 2022). International evidence indicates the gains should be larger, around 0.2 SD, which would make our
results more significant as they increase the expected achievement (Avvisati and Givord, 2021). We also assume students tested in autumn (10th grade in TICKS 2021) have a similar achievement to those tested one grade below in the spring (9th-grade assessment in PISA 2003-2018). Assuming any achievement progress between spring and autumn makes our results even more significant.
Conclusions, Expected Outcomes or FindingsThe Polish success story of rapid social and economic progress relied strongly on human capital improvement. Unfortunately, this factor is now under significant distress. Significant learning losses have been experienced by Polish students due to the COVID-19-induced school closures. In mathematics and science, the learning losses are equal to more than a year's worth of schooling, even though schools were closed for only part of an academic year. In addition, we show that the
2016 reforms also had a negative impact on student learning. These skills losses are likely to affect the future economic success of the students as well as the country as a whole. Future earnings are projected to decline by PLN 74,693 (more than US$15,000) per year for the affected students. The country would then lose the equivalent of 7.2% of GDP over time.
ReferencesAvvisati, F., Givord, P. (2021). How much do 15-year-olds learn over one year of schooling? An
international comparison based on PISA. OECD Education Working Papers No. 257.
Carlana, M., La Ferrara, E. (2021). Apart But Connected: Online Tutoring and Student Outcomes
During the COVID-19 Pandemic. CEPR Discussion Paper No. DP15761.
Donnelly, R., Patrinos, H. A. (2021). Learning loss during COVID-19: An early systematic review.
Prospects 1-9.
Drucker, L.F., Horn, D. & Jakubowski, M. (2022). The labour market effects of the polish
educational reform of 1999. Journal of Labour Market Research 56, 13.
Fryer Jr, R.G., Howard-Noveck, M. (2020). High-dosage tutoring and reading achievement:
evidence from New York City. Journal of Labor Economics 38(2): 421-452
Hanushek, E. A., & Woessmann, L. (2010). The high cost of low educational performance: The
long-run economic impact of improving PISA outcomes. OECD Publishing, France.
Jakubowski M., Gajderowicz T., Wrona S. (2022). Achievement of Secondary School Students
after Pandemic Lockdown and Structural Reforms of Education System. Evidence Institute
and City of Warsaw research report.
Jakubowski M., Patrinos H., Porta E., Wisniewski J. (2016), The Effects of Delaying Tracking in
Secondary School: Evidence from the 1999 Education Reform in Poland. Education
Economics 24(6).
Patrinos, H.A., Vegas, E., Carter-Rau, R. (2022). An Analysis of COVID-19 Student Learning
Loss. Policy Research Working Paper No. 10033, World Bank.
Psacharopoulos, G., Collis, V., Patrinos, H.A. and Vegas, E. (2021). The COVID-19 cost of school
closures in earnings and income across the world. Comparative Education Review 65(2):
271-287.
Electronic copy available at: https://ssrn.com/abstract=4298822
9
UNESCO (2022). UNESCO map on school closures. Retrieved at https://covid19.uis.unesco.org/
on March 2022.
|
