11. Educational Improvement and Quality Assurance
Paper
Challenges in the Implementation of EMIS as a Litmus Test of the Quality of Education
Katarina Mićić, Ana Pešikan
University of Belgrade, Faculty of Philosophy, Department for Psychology
Presenting Author: Mićić, Katarina
Education Management Information System (EMIS) is a necessary tool for educational policy management, education reform, and monitoring and evaluation in education that enables the creation of valuable inputs for improving the quality of education (Abdul-Hamid, 2014; UNESCO, 2018; Bőjte, 2019). Introduction of an EMIS into a complex world of an educational system is a difficult task. One challenging aspect is defining an EMIS that can fit all scenarios and include all data needs and flows (Tolley & Shulruf, 2009), especially in a country such as Serbia, with over million students, three thousands of educational institutions with over 150 thousand of employees, and a diversity of educational programs held in several languages. Another concern is the social context of EMIS introduction, and how such novelty changes working routines and social relations in educational institutions (Kling, 2000).
Implementation of an information systems (IS) often face obstacles due to a gap between its design and the reality. Heeks (2002; 2006) argue that the size of this gap determines the probability of an IS successful implementation and its acceptance among users. He proposed a model for estimating the extent of this gap on six dimensions. Information dimension refers to a discrepancy between the envisioned set of data to be collected through an IS, and the data that users actually work with. Technology dimension assesses the difference between the required infrastructure for an IS to operate and the infrastructure available to users, in this case – to schools. Processes dimension deals with the difference in the processes envisioned to be automatized and the existing ones. Objectives and values dimension deals with the compatibility of the goals and values that an IS brings and enforces with the goals and values of its users. Staffing and skills accesses the gap between the required staff for operating the IS and their competence level and the available number of people and their actual competence level. Management system and structures is a dimension that is concerned with the discrepancy between the required managerial structure needed to guide the use of an IS and the one available. On the basis on this model, we analyzed the Serbian EMIS and looked into the causes of the gaps, which circled back to more broad and general structures in the decision-making process that pose barriers to other aspects of reform success as well.
After four unsuccessful attempts to build EMIS in Serbia in the last two decades, the Government of the Republic of Serbia started implementing EMIS, which has been developing since 2019. New EMIS includes data on all levels of education and collects data on schools, teachers, classes, and students (Rulebook on EMIS, 2019). However, during the first year of implementation, all groups of EMIS users – from those who collect and enter to those who use data – encountered difficulties. We utilized Heeks’ model as a lens to analyze the development and effectiveness of the implementation of EMIS. However, we discovered that challenges in the implementation of EMIS reflect subtle and wider obstacles to ensuring the quality of education. This study uses the case of EMIS implementation unravel these obstacles and to advocate for deconcentrated (Ivić, 2017) and participatory decision-making processes as an opposite to excessive top-down politics with poor communication of the change introduction in educational policy.
Methodology, Methods, Research Instruments or Sources UsedThe study is a part of a project concerning support of reform implementation with a focus on the improvement of evidence-based policy making in Serbia, funded by European Union. The study was conducted during summer 2022. Methodological approach consisted of desk research, focus group discussions and interviews. Desk research was focused on analysis of relevant legal acts and bylaws. Five focus groups were conducted with a total of 104 representatives of educational institutions who enter data in EMIS and customer support administrators. Ten semi-structured interviews were conducted with users of educational data – three representatives of the government, five scientific researchers, and two representatives from other relevant institutions that use educational data. Findings from the desk research were presented to participants in both the focus groups discussions and the interviews and were used to guide the discussion. However, participants were eager to bring up other topics as well, so we would update the focus group and interview guide after each session. Transcripts of focus group discussions and interviews were recorded and transcribed. Transcripts were analyzed based on the Heeks’ dimensions in MAXQDA software.
Conclusions, Expected Outcomes or FindingsOur analyses revealed a moderate gap on the information dimension as the introduction of EMIS envisioned collection of data that haven’t been collected before and whose purpose has not been communicated to the school staff nor the parents. A moderate gap exists on the technology dimension as some rural areas haven’t been recognizes as not having adequate technical resources. We noted resistance regarding the processes the EMIS is envisioned to fulfill, as some actors don’t appreciate automatization of certain decisions, mostly regarding finances. A moderate gap was also noted in terms of objectives and values, as a large number of staff and parents fear being monitored, don’t appreciate the expansion of technology or want to avoid transparency. On the staff and skills dimension - analysis revealed that many persons who are delegated to work with EMIS are not competent for the role or satisfied with it, since their workload has increased while their compensation remained the same. Finally, regarding the management system and structures dimension – our analysis revealed the need for decentralization of some managerial processes, the most important one being data quality assurance.
Looking for the ways of removing the barriers in EMIS implementation and closing the uncovered gaps, we underlined several weak spots in the ways EMIS was design and introduced. Some of the gaps could have been narrower in policy making processes were more participatory and included perspective from the EMIS direct users. Closely related to this is a need for deconcentrating the decision-making processes in order to obtain better insight (e.g., on the available infrastructure or staff skills) and feedback (e.g., on data quality and technical issues). Finally, gaps in the domains of information and objective and values could be shrunk by better transparency and communication of context and intents of implementing different novelties and measures.
ReferencesAbdul‐Hamid, H. (2014). What Matters Most for Education Management Information Systems: A Framework Paper. SABER — SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS EDUCATION MANAGEMENT INFORMATION SYSTEMS, Working Paper Series Number 7, World Bank Group, jun 2014 http://wbgfiles.worldbank.org/documents/hdn/ed/saber/supporting_doc/Background/EMIS/Framework_SABER-EMIS.pdf
Bőjte, D. (2019). Mastering the Skills Gap at Systemic Level–The Education Management Information System: A Key Element for an Effective Digital Transformation in Education. Revista de Management Comparat International, 20(2), 131-143.
Heeks, R. (2002). Information Systems and Developing Countries: Failure, Success, and Local Improvisations. The Information Society, 18(2), 101–112. https://doi.org/10.1080/01972240290075039
Heeks, R. (2006). Health information systems: Failure, success and improvisation. International Journal of Medical Informatics, 75(2), 125–137. https://doi.org/10.1016/j.ijmedinf.2005.07.024
Ivic, I., & Pesikan, A. (2012). Education system reforms in an unstable political situation: the case of Serbia in the first decade of the 21st century. Ceps Journal, 2(2), 31-53.
Kling, R. (2000). Learning about information technologies and social change: The contribution of social informatics. The Information Society, 16(3), 217–232.
Pešikan, A., & Ivić, I. (2016). The sources of inequity in the education system of Serbia and how to combat them. Center for Educational Policy Studies Journal, 6(2), 101-124.
Pešikan, A., & Ivić, I. (2021). The impact of specific social factors on changes in education in Serbia. Center for Educational Policy Studies Journal, 11(2), 59-76.
Pešikan, A. (2022). Analiza stanja informacionog sistema u prosveti i preporuke za njegovo unapređenje. [Situation analysis of Serbian EMIS and recommendation for its improvement.]
Pravilnik o Jedinstvenom informacionom sistemu prosvete [Rulebook on Educational Menagement Information System] Službeni glasnik, 81/ 2019 http://www.pravno-informacioni-sistem.rs/SlGlasnikPortal/eli/rep/sgrs/ministarstva/pravilnik/2019/81/5/reg
Tolley, H., & Shulruf, B. (2009). From data to knowledge: The interaction between data management systems in educational institutions and the delivery of quality education. Computers & Education, 53(4), 1199–1206.
UNESCO. (2019). https://learningportal.iiep.unesco.org/en/glossary/educational-management-information-system-emis
11. Educational Improvement and Quality Assurance
Paper
From National Policy to Local Practices – Systematic Quality Work in Education from the Perspective of Local Authorities
Tuija Muhonen, Anders Edvik, Magnus Erlandsson, Margareta Serder
Malmö University, Sweden
Presenting Author: Muhonen, Tuija;
Serder, Margareta
Over the last 20 years systematic quality work has become the main tool for developing Swedish schools (Håkansson & Adolfson, 2022). According to the Education Act (2010:800), quality work – at the local educational authority level as well as in the schools themselves – should be conducted in a systematic and continuous way, with respect to planning, follow-up, analyses, and actions taken to develop education. By continuous assessments and evaluations, the goal of the systematic quality work is to identify and address issues that need improvement for students to achieve the educational goals (Swedish National Agency for Education, 2012a). A review of the literature reveals the problematic aspects of evaluation practices and quality management, such as the risk to focus on what is measurable rather than what is desirable as well as the diverse definitions of quality (Lundström, 2015). However, what the local quality systems consist of, how they have been designed, and what practices and perceptions of quality they entail is less understood.
Previous research has primarily focused on individual schools’ quality work (Håkansson, 2013; Jarl, et al., 2017) , while less attention has been paid to the way the local educational authorities conduct systematic quality work. Thus, the aim of this study is to fill this knowledge gap by investigating how the ideas of systematic quality work in the Swedish Education Act's requirements are interpreted, translated, and materialized at the local education authority level.
The following research questions will guide our study:
1. How do local educational authorities interpret and translate the systematic quality work regulations and requirements in the Education Act?
2. How do these interpretations och translations materialize in the local quality work practices?
Theoretically we approach the phenomena of systematic quality management within the Swedish school sector from an organizing (Czarniawska, 2014) and practice-oriented perspective (Gherardi, 2019; Nicolini, 2009; 2012). These theoretical perspectives provide us a framework to analyse how the institutionalized ideas (as mental images that are well spread within the society) of systematic quality management - through authorities, policies, regulations, and quality models - are translated and materialized (Czarniawska & Joerges, 1996) into the local quality work organization and practices. The latter refers to the practices of doing and saying something related to the ideas of systematic quality management in different social contexts and time (Gherardi, 2019; Nicolini, 2009; 2012). Although the national guidelines involve the entire school system, these are interpreted, translated, and materialized by actors operating in a local context, which means that quality is understood in different ways and that the systematic quality work is conducted in different ways. An organizational perspective also includes aspects related to the tensions that arise when different interests and logics collide (for example between political, administrative, and professional interests and logics; see Czarniawska, 2014).
Methodology, Methods, Research Instruments or Sources UsedIn this project five Swedish municipalities have participated in a study of what systematic quality work means at a local education authority level and what practices materialize from the national regulations. At the heart of the study is the recognition that quality systems are locally designed to meet the national requirements, thereby allowing diverse interpretations and translations to occur.
The local education authorities can be understood as mediators, partly between state and municipal control, partly between needs and agendas at different levels in the chain of command. This understanding also characterizes the design of the study. The empirical data has been collected through three complementary methods: document studies, observations, and interviews. The document studies consist of analysing different central documents, e.g., quality reports, provided by the local education authorities covering the past two years. We have also observed meetings related to the systematic quality work (so called “quality-dialogues”). Besides the local education authorities, the key actors in these meetings were the principals, assistant principals, and teacher representatives of the school being followed up.
The interviews were conducted with key persons in five different local Swedish education authorities individually by the authors. The duration of the interviews was approximately one hour, and they were conducted either face-to-face, via Zoom, or telephone. The interviews were based on an interview guide including questions about the participants’ role, their experiences, and activities in relation to the systematic quality work, the expected and actual effects, as well as challenges and potential for improvement of systematic quality work. The interviews were recorded with informed consent and were later transcribed verbatim.
All the research material described above is now gathered and will be analysed during the Spring 2023. As a tool for data analyses, we will apply Bacchi´s (2012) method “What is the problem represented to be?”.
Conclusions, Expected Outcomes or FindingsEqual education for all is includes three fundamental aspects: equal access to education, equal quality of education and the compensatory nature of education (Swedish National Agency for Education, 2012b). All students should receive an equivalent education, regardless of the area they live in, the socio-economic conditions they come from, or their functional variations. But study after study shows that Swedish students' schooling is not equal, and that who you are and where you live play a decisive role in the quality of the education you receive. Many of the last decade's school policy reforms and targeted initiatives have had as their overarching goal to address this lack of equality, so far with few concrete results. In order to break this trend, there has been an increasing focus on the local educational authorities’ responsibility for the individual school's shortcomings, quality, and development. Furthermore, lack of equality is a problem within rather than between different local educational authorities. Although there is paucity of research, the limited results show that schools are often isolated with their problems and that there is a lack of supportive structures and a functional systematic quality work (Jarl, et al., 2017; Swedish Schools Inspectorate, 2021).
The paper will present results from the ongoing study, results that we believe will have relevance both in the Swedish, Nordic and in a wider European context. Through our investigation we will contribute knowledge regarding how the National Educational Act's requirements for systematic quality work are interpreted, translated, and materialized at the local level, and how this in turn shapes, promotes or hinders the quality work of individual schools.
ReferencesBacchi, C. (2012). Introducing the ‘What’s the Problem Represented to be?’ approach. Engaging with Carol Bacchi: Strategic interventions and exchanges, 21-24.
Czarniawska, B. (2014). A theory of organizing. Edward Elgar Publishing.
Czarniawska, B., & Joerges, B. (1996). Travels of ideas. Translating organizational change, 56, 13-47.
Education Act, [Skollag] (2010), 2010:800. https://www.riksdagen.se/sv/dokument-lagar/dokument/svensk-forfattningssamling/skollag-2010800_sfs-2010-800
Gherardi, S. (2019). How to conduct a practice-based study: Problems and methods. Edward Elgar Publishing.
Jarl, M., Blossing, U. & Andersson, K. (2017). Organizing for School Success: Strategies for an Equal School. [Att organisera för skolframgång: Strategier för en likvärdig skola.] Stockholm: Natur & kultur.
Håkansson, J. (2013). Systematic quality work in preschools, schools and leisure-time centres: strategies and methods. [Systematiskt kvalitetsarbete i förskola, skola och fritidshem: strategier och metoder.] Lund: Studentlitteratur.
Håkansson, J., & Adolfsson, C. H. (2022). Local education authority’s quality management within a coupled school system: Strategies, actions, and tensions. Journal of Educational Change, 23(3), 291-314.
Lundström, U. (2015). Systematic quality work in Swedish schools: Intentions and dilemmas. Scandinavian Journal of Public Administration, 19(1), 23-44.
Nicolini, D. (2009). Zooming in and out: Studying practices by switching theoretical lenses and trailing connections. Organization studies, 30(12), 1391-1418.
Nicolini, D. (2012). Practice theory, work, and organization: An introduction. Oxford: Oxford University Press.
Swedish Schools Inspectorate (2021). Complaints against education. Organization and procedures of the Local education authorities [Klagomål mot utbildningen Huvudmäns organisation och tillvägagångssätt] file:///Users/imtumu/Downloads/klagomal-mot-utbildningen--kvalitetsgranskningsrapport-skolinspektionen-2020_tg.pdf
Swedish National Agency for Education [Skolverket] (2012)a. Systematic quality work – for the school system. The Swedish National Agency for Education's general advice with comments. [Systematiskt kvalitetsarbete – för skolväsendet. Skolverkets allmänna råd med kommentarer.]
Swedish National Agency for Education [Skolverket] (2012)b. Educational equity in the Swedish school system? A quantitative analysis of equity over time. Summary of Report 374.
|
