09. Assessment, Evaluation, Testing and Measurement
Paper
Evaluating the Implementation of a Nationwide Quality Management System for Schools. Concept and First Results
Erich Svecnik
IQS, Austria
Presenting Author: Svecnik, Erich
Like many other European countries Austria is currently implementing a nationwide Quality Management System for schools (QMS; https://www.qms.at/). Its aim is systematic and targeted school and teaching development based on a quality circle of plan–do–check–adjust (PDCA, ‘Shewhart Cycle’) and thus similar to other countries, especially some German Länder with which there is also a continuous exchange. The most important features are the introduction of a mandatory quality framework for all schools as well as an increased data or evidence orientation in school and teaching development in general. QMS-tools include the definition of a school’s pedagogical guiding principles, a school development plan, a balance and target agreement meeting between the principal and the regional school quality manager (formerly ‘school inspector’) and a quality handbook. To support the data orientation, an internet platform with several hundred instruments for internal evaluation was provided. Formerly different quality management programs for general and vocational schools (and thus different traditions and instruments) are being merged into QMS.
The implementation process and the diffusion of the QMS and its elements into the school system are formatively evaluated in an accompanying process (Rossi et al., 2019; Stockmann, 2011). The overall objective of this evaluation is the generation of knowledge for the optimization of the implementation process as well as its monitoring and documentation of progress.
The theoretical background of this research is based on Rogers’ (2003) ‘Diffusion of Innovations’ describing typical stages of immersion. Accordingly, knowledge of the innovation is the starting point, in concrete knowledge of the QMS model, which should subsequently lead to a positive attitude or acceptance (persuasion). The next stages are the informed decision of the actors to adopt the innovation (decision) and the actual implementation, which in the best case leads to reinforcement and confirmation. Coburn (2003) focuses attention on the depth of change, its sustainability and ownership in the medium and long term, although these are of little importance in the initial phase. A closer look at the context of implementation and the creation of necessary framework conditions follows the approach of implementation research (Petermann, 2014).
The design of the evaluation and the underlying theoretical assumptions lead to the following three guiding questions:
- How deeply has the nationwide Quality Management System already diffused into everyday school life?
- How can the implementation process be further promoted and supported?
- Are there different patterns in this respect in different school sectors, specifically between general education and vocational education?
According to the underlying model of Utilization-Focused Evaluation (Patton, 1997, 2000), the detailed questions are defined in the further course of the evaluation in close coordination with the persons responsible for QMS.
Methodology, Methods, Research Instruments or Sources UsedIn a first step all 43 quality regional coordinators were surveyed using an online questionnaire. The survey focused on the challenges of their work, the need for support and their perceptions of the implementation of QMS to date in terms of Diffusion at school level, Acceptance, and Realization processes. Results showed high acceptance of QMS among respondents, a high level of satisfaction with support from the ministry, but the diffusion at school level is not yet perceived as very far advanced.
The next step is a survey of a representative sample of school principals and quality school coordinators using adapted versions of A-SEW (Carmignola et al., 2021) with the dimensions of meaningfulness, usefulness, and practicality of innovation. Individual items from the Stages of Concern Questionnaire (George et al., 2006) also provide information on personal aspects of implementation. Other content-related aspects refer to supportive and obstructive framework conditions including support and training needs as well as possible need for improvement of the available material. First indications of non-intended effects of the QMS introduction (e.g. Landwehr, 2015) are also to be obtained in order to be able to take countermeasures if necessary.
Conclusions, Expected Outcomes or FindingsFirst results will be available in August 2023. Data will be analysed using descriptive statistics for getting an overview and inferential statistics to examine differences between educational sectors. In addition, (multilevel) regression models will provide explanations and cluster analyses will help in defining tailored support for schools by identifying schools with similar characteristics.
Once the findings are available, answering the research questions will provide the persons/institution(s) responsible for and steering the implementation process of QMS with data/knowledge to optimize the processes and provide appropriate support.
ReferencesCarmignola, M., Hofmann, F. & Gniewosz, B. (2021). Entwicklung und Validierung einer Kurzskala zur Einschätzung der Akzeptanz von Schulentwicklungsprojekten (A-SEW). Diagnostica, 67(4), 163–175.
Coburn, C. (2003). Rethinking Scale: Moving Beyond Numbers to Deep and Lasting Change. Educational Researcher, 32(6), 3–12.
George, A. A., Hall, G. E., & Stiegelbauer, S. M. (2006). Measuring implementation in schools: The stages of concern questionnaire. SEDL
Landwehr, N. (2015). Die institutionelle und kulturelle Verankerung des Feedbacks. In: Buhren, C. G. (Ed.). Handbuch Feedback in der Schule. Weinheim Basel: Beltz.
Patton, M. Q. (1997). Utilization-Focused Evaluation: The New Century Text. Thousand Oaks; London; New Delhi: Sage Publications.
Patton, M. Q. (2000). Utilization-focused evaluation. In: Stufflebeam, D.L., Madaus, G.F., Kellaghan, T. (Eds.) Evaluation Models: Viewpoints on Educational and Human Services Evaluation (pp. 425-438). Boston: Kluwer Academic Publishers.
Petermann, F. (2014). Implementationsforschung: Grundbegriffe und Konzepte. Psychologische Rundschau, 65(3), 122–128.
Rogers, E. M. (2003). Diffusion of innovations. New York: The Free Press.
Rossi, P. H., Lipsey, M. W. & Henry, G. T. (2019). Evaluation: a systematic approach (Eighth edition.). Los Angeles: SAGE.
Stockmann, R. (Hrsg.). (2011). A Practitioner Handbook on Evaluation. Cheltenham, UK ; Northampton, MA: Edward Elgar.
09. Assessment, Evaluation, Testing and Measurement
Paper
Use and Impact of External Evaluation Feedback in Schools in Iceland
Björk Ólafsdóttir, Jón Torfi Jónasson, Anna Kristín Sigurðardóttir
University of Iceland, Iceland
Presenting Author: Ólafsdóttir, Björk
Past findings concerning whether and how feedback from external evaluations benefit the improvement of schools are inconsistent and sometimes even conflicting, which highlights the contextual nature of such evaluations and underscores the importance of exploring them in diverse contexts. Considering that broad international debate, we investigated the use and impact of feedback from external evaluations in compulsory schools in Iceland, particularly as perceived by principals and teachers in six such schools. The research questions guiding the study was “How and to what extent do schools use the feedback presented in external evaluation reports?” and “To what extent do schools sustain the changes made after using the feedback from external evaluations instrumentally?” The framework used for analysing the evaluation feedback use was based on Rossi et al. (2004) and Aderet-German and Ben-Peretz (2020) and distinguishes between instrumental, conceptual, strategic and reinforcement-oriented use.
Methodology, Methods, Research Instruments or Sources UsedTo map the perceived use and long-term impact of the feedback, a qualitative research design was adopted to examine changes in the schools made during a 4–6-year period following external evaluations by conducting semi-structured interviews with principals and teachers, along with a document analysis of evaluation reports, improvement plans and progress reports. Six schools were selected to participate in the research based on the evaluation judgement, school size and geographical location. Six principals and eight teachers were interviewed. In analysing the interview transcripts and documentation a thematic approach (Braun & Clarke, 2006) was followed.
Conclusions, Expected Outcomes or Findings
The findings reveal that feedback from external evaluations has been used in a variety of ways, as the data revealed clear examples of instrumental, conceptual, persuasive and reinforcement-oriented use. Instrumental use could be seen in relation to (1) leadership and management: primarily respecting professional collaboration amongst staff members and the instructional leadership of school leaders; (2) learning and teaching: mainly regarding differentiated strategies for instruction, democratic participation of the student and the use of assessments to improve students’ learning, and (3) internal evaluation: mostly concerning evaluation plans and methods, stakeholder participation and improvement plans. Instrumental use varied between the schools, and they did not all made major changes in all three areas. Conceptual use was also evident at the schools, and in that context the usefulness of obtaining an external view of the school’s function and getting help in identifying where improvements were needed was highlighted. In some cases, the evaluation feedback led to productive discussions and reflections among the professionals and for a three newly appointed principals it gave a useful instructions. Persuasive use of the evaluations feedback was identified in three interviews in the context of supporting changes that the interviewee wanted to bring about. Likewise, reinforcement-oriented use was analysed in three interviews at schools that had received positive evaluation feedback which they experienced as empowering. The findings also showed that both teachers and principals had a positive attitude towards the external evaluation and had generally experienced the evaluation feedback as useful and that it had contributed to changes in practices in the schools. The improvement actions presented in the schools’ improvement plans were generally implemented or continue to be implemented in some way, and the changes made have mostly been sustained.
ReferencesAderet-German, T., & Ben-Peretz, M. (2020). Using data on school strengths and weaknesses for school improvement. Studies in Educational Evaluation, 64, Article 100831. https://doi.org/10.1016/j.stueduc.2019.100831
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.
Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach (7th ed.). Sage.
09. Assessment, Evaluation, Testing and Measurement
Paper
Knowledge Mapping Of Learning Analytics And Professional Capital In Education: a Bibliometric Study
Javier de la Hoz- Ruíz1, Mohammad Khalil2, Jesús Domingo Segovia1
1University of Granada, Spain; 2SLATE, University of Bergen, Norway
Presenting Author: de la Hoz- Ruíz, Javier
To reach objective 4 and 17 adopted by the General Assembly in September 2015, go through working together, teachers and students need to form a community of knowledge seekers and builders as the UNESCO (2021) affirms, so one of the ways to achieve these two sustainable development goals is to form communities of professional practice.
Thus, improving education is significantly dependent on the ability of school leaders to connect everyone (teachers, families and local community) into a professional community of practice, which increases their professional capital. It has three dimensions; human capital is acquired and useful skills of all inhabitants or members of society (Smith, 1776); social capital, those characteristics of social organization, such as trust, standards and networks that can improve the efficiency of society by facilitating coordinated actions (Putnam, 1995) and decision capital, is what professionals acquire and accumulate through structured and unstructured experiences, practice and reflection , capital that allows them to make judgements.
In complex contexts, a new governance of the school is required, with horizontal leaders (leadership from the middle) that build projects, cultures and environments with a community vocation of commitment to educational improvement, while expanding the social capital of the Professional Practice Community.
COMMUNITY OF PROFESSIONAL PRACTICE
Interestingly, Domingo-Segovia et al. (2020) have used the term of “Professional Practice Community” as a broader term to “professional learning community” that includes the school and the local community context. Community that requires the emergence of fluid networks of interrelationship, communication and support for the learning of all and for all, with a shared and networked leadership, articulated from a broad perspective of “middle leadership” (Rincón, 2019).
Therefore, as an “extended” community, the stage and actors must be linked and expanded with the collective goal of educational improvement, expanding networks of influences and opportunities. Hence the importance of the “increase of professional capital” (Hargreaves & Fullan, 2014), the next point to be discussed.
PROFESSIONAL CAPITAL AND LEARNING ANALYTICS
The key to this term is the systemic development and integration of three types of capital – human, social and decision-making – into the teaching profession. Professional capital has to do with collective responsibility, not individual autonomy; with rigorous training, continuous learning, going beyond the evidence, being open to the needs and priorities of students and society “ (Hargreaves & Fullan, 2012).
In addition, the use of learning analytics defined as “the measurement, collection, analysis and presentation of data about students and their contexts, in order to understand and optimize learning and the environments in which it occurs” acquires a relevant value as it can be used as a means to extract the most effective methodologies, processes and tools in data measurement, collection, analysis and reporting of professional capital (Khalil & Ebner, 2016).
Thus, The rationale for this work was to understand the structure of how learning analytics can help the improvement and understanding of professional capital in the field of education by analyzing its scientific output. For this purpose, bibliometric maps offering a better understanding of the structure of a scientific domain through the graphical representation of the different units of analysis and their relationships (Small, 2006)
Thus, this research will answer the following questions:
RQ1: What are the key themes or knowledge grouped around the use of learning analytics for the improvement of professional capital?
RQ2: What is the research trend of the field under study?
RQ3: What are the research boundaries extracted from the network analysis?
The steps and tools used in this process are explained more specifically in the methodology.
Methodology, Methods, Research Instruments or Sources UsedFirstly, to guide the first part of the study and to check the scientific production more in order to make our maps, the guides were used. and PRISMA checklist (http://www.prisma-statement.org) to ensure transparency in both the research process and analysis (Moher et al 2009).
Second, the cluster-based VOSviewer (van Eck & Waltman, 2011) was used to perform the analysis. It presents the structure, evolution, cooperation and other relationships of the field of knowledge for literature data. It also allows the user to view and explore scientific data mapping in cluster format based on data from scientometrics networks.
Both are explained more specifically below.
PRISMA
Web of Science (WOS), Scopus and Association for Computing Machinering (ACM) were used as data sources in this document.
In order to provide rigor to our search process, we proceeded to establish keywords extracted from the ERIC thesaurus (Hertzberg and Rudner, 1999). This document retrieves the information using the search formula ALL FIELDS (ALL) / (ALL=“learning analytic*" OR ALL="academic analytic*" OR ALL="teaching analytic*") AND (ALL="social capital" OR ALL=“human capital“ OR ALL=“decisional capital” OR ALL=“professional capital”) and the results after the duplicate citations were 657 papers at this initial stage . The search was conducted on Feb. 23, 2022.
The process of inclusion of the studies was conducted through peer review (Sarthou, 2016). We proceeded to read the items returned by our search to identify only those that were relevant to our research questions and objectives. Of the 93 items returned, we selected 84. In this, the reason for the exclusion of these studies was the thematic inadequacy in relation to our study.
Thus, the articles under study have been obtained, they are analysed with the software described below.
VOSVIEWER
Once the 84 articles have been imported in zotero, a multiplatform bibliographic reference manager, free, open and free, its main objective is to help us collect and manage the resources needed for our research (Alonso-Arévalo, 2015).
However, we recommend also exploring the effect of excluding a smaller or higher percentage of terms, in our case we eliminated the words "study" and "analysis" as we believe they distort the results. We refer to Van Eck and Waltman (2011) for a brief explanation of the calculation of relevance scores.
The network visualization maps or cluster density maps were produced by the VOSviewer, maps that show the results of the analysis.
Conclusions, Expected Outcomes or FindingsThis paper assessed global research trends from 2012 to 2019. The subject of learning analytics in the improvement of professional capital has been a field with a lot of research over the last 10 years, but it should be noted that scientific output has increased exponentially in recent years. There is a growing interest in research related to both learning analytics and business capital independently, which corresponds to the urgent need to jointly develop and improve these research fields.
Therefore, this study will provide us with three key points: a) It helps to better understand how learning analytics studies are carried out in the improvement of professional capital, as well as the fields and disciplines in which they are carried out, specifically 6 clusters or clusters were detected (research on the influence of community improvement program, research in the analytics of learning, analysis of collaborative networks, relationship models for performance improvement and theoretical background) b) The evolution over time of the studios, where it is possible to appreciate a current interest in platforms for the improvement of professional capital; and c) a frontier of analysis is proposed, with content studies to try to observe and use more specific information of these articles.
Limitations of this bibliometric study should be addressed, it is suggested to expand the research using other databases such as EBSCO, ProQuest, Emerald, SAGE, or others; however, the study agrees with the objective of the research, providing. In addition, future studies could consider other types of maps offered by the software such as co-citation between authors, journals...
In future work, the construction and comparison of two-dimensional bibliometric maps corresponding to several time periods would show where trends and research fronts are evolving.
ReferencesThe results of this publication are part of three research projects:
1) "Communities of professional practice and learning improvement: intermediate leadership, networks, and interrelationships. Schools in complex contexts" (Ref.: PID2020-117020GB-I00), funded by MCIN/AEI/10.13039/501100011033/ and ERDF "A way of doing Europe"; and
2) "Communities of professional practice and learning improvement" (Ref.: P20_00311), funded by the Andalusian Plan for Research, Development, and Innovation (PAIDI 2020).
Bolam, R, et al. (2005). Creating and Sustaining Effective Professional Learning Communities, DfES Research Report RR637, University of Bristol, Bristol.
3) "Extended Professional Learning Communities and Collaboration Networks for Sustainable Development and Inclusion: New Governance and Social Capital" (Reference: B-SEJ-234-UGR20), financed by the FEDER 2020 Operational Program (Andalusia 2014-2020)"
Domingo-Segovia, J., Bolívar-Ruano, R., Rodríguez-Fernández, S., & Bolívar, A. (2020). Professional Learning Community Assessment-Revised (PLCA-R) questionnaire: Translation and validation in Spanish context. Learning Environments Research, 23(3), 347-367. https://doi.org/10.1007/s10984-020-09306-1
Hargreaves, A., Fullan, M., & Pruden, J. (2012). Professional Capital.
Hargreaves, A., & Fullan, M. (2014). Capital profesional. Transformar la enseñanza en cada escuela. Madrid: Morata.
Khalil, M. & Ebner, M. (2016). “What is Learning Analytics about? A Survey of Different Methods Used in 2013- 2015”. In Proceedings of Smart Learning Conference, Dubai, UAE, 7-9 March, 2016 (pp. 294-304). Dubai: HBMSU Publishing House.
Leana, C. R. (2011). The missing link in school reform”, Stanford Innovation review, 34.
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D.G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA Statement. Annals of Internal Medicine, 51, 264-269.
Putnam, R. D. (1995). "Bowling Alone: America's Declining Social Capital", Journal of Democracy 6:65-78.
Rincón, S. (2019). Las redes escolares como entornos de aprendizaje para los líderes educativos. En J. Weinstein & G. Muñoz (eds.). Cómo cultivar el liderazgo educativo. Trece miradas. (pp.355-388) Santiago: Universidad Diego Portales
Small, H. (2006). Tracking and predicting growth areas in science. Scientometrics, 68(3), 595-610. https://doi.org/10.1007/s11192-006-0132-y
Smith, A. (1776), An inquiry to the nature an causes of the wealth of nations, Book II: of the nature, accumulation, and employment of stock Nueva York. Classic house books.
UNESCO (2021). Reimagining our futures together: a new social contract for education. Paris. Unesco.https://unesdoc.unesco.org/ark:/48223/pf0000379707
United Nations General Assembly. (2015). Transforming our world: The 2030 agenda for sustainable development.
Van Eck, N.J., & Waltman, L. (2011). Text mining and visualization using VOSviewer. ISSI Newsletter, 7(3), 50–54
|
