Conference Agenda

Complexity in the education sector, as well as continuing societal transformation, and the diversity of interdisciplinary and mutually influencing thematic areas within the field, all call for a reflection on methodological approaches to assess, represent and analyse evidence. Research syntheses offer one opportunity to investigate such dynamic research processes. This panel contributes to this complex issue by interrogating how diversity can be processed through methods of research synthesis in the field of educational technology. In this context, diversity relates to the scope of topics in digitising, as well as methods that are internationally applied to the research field of digitalisation and education. Furthermore, the topic of digitalisation implies an inherent form of diversity, as it does not only affect one single area in isolation, but rather impacts and influences all aspects of society, leading to various approaches of how to understand and study digitalisation.

Since the 2000s, research syntheses have reached a growing significance in social sciences generally (Booth et al., 2016; Petticrew & Roberts, 2006) but also within educational research in particular (Gough et al., 2017; Zawacki-Richter et al., 2020; Bedenlier et al., 2023). One reason for this is that by conducting research syntheses, scientists aim to systematically categorise a constantly growing body of literature. In this panel, we will take digitalisation in education as an example to discuss how far the method can be used to assess a dynamic research field – given a research situation that is highly international, interdisciplinary, heterogeneous (diversity of research designs) and often diffuse (unclear body of literature sources). Such a state of research seems typical for literature on transformation processes in general.

The panel will also query the complexity of assessing such research fields with profound methods and categorizing them qualitatively. A further methodological challenge emerges from the handling of a topic that has global application, like many current transformation processes, but is nonetheless shaped by cultural contexts and regional or national developments. For example, an investigation of factors that influence digitalisation in schools cannot do without international best-practice experience. At the same time, conditions of schools and school systems vary greatly within and across countries. These constellations require both an intensive treatment of international discourses and of local contexts, as well as a critical reflection on the benefits and challenges of translation and transfer of research, including bias within research syntheses.

The panel contributions focus on reviews of digitalisation in education from varying perspectives and will consider a range of review types (Sutton et al., 2019), discussing their advantages and challenges: (1) a series of critical reviews which systematically assess and then categorise and discuss literature, (2) a review of reviews, wherein statements from selected reviews are synthesised, (3) a rapid living review, where systematic methods are used within a shorter time frame, and the corpus is updated regularly to reflect extant literature. All three papers reflect international research and were conducted by scientists based in Norway, Germany and Australia. Within the field of digital education the thematic scope from the three examples ranges from different digital teaching aids, tools and resources to questions of management and organizational structures.

Moreover, all the three contributions will investigate the role reviews might play in societies that are characterised by diversity and dynamic transformation processes. To pursue this issue, a discussant from the United Kingdom will finally reflect the examples to assess the role of research syntheses in the context of knowledge societies and Open Science.

Education in an increasingly digitalised world represents dynamic social developments and a rapidly developing field of research in which new innovative concepts, technologies and transformation strategies are continuously being tested. Researchers are therefore finding it challenging to keep track of the latest findings and ongoing trials and to identify current research desiderata. Research syntheses, e.g. in the form of critical reviews, can make an important contribution to summarising the state of research on a clearly defined topic in a structured way and thus making it useful for science and practice (Wright & Michailova 2022). Critical reviews, like other research syntheses, involve a systematic search, description and analysis of the literature available in relation to a specific research question. They focus primarily on conceptually grasping the research field and can thus contribute to further theory or model building in the field (Wilmers et al. 2020; Grant and Booth 2009). The focus is on a detailed search of existing literature, a narrative classification of this literature in the research context and an assessment of the significance of the literature with regard to its contribution to answering a respective research question, as well as the identification of research desiderata. Critical reviews are characterised by rather broad questions and can be used to synthesize studies qualitatively (Wetterich & Plänitz 2021). The presentation is based on a critical review of curriculum development in the school sector in the context of digitalisation. The search resulted in 2444 titles. Content, theoretical, empirical and methodological criteria were considered for the inclusion of a publication (Siddaway, Wood & Hedges 2019). Following further exclusion criteria on a formal (e.g. non-matching literature type or other language than English or German) and on a qualitative level (e.g. matching the research question or meeting basic scientific standards), 29 publications were finally included in the critical review. These included studies collected primary data using interviews, questionnaires and document analyses. In order to systematically assess the studies and their contents in the review, a theoretical framework was used as the underlying construct (Newman & Gough 2020). In this contribution, we would like to discuss the potentials and challenges of the methodology of critical reviews in educational research, particularly in the field of education in an era of digital change. This implies specific challenges such as dealing with an unpredictable number of literature findings or handling very heterogeneous sets of literature.

Grant, M. J. & Booth, A. (2009). A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies. Health information and libraries journal 26 (2): 91–108. Newman, M. & Gough, D. (2020). Systematic Reviews in Educational Research: Methodology, Perspectives and Application. In O. Zawacki-Richter, M. Kerres, S. Bedenlier, M. Bond & K. Buntins (eds.) Systematic Reviews in Educational Research: Methodology, Perspectives and Application, 3–22. Wiesbaden: Springer VS. Siddaway, Andy P., Wood, Alex M. & Hedges, Larry V. (2019). How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses. Annual Review of Psychology, 70, 747–770. Wetterich, C. & Plänitz, E. (2021) Systematische Literaturanalysen in den Sozialwissenschaften: Eine praxisorientierte Einführung. Opladen, Berlin, Toronto: Verlag Barbara Budrich. Wilmers, Annika et al. (2020). Reviews zur Bildung im digitalen Wandel: Eine Einführung in Kontext und Methodik. In A. Wilmers, C. Anda, C. Keller, M. Rittberger (eds.), Bildung im digitalen Wandel. Die Bedeutung für das pädagogische Personal und für die Aus- und Fortbildung, 7-29. Münster, New York: Waxmann. Wright, A. & Michailova, S. (2022). Critical literature reviews: A critique and actionable advice. Management Learning. https://doi.org/10.1177/13505076211073961

The use of digital learning aids, resources and tools in education consists of several different research fields encompassing a wide range of research designs, contexts, and perspectives (Lai & Bower, 2019). As part of a project commissioned by the Norwegian Directorate of Education, we aimed to scope the field as broadly as possible by conducting a systematic review of systematic reviews by answering the question: What is known about digitalization in compulsory education? (Munthe et al., 2022). We followed the PRISMA-guidelines (Page et al., 2021) in searching, screening, and coding the included articles. We had a total of 262 included international systematic reviews, which we synthetized thematically in our research report, targeted to the Directorate, and directed to school stakeholders, teachers, policy makers and researchers. In our presentation we will lift advantages and challenges in conducting and using review of reviews as a method to scope the field of digitalization in education. According to Gough et al. (2017) reviews of reviews able broader approaches than individual systematic reviews and they can be relevant for decision-makers to save them to assimilate multiple systematic reviews themselves. They can also be timesaving by enabling the processing of many primary studies in the same review. The included systematic reviews in our study included totally thousands of primary studies. Reviews of reviews thus enable sustainable use of research resources by systematizing and synthetizing the existing research and highlighting the research gaps. The deeper approach can be challenging with review of reviews. In our study, the results showed the huge potential of digital resources to transform learning processes in classrooms and contribute to students learning. However, a majority of the included systematic reviews pointed out it is not self-evident, and a successful transformation mostly handles the role of the teacher, without deeper discussions about teachers pedagogical and didactical choices. A deeper understanding about the role of the teacher was not possible with our approach. Another challenge handles the quality of systematic reviews, and the quality check of the primary studies included the reviews. In our study, only a few systematic reviews documented the quality check of the included primary studies. A challenge also lies in the time-consuming investigation of potential overlap of included primary studies. Unfortunately, not all reviews provide the references of included primary studies.

Gough, D. A., Oliver, S., & Thomas, J. (2017). An introduction to systematic reviews (Second edition.). SAGE. Lai, J. W. M. & Bower, M. (2019). How is the use of technology in education evaluated? A systematic review. Computers & Education, 133, 27-42. https://doi.org/10.1016/j.compedu.2019.01.010. Munthe, E., Erstad, O., Njå, M.B., Forsström, S., Gilje, Ø., Amdam, S., Moltudal, S., Hagen, S.B. (2022). Digitalisering i grunnopplæring; kunnskap, trender og framtidig forskningsbehov. Kunnskapssenter for utdanning: Universitetet i Stavanger. Page M. J., McKenzie J. E., Bossuyt P. M., Boutron I., Hoffmann T. C., Mulrow C. D., et al. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Systematic Reviews 2021;10:89

During the height of the COVID-19 pandemic, teachers, school leaders and policymakers needed evidence quickly, in order to inform emergency remote education policy and practice. This paper reports on a rapid review of 89 K-12 studies from around the world, undertaken following the first 7 months of the pandemic. Owing to the frequently changing landscape of the pandemic, a rapid review was chosen over a more extensive systematic review, as this allowed for streamlining and omitting aspects of the reviewing process to enable quicker dissemination (Hamel et al., 2020; Tricco et al., 2020). The rapid review was still undertaken using a transparent and replicable search strategy (Gough et al., 2012), but the number of databases searched was limited to four and a formal quality assessment was not undertaken, although any studies that did not include explicit details of participants with clear empirical data were excluded. This review was also intended to be a living rapid review (Elliott et al., 2014), updated regularly with new studies meeting the inclusion criteria, particularly through the use of machine learning via Microsoft Academic Graph within the evidence synthesis software EPPI-Reviewer (Thomas et al., 2023). Whilst this was initially achievable, the sheer volume of research that has been published on teaching and learning during the pandemic quickly became overwhelming for only one researcher to keep up with, coupled with a changeover of machine learning provider within EPPI-Reviewer to OpenAlex (Priem et al., 2022). This paper will therefore discuss the benefits and challenges of conducting both rapid reviews and living reviews, reflecting on this COVID-19 example, and provide advice for conducting similar reviews in the future.

Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins, J. P. T., Mavergames, C., & Gruen, R. L. (2014). Living systematic reviews: An emerging opportunity to narrow the evidence-practice gap. PLoS Medicine, 11(2), e1001603. https://doi.org/10.1371/journal.pmed.1001603 Gough, D., Oliver, S., & Thomas, J. (Eds.). (2012). An introduction to systematic reviews. Sage. Hamel, C., Michaud, A., Thuku, M., Skidmore, B., Stevens, A., Nussbaumer-Streit, B., & Garritty, C. (2020). Defining Rapid Reviews: a systematic scoping review and thematic analysis of definitions and defining characteristics of rapid reviews. Journal of Clinical Epidemiology. Advance online publication. https://doi.org/10.1016/j.jclinepi.2020.09.041 Priem, J., Piwowar, H., & Orr, R. (2022). OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts. ArXiv. https://arxiv.org/abs/2205.01833 Thomas, J., Graziosi, S., Brunton, J., Ghouze, Z., O'Driscoll, P., Bond, M., & Koryakina, A. (2023). EPPI-Reviewer: advanced software for systematic reviews, maps and evidence synthesis [Computer software]. EPPI-Centre Software. UCL Social Research Institute. London. https://eppi.ioe.ac.uk/cms/Default.aspx?alias=eppi.ioe.ac.uk/cms/er4 Tricco, A. C., Garritty, C. M., Boulos, L., . . . Straus, S. E. (2020). Rapid review methods more challenging during COVID-19: Commentary with a focus on 8 knowledge synthesis steps. Journal of Clinical Epidemiology, 126, 177–183. https://doi.org/10.1016/j.jclinepi.2020.06.029