Conference Agenda

Research syntheses gather, aggregate and analyse results of scientific publications on a specific subject to map past and current research and – in some cases – extract implications for research, policy and practice (Newman & Gough, 2020). They enable guidance based on stable evidence across a multitude of settings and samples, rather than on anecdotal results from single studies with only limited generalizability. The reliability and validity of the results, and therefore the scientific basis of its implications, are ensured by following the principles of open science and good scientific practice (Crüwell et al., 2019; Open Science Collaboration, 2015).

Adhering to the principles of open science while conducting research syntheses requires researchers for example to enable open/FAIR access to their research data (Stall et al., 2019), the use of open source software or the publication under open access licensing, including through ensuring transparency, replicability and reproducibility of each single step taken in conducting research syntheses. If those principles are disregarded, potential biases, blind spots or limitations of the results may be overlooked, which can lead to the incorrect transfer of the extracted implications. Therefore, each single step in conducting research syntheses has to be held to the highest standard of scrutiny regarding good scientific practice. This includes – but is not limited to – the development of (a) the initial search string and its application for the literature search, (b) the inclusion and exclusion criteria and the ensuing screening processes and (c) the coding/data extraction scheme, data analysis and syntheses processes and methods and presenting the review’s results. For (a) in conjunction with uploading the database hits to an accessible repository, the actual search terms for each single database have to be reported including the Boolean operators, the applied filters and the search date. Requirements for enabling transparency, replicability and reproducibility during screening (b) include clearly stating the inclusion and exclusion criteria and their application and reporting on resolving potential intercoder differences. In parallel to (b), the coding/data extraction schemes and potentially arising differences in the analysis and synthesis have to be reported transparently as well. In addition, if possible, all scripts applied during the analysis and visualisations should either be attached to the review or uploaded to relevant repository to enable the replicability and reproducibility of the results.

In this symposium, four research syntheses are presented with a focus on their adherence to the principles of open science and good scientific practice. While each presented project contained all relevant steps of conducting research syntheses, their goals, research questions and utilized methodology varied greatly: The first focuses on the development and application of a comprehensive yet precise search string for an overview of reviews on digital competences of young learners. Presentation two focuses on structuring and sharing the data extraction coding tool, as well as presenting the review’s results through various openly accessible means for researchers, policymakers and educators. The third presentation focuses on documenting the processes of analysis and synthesis by presenting the results of a methodological review of metasyntheses in educational research. The fourth presentation is concerned with potential biases in systematic reviews due to most included research originating in WEIRD-countries, i.e. western, educated, industrialized, rich and democratic countries.

The summarising discussion focuses on the general role of the principles of open science and good scientific practice while conducting research syntheses and how they represent requirements for evidence-based guidance of future research, policy and practice. In addition, it will highlight the potentials and challenges of transferring implications from syntheses into educational policies, practice and administration in regard to the heterogeneity of national educational systems, policies and curricula.

In line with the principles of open science and good scientific practice described in the description of the symposium, this contribution provides in-depth information on and results of a post-hoc analysis of a comprehensive literature search strategy for the identification of published research syntheses on digital competences of young learners (Carratero et al., 2017). Literature search strategies in systematic reviews consist of the development of the initial search string rooted in the theoretical background of the review, the searched databases and processing the database hits (Hausner et al., 2012). In addition, the efficiency of the search strategy can be determined via post-hoc analyses of the search’s precision and sensitivity (Pastore & Scheirer, 1974). In this project, two different search strategies were applied by searching for relevant phrases (i.e. “digital competence” instead of “digital” and “competence”, see Vechtomova & Karamuftuoglu, 2004) and traditional term-based searches in two languages in multiple databases. Furthermore, a manual search was conducted to identify additional papers for the screening process not found by the search strategy. Combining the search strategies, the databases and the languages resulted in a total of eight searches which led to n = 4,871 unique database hits. After title/abstract-screening and full-text-screening, n = 120 documents were included for the subsequent analyses. Subsequently, the search strategies were evaluated by determining the duplication and exclusivity rates between the searches as well as for the included and excluded papers. Additionally, we determined the document-frequencies of each single search term dependent on the inclusion decision; by determining the logarithmic difference between the document-frequencies of the search terms, information on the relevancy of the search terms for the inclusion decision was extracted. All in all, this presentation provides insights into the development of a comprehensive search strategy which aimed at finding a compromise between exhaustiveness and precision. Each central step of search strategies is presented (i.e. development search string, database search, data export) and evaluated always under the premises of replicability, reproducibility and research transparency. Implications for search strategies for research syntheses in regard to the principles of open science and good scientific practice are discussed such as reporting the search strings in their applied form, reporting on the distribution of database hits and post-hoc evaluating the search to identify potential thematic biases based on skewed distributions within the search string.

Carretero, S., Vuorikari, R., & Punie, Y. (2017). DigComp 2.1. The Digital Competence Framework for Citizens. With eight proficiency levels and examples of use. Publications Office of the European Union. Hausner, E., Waffenschmidt, S., Kaiser, T., & Simon, M. (2012). Routine development of objectively derived search strategies. Systematic reviews, 1(1), 19. Pastore, R. E., & Scheirer, C. J. (1974). Signal detection theory: Considerations for general application. Psychological Bulletin, 81(12), 945. Vechtomova, O., & Karamuftuoglu, M. (2004, November). Approaches to High Accuracy Retrieval: Phrase-Based Search Experiments in the HARD Track. In TREC

Sub-paper had to be withdrawn

.

Open science principles require research syntheses to make analytic decisions visible and traceable, yet how qualitative meta-syntheses in education document these processes remain underexplored. Meta-synthesis aims to systematically synthesize findings across qualitative studies and produce configurative interpretations. However, meta-synthesis in education is often characterized by methodological diversity, limited transparency, and weak theoretical anchoring, possible partly because many meta-synthesis methods originate from the health field, where research questions and theoretical traditions differ from those in education (Maeda et al., 2022). This paper contributes to the symposium’s focus on open science and good scientific practice in research syntheses by examining how research questions and epistemological positioning shape methodological choices, synthesis methods, and the use of theory, and how these processes are documented in educational meta-syntheses. The study is conducted as a methodological systematic review, following principles for systematic reviews (Gough et al., 2017). A systematic search was carried out in ERIC, Scopus, and Web of Science. Peer-reviewed systematic reviews published in English and focusing on education were included if they reported a meta-synthesis process. Double-blind screening was used in the title and abstract and full-text phases, resulting in 164 included systematic reviews. Data extraction and analysis are guided by Brunton et al.’s (2020) framework to examine research questions, analytic approaches, epistemological positioning, and the role of theory. Based on preliminary findings, thematic analysis with an inductive approach dominates meta-synthesis in educational research. However, many studies provide limited descriptions of how analysis and synthesis are conducted. In several studies, analysis methods seem to be taken from primary qualitative studies rather than from established meta-synthesis approaches. Furthermore, theory is used primarily to organize coded findings rather than to guide the synthesis process, and explicit epistemological reflection is rarely articulated. By the time of the conference, coding will be completed. The presentation will report the main results and reflect on implications for transparency, methodological clarity, and open scientific practice in educational meta-synthesis.

Brunton, G., Stansfield, C., Caird, J., & Thomas, J. (2020). Innovations in framework synthesis as a systematic review method. Research Synthesis Methods, 11(3), 316–330. https://doi.org/10.1002/jrsm.1399 Gough, D., Oliver, S., & Thomas, J. (2017). An introduction to systematic reviews (2nd ed.). SAGE. Maeda, Y., Hunsu, N., Lee, M., & Horan, E. M. (2022). A methodological systematic review of qualitative synthesis research in education. Review of Educational Research, 92(3), 407–446. https://doi.org/10.3102/00346543221083297

Research on educational technologies continues to be predominantly shaped by Western research traditions and learning theories (Bulfin et al., 2013; Hew et al., 2019). In addition, much of the extant empirical evidence originates from so-called WEIRD countries (Henrich et al., 2010), meaning western, educated, industrialized, rich, and democratic contexts. This pattern is also visible in recent work on learning analytics (Baek & Doleck, 2024) and bibliometric studies in human–computer interaction (Linxen et al., 2021). Drawing back on empirical research on artificial intelligence in higher education as an example, the presented mapping review (Sutton et al., 2019) examines the extent to which this field remains influenced by WEIRD publications. Following the PRISMA guidelines for systematic reviews (Page et al., 2021), the authors developed an extensive search string and applied pre-determined inclusion and exclusion criteria. Studies were included if they were primary studies that addressed students in tertiary education, clearly focused on AI in higher education, implemented a (quasi)experimental study design, incorporated a distinct method section, and were written in languages that were translatable by DeepL. Following the rigorous application of exclusion criteria during multiple rounds of screening titles and abstracts, as well as the full texts, 249 studies were deemed eligible for inclusion and data extraction. The preliminary results of this review indicate that publication structures have shifted, with almost one third of studies originating from China. However, most studies do not report participant characteristics, raising doubts about whether this geographic shift necessarily reflects more inclusive research practices. The presentation will discuss these findings and their implications for both future research and the responsible use of AI in education.

Baek, C. & Doleck, T. (2024). Learning Analytics. A Comparison of Western, Educated, Industrialized, Rich and Democratic (WEIRD) and Non-WEIRD Research. Knowledge Management & Learning, 16(2), 217-236. Bulfin, S., Henderson, M., & Johnson, N. (2013). Examining the use of theory within educational technology and media research. Learning, Media and Technology, 38(3), 337–344. Henrich, J., Heine, S. J., & Norenzayan, A. (2010). Most people are not WEIRD. Nature, 466(7302), 29–29. Hew, K.F., Lan, M., Tang, Y., Jia, C., &Lo, C.K. (2019). Where is the “theory” within the field of educational technology research? BJET, 50, 956-971. Linxen, S., Sturm, C., Brühlmann, F., Cassau, V., Opwis, K., & Reinecke, K. (2021). How WEIRD is CHI? CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 143, 1-14. Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., ... & Moher, D. (2021). Updating guidance for reporting systematic reviews: development of the PRISMA 2020 statement. Journal of clinical epidemiology, 134, 103-112. Sutton, A., Clowes, M., Preston, L., & Booth, A. (2019). Meeting the review family: exploring review types and associated information retrieval requirements. Health Information and Libraries Journal, 36(3), 202-222.