Conference Agenda

In times of ever-increasing numbers of publications and easier accessibility, e.g. through Open Access publishing, the overview of existing research and secured knowledge can be lost (Brooth et al., 2016). Systematic reviews are a suitable method to synthesize research knowledge in a criterion-guided and transparent way and to provide a structured overview of the research field under investigation. Thus, with systematic reviews reliable findings can be bundled and at the same time the need for further research can be identified. The transfer of synthesized knowledge takes place in the course of evidence-based policy advice (Pawson, 2006) as well as by reflecting the results back to the research community (Gough et al., 2017). As a part of the Open Science movement, systematic reviews objectively select and combine available studies and thus promote the accessibility of scientific knowledge to the interested public.

Originally invented in the field of medicine (see evidence-based medicine; Sackett et al., 1996; Thoma & Eaves, 2015), systematic reviews are increasingly being implemented in educational research (Zawacki-Richter et al., 2020). However, there is the challenge of transferring the methodological approach to the studies conducted in educational research. Whereas in medicine most studies are based on randomised control group trials with a similar research design, the strength of educational research lies in its plurality of empirical approaches, both from a qualitative or quantitative paradigm. But how can a structured and criteria based synthesis based on such a heterogeneous background of studies be achieved? The symposium addresses the challenges to this transfer of the method in the preparation of a systematic review. Furthermore we will discuss how methodological guidelines and quality standards for systematic reviews in the field of educational research can be implemented.

The four papers in the symposium show how a systematic aggregation of studies in a research field can succeed and focus on challenges to the respective methodological steps. They stem from different disciplinary fields of educational research (sociology, psychology, and educational sciences) and contribute to an international perspective, covering literature from various countries and publication languages. The first presentation concentrates on the challenges in the process of the literature search and identification of relevant studies for a systematic review. Examining the case of Germany, the paper deals with the question how educational inequality can be reduced and addresses educational barriers in formal, non-formal and informal learning contexts. The second paper covers the theme of digitalisation in cultural education – based on a literature search at an international level. The authors take a look at the support of selection and categorization of literature using text mining methods. The third paper presents a systematic review of social inequality in educational attainment from preschool to higher education that covers the state of research from the German-speaking area (Germany, Austria, and Switzerland). The presentation focusses on the collection of relevant information (coding) as well as the synthesis of evidence by means of research maps (gap maps). The fourth paper deals with the language bias in international systematic reviews in the field of educational technology. The trilingual team of authors presents a mapping review based on existing systematic reviews including the languages of English, Spanish and German.

Children, adolescents and young adults from socially disadvantaged families face educational barriers in the acquisition of competences or the access to learning opportunities (Hadjar & Gross, 2016; Maaz & Dumont, 2019; among others). While there is broad research on how and where educational barriers occur and arise, fewer studies exist on how disadvantages in education can be compensated or prevented in Germany. Also, researchers have mainly investigated the overcoming of educational barriers in formal education settings (Autorengruppe Bildungsberichterstattung, 2020, p. 79). Most importantly, comprehensive systematic reviews on overcoming educational barriers do not exist for Germany. Our aim is thus to systemise research on processes, measures and programs for overcoming educational barriers in different learning environments ─ formal, non-formal and informal ─ in Germany. We will include studies concerning educational inequality from birth up to the transition to post-secondary education (age range 0-27). We will consider empirical qualitative and quantitative longitudinal and cross-sectional studies from the social and educational sciences and related disciplines. We focus on educational barriers which are (re-)produced or increased, e.g., by the social or ethnic background, gender, physical or mental impairment (Wenzel, 2008, p. 430). We intend to identify measures which prevent such mechanisms of exclusion and which consequently promote diversity. For the intended review, we need to consider a broad range of criteria of finding and selecting relevant studies (Newman & Gough, 2020). Resulting challenges in the literature search and screening processes will be the focus of the presentation. Central questions are: How can we generate a systematic search strategy to produce a resource-oriented outcome? How to deal with technical restrictions of the literature database? How can we define selection criteria so that different reviewers follow a unified screening procedure? How to document screening decisions in a detailed and transparent way? The paper will illustrate innovative automated solutions to these questions. The literature found with search terms related to ‘overcoming educational barriers’ was post-hoc filtered for search terms related to social inequality and social background using R (R Core Team, 2020; package stringr, Wickham, 2019), which allowed for a more economic search procedure. An electronic questionnaire guides the reviewer through the screening process and documents each decision. These new methods facilitate and systemize literature search and screening processes. As an outlook, we will present first results and discuss implications for educational research.

Autorengruppe Bildungsberichterstattung (Hrsg.). (2020). Bildung in Deutschland 2020. Ein indikatorengestützter Bericht mit einer Analyse zu Bildung in einer digitalisierten Welt. W. Bertelsmann Verlag. Hadjar, A. & Gross, C. (Hg.) (2016). Education Systems and Inequalities. Bristol: Policy Press. Maaz, K., & Dumont, H. (2019). Ungleichheiten des Bildungserwerbs nach sozialer Herkunft, Migrationshintergrund und Geschlecht. In O. Köller, M. Hasselhorn, F. W. Hesse, K. Maaz, J. Schrader, H. Solga, C. K. Spieß & K. Zimmer (Hrsg.), Das Bildungswesen in Deutschland: Bestand und Potenziale (S.299-332). utb. Verlag Julius Klinkhardt. Newman, M., & Gough, D. (2020). Systematic Reviews in Educational Research: Methodology, Perspectives and Application. In O. Zawacki-Richter, M. Kerres, S. Bedenlier, M. Bond & K. Buntins (Hrsg.), Systematic Reviews in Educational Research: Methodology, Perspectives and Application (S.1-22). Springer Nature. R Core Team (2020). R: A language and environment for statistical computing. Version 4.0.2. R Foundation for Statistical Computing, Vienna, Austria. URL: https://www.R-project.org/. Wenzel, H. (2008). Studien zur Organisations- und Schulkulturentwicklung. In W. Helsper & J. Böhme (Hrsg.), Handbuch der Schulforschung (2. Aufl.) (S. 423-447). VS Verlag für Sozialwissenschaften. Wickham, Hadley (2019). stringr: Simple, Consistent Wrappers for Common String Operations. R package version 1.4.0. https://CRAN.R-project.org/package=stringr

Systematic reviews are an essential tool to obtain an overview of interdisciplinary research fields. However, they are often hampered by broad search strings leading to many false negative results. Such literature searches can lead to several 100,000 papers and manual screening and categorization are no longer feasible. In such cases, text mining can support screening and categorizing of large literature corpora. In our study, we used a corpus of n > 250,000 papers from a literature search on international research on digitalisation in arts, aesthetic and cultural education (D-ACE). After cleaning the exported texts (title, journal, keywords, abstracts), we applied an iterative procedure of predictive modeling and prioritized screening (as described in detail by Christ et al., 2021), followed by topic modelling. Predictive modelling was utilized to avoid having to screen all texts. Finally, topic modeling of the included papers was performed to determine content and size of hot topics of research on D-ACE. Across iterations, the inclusion rate decreased from 85.8% to 1.8% in the 17th iteration. In total, n = 3,846 papers (including n = 1786 from the training set) were included. Most of them were from authors from the USA, followed by authors from the UK, Taiwan, Australia, Spain, Canada and Germany. The publication rate has increased at an accelerated rate since 2012, which did not differ for the affiliation country or continent of the authors. The included papers were first analyzed via topic modelling to determine the latent research topics within the corpus, followed by a more specific analysis of investigated cultural activities. In total, k = 31 latent topics were covered in the included papers. Hot topics included “formal education and learning”, “serious games and gamification of learning”, “popular games and engagement”, “user experience and interface design” and “effects on personality and behavior”. Focusing on the investigated cultural activities resulted in k = 17 topics containing all major facets of cultural activities i.e. music, literature, visual arts, performing arts and games. With k = 8, a majority of topics related to video games. The distribution of articles in the topic models did not differ according to the affiliation countries or continents. Overall, predictive modelling and priority screening turned out to be well suited for efficiently identifying hot topics of international research during preparation of systematic reviews. Implications for further development of these methods as well as for conducting systematic reviews and original work are discussed.

Christ, A., Penthin, M., & Kröner, S. (in revision). Two decades of research syntheses on digital cultural education: A tertiary review. Christ, A., Penthin, M., & Kröner, S. (2021). Big data and digital aesthetic, arts, and cultural education: Hot spots of current quantitative research. Social Science Computer Review, 39, 821-843. https://doi.org/10.1177%2F0894439319888455 Kröner, S., Christ, A., & Penthin, M. (2021). Stichwort: Digitalisierung in der kulturell-ästhetischen Bildung–eine konfigurierende Forschungssynthese [Digitalization in aesthetics, arts and cultural education – a scoping review]. Zeitschrift für Erziehungswissenschaft, 24, 9-39. https://doi.org/10.1007/s11618-021-00989-7

Social inequalities can be found in various aspects of educational attainment at different points in the educational biography, from early childhood education to higher education. There is a positive effect of a privileged parental background on participation in institutionalized early childhood education and care (e.g., Stahl & Schober, 2018), academic-track attendance at secondary school level (“Gymnasium”, e.g., Dumont et al., 2019) and university entrance (e.g., Reimer & Pollak, 2010). Social background effects can also be found in children’s and adolescents’ competencies (e.g., Linberg et al., 2019) and their later degrees (Becker, & Meyer, 2019). Still, the existing research needs to be synthesized in order to assess the entire evidence base. This is where the present systematic review sets in. For the first time, the available research on social inequalities in educational attainment in predominantly German-speaking countries (Germany, Austria & Switzerland; included publication languages German, English, French) was systematized for the a) pre-school level, b) primary and secondary school level as well as c) higher education and d) vocational training for a period of 20 years (2000-2020). The aspects of educational participation, competencies and grades were included as well as educational qualifications. The present systematic review follows the methodical steps of 1) an extensive literature search, 2) a two-step criteria-based literature selection (screening) and 3) coding of relevant information (Gough et al., 2017). A total of 33,662 literature references were identified using online database searches (FIS Bildung, Google Scholar, ERIC, DNB and educational sector specific databases) and screened regarding the studies’ relevance for the research question and their scientific quality. 568 studies could be included in the coding process. The general results show that social inequality is persistent throughout all educational sectors (Bachsleitner et al., 2022). The strongest evidence can be found for the primary and secondary school level based on n= 318 included studies. The present study manages thus to combine robust findings on social inequalities in educational attainment and highlight further research potentials. Furthermore, conclusions can be drawn about the field of inequality research itself. The evidence on social inequality effects will be presented using gap maps, which provide a way of illustrating the found evidence. The methodological procedures for quality assurance as well as the challenges of a broadly based systematic review will be outlined.

Bachsleitner, A., Lämmchen, R., & Maaz, K. (2022). Soziale Ungleichheit des Bildungserwerbs von der Vorschule bis zur Hochschule. Eine Forschungssynthese zwei Jahrzehnte nach PISA. Waxmann. Becker, R. & Mayer, K. U. (2019). Societal change and educational trajectories of women and men born between 1919 and 1986 in (West) Germany. European Sociological Review, 35(2), 147–168. Dumont, H., Klinge, D., & Maaz, K. (2019). The many (subtle) ways parents game the system: Mixed-method evidence on the transition into secondary-school tracks in Germany. Sociology of Education, 92(2), 199–228. Gough, D., Oliver, S., & Thomas, J. (2017). An introduction to systematic reviews (2nd Edition). Sage. Linberg, T., Schneider, T., Waldfogel, J., & Wang, Y. (2019). Socioeconomic status gaps in child cognitive development in Germany and the United States. Social Science Research, 79, 1–31. Reimer, D., & Pollak, R. (2010). Educational expansion and its consequences for vertical and horizontal inequalities in access to higher education in West Germany. European Sociological Review, 26(4), 415–430. Stahl, J. F., & Schober, P. S. (2018). Convergence or Divergence? Educational Discrepancies in Work-Care Arrangements of Mothers with Young Children in Germany. Work, Employment and Society, 32(4), 629–649.

In recent years, the interest in and use of secondary research to synthesise evidence in education has grown considerably. While systematic reviews often aim to analyse a large corpus of studies, educational technology research often focuses on very specific and locally targeted contexts, with nationally or linguistically defined communities (Marín et al., in press). As such, in their methodological development and adaptation to educational research, evidence syntheses have not yet sufficiently taken into account and reflected key aspects such as research context and the language of publication (Zawacki-Richter et al., 2020). This implies potential biases with regard to the associated visibility, validity and discoverability of results. This paper addresses these content-related and methodological challenges through a mapping review of reviews (Sutton et al., 2019), by exploring how (non)English-language primary studies have been synthesised in educational technology research. Search strings were developed in English, Spanish and German, and were used to search in the databases ERIC, Scopus, Web of Science, Dialnet, FIS Database and Google Scholar, yielding 7,275 items in the initial search. A sample of articles was drawn from the corpus, using methods for estimating sample size in the social sciences. 446 educational technology systematic reviews, published since 1983 in article, conference paper, chapter, or report form, in English, Spanish or German, were analysed using a pre-defined coding scheme. Among other aspects, the languages of publication of the primary studies and the reasons for or against the exclusion of certain languages were coded. The coding scheme can thus be used to demonstrate how existing systematic reviews deal with primary studies in different publication languages, and the extent to which bias exists with respect to consideration of English-language and peer-reviewed articles (e.g., Jackson & Kuriyama, 2019). Of the 446 systematic reviews, only 17% included studies in more than one language, and a further 42% did not provide any information about the language of inclusion (n=75). Multilingual searches were even less common, with only 8% of studies (n=35) searching for publications in more than one language, and only 7% (n=30) included articles with different languages. The findings of the review, including consideration of its own limitations (e.g., exclusion of reviews published in languages other than English, Spanish or German, choice of databases), are situated within the evolving methodological discussion of educational systematic reviews (e.g., Chong et al., 2023) and reflected against the larger context of educational publication structures (Beigel, 2021).

Beigel, F. (2021). A multi-scale perspective for assessing publishing circuits in non-hegemonic countries. Tapuya: Latin American Science, Technology and Society, 4(1), 1845923. https://doi.org/10.1080/25729861.2020.1845923 Chong, S. W., Bond, M., & Chalmers, H. (2023). Opening the methodological black box of research synthesis in language education: where are we now and where are we heading? Applied Linguistics Review. https://doi.org/10.1515/applirev-2022-0193 Jackson, J. L., & Kuriyama, A. (2019). How often do systematic reviews exclude articles not published in English? Journal of General Internal Medicine, 34(8), 1388–1389. https://doi.org/10.1007/s11606-019-04976-x Marín, V. I., Buntins, K., Bedenlier, S., & Bond, M. (in press). Invisible borders in educational technology research? A comparative analysis. Educational Technology Research and Development. Sutton, A., Clowes, M., Preston, L., & Booth, A. (2019). Meeting the review family: Exploring review types and associated information retrieval requirements. Health Information and Libraries Journal, 36(3), 202–222. https://doi.org/10.1111/hir.12276 Zawacki-Richter, O., Kerres, M., Bedenlier, S., Bond, M., & Buntins, K. (Eds.). (2020). Systematic Reviews in Educational Research. Springer Fachmedien Wiesbaden. https://doi.org/10.1007/978-3-658-27602-7 Victoria I. Marín acknowledges the support of the Grant RYC2019-028398-I funded by MCIN/AEI/ 10.13039/501100011033 and FSE “El FSE invierte en tu futuro”.