ID: 246 / Workshop B-04: 1
Topics: Technology Uptake
Keywords: systematic review, machine learning, crowdsourcing, artificial intelligence, technology
Human and artificial intelligence: new technologies and processes to find studies for systematic reviews ( 2 x 75 min)
1EPPI-Centre, UCL, United Kingdom; 2Radcliffe Department of Medicine, University of Oxford, United Kingdom
BACKGROUND: The large and growing number of research publications, coupled with poor search precision, can make identifying all studies eligible for inclusion in a systematic review both challenging and time consuming. Machine learning and text mining technologies have great potential, but may best be considered as aids to human effort, rather than replacements. Emerging approaches to finding research are not limited to technological solutions though, and new human processes – including ‘crowdsourcing’ - are showing that it is possible to make the study identification process more efficient.
AIMS: To present, and for participants to have hands-on experience with, some of the latest automation and crowdsourcing tools to support study identification in systematic reviews. To consider critically the evidence base that supports the use of the tools. To discuss their use as a group, and how users might contribute to their further development and evaluation
CONTENT: We will outline and experience the ways in which new technologies are being applied to searching and study selection in systematic reviews. We will provide overviews of: Current applications for searching, including approaches that aim to improve sensitivity and/or precision, or to aid database translation; Current applications for study selection, including approaches that aim to reduce the number needed to screen or expedite quality assurance; Living systematic reviews: how we can utilise new technologies to maintain the currency of a given review – or suite of reviews; How some study identification tasks can be carried out at scale – outside the scope of individual reviews – making study identification much more efficient, and reducing duplication of effort on a global scale.
We will also summarise and discuss the current evidence base to consider as a group how mature particular technologies are, whether they are ready for use, or what additional development and evaluation is necessary.
Learning outcomes : Participants should be able to: Differentiate some ways that new technologies and processes – including machine learning, text mining and crowdsourcing - help with study identification; Be familiar – and have interacted – with some of the latest tools which utilise these new technologies and processes; Be developing a critical awareness of the evidence base and the issues that need to be borne in mind when using these tools; Have an introductory understanding of how some of the new technologies work.
Type of interactivity : Most of the time will be devoted to hands-on experience with tools, and discussion about their use. Please bring a laptop / tablet with you to try the online tools for yourself. We will adopt the following pattern of activity for each technology we cover:
For those who attended our EAHIL workshop in 2018, this year’s workshop will additionally cover crowdsourcing as well as providing up-to-the-minute overviews of the latest technologies and their evaluations. A new theme will be a focus on human-machine interaction: rather than thinking that the machine will be able to do all the work, we consider how the human and machine together are able to achieve more than either operating alone.
Level : Intermediate
Target audience : Information specialists, librarians, and review authors; also of relevance for commissioners and users of reviews
Preparation for the session : No
Biography and Bibliography
James Thomas is Professor of Social Research and Policy at the EPPI-Centre, UCL, London. His research is centred on improving policy and decision-making through the use of research. He has written extensively on research synthesis, including meta-analysis and methods for combining qualitative and quantitative research in mixed method reviews. He also designed EPPI-Reviewer, software which manages data through all stages of a systematic review, which incorporates machine learning/AI. He is principal investigator of the Evidence Reviews Facility for the Department of Health and Social Care, England, a large programme of policy-relevant systematic reviews with accompanying methodological development. James is co-lead of Cochrane ‘Project Transform’ which is implementing new technologies and processes to improve the efficiency of systematic reviews. He is also co-investigator on a major Collaborative Award from the Wellcome Trust, led by Susan Michie (UCL), to develop novel technologies to organise, synthesise and present the behavioural science literature.
Anna Noel-Storr has worked for Cochrane since 2008 as an information specialist for the Cochrane Dementia and Cognitive Improvement Group based at the University of Oxford. During that time she has played a leading role in the development and implementation of crowdsourcing in health evidence production. This began with the 'Trial Blazers' study for which she won the Thomas C Chalmers Award in 2013. Since then, she has led a number of initiatives exploring the role of crowdsourcing and citizen science in systematic review production and evidence synthesis. She currently leads Cochrane Crowd, a component of Cochrane ‘Project Transform’.This work involves the development of a crowd platform offering willing contributors a range of micro-tasks to dive into, all of which are designed to enhance Cochrane’s content and speed up the review production process without any compromise on the exceptionally high quality expected of Cochrane systematic reviews.
Claire Stansfield is an Information Scientist at the EPPI-Centre, UCL Institute of Education, London and is involved in developing and applying research methods for systematic literature searching across a range of policy areas in health promotion, public health, social care and international development. She also supports research groups internationally to learn and use literature searching methods for systematic reviews, particularly within the international development field.
Thomas J, Noel-Storr A, Marshall I, Wallace B, McDonald S, Mavergames C, Glasziou P, Shemilt I, Synnot A, Turner T, Elliott J; Living Systematic Review Network. Living systematic reviews: 2. Combining human and machine effort. J Clin Epidemiol. 2017 Nov;91:31-37. doi: 10.1016/j.jclinepi.2017.08.011