16. ICT in Education and Training
Paper
A Systematic Review of Empirical Research on Students’ ChatGPT Use in Higher Education
Kenan Dikilitas, May Irene Klippen, Serap Keles
University of Stavanger, Norway
Presenting Author: Klippen, May Irene
This review aims to synthesize empirical research evidence on student’s use of ChatGPT in higher education, emphasizing pedagogical possibilities and addressing emerging threats and challenges. Chat Generative Pre-Trained Transformer (ChatGPT) swiftly gained prominence as an open-access tool in higher education since its introduction in November 2022. It has rapidly become widely used across various domains, including higher education. The use of ChatGPT is still an emerging area, with a surge in studies reflecting its widespread adoption. As higher education institutions grapple with the integration of ChatGPT, concerns and opportunities abound. This review focuses on understanding the impact dimensions on students' use of ChatGPT, particularly considering the evolving landscape of their learning processes.
While ChatGPT use is a relatively new practice, research into it is an emerging area for researchers. However, there are still several studies that have been published in such a short time, because of the substantial use of it across the world and in every domain of life including higher education institutions by teachers, students, and administrators. In our systematic review, we examine the impact dimensions on students’ use mainly because of the increasing concerns about how they use it and how this might influence their learning. Initial studies have also explored potential benefits of ChatGPT in language learning within higher education contexts (Baskara,2023). While educational technologies driven by artificial intelligence (AI) are progressively used to automate and provide support for various learning activities (Cavalcanti et al., 2021;), recent research has focused on the impact of ChatGPT, identifying challenges and opportunities in learning, but they have not examined this within the higher education sector (Lo, 2023).
The ongoing debate surrounding ChatGPT's use in higher education presents varying perspectives. These concerns and benefits create different perspectives where some argue for its use freely and suggest that graders need to create more critical assigned tasks that require personalized and contextualized examples and justifications which may not directly be generated by ChatGPT, while others argue against its use or its use with caution by students (Tlili et al., 2023). Also, many higher education institutions have started to apply restrictions or ban ChatGPT’s use by students in their updated policy documents. On the other hand, a review of media news articles on how ChatGPT use can disrupt students’ learning and teaching in universities also revealed that the sentiment in media news is on more into the negative discourse than a positive one, hence highlighting the public discussions and university responses on such controversies about academic integrity (Sullivan et al., 2023). There are also those who believe we need to add new components in the process of assessment including verbal exams where students demonstrate their verbal ability to present the assignment that they generate (Rudolph et al., 2023).
There are several issues that emerge in the first year of the use of ChatGPT reported and discussed in the published research. However, despite the increasing body of research on ChatGPT in higher education, there is no systematic review that provides a comprehensive overview of what research has found. Therefore, it is timely to present a consolidated overview of the impact dimensions of the ChatGPT’s use and the potential implications for higher education. More specifically, in this review, we sought answers to the following research questions:
RQ 1: What are the defining characteristics of empirical research on ChatGPT in higher education?
RQ 2: What pedagogical possibilities and insights can we gain from the students’ use of ChatGPT in the context of higher education?
Methodology, Methods, Research Instruments or Sources UsedTo address our research questions, we employed a systematic review approach, following guidelines by Page et al., (2022). The methodological framework guided our process, involving literature search, study identification, data extraction/study coding, study quality appraisal, and thematic analysis.
The literature search, conducted on November 10th, 2023, targeted three databases—ERIC, Scopus, and Web of Science—chosen for their extensive coverage of educational studies. The search string, incorporated terms such as "chat generative pre-trained transform*" OR "gpt*" AND "higher education*" OR "universit*" OR "college*."
The following inclusion and exclusion criteria were used to identify relevant studies:
• Population: Students in higher education.
• Concept: Students' use of ChatGPT.
• Context: Higher education settings.
• Types of studies: Primary research with data.
• Publication language: Studies presented in full text in English.
• Time of publication: Studies published after the introduction of ChatGPT in November 2022.
Studies addressing other aspects, like performance testing or comparisons between teacher and ChatGPT feedback, were excluded. After eliminating duplicates, a two-stage screening process involved reviewing titles and abstracts, followed by full-text examination, with disagreements resolved through discussion.
Using EPPI-Reviewer Web, the second author extracted information about each study, including characteristics such as country, research question, study design, research method, study informants, field of study, and study purpose. Findings were also extracted to identify common themes, and the third author reviewed and updated the extracted data for accuracy.
Thematic analysis facilitated data synthesis and theme derivation. The analysis team, consisting of three authors, undertook a stepwise process, beginning with data extraction, followed by inductive coding, and subsequent theme generation through co-author discussions. Rigor was maintained through continuous challenge and validation of assumptions and potential biases by the third author.
The Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018) in EPPI Reviewer assessed the methodological quality of each included article. This tool, designed for various study types, involved screening questions and additional criteria for assessing quantitative, qualitative, and mixed-method studies. Ratings ('yes,' 'no,' or 'can't tell') were independently assigned by the second and third authors, with disagreements resolved through discussion. Studies with quantitative (randomized control trial), quantitative (non-randomized), and mixed-method designs were omitted from the MMAT's checklist as they were not present in the reviewed studies.
Conclusions, Expected Outcomes or FindingsEight studies were identified through a comprehensive literature search in three databases in October 2023, employing various research designs. The analysis revealed four overarching themes: 1) promoting students' learning and skill development; 2) providing content and immediate feedback; 3) activating motivation and engagement; and 4) addressing ethical aspects of ChatGPT use.
The results in our review show that ChatGPT might function as an effective tool to provide timely scaffolding by offering precisely enough assistance to empower students to eventually complete their tasks autonomously. Consistently, studies highlight positive impacts, acknowledging ChatGPT for improving writing skills, promoting personalized learning, and facilitating self-directed learning.
ChatGPT's role in providing feedback is essential, offering real-time assistance to enhance writing and deepen understanding. This feedback enriches the teaching and learning experience, fostering connection.
Findings indicate students view ChatGPT as a motivational tool, recognizing its role in minimizing affective barriers, reducing stress during assignments. Positive perceptions encourage usage, emphasizing teachers' role in enhancing perceived usefulness.
However, concerns include potential ethical issues, plagiarism, unauthorized information ownership, and the risk of impeding creativity and critical thinking. Some studies express concerns about blind reliance, potentially slowing actual learning progress.
The systematic review suggests practical implications. Clear guidelines, workshops, and ethical ChatGPT use promotion in higher education institutions are recommended. Essential training programs for students and teachers, emphasizing responsible use, are crucial. Redefining assessment policies, aligning with the assessment for learning approach and incorporating multiple evaluation points throughout the course, is advised.
In conclusion, the systematic review recognizes the evolving landscape of ChatGPT's integration into higher education and aims to provide a consolidated overview of its impact dimensions and potential implications. By addressing critical research questions, the review endeavors to contribute valuable insights for higher education decision-makers and policymakers navigating the complex terrain of AI-driven tools in the educational landscape.
ReferencesBaskara, R. (2023). Exploring the implications of ChatGPT for language learning in higher education. Indonesian Journal of English Language Teaching and Applied Linguistics, 7(2), 343-358.
Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gašević, D., & Mello, R. F. (2021). Automatic feedback in online learning environments: A systematic literature review. Computers and Education: Artificial Intelligence, 2, 100027.
Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., & Nicolau, B. (2018). Mixed methods appraisal tool (MMAT), version 2018. Registration of copyright, 1148552(10). Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), 410.
Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), 410.
Page, M. J., Moher, D., & McKenzie, J. E. (2022). Introduction to PRISMA 2020 and implications for research synthesis methodologists. Research synthesis methods, 13(2), 156-163.
Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6(1), 342-363.
Sullivan, M., Kelly, A., & McLaughlan, P. (2023). ChatGPT in higher education: Considerations for academic integrity and student learning.
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10(1), 15.
16. ICT in Education and Training
Paper
What do Teachers Think about the Use of Generative Artificial Intelligence (GAI) in their Classrooms?
Carlos De Aldama1, Beatriz Cabellos2, Juan-Ignacio Pozo3
1Complutense University of Madrid, Spain; 2Cardenal Cisneros Higher Education Centre, Spain; 3Autonomous University of Madrid, Spain
Presenting Author: De Aldama, Carlos
Recently, the advent of Generative Artificial Intelligence (GAI) has sparked significant interest and debate in the field of education. GAI technologies, characterized by their ability to generate new content and provide personalized learning experiences, are reshaping educational paradigms (Dai et al., 2023). These technologies, including advanced language models and adaptive learning systems, offer unique opportunities and challenges for teaching and learning processes (Lo, 2023).
Despite the growing interest of GAI as educational tools, there is a lack of research focusing on teachers' perceptions and beliefs about using these technologies in educational settings. In a study conducted by Kaplan-Rakowski et al. (2023), the authors found that, in general, teachers hold favorable views toward the use of GAI in educational settings, irrespective of their individual teaching methodologies. The study revealed a correlation between the frequency of GAI usage by teachers and the positivity of their attitudes towards it. Similar results were found in a recent report with Spanish teachers and families (GAD3, 2024). Moreover, younger teachers hold a more positive view concerning the use of GAI in educational contexts than older ones.
In another study conducted by Al-Mughairi and Bhaskar (2024), the factors affecting the adoption AI techniques in higher education were explored. Applying a thematic analysis, the authors found both encouraging and inhibiting factors for the adoption of GAI in educational settings. In particular, four key themes that drive teachers to integrate ChatGPT into their educational practices were identified: 1) The pursuit of innovative educational technologies, 2) Customization of teaching and learning experiences, 3) Efficiency in terms of time management, and 4) Opportunities for professional growth. Conversely, five factors that pose as barriers to adopting ChatGPT were found: 1) Concerns about the tool's reliability and accuracy, 2) A decrease in human-to-human interaction, 3) Issues related to privacy and data security, 4) The absence of adequate support from educational institutions, and 5) The risk of becoming overly dependent on ChatGPT.
Teachers' beliefs play a crucial role in the adoption and effective integration of new technologies in teaching practices. Although there is a growing body of research in this regard, there is still a lack of evidence analyzing these views under theoretical lenses (i.e. to what extent are these beliefs more teacher or student-centred?). Understanding these beliefs is essential for developing strategies that support teachers in navigating the challenges posed by GAI and leveraging its benefits effectively.
Purpose of Study
This study aims to fill this gap by exploring Spanish teachers' beliefs about the use of Generative AI in educational contexts. To this end, we have developed a comprehensive questionnaire comprising 38 items, designed to explore what teachers think about how GAI could affect four dimensions of teaching/learning practices: (1) the kind of learning processes activated by students (more content o process centred) (2) the type of information management performed by students, (3) the evaluation processes designed by teachers and (4) the changes in teachers’ roles and identity as a consequence of the introduction of GAI.
The objective is to validate this instrument and collect data from Spanish teachers, providing insights that could inform the development of pedagogical strategies and technological tools that align with teachers' perspectives and educational goals.
Methodology, Methods, Research Instruments or Sources UsedAt the current stage of this research, we are focused on establishing the content validity of the instrument. Twelve subject matter experts, with extensive knowledge in the fields of education, technology, and psychometry, have been engaged to review the questionnaire. They have provided feedback on the relevance, clarity, and appropriateness of each item.
Following this, we plan to assess the questionnaire's construct validity using Exploratory and Confirmatory Factor Analysis (EFA & CFA). EFA will be used to uncover the underlying structure of the questionnaire and to identify the interrelationships among the items. CFA will follow to confirm the structure and test our hypotheses about the underlying constructs that the questionnaire is intended to measure.
To evaluate the internal consistency of the questionnaire, a reliability analysis will be conducted, employing appropriate methods such as Cronbach's Alpha and/or McDonald's Omega. These statistical techniques will measure the extent to which the items within each dimension are correlated, thus providing an indication of the reliability of the scales.
Once the instrument has been piloted and refined based on feedback and statistical analysis, we aim to collect data from at least 200 teachers in higher education. This sample size is chosen to ensure a diverse and representative dataset, enhancing the generalizability of our findings. Both the validation process and the preliminary results will be showcased at the ECER 2024.
Conclusions, Expected Outcomes or FindingsThe aim of this study is twofold. On the one hand, it focuses on the validation of a new instrument designed to measure teachers' beliefs about Generative Artificial Intelligence (GAI) in educational contexts. On the other, it aims to present initial findings on these beliefs, shedding light on how Spanish educators perceive the integration of GAI into their teaching practices and the broader educational landscape.
This research carries practical implications for the responsible and effective integration of GAI in educational contexts. For instance, increasing our understanding of teachers ‘beliefs may enhance educators' digital literacy and competency in using GAI for personalized learning. In a rapidly evolving educational landscape, understanding, and aligning with educators' perspectives are essential for harnessing the full potential of AI.
ReferencesAl-Mughairi, H., & Bhaskar, P. (2024). Exploring the factors affecting the adoption AI techniques in higher education: insights from teachers' perspectives on ChatGPT. Journal of Research in Innovative Teaching & Learning.
Dai, Y., Liu, A., & Lim, C. P. (2023). Reconceptualizing ChatGPT and generative AI as a student-driven innovation in higher education. https://doi.org/10.35542/osf.io/nwqju
GAD3 (2024). El impacto de la IA en la educación en España. https://empantallados.com/ia/
Kaplan-Rakowski, R., Grotewold, K., Hartwick, P., & Papin, K. (2023). Generative AI and teachers’ perspectives on its implementation in education. Journal of Interactive Learning Research, 34(2), 313-338.
Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), 410.
16. ICT in Education and Training
Paper
Artificial Intelligence Readiness in Education: the Student Teachers’ Journey
Rachel Farrell1, Pamela Cowan2
1University College Dublin, Ireland; 2Queen's University Belfast, United Kingdom
Presenting Author: Farrell, Rachel;
Cowan, Pamela
Five key ideas exist in the area of Artificial Intelligence: Perception, Representation and Reasoning, Learning, Natural Interaction and Societal Impact. This research project aims to consider the contribution (positive and negative) of AI in the field of education and address the ‘readiness’ of Teacher Education to utilise AI as a tool for supporting student teachers’ development in teaching and learning.
It is acknowledged that Generative-AI (GAI), such as ChatGPT, lacks the ability to provide criticality, depth and accuracy needed for Masters level writing on PME and PGCE programmes, however as a tool for providing formative feedback or acting as an ‘intelligent tutoring system’, AI could offer exciting opportunities in terms of supportive, personalised, ‘just-in-time’ assistance to student teachers if they were taught properly in how to use generative-AI tools. However this goal requires student teachers to be confident and comfortable in the ethical and effective use of AI. Luckin et al. (2022) refer to “AI Readiness” as the journey that students (and faculty) must take to move from a lack of understanding about the nature of AI and its potential, to comprehending AI’s capabilities and shortcomings, with an awareness of the ethical, social and legal implications of engaging with such a complex technology (Harvard Business Review, 2023). This research study addresses DigiComp 2.2 – the European Digital Competence Framework (Vuorikari et al., 2022) - which was updated in 2022 to include a competence focusing on knowledge, skills and attitudes related to citizens interacting with AI systems, as opposed to technical knowledge about AI itself.
● What are student teachers’ attitudes towards AI and GAI?
● What is the connection between AI anxiety and learning motivation?
● What AI is currently be utilised for educational purposes?
Methodology, Methods, Research Instruments or Sources UsedThis research project aims to investigate the integration of artificial intelligence (AI) in Initial Teacher Education (ITE) programs, focusing on student teachers in two partner institutions in two neighbouring countries. The study will unfold in three phases, employing an exploratory sequential mixed methods approach.
In Phase 1, a literature-based review will identify various types of AI implementation in curricula. The analysis will be aligned with the 5 key ideas of AI, guiding the development of materials for the AI Readiness Journey, intended for online delivery.
Moving to Phase 2, the AI-Readiness Journey in ITE will commence with a survey gauging student teachers' attitudes towards AI before undertaking the journey. Building on previous work regarding Technology Readiness, the survey will incorporate an AI Attitude scale. This phase aims to correlate AI attitudes with measures of Technology Readiness, following research by Schepman & Rodway (2022). Participants will engage with AI-Readiness Journey materials to enhance their understanding of AI's potential in education.
Phase 3 focuses on Generative-AI as an Intelligent Tutoring System (ITS). Student teachers will be trained in utilizing ChatGPT (or other Generative AIs) to support their knowledge development in teacher education. This phase specifically targets core terminology, theory-practice links, applications of Generative AI for planning, and reflection.
Throughout the study, an analysis of survey data will be conducted using SPSS, while qualitative comments will undergo thematic analysis based on Braun & Clarke's framework (2020). Any patterns discerned across subject disciplines or between the two countries will be reported. Although the participant pool might not support robust inferential statistical analysis, this option remains open depending on uptake in Phases 2 and 3. The research aims to shed light on the integration of AI in teacher education and its impact on student teachers' attitudes and readiness.
Conclusions, Expected Outcomes or Findings The project's focus on identifying models of AI practice in schools across Northern Ireland (NI) and Ireland is expected to yield valuable insights into the diverse landscape of AI applications in curriculum-based learning. The cross-border cooperation among researchers is crucial in navigating the rapidly changing technological landscape and providing alternative perspectives. The AI-Readiness Journey materials will be instrumental in showcasing how Generative-AI (GAI) can serve as an Intelligent Tutoring System (ITS), supporting student teachers' pedagogical practices during school-based placements. Initial findings suggest the transferability of ITS processes across curricula in both regions and within Europe, emphasising the potential harmonisation of AI implementation in teacher education.
The expected outcomes of the project include substantial capacity building in Initial Teacher Education (ITE) programs, primarily benefiting student teachers and potentially extending to teachers in placement schools. Measurable outcomes, such as exemplars of AI in the curriculum, an online AI-Readiness Journey toolkit, and examples of GAI as an ITS, will be shared electronically. These resources aim to modernise ITE programs, providing practical skills in AI and GAI for future educators.
Student teachers stand to gain awareness and practical skills in AI and GAI usage, fostering a community of practice within their institutions. Policymakers, including Ireland's Teaching Council and NI's Education Authority, are positioned to receive valuable insights for policy formulation. Institutional benefits extend to the modernization of ITE programs, potentially impacting placement schools through capacity building.
The project's outcomes are expected to be well-received, fostering interest and enthusiasm for experimenting with new AI technologies without fear of failure. This approach aligns with the overarching goal of enhancing AI literacy in teacher education, benefiting not only the immediate participants but also the wider academic community across the island of Ireland and Europe.
ReferencesBraun, V., & Clarke, V. (2020). Thematic Analysis: A Practical Guide. Sage Publications.
Luckin, R., Pritchard, A., Ainsworth, S., Akpan, J., & Law, N. (2022). Artificial Intelligence and Education - A summary of the discussions at the Global Education Leaders’ Partnership AI in Education Symposium. Harvard Business Review. https://hbr.org/sponsored/2022/05/artificial-intelligence-and-education
Schepman, A., & Rodway, P. (2022). Exploring the Relationship between Attitudes towards Artificial Intelligence and Technology Readiness. Journal of Technology in Human Services, 40(1), 55–75. https://doi.org/10.1080/15228835.2022.2068361
Vuorikari, R., Kankaanranta, M., Ala-Mutka, K., Bacigalupo, M., & Manganello, F. (2022). DigiComp 2.2 - The Digital Competence Framework for Citizens with eight proficiency levels and examples of use (Joint Research Centre Science for Policy Report). Publications Office of the European Union. https://publications.jrc.ec.europa.eu/repository/bitstream/JRC109361/jrc109361_2017%20digcomp%202.2.pdf
|
