Conference Agenda

Generative artificial intelligence (GenAI) is rapidly reshaping assessment practices in higher education, intensifying debates about validity, fairness, transparency, and learner agency. While many discussions of GenAI and assessment focus on tool adoption, efficiency gains, and automation, such as scaling feedback and reducing marking workload, less is known about how educators conceptualize AI-supported assessment before implementation. This conceptual phase is particularly salient in teacher education and graduate preparation, where educators must integrate emerging technologies with established principles of assessment literacy, ethical responsibility, and pedagogical coherence.

To support principled decisions about AI involvement, the Artificial Intelligence Assessment Scale (AIAS) offers a graduated framework for specifying levels of AI use in assessment, from limited involvement to higher degrees of automation (Perkins, Roe, & Furze, 2025). Human-AI interaction scholarship similarly emphasizes augmentation, collaboration, and delegation rather than replacement, foregrounding how authority and accountability are distributed between humans and AI systems (Jarrahi, 2018). Building on these perspectives, this study examines how graduate-level educators conceptualize AI-supported assessment when designing assessment experiences without using AI tools.

Data consisted of eight lesson plans produced by graduate students (most of whom were practicing teachers or instructors) enrolled in a graduate course on instructional design and educational technology in higher education. As part of the assignment, each participant designed a two-hour lesson plan that included an assessment component aligned with a self-selected AIAS level (Levels 1–5). Participants were explicitly instructed not to use AI tools during the task, ensuring that the resulting designs reflected conceptual reasoning about AI involvement rather than tool-driven experimentation. The analytic corpus included lesson plans (learning outcomes, instructional sequence, and timing), a written justification for the chosen AIAS level, ethical and pedagogical reflections, and assessment rubrics.

Using qualitative document analysis followed by cross-case thematic analysis, I examined (a) participants’ interpretations of AIAS levels, (b) pedagogical alignment among learning outcomes, instructional activities, and assessment criteria, (c) the scope and depth of ethical reasoning, and (d) implied models of human–AI collaboration and accountability. Findings indicate a strong preference for collaborative AI uses (AIAS Level 3), with AI positioned as supporting feedback provision, assisting evaluation, or offering alternative perspectives while final responsibility remained with the educator. Fewer lesson plans selected Level 2, two approached Level 4, and none selected Level 1 or Level 5. Across cases, collaboration framings clustered around AI as a co-feedback provider/co-rater, a planning partner, a creative producer, or an authoritative evaluator.

Pedagogical coherence varied across cases. Strong alignment was evident when learning outcomes, activities, and criteria targeted the same constructs and AI was positioned to support disciplinary learning rather than becoming the object of assessment. However, some plans showed construct drift when criteria emphasized the polish or novelty of AI-generated artefacts over students’ conceptual understanding. Ethical reflections were also uneven: academic integrity, authorship, and reliability were frequently emphasized, whereas algorithmic bias, data privacy, and transparency were less consistently addressed.

Overall, the study provides evidence that AIAS-guided, non-technological design tasks can surface educators’ emerging AI assessment literacy and identify where teacher education and professional development may require stronger scaffolding for alignment, accountability, and ethical depth.

The rapid uptake of generative AI tools such as ChatGPT has intensified debate about their role in education, with much existing research evaluating their impact primarily through performance outcomes (Luckin, 2025). However, performance-based indicators provide a limited account of learning in contexts where AI systems readily generate outputs (Corbin, Dawson & Liu, 2025). Educational research has long argued that learning is better understood through the processes by which learners engage with difficulty (Bjork & Bjork, 2011; Dweck, 2006; Duckworth et al., 2007; Young, Bevan, & Sanders, 2024). Responses to challenge, including effort, persistence, perceived difficulty, and affective experience, are associated with deeper learning and broader outcomes (Porter et al., 2020; Ennion & McLellan, 2025). Yet little empirical work has examined how generative AI shapes these behavioural and psychological processes when students encounter challenging tasks.

This study addresses this gap by examining how different forms of support, human tutoring, generative AI chatbot support, and independent internet search, shape students’ responses to academic challenge. The analysis is grounded in theories of scaffolding (Gaffney & Rodgers, 2018; Vygotsky, 1978) and co-regulation (Hadwin, Järvelä, & Miller, 2017), which conceptualise learning as an adaptive process involving cognitive, behavioural, motivational, and emotional adjustment in response to difficulty. From this perspective, learning is socially shaped through interaction with others. Human tutors are typically understood as providing rich scaffolding and co-regulatory support. Generative AI occupies a more ambiguous position (Lodge, de Barba, & Broadbent, 2023; Nguyen, 2025). It offers interactive and contingent responses but lacks the social and emotional attunement characteristic of human interaction. Independent internet search represents a further contrast, providing access to information without adaptive or dialogic support and placing greater self-regulatory demands on learners. These forms of support can therefore be understood as distinct regulation ecologies that shape how learners experience and manage challenge.

A randomised controlled study was conducted with 149 students aged 16 to 18 in a sixth-form college in England. Participants were randomly assigned to one of three support conditions, one-to-one human tutor, one-to-one generative AI chatbot, or independent internet search, while working through a set of deliberately challenging academic tasks. The session consisted of two successive task sections of increasing difficulty, followed by a challenge-seeking activity. The study treated responses to challenge as outcomes in their own right rather than proxies for attainment. Measures included self-reported effort, perceived task difficulty, number of attempts, challenge-seeking behaviour, and self-reported stress. Non-parametric analyses were used to compare patterns across support conditions and to examine change across the two task sections.

Across the sample, students showed adaptive responses, with increased engagement and reduced perceived difficulty in the second task section. However, support type significantly shaped how challenge was experienced and managed. Students working with a human tutor reported the lowest perceived difficulty and stress and made the greatest number of attempts, alongside lower, slightly declining self-reported effort. Students supported by the AI chatbot reported the highest and most sustained levels of effort across both task sections, continued perceptions of challenge, and moderate levels of stress. This group also showed the largest increase in attempts across sections, suggesting persistence under difficulty. In contrast, students relying on independent internet search reported the highest difficulty and stress and showed the least adaptive change across measures.

Taken together, the findings suggest that generative AI reshapes the conditions under which learners engage with challenge rather than simply improving or undermining performance. Different forms of support shape how students regulate effort, experience difficulty, and adapt over time. The study highlights the need to move beyond attainment-focused evaluations of AI in education and toward empirical investigation of learning behaviours and regulatory processes.