1. Introduction: Theoretical framework, aims, and research questions

The gradual yet extremely rapid integration of Artificial Intelligence (AI) into educational contexts is having a substantial impact on assessment practices, calling into question traditional docimological models grounded in human judgment, transparency, and pedagogical intentionality. Within the field of experimental pedagogy, assessment is increasingly understood not as a purely technical act of measurement, but as a situated and interpretative process, capable of guiding and orienting learning trajectories. In this perspective, generative AI systems may function as decision-support tools, as they introduce new forms of mediation that reshape assessment cultures, redistribute agency, and require a rethinking of assessment-related competences.

The underlying assumption is that AI does not simply optimize existing assessment procedures, but actively contributes to transforming how assessment principles are understood, negotiated, and revised—both by those who are assessed and by those who are preparing to take on a teaching role.

The aim of the study is to empirically explore how graduates enrolled in the 60 ECTS university programme qualifying for teaching reinterpret assessment principles when these are mediated by AI.

The research addresses the following questions:

1. How are traditional assessment principles reinterpreted by prospective teachers in AI-mediated contexts?

2. What tensions emerge between automation, professional judgment, and ethical responsibility?

3. Which metacognitive and reflective dimensions characterize these assessment representations?

2. Methods

The study adopts an exploratory mixed-methods research design. Data were collected through a structured questionnaire administered to 114 participants who already hold a university degree and are enrolled in 60 ECTS programmes aimed at qualification for secondary school teaching. The sample thus consists of individuals with a consolidated academic background and an explicit professional orientation.

The research instrument consisted of a questionnaire combining closed-ended items (Likert scales and evaluative statements) with open-ended reflective questions focusing on assessment principles, the use of AI, and evaluative responsibility in educational contexts.

Quantitative data were analysed descriptively in order to identify distribution patterns and convergences across key assessment dimensions (transparency, fairness, objectivity, feedback, and personalisation). Qualitative data were subjected to thematic analysis following an inductive–deductive approach informed by docimological literature and assessment literacy frameworks.

3. Results and discussion

The findings point to a significant reconfiguration of assessment representations in the presence of AI, consistent with participants who already adopt a professional perspective. From a quantitative standpoint, a strong convergence emerges around formative principles: a substantial proportion of prospective teachers associate AI primarily with enhanced feedback, greater personalisation of assessment processes, and diagnostic support. AI is therefore predominantly interpreted as a tool that supports learning and instructional regulation, rather than as an autonomous evaluative authority.

Participants’ responses reflect a persistent tension between recognition of AI’s potential in terms of efficiency, systematisation, and decision support, and concerns related to algorithmic opacity, loss of sensitivity to educational contexts, and the risk of inappropriate delegation of evaluative responsibility. This ambivalence appears coherent with a form of reflection already oriented towards professional practice.

A particularly relevant empirical result concerns the centrality of teacher judgment. Participants consistently stress that final assessment decisions must remain the responsibility of the education professional. AI is positioned as a support for interpretation, comparison, or feedback provision, but not as a legitimate decision-maker.

A further dimension emerging from the data concerns metacognitive activation. Engagement with AI-mediated assessment scenarios prompts explicit reflection on the validity of criteria, the quality of data, and the need to critically evaluate outputs produced by automated systems. In this sense, AI acts as a productive problematising factor, making the underlying assumptions of assessment more visible and open to scrutiny.

4. Educational significance of the research

This study provides empirical evidence relevant to the debate on AI and educational assessment by showing how prospective teachers negotiate principles, roles, and responsibilities within technology-mediated assessment contexts. The findings highlight the urgency of integrating assessment literacy and AI literacy within teacher qualification programmes, with particular attention to the critical, ethical, and metacognitive dimensions of assessment.

From a pedagogical perspective, the study supports the development of human-centred assessment models in which AI operates as a co-agent under the supervision of professional judgment, strengthening the formative function of assessment without undermining its interpretative and relational nature.

Introduction

Higher education increasingly utilizes learning tools that draw on artificial intelligence (AI) and large language models (LLMs). AI-based tools have great potential to benefit student learning, for example by providing personalized and on-demand learning support (Fuchs, 2023). Despite their growing use, research on students’ trust in LLM-based educational tools remains limited. Trust is a key prerequisite for meaningful learning with such tools, as it reflects the extent to which students are willing to follow the tools’ recommendations (Madsen & Gregor, 2000). Both overtrust and distrust can hinder learning through misuse or disuse of these tools (Lee & See, 2004). Therefore, students should calibrate their trust to a level that aligns with the tools’ actual capabilities (Lee & See, 2004). To foster trust calibration, we need to understand which factors shape students’ trust. Prior research (e.g., Hohenberg & Guess, 2023; Wang & Ding, 2024) suggests that explainability (transparency about how a system works) and source credibility (credibility of the person recommending a system) are factors that can influence trust. While these factors have been linked to trust in domains such as automation, health or commerce, their role in (AI-based) educational technology remains largely unexplored. This study therefore addresses the following research question: How do explainability and source credibility influence students’ trust in LLM-based programming learning tools?

Methods

We conducted an online 2×2 between-subjects experiment with university students (N = 201) enrolled in programming courses. Programming was selected as the learning context, as AI-based tools are widely used in this domain. In the experiment, participants were introduced to a fictitious LLM-based programming learning tool called CodeCara. The pedagogical purpose of CodeCara is to support students’ planning skills by providing LLM-based feedback on their plans of solving programming problems. The introduction to CodeCara varied systematically in explainability and source credibility: First, participants read a recommendation for CodeCara presented as either an email from a professor (high-credibility condition) or a WhatsApp message from a fellow student (low-credibility condition). Subsequently, participants watched a video demonstrating CodeCara. Compared to the unexplained condition, the explained condition featured an enhanced version of CodeCara which additionally included information on its background processes. Afterwards, participants completed questionnaires measuring their trust in CodeCara, perceived explainability of CodeCara, and perceived credibility of the recommender.

Results and Discussion

While the explained version of CodeCara increased perceived explainability, it did not alter students’ trust in CodeCara. In contrast, students’ trust in CodeCara was higher when CodeCara was recommended by a professor than by a student, although perceived source credibility did not differ. Beyond the experimental manipulations, perceived explainability and perceived source credibility were positively correlated with trust. These findings indicate that source credibility can impact trust, but possibly more through implicit than conscious processes. However, explainability seems to not always influence trust. Its effect may depend on moderating factors such as the type of explanation provided (e.g., about the tool’s process, intent, or performance), their comprehensibility, the perceived task risk, students’ initial trust levels, or the cognitive load induced by the additional information. Overall, learners’ individual perceptions of explainability and source credibility might additionally affect trust, independent of whether these cues are added.

Educational Significance of the Research

This study contributes to the growing field of AI in education by examining factors that shape students’ trust in LLM-based learning tools. The findings show that who promotes an LLM tool can influence students’ trust, even when the tool itself is unchanged. Therefore, guidance from educators may foster students’ successful use of these tools. However, credibility can also increase vulnerability to misinformation if trusted sources provide biased recommendations. This highlights the need to strengthen students’ critical thinking as well as educators’ awareness of their influence. While explainability might also be an important factor for trust calibration, its impact is not yet fully understood and likely depends on multiple, contextual factors. Poorly designed explanations may even undermine learning outcomes. Therefore, educators should be cautious when integrating explainable AI systems into education. Overall, the findings show that trust calibration in educational AI contexts is complex and requires further research. Such work is crucial to establish an empirical foundation for the use of LLM tools in education.

1. Introduction

Generative artificial intelligence (AI) is rapidly reshaping teaching and learning in Higher Education (HE). Teachers are expected to design AI-aware learning activities, critically appraise AI-generated content, address ethical and legal implications, and support students’ responsible use of AI tools. Existing digital competence frameworks for HE Teachers address generative AI only indirectly and do not yet specify which AI-related competences HE teachers need.

In parallel, research on generative AI literacy has produced the Generative AI Literacy Assessment Test (GLAT), a performance-based instrument that models generative AI literacy as a set of knowledge- and action-oriented competencies.. At the same time, the Frankfurt Model of Digital Competencies for HE Teachers offers a HE-specific framework of digital teaching competences, structured into eight dimensions and several subfields, and has been used in empirical studies on university teachers’ digital skills.

The present study aims to connect these strands by extending the Frankfurt Model with AI-related competences derived from GLAT. The study addresses three research questions:

(1) Which AI-related competences do experts consider most relevant for university teachers in the next three to five years?

(2) How can these competences be mapped onto the dimensions and subfields of an established digital teaching competence framework?

(3) Where do experts converge or diverge in their judgments about the relevance and placement of AI-related competences?

2. Methods

To address these questions, the study employs a multi-round Delphi design, which is suitable for structuring expert judgment in complex, future-oriented domains and for refining conceptual frameworks. The Delphi panel will consist of approximately 25–30 experts from several European countries, including researchers in HE pedagogy and educational sciences, specialists in technology-enhanced learning and academic development, and experienced HE teachers.

The study is organized in at least two rounds. In Round 1, participants receive a concise overview of the eight dimensions and subfields of the Frankfurt Model, presented in brief, accessible descriptors, and 25 GLAT-derived competence concepts, reformulated as short, teacher-focused statements. For each statement, experts rate its relevance for HE teachers on a Likert scale, assign it to up to two dimensions of the Frankfurt Model and optionally to specific subfields, and provide open comments on wording, placement, and missing competences. Additional free-text prompts invite experts to describe further AI-related competences they regard as important but not represented.

Round 1 quantitative data are analyzed using descriptive statistics (e.g., medians, interquartile ranges, distributions of dimensional mappings) to identify areas of consensus and divergence. Qualitative responses are analyzed through thematic content analysis to cluster suggestions for rephrasing, re mapping, and adding new competences.

In Round 2, participants receive structured, anonymized feedback for selected competence statements, including summary statistics and condensed argument clusters, and are invited to reconsider their relevance ratings and mappings in light of the group feedback. New competences synthesized from Round 1 comments are added as additional statements.

If needed, a brief third round will focus on persistent outliers to distinguish between possible convergence and stable, theoretically informative dissensus.

3. Results and Discussion

At the time of the conference, Round 1 is expected to be completed and Round 2 ongoing or finalized. We expect to present a profile of AI-related competencies ranked by perceived relevance for HE teachers, mapping patterns showing how different AI competences align with or extend the dimensions of the Frankfurt Model, and clusters of expert commentary highlighting contested areas.

We anticipate that some competences will cluster around digital teaching and assessment, while others will stretch the model toward ethics, data literacy, and scholarly practice. Divergent mappings may illuminate disciplinary and national differences in how AI competences are conceptualized and prioritized.

4. Educational Significance of the research

Theoretically, the study contributes by explicitly embedding AI-related competencies into an existing, HE specific digital teaching competence framework. It links broader digital competence work with emerging research on generative AI literacy. Educationally, an AI sensitive extension of the Frankfurt Model can inform the design of professional development programs and self assessment instruments for teachers. It can support institutions in articulating expectations for staff in AI rich learning environments. Finally, the international Delphi process highlights areas of consensus and contestation that can guide further empirical research and policy discussions on AI in HE.