Conference Agenda
Overview and details of the sessions of this conference. Please select a date or location to show only sessions at that day or location. Please select a single session for detailed view (with abstracts and downloads if available).
|
Daily Overview |
| Session | ||
Track 6-05: Evaluation in Higher Education
| ||
| Presentations | ||
Evaluating AI competency needs in higher education: Implications for curriculum development and trust Humboldt-Universität zu Berlin, Germany As generative AI becomes embedded in an increasing number of professional domains, higher education institutions face growing expectations to demonstrate the relevance of their programs and to respond effectively to changing competency demands (Budde & Tobor, 2025). This shift not only raises questions about curriculum development but also about how the value of higher education is assessed and how trust in its capacity to prepare graduates for uncertain futures can be maintained. In this context, graduate feedback represents a valuable resource for understanding how academic programs align with labor market requirements. Graduate surveys occupy a distinctive position at the interface between higher education and the labor market. By enabling retrospective evaluations of study experiences alongside detailed information on employment trajectories, they provide insights into whether and how competencies acquired during higher education translate into professional practice. They also offer valuable evidence on fields of employment, sectoral distribution, and levels of job satisfaction. Taken together, these data allow institutions to assess whether program goals align with actually acquired competencies as well as competency requirements and to identify areas where curricula may need to adapt (Janson, 2014). In doing so, graduate surveys contribute to the construction of evaluation regimes that define what counts as valuable knowledge and relevant competencies in higher education. They play a key role in shaping how competencies and knowledge are recognized as valuable within institutions and in fostering trust between higher education and the labor market. We present findings from an annual, nationwide graduate survey that was expanded to include a dedicated AI module for the first time. The study is situated at Humboldt University of Berlin, a major German research university, and embedded in a collaborative project focused on strengthening AI competencies in higher education using open educational resources (project KI-Kompetenzen an Hochschulen stärken), funded by the BMFTR (German Federal Ministry of Research, Technology and Space) through 2029. The survey was conducted in winter 2025/26, addressing graduates one to one and a half years post-graduation, i.e., respondents were the first cohort experiencing broad availability of generative AI tools during their thesis phase. The questionnaire examines AI use in final theses, frequency of AI use in current professional practice, current competency requirements, and graduates’ perceptions of how well their studies prepared them for evolving demands. Respondents also assess whether competencies acquired during their studies have gained or lost relevance due to the growing presence of AI in their professional environments. In our analysis, we will identify which AI-related competencies graduates consider most important in their professional contexts and explore potential gaps between perceived preparation during their studies and current workplace expectations. By linking responses from the AI module with broader information on employment situations and career paths, we also seek to better understand how technological change shapes competency demands across different sectors and professional roles. The survey findings will feed into the further development of an AI literacy framework intended to guide the evaluation of measures developed within the collaborative project. We further reflect on how evidence generated through graduate surveys can inform institutional strategies aimed at integrating AI-related competencies into study programs and at supporting continuous curriculum development. By presenting selected measures to be developed within the collaborative project context, we illustrate how evaluation data can be translated into concrete actions and used to guide academic program enhancement. Beyond its empirical contribution, the study highlights the potential of graduate surveys not only as monitoring instruments but as strategic tools for understanding changing competency landscapes and supporting evidence-informed decision-making. We discuss implications for quality assurance, curriculum responsiveness, and the evolving relationship between higher education and the labor market and consider how evaluation can contribute to maintaining trust in higher education in an era characterized by rapid technological change and shifting expectations. Trust as a Precondition for Meaningful Student Evaluation Central European University, Hungary Trust is an essential, yet often overlooked, precondition for effective evaluation in higher education. While student surveys, such as course evaluations and targeted experience surveys, remain central tools for quality assurance, their usefulness very much depends on students’ willingness to participate and their confidence in the integrity of institutional mechanisms. Across higher education systems, declining response rates raise concerns about the legitimacy, representativeness, and practical utility of evaluation data. This trend aligns with a broader, well‑documented decline in public trust in major institutions, including those within higher education systems. [oecd.org] [truedemdata.eu] [eurofound.europa.eu] Empirical research consistently shows that willingness to participate in surveys is strongly linked to trust in institutions. Silber et al. (2022) demonstrate that institutional trust increases individuals’ intent to participate in surveys, both directly and through more positive perceptions of survey legitimacy. This relationship provides a compelling explanation for the long‑term decline in higher‑education survey participation documented by Fosnacht et al. (2017), who report substantial reductions in student response rates across multiple major survey instruments over several decades. Although “survey fatigue” is often cited as the primary cause of low response rates, the documented erosion of institutional trust suggests a broader set of factors influencing student engagement with evaluation processes. Two trust‑related dimensions are particularly important for students: confidentiality and responsiveness. Concerns about anonymity, especially in small cohorts common in many European institutions, can hinder honest participation even when anonymity is formally guaranteed. Equally important is students’ belief that their feedback is heard and have meaningful impact. Evidence from a scenario‑based experiment by Zheng and Ma (2025) demonstrates that participation increases when students understand how evaluation outcomes influence teaching practices and institutional decision‑making. This paper examines these dynamics through an emerging case study at Central European University (CEU), where declining student response rates have become evident in both course evaluations and broader survey participation in the last few years especially among younger cohorts. While comprehensive institutional data will be available by the time of the conference, preliminary observations suggest that confidentiality concerns, perceived lack of impact, and a wider erosion of trust in institutions more generally may all contribute to this decline. The CEU case therefore offers a concrete context for exploring how trust is constructed, strained, and potentially restored within higher‑education evaluation regimes. Building on the literature and early CEU insights, the paper proposes several strategies for strengthening trust in evaluation mechanisms: transparent communication about confidentiality safeguards; participatory co‑design of survey instruments with students; and systematic and visible follow‑up on student feedback. Together, these measures move evaluation beyond a compliance‑oriented exercise toward a process grounded in recognition, transparency, and mutual accountability. Ultimately, the paper argues that trust is not merely a contextual variable but a structural condition for meaningful evaluation. Without trust, even the most sophisticated assessment instruments risk producing incomplete, distorted, or unusable data—undermining both quality assurance and institutional legitimacy. Managing Meaning in Higher Education: Ethical Leadership and Autonomy‑Supportive Teaching as Drivers of Student Purpose, Moral Orientation, and Sustainable Behaviour 1: Vilnius Business College, Lithuania; 2: Vilnius Business College, Lithuania; 3: Vilnius Business College, Lithuania Universities increasingly embed sustainability goals in curricula, yet student engagement in sustainable and prosocial practices remains uneven. A common response is to add more sustainability content; a less examined lever is the everyday motivational climate of programmes, how leadership and teaching practices manage meaning by making learning feel purposeful, ethically grounded, and socially connected. This paper tests a climate‑to‑meaning‑to‑behaviour model that links programme leadership and didactic environments to students’ purpose, moral orientation, and sustainable behaviour. We advance a Meaning‑Supportive Educational Climate (MSEC) model with three contextual inputs: (1) ethical programme/department leadership (integrity, fairness, accountability), (2) autonomy‑supportive teaching (choice, rationale, perspective‑taking, and responsiveness), and (3) perceived course value/interest. We hypothesise that MSEC strengthens students’ psychological need satisfaction in their programme (autonomy/authenticity, competence, relatedness) and prosocial/purposeful motives for studying. In turn, need satisfaction and purpose motives are expected to cultivate moral orientation (prosocial orientation and respect for diversity) and belonging/collaboration, which together relate to students’ everyday sustainable behaviours. Data come from a cross‑sectional student survey administered via Qualtrics (15–18 minutes) in Lithuanian higher education. Three embedded attention checks were used to identify inattentive responding. After excluding failed attention checks, the analytic sample comprised N = 1,115 students (approximately 74% Bachelor, 24% Master, 2% PhD; 53% in‑person, 27% hybrid, 20% online formats). The questionnaire included Likert blocks for ethical leadership (10 items), autonomy‑supportive teaching (6), course value/interest (6), prosocial/purpose motives for studying (6), psychological need satisfaction (12), prosocial orientation (5), respect for diversity (4), belonging/collaboration (4), and sustainable behaviours in the past three months (13 frequency items; e.g., energy saving, recycling, transport choices, and stewardship actions). Internal consistency was adequate to high across constructs (Cronbach’s α = .78–.91). We estimated a path model using composite scores, controlling for background and engagement covariates (age, gender, study level, year, course format, employment, income, sustainability coursework, service learning, organisation membership, volunteering, nature access, political orientation, religiosity). Indirect effects were tested with 1,000 bootstrap resamples. Results support the proposed “managing meaning” mechanism. The composite MSEC index (ethical leadership + autonomy‑supportive teaching + course value) strongly predicted psychological need satisfaction (β = .79, p < .001; R² = .65). When decomposed, perceived course value/interest showed the largest unique association with need satisfaction (β = .48), with autonomy‑supportive teaching (β = .22) and ethical leadership (β = .20) also significant (all p < .001). Need satisfaction predicted prosocial/purpose motives (β = .52, p < .001) and belonging/collaboration (β = .44, p < .001). Purpose motives predicted prosocial orientation (β = .46, p < .001) and respect for diversity (β = .41, p < .001). In the final outcome model, sustainable behaviour was associated with belonging/collaboration (β = .22, p < .001), need satisfaction (β = .17, p < .001), prosocial orientation (β = .13, p = .002), respect for diversity (β = .08, p = .018), and a remaining direct effect of MSEC (β = .18, p < .001), explaining 42% of variance. The total effect of MSEC on sustainable behaviour was β = .52; the indirect effect was β = .35 (bootstrap 95% CI [.28, .41]), indicating that roughly two‑thirds of the association operated through meaning‑ and relationship‑based mediators. For teaching, learning, and student experience, the implication is blunt: sustainability behaviour is not only a knowledge problem but a meaning‑and‑belonging problem. Designing courses that students experience as valuable and autonomy‑supportive, and cultivating ethical leadership signals about “how we do things here,” appears to be a practical route to foster purpose‑driven, morally oriented learners who act sustainably beyond the classroom. Limitations include cross‑sectional self‑report data and a single‑country context; future research should use longitudinal, multi‑level designs and behavioural indicators of sustainability. | ||

