Conference Agenda

Student Evaluation of Teaching (SET) is the procedure by which students evaluate and rate teaching performance. Usually, during a SET procedure students complete rating forms or questionnaires about different aspects related to their teachers, but mostly about their teaching practices. Universities or higher education institutions from all over the world implement SET procedures to achieve 3 main purposes. Generally, and from a practical point of view, the main purpose of implementing this type of procedure in most universities is the necessity of reporting SET results to quality assurance agencies. The other main goal of SET procedures, and surely the most important one from an educational perspective, is to provide feedback to academics about their teaching practices and/or to design teacher training programs focused on developing teaching skills. Another important use of SET results is related to evaluating evidence of teaching performance to use the results for academic career advancements or other ways of rewarding teaching effectiveness.

The topic of Student Evaluations of Teaching is one of the most researched ones in the domain of educational research, with over 2000 studies published in peer-reviewed journals over a period of a little more than 100 years (Spooren et al., 2017). One of the earliest debates in this field of research is about the validity of the SET scales and procedures. The main question was whether the measurement instruments applied to students during these procedures can accurately measure teaching effectiveness. Even if this debate was most active in the 1970s and the evidence was inclining more towards the affirmative answer to the question in case (see reviews from Richardson, 2005 and Marsh, 2007), a recently published meta-analysis (Uttl et al., 2017) presented some evidence which seriously threatens the validity of SET results. The results of the mentioned study strongly suggest that there is no relationship between the SET results of a teacher and the level of their students’ achievement/learning.

The existence of this relationship is vital to the SET validity debate starting from the premise that if SET results accurately reflect teaching effectiveness, then teachers identified as more effective should facilitate a higher level of learning and achievement among their students. Put simply, good teachers can help their students learn more and if SET results are valid, they should correlate with student achievement.

At the same time, several SET scales were rigorously developed from a theoretical and psychometrical point of view (e.g., SEEQ, CEQ, ETCQ). Also, there is a lot of evidence that those specific SET scales can accurately measure and offer support in developing teaching skills (Marsh, 2007; Richardson, 2005).

Starting from these, and referring to the meta-analytic results presented above, the main question that arises is whether the relationship between SET results and student learning is stronger when the utilized SET scale is more rigorously developed and validated.

Thus, the research questions that guide the present study are the following:

1. What is the average effect-size of the relationship between SET results and student achievement, in all the multi-section SET studies published to date?

2. Is the average effect-size of the relationship between SET results and student achievement different as a function of the SET measure validity evidence?

Academic integrity in higher education has been an increasing focus in recent years (Macfarlane et al. 2014), and this interest intensified with the pivot to emergency remote teaching and assessment during COVID-19 (Holden et al. 2020; Eaton et al. 2021). For those working and studying in Irish higher education, a perception of a rapidly expanding range and types of academic misconduct has prompted renewed institutional and sectoral efforts to safeguard academic integrity (e.g. QQI 2018a, 2018b; NAIN 2021a, 2021b). The project informing this paper explores the costs and benefits of using the ICAI-McCabe surveys, which are used in other countries to research students’ and academic staff’s behaviours, attitudes, and beliefs about academic integrity.

Central to dealing with the question of academic integrity, given the implications it can have on students and academic staff alike, is to initially explore what those working and studying in Irish higher education know and believe about academic integrity. As such, this talk will evaluate the challenges, costs, and benefits of adapting the ICAI-McCabe surveys of faculty and student attitudes and beliefs about academic integrity for use within the HEI sector in Ireland. The goal here is to explore how ‘academic integrity’ is understood in the Irish context specifically, and how these understandings may differ from the institutional and national settings in which these ICAI-McCabe surveys were developed and are currently in use.

The proposed approach to a cost-benefit analysis of using AIS in Ireland is informed by a virtue ethics philosophical perspective, extending the question of academic integrity beyond institutional and/or procedural viewpoints (Williams 2010). Such a perspective aligns with the ‘values-based’ approach to academic integrity, and draws on a body of work within academic integrity research that focuses on ‘virtue ethics’ approaches to what is consequently termed ‘educational integrity’ (Bretag 2016).

This open perspective avoids reducing ‘costs’ and ‘benefits’ to quantitative or monetary notions, and explores beliefs, attitudes and values. This innovative approach addresses the range of cultural and philosophical underpinnings of academic integrity in what would be a new national setting for the AIS, and with a view to the diversity of the Irish higher education context. Thus we propose a mixed-method approach with a significant qualitative component and a wider sampling perspective of the responsible actors - beyond those charged with institutional responsibility for quality assurance.

One of the main challenges faced by university graduates is that of achieving early labor market insertion adjusted to the professional profile for which they have been trained. Not achieving this objective causes them to have significantly low expectations for the future (Gómez Acuñas et al., 2009), in which factors such as the development of training and labor market insertion actions (professional orientation), the possibilities of the labor market in terms of accessing and maintaining a job (employability factors) or the promotion of transversal competencies linked to greater employability, such as motivation for success (García-Álvarez et al., 2023)

This requires an assessment of the conditions of the labor market, since important changes have gradually been introduced in the logic of access to the world of employment. In a changing and competitive scenario such as the current one, companies require people not only capable of intervening at a technical level, but also with skills that allow them to relate to the environment, work in teams, adapt to change, self-motivate and be proactive (Riaño, 2012). In other words, they are expected to have certain skills that help them to define and manage a professional project in the short and medium term, for which they must set realistic academic goals, be perseverant, or develop tolerance to stress, issues linked to adequate achievement motivation (Preckel and Brunner, 2015).

It is a fact that motivation has a direct impact on students' education, constituting one of the most valuable competencies in order to achieve an effective academic and professional promotion (Polanco, 2005). Traditionally, students' effort has been linked to the overcoming of certain situations involving performances (Eison, 1979), as well as the establishment of educational standards that are oriented towards deep learning strategies and that allow achieving a remarkable mastery of skills for personal autonomy and adaptation to the productive environment (Cervantes et al., 2018; Kadioglu & Uzuntiryaki-Kondacki, 2014).

So much so that most of the studies developed have focused on the perspective of results, analyzing the influence of motivation on academic performance, taking into account variables such as learning situations, the type of content or the characteristics of the assessment (Cox, 2012). On the contrary, we did not find references that investigate the factors that determine the degree of motivation of students, such as previous education, social capital, personal characteristics or the socio-economic context.

Considering this last approach, and given the existing differences between degrees (training fields, professional opportunities, insertion logics, access channels, etc.), the aim of the work we present here is to analyze the factors that may influence the perception of a greater or lesser motivation for success in university students of education, selecting academic, personal and work-related variables.