Conference Agenda

Education across the globe, including Europe, the United Kingdom and Australia, has been fuelled by a resurgence of high-stakes testing, exacerbated by outcome-defined policies and competing factors, including the ongoing demands of accountability, performativity, and the rigid alignment between high-stakes exams and curricula (Lingard & Sellar, 2016; Löfstedt et al., 2020; Verger, Parcerisa & Fontdevila, 2019). The performance-based nature of education has turned education into schooling. Kemmis et al. (2014) describe schooling as the opposite of education; it is a ‘technical tool’ (p.25) used to impart and monitor information as outcomes. These influence assessments' objectives, methods, and outcomes, in addition, to teachers' pedagogical assessment practices (Cairns, 2020). Given current political preferences for teachers' accountability and measurable outcomes, students undertake multiple assessments measuring their learning at single points in time, each affecting their depth of understanding, engagement with the subject, and sense of who they are as a student (Andrade & Brookhart, 2020). Further, teachers' pedagogical practices may be modified by the effects of high-stakes testing as they move from collaborative work in the classroom to more 'teach to the test' methods, further reducing student engagement (Resnick & Schantz, 2017). A significant number of studies investigating the advantages of collaborative learning have been undertaken, promoting a constructivist approach to collaboration in the classroom. However, behaviourist methods are still prevalent when assessing students. This discrepancy leads to the question: if collaboration is widely recognized as enhancing learning, why does the reliance on individual summative testing persist? Broader recognition of multiple assessment methods is required to improve engagement in high school STEM classes and abate the documented effects of individual summative testing. Leading to the research questions –

• How do students perceive collaborative testing?

• How do students and teachers discern collaborative testing's impact on student learning?
• What are teachers' perceptions of the utility of collaborative testing?

Perspective

Humans exist in a dynamic world where we are a part of the world and being of that world interact with other humans and non-humans. Participating in the environment entangles or intertwines us with the material and others, all affecting each other as we understand our place, meaning and knowledge of the world (Barad, 2007; Muris, 2022; Plauberg, 2018). Knowledge is not separated from the learner; it is interwoven and interconnected; it affects and is affected by interactions with non-human and human interactions (Barad, 2007; Guillion, 2018).

Teaching science involves a dynamic intra-mingling between students, teachers, surroundings, and our classroom's tools. In this intra-play, students attained their understanding and knowledge of science. They did not do so alone; students did not sit apart, and they did not sit quietly; they participated, interacted and enhanced their understanding of science. These interactions influence our meaning-making (Koro- Ljungberg, 2015). The learner’s thinking and actions act upon the world equally as the thinking of the learner acts on the world. There can be no separation between object or subject, from human and non-human; all are intertwined in the knowing. Barad (2007) termed this interconnecting of theory and knowledge as onto-epistemology.

Onto-epistemology does not separate the object/subject, human/non-human, world/us, knowledge/learning. This theory places equal importance on the material world, ‘matter matters’ (Barad, 2007). Matter is an active participant in the entanglement of meaning, enfolded alongside material and discursive practices; they are constantly reconfiguring to forge reality (Guillion, 2018).

In December 2020, the Italian Ministry of Education adopted a new assessment modality based on predominantly discursive communication and the abolition of numerical grades. This means was introduced to incentivize greater equality and support students in becoming aware of how they develop their competencies (Girelli, 2022). One of the core elements of this change is the shift from a "normative" way of assessing pupils - that implies the comparison between a single student and the rest of the class - to a criteria-based one. This latter mode of assessment requires the teacher to consider pre-established criteria that help the students better understand their potential (Black, Wiliam, 1998) by indicating the level of achievement of a learning objective. Four fundamental elements must be considered for this purpose: the level of autonomy with which the pupil can achieve the set goals, his or her ability to find resources spontaneously, the fact that learning takes place in an unfamiliar situation, and finally whether the acquisition of the set goals is continuous (i.e. it occurs in more than a single isolated case) or not (MIUR, 2020).

The theoretical framework of this research is the Capability Approach developed by Amartya Sen and Martha Nussbaum (Sen, 1999; Nussbaum, 2006). It is «an intellectual discipline that gives a central role to the evaluation of a person’s achievements and freedoms in terms of his or her actual ability to do the different things a person has reason to value doing or being» (Sen, 2009). The attention devoted to the valorization of non-quantifiable processes in terms of accountability and rethinking the role played by the concept of measurement (Unterhalter, 2019) makes this framework a proper tool for proposing a new look at the evaluation methods currently in use in Italian primary schools. Since the right to education «requires a deep commitment to building human capabilities» and «Education nurtures understandings and builds capabilities that can help to ensure that our futures are more socially inclusive (UNESCO, 2021)», it is necessary to understand whether the learning processes promoted at school can promote the development of such resources.

Therefore, the question from which this research stems is how can formative assessment help students develop their capabilities. To date, few studies within the capability approach have focused on primary schools and formative assessment (Robeyns, 2017; Unterhalter, 2019; Palumbo, Pandolfini, 2019). The Italian context lends itself particularly well to investigating the relationship between these realities since the main document used by teachers to guide their teaching (MIUR, 2012) was formulated by Edgar Morin and the theory of complexity is also a fundamental factor for the capability approach (Szekeley. Mason, 2018).

Student Evaluation of Teaching (SET) is the procedure by which students evaluate and rate teaching performance. Usually, during a SET procedure students complete rating forms or questionnaires about different aspects related to their teachers, but mostly about their teaching practices. Universities or higher education institutions from all over the world implement SET procedures to achieve 3 main purposes. Generally, and from a practical point of view, the main purpose of implementing this type of procedure in most universities is the necessity of reporting SET results to quality assurance agencies. The other main goal of SET procedures, and surely the most important one from an educational perspective, is to provide feedback to academics about their teaching practices and/or to design teacher training programs focused on developing teaching skills. Another important use of SET results is related to evaluating evidence of teaching performance to use the results for academic career advancements or other ways of rewarding teaching effectiveness.

The topic of Student Evaluations of Teaching (SET) is one of the most researched ones in the domain of educational research, with over 2000 studies published in peer-reviewed journals over a period of a little more than 100 years (Spooren et al., 2017). One of the earliest debates in this field of research is about the validity of the SET scales and procedures. The main question was whether the measurement instruments applied to students during these procedures can accurately measure teaching effectiveness. Even if this debate was most active in the 1970s and the evidence was inclining more towards the affirmative answer to the question in case (see reviews from Richardson, 2005 and Marsh, 2007), a recently published meta-analysis (Uttl et al., 2017) presented some evidence which seriously threatens the validity of SET results. The results of the mentioned study strongly suggest that there is no relationship between the SET results of a teacher and the level of their students’ achievement/learning.

Analyzing the SET scales utilized in the studies included in the last-mentioned meta-analysis, we observed that lots of them contain items that are either too vague, either unclear, or even completely unrelated to observable teaching actions and behaviors. Moreover, many SET instruments had only 1 or 2 overall questions about teaching quality.

Some of the main reasons why we chose to develop a new SET scale are the following: (1) most SET scales are not rigorously developed from the theoretical and psychometrical point of view, and those that are, were developed in older times when digital environments were not so common; (2) the majority of SET instruments lack available and published psychometrical evidence for the validity of the results; (3) there are very few instruments that contain questions strictly related to the observable behaviors and actions of teachers; and (4) there are not many SET instruments than can be adapted to different teaching contexts by adapting the selection and number of dimensions to be measured.

Thus, we propose the development and validation of a new multi-dimensional instrument that is based on a systematic mixed-methods process of defining effective teaching in higher education, consists of questions related to the observable actions and behaviors of teachers, and contains a large number of dimensions suitable for several teaching contexts.

Given that the purpose of the study is to develop and validate a SET scale, the research objectives are the following:

1) To perform the theoretical substantiation of the construct and related dimensions.

2) To develop and refine items for each dimension of the scale.

3) To gather evidence in favor of validity (content, construct and criterion) and fidelity (test-retest, internal consistency) of the scale.

Formative assessment places greater emphasis on bidirectional contributions between the students in the assessment process (Yorke, 2003), as well as advocates an equal teacher-student relationship. To promote student learning, both stakeholders are expected to be fully engaged in the learning process (Black & Wiliam, 1998), instead of measuring student performance unilaterally (Wiggins, 1998). Although formative assessment is well established in western classroom practices and is required as one underpinning literacy for teachers (Popham, 2011), when the idea of formative assessment was introduced to Confucian heritage countries in Asia, tensions for K‒12 teachers arose due to the perennial convention of overemphasizing summative assessment in school.

Profoundly influenced by Confucianism, teachers in China may perceive formative assessment as a novel assessment paradigm that contradicts their traditional mindset on assessment. Take Chinese teachers as an example, many of them seem to be reluctant to this reform (Zhan, 2021). Moreover, schooling in these countries has been constantly shifted to virtual online classroom platforms since the COVID-19 pandemic. Given that online classrooms largely constrain the effectiveness of the implementation of assessment (Khan & Khan, 2018), the different classroom modes complicate the teacher formative assessment practices.

However, there is a paucity of research on teachers’ overall literacy of formative assessment in the eastern Confucian heritage countries (Zhan, 2019). There is also a need to design and develop a psychometrically sound and culturally responsive inventory instrument in the field of formative assessment literacy in the Confucian cultural context. Thus, I attempt to design and validate a first easternized online inventory to evaluate teachers’ capacity for formative assessment acquisition and authentic application. The whole process includes four phases under the guide of an exploratory sequential mixed-methods design. Building on the interviews and the prior inventories on relevant domains (e.g., Campbell & Mertler, 2004; Yan & Pastore, 2022), I plan to categorize the dimensions as five scenarios, consisting of 1) face-to-face classroom, 2) online classroom, 3) after-school assignments and tasks, 4) feedback, 5) ethical concerns. Each contains six multiple-choice questions and each question has one best answer. The inventory will be uploaded to www.wjx.cn, a Chinese online open survey website, and automatically code the correct answer as 1 point while the incorrect answers as 0 points. Thus, the total grade for one participant is 30 points and each subsection values 6 points.

In particular, I am fully aware of the unique characteristics of Confucian heritage culture and its ubiquitous impact on teacher assessment practices. Therefore, all the questions are edited in line with the authentic local classroom settings. For example, I will emphasize student dignity considerations to replace the typical western inventory questions about privacy risks.

The expected findings will contribute to the initial investigation of the status of formative assessment literacy for K‒12 teachers in Confucian heritage culture and locate the specific factors influencing the building and practice of teachers in this domain. Eventually, the inventory will be used to gather evidence to inform in-service and preservice teacher professional development programs.