Conference Agenda

Without doubt, everyday assessment and responses to student performance are central facets of school quality and an important field of innovations in school. The development of a “new” and formative performance resp. assessment culture, which is suitable for initiating and supporting the acquisition and development of both subject-specific and interdisciplinary/generic competences of students (e.g. self-regulated learning, social competences), seems to be a necessary condition for the development of teaching and learning in general (cf. MfSW NRW 2009; Beutel et al. 2017; Wiliam 2018). In this context, the term “culture” refers to a fundamental change of assessment practice that is not limited to a selective use of some additional or alternative diagnostic instruments by only a few teachers (cf. Jürgens & Diekmann 2006; Box 2019, 42/143). According to Sacher (2014, 264) the development of a new resp. formative performance and assessment culture in schools will only succeed, if it is based on a jointly formulated assessment concept in which the teaching staff fixes objectives, guiding principles and concrete agreements on assessment practice. Such an assessment concept – Sacher points out – must be implemented, regularly evaluated, discussed, and revised as an essential part of the school program. In which way, to what extent and how successful individual schools undertake efforts regarding the requested change and development of performance and assessment culture(s) (cf. e.g. Winter 2012) has hardly been empirically investigated (in Germany). Most of the findings on performance and assessment culture and associated innovations relate to schools that can be described as “extraordinary”. Extraordinary in the sense that they, for example, have been nominated for the (nationwide) German School Award (cf. Porsch et al. 2014, Beutel & Pant 2020) or have a special pedagogical profile (e.g. Montessori, cf. Diekmann 2018). There are almost no empirical findings that give a broader impression of focal points, achievements, or school form/grade specific characteristics of the change in performance/assessment culture at “ordinary” schools. One exception are the findings obtained in the context of an external evaluation (“Qualitätsanalyse”) of schools in North Rhine-Westphalia (federal state of Germany). During this evaluation, various methods were used to gain a comprehensive impression of the work and quality of schools. Among other things, classroom observations were conducted, and so-called school portfolios were reviewed. The school portfolios contained various documents specific to individual schools, such as school programs. A mandatory component of the school portfolios were the performance concepts developed by the individual schools. In summary the performance resp. assessment concepts schools had to submit during the evaluation are characterized as unsatisfying and in need of development (cf. MfSW NRW 2009, 34). Unfortunately, this conclusion is not really explained in detail. Regarding specific differences between school types and levels, only a few findings are reported. For example, it is pointed out that the performance/ assessment concepts at secondary schools are comparatively subject-specific (compared to the performance/ assessment concepts at primary schools). In contrast, performance/assessment concepts at primary schools apparently prove to be more elaborate about the formative use of individual diagnostics. (cf. MfSW NRW 2016, 30-32). The following questions arise from this:
Research question 1: How are performance/assessment concepts designed in terms of scope and content? Are there any school level/form-specific priorities or features?
Research question 2: Are performance/assessment concepts embedded in a whole school approach of school development?

Based on the policy enactment perspective (Ball et al., 2011), this study investigates how primary citizenship teachers do assessment policies in their practice and discusses its influencing factors. While the citizenship programme remains non-statutory at the primary level in many countries, such as England (Richardson, 2010), the programme is mandatory in China due to the emphasis on fostering socialist identity and moral cultivation. The programme standards mandate an assessment approach that focuses on students' 'values' and 'process performance' (Ministry of Education, 2022, p.49) rather than test scores. Advocating 'assessment for learning' (ibid., p. 50), the standards call for a greater emphasis on formative assessments. To ensure the full implementation of national education policies (Lu et al., 2018, p.113), China has established an internal agency system - the System of Pedagogical Research Officer. Primary citizenship programme pedagogical research officers are appointed by the district education authorities and have administrative powers. As intermediaries, they hold the authority and responsibility to interpret, translate, and organise citizenship assessment policies in practice.
Contrasting with the popular perception in China that relegates teachers to the role of policy implementers, scholars (Braun, et al., 2011; Ball, 2011) acknowledge teachers as policy enactors. As Ball (1994, p. 19) asserts, policies do not typically provide a set course of action, but rather create situations where the choices for what to do are limited or altered, or specific aims or outcomes are established. The majority of educational policies depend on their realisation through teaching, positioning teachers not merely as implementers, but as interpreters and 'translators' of policy (Perryman et al., 2017, p.745). This act of 'translation' suggests that while teachers adhere to policy, they also make adaptive modifications.
The study reveals that all schools employ standardised tests—developed by the district's education authorities—as summative assessments for students from Year 3 onwards, despite the absence of such tests for Year 1 and 2. Notwithstanding the stipulation in assessment policies that 'assessment results should be graded rather than scored' (Ministry of Education, 2022, p. 52), test results are ultimately rendered in the form of scores. Formative assessment post Year 3 is notably sparse, predominantly consisting of verbal feedback within classes. This is the case despite a unanimous acknowledgment that the curriculum standards advocate against basing judgments of students’ learning performance solely on test results.
Teachers perceive this disjunction—being aware of but not adhering to the assessment policies—as an outcome of the 'internal disintegration of the policies', a consequence of intermediary influences. In a manner akin to the role of medieval bishops interpreting the Bible, the pedagogical research officer wields unassailable authority within their community to interpret and translate official assessment policies. Their instructions and guidelines are considered the truly applicable policies, while the national assessment policies are often disregarded as overly 'idealistic' and 'abstract'. Furthermore, formative assessment is decried as a privilege available only to economically developed regions, which have the financial means to engage national and international experts for knowledge dissemination and practical guidance. Teachers also face considerable pressure from parents. As primary schools increasingly serve as childcare providers, teachers interact more directly and frequently with parents, many of whom express scepticism towards formative assessment due to its absence of tangible scores and rankings. The teachers identify the prevailing culture of competition or the 'rat race' endemic in nowadays China as the root of these challenges. As the country’s economy slows, societal pressure to compete escalates, underscoring the view that ‘excellence is not the sole goal, but more importantly, to be better than others'.

The use of scores from self-report measures are very common in several areas of research. Since those instruments provide researchers to measure some psychological constructs such as personality, attitudes, beliefs, emotions of too many respondents in a short time, they are preferred widely for data collection process (Alarcon & Lee, 2022; Curran, 2015; Ulitzsch et al., 2022). However, some important problems may occur when responders do not give their best effort to select the response correctly reflecting themselves (which is very common especially with the unmotivated responders) (Rios & Soland, 2021; Schroeders et. al., 2022). Individuals may respond to items without reading them, by misinterpreting them or be unmotivated to think about (Huang et al., 2012; Ward & Meade, 2022). This type of responding behaviour have been stated as random (Beach, 1989), careless (Meade & Craig, 2012), insufficient effort (Huang et al., 2012), disengaged responding (Soland et al., 2019) in the literature. In the context of this study, the term ‘careless responding’ with CR abbreviation is preferred. Careless responding (CR) behaviour is a major concern based on the data taken from self-report scales in any type of research (Meade & Craig, 2012). Even the amount is small, it may affect the data quality and results of the study severely. Careless responses may introduce a measurement error, weaken the relationship between variables and inflate the Type II error. It may also introduce a new source of construct-irrelevant variance to the process and end up with an undesirable effect on psychometric properties of the scale (item difficulty, average scores, test reliability, factor structure etc.). Briefly, CR has the potential to weaken the test scores’ validity in different ways (Beck et. al., 2019; Rios & Soland, 2021).

Considering the factors stated above, CR have become an important and interesting research topic for researchers with a growing interest. One of the most important aspects on CR research is the way how we can detect and cope with them to ensure the quality of survey data. Identifying careless responders and removing them from the dataset is one of the suggested ways to increase data quality. In the literature, there are several data screening methods mainly classified in two groups; priori and post-hoc. Priori methods are the ones that are planned and incorporated into data collection process before the administration of survey. On the contrary, post-hoc methods get involved in the process after data collection. They are implemented on the collected dataset and typically based on a statistical calculation.

While there are several studies focusing on the effect of careless responding on datasets and comparison of the efficacy of CR identification methods, there is still no clear answer about the detection accuracy of CR identification methods (Goldammer et al., 2020). Besides, this study will focus on prior methods that have been studied and focused less on previous studies.

The present study will handle three different ways of prior methods (instructed response items, reverse items and self-report items) which will be explained in method part in detail. In the context of this study, these three ways of CR identification will be used, examinees will be removed from dataset according to those methods separately and their effects on psychometric properties of data will be investigated. This study addresses the following research questions;

- How was the distribution of careless responders with respect to three different CR identification methods?

- How did psychometric properties of the data (scale mean, reliability, correlation between factors, factor structure etc.) change when careless respondents were removed from the data with respect to different CR identification methods?

How can we activate and create assessment systems that lead to a flourishing school where everyone is able to fulfil their potential and achieve both success and well-being? How might we shift assessment practices toward equity and justice/fairness? How do the assessment methods meet the diversity of the students? The research project SHIFT (Shaping and Inspiring a Fair Thinking in assessment) aims to investigate how a range of emerging trends within the international community can be used to answer these questions. These trends concern the literature on: (1) human capabilities (Sen, 1999) as a framework for ‘social justice’, (2) Assessment for Learning (Swaffield, 2011) as the horizon for understanding assessment, (3) Universal Design for Assessment (CAST, 2011) as the philosophy that attempts to go beyond the ‘model of adjustment’ and (4) such approaches as: fair and equitable assessment (Tierney, 2013; Montenegro & Jankowski, 2020), culturally-responsive assessment (Nortvedt et al., 2020), inclusive and universal assessment (Waterfield & West, 2006; Nieminen, 2022; Tai et al., 2021).

An increased focus on equity and justice emerges from the 2030 Agenda for Sustainable Development, where the commitment is to provide inclusive and equitable quality education at all levels, as well as from other European and international documents (OECD, 2012, 2005; UNESCO, 2015, 2022). The same concern is evident in empirical studies focused mainly in higher education contexts (Nieminen, 2022; Tai et al., 2021), and what has become evident as more and more assessment researchers and practitioners engage with the equity conversation is the desire for considering these issues also in school context. Moreover most assessment research is based on what can be described as a ‘technical perspective’, looking at whether assessment is efficient, reliable, valid, leaving less space for a “humanistic” perspective that highlights assessment to foster learning for human flourishing and for responsibility toward and within society (Swaffield, 2011; Fuller, 2012; Gergen & Gill, 2020; Hadji, 2021).

Dialogue into the paradigms of assessment is of paramount importance if assessment aims at embracing a focus on equity, ethics, and humanization and meeting the challenges of these times. The paradigm shift was initiated many years ago, moving from assessment of learning towards assessment for learning, giving greater attention on the role of learners (opening the way to participatory approaches connecting school and community), on a shift from product to process-focused assessment and on a view of learning as a lifelong process rather than something done to prepare for an exam. Although these changes have been partially incorporated into the debate about educational assessment, work remains to be done to ensure the necessary attention to the issue of diversity among learners. Such an approach would strengthen the value of the shift and enlarge the potential of the assessment process towards the promotion of all students’ learning and growth moving away from a model of adjustments, which makes specific reasonable accommodations for some students towards assessment models that allow all students to fully participate and learn in the most equitable way.

Coherently with the theoretical framework, the research design addresses the importance of engagement, participation and opportunities for access, choosing a community-based approach interconnected with the appreciative one, seeking to produce a new imaginary for approaches to assessment, with implications for both cultures and practices. The aim of the Programme is to connect assessment with justice and equity through a participatory process sustaining the shift towards a fair thinking in assessment.