Conference Agenda

As the education agenda of global agencies changed after 2015 to emphasise minimum standards of quality for all countries to be delivered by 2030, the OECD has sought to expand its most successful comparative instrument, the Programme for International Student Assessment (PISA), to include low- and middle-income countries (LMICs). In 2014, it introduced PISA for Development (PISA-D) as the means to establish PISA as a universal measure of learning, and in 2020, it declared PISA-D a success. The most innovative feature of PISA-D was that the assessment included out-of-school youth (OOSY); that task was sub-contracted to Educational Testing Service (ETS). The OOSY population is a geographically dispersed group which present considerable challenges to any researchers seeking to access them. Given this, we ask: who did the OECD assess? More specifically, how did the OECD define the target population of PISA-D out-of-school assessment, what was the sampling frame, and were they accurately represented in the PISA-D OOSY sample?

Much of the existing literature on the OECD influence is based on different theoretical positions. These differences in perspective have real consequences, however, often determining which legitimation dynamics researchers see and which they overlook. In this paper, we seek to adopt a holistic approach by drawing on Suchman’s (1995) framework for analysing the multiple sources of organisational legitimacy and the means by which it is promoted and repaired.

In applying Suchman’s framework, we argue that PISA-D was a macro level exercise designed to legitimate the OECD’s extension of PISA into LMICs and to establish its role in a new arena. The incorporation of OOSY in the assessment was a key micro level endeavour which would allow the OECD to achieve that end and, if not done properly, would challenge aspects of its legitimacy. For example, not assessing sufficient OOSY would debase the quality of the OECD’s products and services; this would also damage the OECD’s moral claims with regard to monitoring the attainment of the SDGs and promoting an inclusive approach. In parallel, at the cognitive level, this would challenge the whole logic of the novelty and value of PISA-D. Overall, the successful identification and assessment of OOSY was vital to ensuring its legitimacy. This would require the OECD to either address the considerable difficulties of accessing OOSY or find a tactical solution which obscured the many challenges to its legitimacy.

Suchman (1995) also analyses how organisations respond to challenges to their legitimacy and identifies three broad approaches: (a) offer normalising accounts; (b) restructure, and (c) don’t panic. He suggested that although legitimacy crises may coalesce around performance issues, most challenges ultimately rest on failures of meaning, where ‘audiences begin to suspect that putatively desirable outputs are hazards, that putatively efficacious procedures are tricks, or that putatively genuine structures are facades’ (1995, 597). Consequently, the initial task in mending a breach of legitimacy usually will be ‘to formulate a normalising account’ that separates the threatening revelation from larger assessments of the organisation as a whole. He identified ‘justifications’ and ‘explanations’ as the two main types of normalising accounts. Suchman also noted that organisations may also re-establish legitimacy through micro-level strategic restructuring, in the sense that ‘narrowly tailored changes that mesh with equally focused normalising accounts can serve as effective damage-containment techniques’ (ibid., 598).

PISA has become the “world’s premier yardstick” against which the “quality, equity and efficiency” of national education systems are evaluated (Gurría in OECD, 2018a, p. 2). PISA claims to be able to compare, on a single scale, the performance of education systems around the globe. These comparative measures have “contributed to the constituting of a global commensurate space of educational performance” (Rizvi & Lingard, 2010, p. 135), regardless of the varying political, economic, social, and cultural contexts of participating nations. Data from PISA are being used by countries to identify “gaps” in their education systems and to develop policies to “move up” on the league tables (Meyer & Benavot, 2013; Wiseman, 2013). A recent development in this space has been the OECD’s development of an internationally comparable measure of Global Competence.

The use of PISA measures as benchmarks for shaping global policies and governing education makes it important to examine the process of how “PISA knowledge” is arrived at. Developing a set of measures requires normative decisions about what the concept encompasses. The assessment of global competence provides a useful example of examining the development of this particular form of global knowledge. Given the multifaceted definitions and understandings of this term, this paper empirically examines the challenges such efforts at stabilising the definition faced, and the ways in which these were negotiated. Locating our study in the interdisciplinary field of Science and Technology Studies (STS), and deploying the concept of tinkering (Knorr Cetina, 1981), we attend to the practices that stabilised the assessment of Global Competence in PISA 2018.

To make globally acceptable knowledge, various epistemic, cultural and political perspectives are brought together in relations of mutual learning and construction, and through iterative processes of expert consultation, country feedback, committee endorsement, etc. These encounters, where diverse perspectives are brought together, have the potential to be hijacked by more outspoken or forceful participants. Moreover, these processes typically take several months during which a range of unexpected events may occur or challenges posed to the successful completion of the endeavour. Tinkering is the way in which actors and events are managed – through cajoling, placating, compromising, modifying, etc. to ensure that the project does not collapse.

Drawing on empirical data relating to the development of the assessment of global competencies, we provide examples tinkering in the development of PISA’s tests of global competency. We highlight three key tinkering moves by the OECD during the process of developing the assessment. In the first move, the OECD replaced the initial Global Competence Expert Group with another group of experts to placate the PISA Governing Board, which objected to the heave economic slant of the first expert presentation. In the second tinkering move, the OECD retrospectively aligned the PISA assessments of global competency with the global competence framework with the UN SDGs. This enabled the OECD to gather more (of the right) allies to support its efforts, and provided “a moral legitimacy the OECD has not enjoyed with the traditional PISA initiative and its narrow economic focus” (Auld & Morris 2019b. p.11). A third tinkering move was the push by the OECD to administer the assessment even when only a minority of the countries decided to participate, arguing that more nations might join subsequent rounds.

International large-scale assessments (ILSAs) are playing an increasingly important role in decision-making and reforms, both nationally and internationally (e.g. Grek, 2009; Lindblad et al., 2018). One of the most influential ILSAs is the Programme for International Student Assessment (PISA). Given the impact PISA has on educational debate and policy, it is crucial that results are trustworthy. Yet, parallel to an increase in the number of ILSAs, there has been growing validity concerns regarding for example the content being tested, the influence on national educational systems and potential bias due to lack of sample representativeness (Grek, 2009; Jerrim, 2021; Meyer & Benavot, 2013). Relatively few studies, however, have focused on whether students are motivated to do their best in ILSAs as compared with high-stakes tests.

A motive for our research is international evidence suggesting that tests of low stakes impact student motivation and effort (Finn, 2015; Wise & DeMars, 2005). Whereas the relation between high- and low-stakes testing has been studied previously, findings are inconsistent, and we know little about this relationship in Sweden. Therefore, the following study examines whether there is evidence for the hypothesis that the lack of personal consequences may bias PISA test scores downwards. Indeed, self-reports from Swedish students indicate that they do not do their best in PISA (Eklöf & Hopfenbeck, 2019). However, PISA test scores have not yet been compared to external criteria such as national test scores. The theoretical framework used to interpret the results of the present study is the expectancy-value theory (Eccles & Wigfield, 2002; Wigfield & Eccles, 2000), postulating that test motivation depends on the student's expectations of succeeding at a particular task, the value the student places on the task, and the interaction between the two (Eccles & Wigfield, 2002). The expectancy-value theory has successfully been used in previous studies to explain the test-taking motivation construct (e.g., Eklöf & Knekta, 2017).

Previous research on the relationship between national tests and PISA/TIMSS revealed moderate to high but imperfect correlations (Skolverket, 2022; Wiberg, 2019; Wiberg & Rolfsman, 2019). One possible explanation is that ILSAs have low stakes for students while national tests have high stakes. In order to test the assumption that motivation influences student achievement, we will examine whether test motivation moderates the relationship between PISA scores and the national test scores. Skolverket (2022) found a correlation of .61 between the two measures but our hypothesis is that the relationship is different for different levels of motivation to take the PISA test. With reference to the expectancy-value theory we assume that the average level of test motivation is higher for national tests since this is a test with higher stakes. For students that were particularly unmotivated to do the PISA test, the correlation with their national test scores could therefore be lower. Consequently, the study examines the following research questions: (1) What is the correlation between test motivation in PISA and PISA achievement? and (2) Is the relationship between low-stakes PISA test scores and high-stakes national test scores moderated by students’ test motivation in PISA?

The effectiveness of initiatives in an educational context is often interpreted in terms of their impact on learning outcomes for every unit of investment. Governments invest a significantly high amount of money and effort in large-scale assessments (LSAs) with the intent to provide data-driven evidence to policymakers and researchers. Such evidence indicates the quality parameters of the education system in terms of learning level, equity, sustainability, and other predefined dimensions. Validity of such information is paramount due to its crucial role in decision-making for inputs, functional strategies, and goal setting for intended outputs, which if implemented as intended are most likely to lead to improvement in learning outcomes. Therefore, it is not an exaggeration to say that the effectiveness of LSAs can also be interpreted in terms of gain in learning outcomes or other dimensions, like any other measure. However, measuring and demonstrating effectiveness of LSAs remains a challenge due to multiple reasons besides the complexities involved in efficacy and effectiveness research, like the notion that assessment data themselves offer solutions. Authors make a compelling argument that data including assessment data do not provide solutions, rather they assist the policy and research community by providing valid evidence and insights enabling informed policy formulation and implementation decisions. Rich data and information provided by large-scale assessments can further be used to analyze the impact of those policy decisions.

The paper consists of three major parts. The first part reviews existing initiatives and proposes a logic model based on reasoning to estimate the effectiveness through evidence and/or counterevidence. A logic model depicts how an initiative is expected to make a difference, using explicit statements of the activities that are likely to bring about the intermediate changes and the impact the initiative intends to make. The proposed model postulates that if evidence generated by LSAs at T1 point of time were utilized to make appropriate modifications in policy and interventions regarding inputs, processes, organizational functioning, governance, monitoring mechanism, and outputs can lead to lead to learning gains per unit of investment at T2 point of time.

The second part builds on policy research and secondary analyses of large-scale assessments conducted in India to generate insights into the policy and practice that emerged from large-scale assessments. While the study primarily uses the assessment data and information from the National Achievement Survey (NAS) and the Annual Status of Education Report (ASER), it makes an effort to corroborate the findings with International and national LSAs in a similar context.

The third part recommends a policy implementation framework consisting of a series of steps to design system-specific strategies and monitor efforts. These steps are organized into two main phases: i) a ‘diagnostic’ phase to identify priority areas or enabling outcomes; ii) an ‘action’ stage to devise, implement and evaluate concrete policy interventions. The diagnostic stage mainly consists of cost-effective action-oriented surveys with a tiered approach, while the action stage consists of evidence-driven developmental goals and action plans for various levels of the system, alignment between all actors involved, customized interventions at school level with continuous monitoring in the cycle of 'assess-act-assess'

Education systems around the world have emphasized the need to transform assessments to improve learning. Proposed framework and model may be vital in designing learning systems to improve learning outcomes through effective systemwide assessments. However, there cannot be a one-size-fits-all policy mix. Feasible policy choices depend on contexts, social preferences, and political constraints. A robust and independent institutional framework, stakeholder engagement, and credible communication strategies are vital to enhancing the effectiveness of LSAs and eliminating learning poverty to achieve sustainable development goals.