23. Policy Studies and Politics of Education
Paper
The Politics of Resilience - the Case of England's Qualification System
Michelle Meadows1, Jo-Anne Baird1, Neil Stringer2, Thomas Godfrey-Faussett1
1Oxford University, United Kingdom; 2Ofqual, United Kingdom
Presenting Author: Baird, Jo-Anne
The UK Government has produced a ‘Resilience Framework’, which aims to ensure the country’s prosperity by having a national infrastructure that is better equipped to tackle adverse events such as, but not limited to, pandemics, weather events and cybersecurity attacks. The qualification system is now being seen as an essential part of government infrastructure. Delivering qualifications is a complex, high-volume, distributed activity involving multiple actors across organisations with a range of relationships. Over 26 million examination scripts and coursework tasks are dealt with in the system in England annually, for qualifications taken by 16- and 18-year-olds. There is a quasi-market of four examination boards who offer academic qualifications and there are hundreds of organisations offering vocational qualifications. In this paper, we report on a project that sought to investigate not only how resilient the qualification system in England is, but what such resilience might mean. Using publicly available documents, input from an expert advisory group (10 people) and elite interviews with 21 assessment insiders, we analysed the resilience of England’s qualification system. The recent exams crisis created by the pandemic was one focus, but we explored resilience more broadly. To define resilience, we drew upon definitions published in the literature for other complex, distributed systems (food, healthcare and utilities). Systematic reviews of the term in other fields pointed out that the term ‘resilience’ is fluid in its meaning. For the purposes of this research project, we defined resilience as, ‘The capacity of the qualification system and its units at multiple levels to actively engage with, manage and learn from periods of change and unforeseen disturbances to deliver timely and sufficiently accurate, trusted, and valid grades to fulfil their purpose(s) now and in the future.’ Interviewees included regulators (3), civil servants (2), academics (2), teacher leaders or union representatives (5), individuals with think tanks, communications or PR perspectives (4), and exam board or awarding organisation insiders (5). We explored how qualification system resilience might be defined, its characteristics, resilience of the qualification system during the pandemic, threats to resilience and what countermeasures might be taken to them. Our interview data showed no consensus on the definition of resilience amongst the industry insider participants. Nor was there agreement on whether the system is currently resilient. Various proposed countermeasures for perceived lack of resilience have been publicly debated (teacher assessment, modular examinations, digitalisation). Our analysis outlines the risks, as well as potential benefits of each of these proposals. We conclude that the term resilience must be defined in relation to specified aims. Many threats to resilience were identified, including political pressure - a key feature of the 2020 exam policies. Fundamentally redesigning the system for resilience to unlikely catastrophic events would be a mistake. The cause of the 2020 crisis is best described as poor policy rather than as system fragility. Prospects for managing policy mistakes through government agencies (‘quangos’) are not encouraging due to the relationship with government. This case demonstrates fundamental weaknesses for the UK in delivering resilience, in the qualifications system and beyond. The role of politics in educational assessment policy differs across nations. This case serves to illustrate how the management of political agendas and policy mistakes is integral to managing education systems. This nebulous concept is useful in political terms, as policymakers can point to a lack of system resilience, rather than identifying issues as policy failures. Pointing to resilience is a useful vehicle for shifting policy evaluation criteria and responsibility for those. Qualification systems may be at particular risk of political pressures because examination grades are symbolic and intangible; their value is socially constructed.
Methodology, Methods, Research Instruments or Sources UsedDocumentary analysis included journal publications, grey literature, parliamentary Select Committee transcripts and reports and statistical publications. This led to a working project definition of resilience; depiction of the qualifications system; understanding of previously documented qualification crises; and consideration of various potential countermeasures to the problems encountered during the pandemic.
An advisory group comprising 10 experts was formed to advise on methodology, conceptual development of the project and interpretation of research findings. Members were selected for their knowledge of regulation, awarding body research, government policy, understanding of the school and college sector or for their academic expertise. An innovative use of this expert group was in collecting data through the advisory meetings.
Twenty-one interview participants were recruited. Participants were selected to give a range of political and ideological perspectives, including individuals openly supportive of the system, as well as those calling for reform. Interviewees included regulators (3), civil servants (2), academics (2), teacher leaders or union representatives (5), individuals with think tank, communications or PR perspectives (4), and exam board or awarding organisation insiders (5).
Interviews were conducted online and were transcribed. Whilst an interview schedule was used to guide the interviews, this was used flexibly. Perspectives of elites – particularly bureaucratic elites – were considered when reflecting on the positionality of the data and of us as insider-outsider researchers. At the end of each interview the main themes of the interview were summarised, giving the participant an opportunity to correct, clarify or extend ideas.
Transcripts were coded deductively by three researchers using a codebook. In a training phase, the researchers independently coded the same three transcripts, randomly selected from the sample of 21. These were then compared for inter-rater agreement, and a coding meeting was held to reflect on the process and the clarity and comprehensiveness of the codebook. One code was revised for clarity. The remaining 18 transcripts were divided randomly between the researchers who again coded independently before a final analysis meeting to discuss the results. There was near-perfect agreement between the coders at each stage. Interrater agreement was calculated as between 95-100% for all but one code - “threats to resilience”. The interrater reliability for this code ranged from 80-88% and disagreements were straightforward to resolve in coding meetings. Data was synthesised across the datasets by code and research question.
Conclusions, Expected Outcomes or FindingsOur definition of qualification system resilience was broadly supported by the interviewees. They commented on threats related to unforeseen circumstances and periods of planned change, including qualification reform. These experts discussed the need to manage the timely delivery of sufficiently valid and reliable grades. Some experts also reflected on the need for public trust in – or at least societal acceptance of – grades. None of the interviewees questioned the very notion of resilience, but some recognised the political capital to be gained from claims about the weaknesses of the system. Having a clear definition of resilience is one step towards being able to debate what are realistic expectations of the system. Afterall, as some interviewees pointed out, there is a limit to the extent to which the delivery of qualifications can be resilient to all potential threats, and a system that functions well under extreme circumstances is unlikely to be suitable in normal times. Nonetheless, interviewees suggested changes to the system that would, in their view, improve resilience: teacher assessment, modular examinations and digitalisation. Political interference in the system was identified as a significant threat to resilience. Although government agencies have been established to manage activities where direct political control is undesirable, they are nonetheless still under political control. The concept of resilience meant different things to different stakeholders, which was a lever for creating change agendas aligned with interviewees’ values and ideologies. None of the suggested countermeasures for improving resilience come without their own risks to resilience. Selecting between these policies is therefore a matter of values and politics, rather than a neutral, technocratic procedure. As insider researchers ourselves, we conclude that who defines the term resilience is key to interpretation of the resilience of the system.
ReferencesBaird, J.-A., & Coxell, A. (2009). Policy, Latent Error and Systemic Examination Failures. CADMO, XVII(2), 105–122.
Baird, J.-A., Isaacs, T., Opposs, D., & Gray, L. (2018). Examination Standards: How Measures and Meanings Differ Around the World. UCL, IOE Press.
Baird, J.-A., & Lee-Kelley, L. (2009). The dearth of managerialism in implementation of national examinations policy. Journal of Education Policy, 24(1), 55–81.
Bergström, J., van Winsen, R., & Henriqson, E. (2015). On the rationale of resilience in the domain of safety: A literature review. Reliability Engineering & System Safety, 141, 131–141. https://doi.org/10.1016/j.ress.2015.03.008
Biddle, L., Wahedi, K., & Bozorgmehr, K. (2020). Health system resilience: A literature review of empirical research. Health Policy and Planning, 35(8), 1084–1109. https://doi.org/10.1093/heapol/czaa032
Birkmann, J., Dech, S., Hirzinger, G., Klein, R., Klüpfel, H., Lehmann, F., Mott, C., Nagel, K., Schlurmann, T., Setiadi, N. J., Siegert, F., & Strunz, G. (2006). Measuring vulnerability to promote disaster resilient societies: Conceptual frameworks and definitions. http://collections.unu.edu/view/UNU:2793
Bowe, R., Ball, S., J., & Gold, A. (1992). Reforming education and changing schools. Routledge.
Cabinet Office. (2022). The UK Government Resilience Framework. UK Government. https://www.gov.uk/government/publications/the-uk-government-resilience-framework/the-uk-government-resilience-framework-html
Hammerstein, S., König, C., Dreisöner, T., & Frey, A. (2021). Effects of COVID-19-related school closures on student achievement—A systematic review. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.746289
Hayward, L., Baird, J.-A., Allan, S., Godfrey-Faussett, T., Hutchinson, C., MacIntosh, E., Randhawa, A., Spencer, E., & Wiseman-Orr, M. L. (2023). National qualifications in Scotland: A lightning rod for public concern about equity during the pandemic. European Journal of Education, 58(1), 83–97. https://doi.org/10.1111/ejed.12543
Humbert, C., & Joseph, J. (2019). Introduction: The politics of resilience: problematising current approaches. Resilience, 7(3), 215–223. https://doi.org/10.1080/21693293.2019.1613738
Kelly, A. (2014). Monopolising the examining board system in England: A theoretical perspective in support of reform. Journal of Education Policy, 29(1), 44–57. https://doi.org/10.1080/02680939.2013.790078
McCaig, C. (2003). School Exams: Leavers in Panic. Parliamentary Affairs, 56(3), 471–489. https://doi.org/10.1093/parlij/gsg101
Opposs, D., Baird, J.-A., Chankseliani, M., Stobart, G., Kaushik, A., McManus, H., & Johnson, D. (2020). Governance structure and standard setting in educational assessment. Assessment in Education: Principles, Policy & Practice, 27(2), 192–214. https://doi.org/10.1080/0969594X.2020.1730766
Ozga, J., Baird, J.-A., Saville, L., Arnott, M., & Hell, N. (2023). Knowledge, expertise and policy in the examinations crisis in England. Oxford Review of Education, 1–19. https://doi.org/10.1080/03054985.2022.2158071
Wisner, B., Blaikie, P., Cannon, T., & Davis, I. (2003). At Risk: Natural Hazards, People’s Vulnerability and Disasters. Routledge.
23. Policy Studies and Politics of Education
Paper
The Politics of Assessment as Experienced and Enacted by Teachers and Guidance Counselors in the Finnish Comprehensive Education
Hannele Pitkänen
University of Jyväskylä, Finland
Presenting Author: Pitkänen, Hannele
In recent years, Finnish basic education has undergone significant transformations in its policies and practices related to student assessment. For instance, reflecting the international trends (see e.g. Birembaum et al. 2015) the role of student self-assessment has been emphasized. This paper delves into the exploration of the current 'politics of assessment' and its historical evolution since the late 1990s, as perceived and experienced by Finnish basic education teachers and guidance counselors working with students in their final years of basic education.
The study employs the theoretical concept “the politics of assessment”. “The politics of assessment” draws on the poststructural stance on governance, characterized as the 'conduct of conduct' (Foucault, 1982, 1988; Fejes & Dahlstedt 2012; Rose, 1999/2009). The term encompasses the role of assessment policies, embedded in legislative and curricular documents, not only playing part in the governing of the practices of evaluation within educational settings but significantly, the part they play in shaping of subjectivities and future perspectives for students involved in assessment. Deriving from these theoretical starting points, 'the politics of assessment' captures the dynamic interplay of the governance of others, being governed, and self-governance within the realm of student assessment policies and practices (Pitkänen 2022).
Against the backdrop of the contemporary landscape of assessment policies, two parallel and globally impactful trends come into sharp focus. The first trend, referred to as 'the politics of self-evaluation,' emphasizes formative assessment and student self-assessment (Pitkänen 2022). In the transnational and European policy discourse and educational theory it has been widely discussed under theme of assessment-for-learning (e.g. Birenbaum et al 2014, OECD 2008). This trend found its way into the Finnish basic education landscape, particularly with the introduction of the 1994 curriculum. Subsequently, the idea of formative assessment and student self-evaluation has firmly taken root in both the curricula of basic education and educational legislation. More recently, a second trend, termed here as 'the politics of standardization,' has emerged within Finnish assessment policy and practice. This trend is deeply intertwined with the international movement toward educational standardization (e.g. Riese et al 2022) including the emphasis of the standard assessment-of-learning (e.g. Sahlberg 2016). In Finland, the prevailing trend towards standardization is deeply rooted in the increasing policy emphasis on ensuring fair and equitable assessment practices. Studies, for example, have indicated that students with similar proficiency levels could receive significantly different grades in different schools (e.g. Hildén ym. 2017; Ouakrim-Soivio 2013), undermining the fundamental premise of comprehensive education to provide students with equal opportunities for further education. As a solution to this problem the assessment criteria have been introduced. They were first presented in the late 1990’s as recommendation-type guidelines. Since 2004, the criterion-based assessment has become established in the Finnish basic education curricula. Currently, in assessments at the end of sixth and ninth grade, the criteria have been specified for grades 5, 7, 8, and 9.
This paper delves into the dual trajectory of assessment politics and its impact on everyday school life. The research analyses 1) teachers' and guidance counselors’ perspectives on the history and formation of the current assessment policy. The study is interested in how teachers and guidance counselors describe and perceive the change. Secondly, the research analyses 2) how the current assessment policy, as narrated by teachers and guidance counselors, has been enacted in the school, how it has been experienced, and how its role is perceived from the perspective of governing teachers' and guidance counsellors work, students' schoolwork, and the students' understanding of themselves.
Methodology, Methods, Research Instruments or Sources UsedThe research presented in this paper is part of an ongoing ethnographic project that explores the politics of assessment and its manifestation in the everyday practices of education and guidance counselling within the context of Finnish comprehensive education. This sub-study is conducted within a single comprehensive school. For this paper, a series of 10-15 thematic interviews will be conducted with teachers and guidance counselors during the spring term of 2024. Teachers and guidance counselors play a pivotal role in this context, actively participating in the enactment of the politics of assessment in the day-to-day reality of education. Additionally, they serve as interpreters or 'translators' of these 'politics' to the students, shaping the students' understanding and engagement with the assessment processes.To provide a historical perspective, emphasis will be placed on inviting participants with extensive experience in final assessment and/or guidance counselling within comprehensive education.
The analysis of the interview data will be twofold. Firstly, we will examine the narratively constructed histories of policy change. The focus in this phase of the analysis will be on understanding how these changes are rationalized by the informants and lived by the participants in the research. Secondly, the study investigates how recent assessment policies have been implemented within the school. This phase of the analysis is guided by the theory-based hypothesis that the policies are not mere implementations but active enactments by actors in the local school contexts (Ball et al 2011). They do not solely impact the actual assessment practices but also significantly contribute to shaping pupil identities, subjectivities, and their understanding of themselves.
The analysis of the enactment of the politics of assessment is approached through the lens of participants' experiences as narrated during the interviews.
Conclusions, Expected Outcomes or FindingsThe study will examines the recent transformations student assessment policies and practices in Finnish comprehensive education, by focusing on the historical narratives and lived experiences of the ones responsible for the implementation of these policies in the grassroot level. This paper will present the preliminary insights derived from interviews with teachers and guidance counselors regarding their experiences with the current politics of assessment and its implementation in the daily routines of education and guidance counseling. he study will offer a rich and nuanced perspectives of those actively involved in policy enactment at the school level and offer intimate accounts on their daily interactions with their students, directly affected by central policies and politics of assessment and guidance. While the study is situated within the context of Finland, the parallel aspects underlying the contextual specificities are universal and global. With the example of Finland, this paper argues that the politics of students' self-evaluation and standardization are prevalent in the common European policy discourse and solutions at large, making the results of this study relevant to an international audience.
ReferencesBall, S. J., Maguire, M. & Braun, A. (2011) How Schools Do Policy. Policy Enactments in Secondary Schools. London: Routledge.
Birenbaum, M., DeLuca, Christopher, Earl, Lorna, Heritage, Margaret, Klenowski, Val, Looney, Anne, Smith, Kari, Timberley, Helen, Volante, Louis & Wyatt-Smith, Claire. 2015. International trends in the implementation of assessment for learning: Implications for policy and practice. Policy Futures in Education 13 (1), 117–140. https://doi.org/10.1177/1478210314566733
Fejes, A. & Dahlstedt, M. (2012). The Confessing Society: Foucault, Confession and Practices of Lifelong Learning. Taylor and Francis Group.
Foucault, M. (1982). The Subject and Power. Critical Inquiry 8(4), 777–795.
Hildén, R., Rautopuro, J., & Huhta, A. (2017). Arvosanan ansaitsemme : asteikolla vai ilman?. In V. Britschgi, & J. Rautopuro (Eds.), Kriteerit puntarissa (pp. 63-80). Suomen Kasvatustieteellinen Seura, FERA. Kasvatusalan tutkimuksia
Pitkänen, H. (2022a). The Politics of Pupil Self-evaluation: A case of Finnish assessment policy discourse. Journal of Curriculum Studies. https://doi.org/10.1080/00220272.2022.2040596
Popkewitz, T. S. (1997). The production of reason and power: Curriculum history and intellectual traditions. Journal of Curriculum Studies 29(2), 131–164.
Popkewitz, T. S. (2004). Educational Standards: Mapping Who We Are and Are to Become, The Journal of the Learning Sciences, 13:2, 243-256, DOI: 10.1207/s15327809jls1302_7
Popkewitz, T. S (2017). Reform and making human kinds: the double gestures of inclusion and exclusion in the practice of schooling. In E. Hultqvist, S. Lindbland & T. S. Popkewitz (eds.) Critical analyses of educational reforms in an era of transnational governance. (pp. 133–150). Springer.
Riese, Hanne & Hilt, Line & Søreide, Gunn. (2022). Educational standardisation in a complex world.
Rose, N. (1999/2009). Powers of freedom: Reframing political thought (2nd ed.). The Press Syndicate of the University of
Cambridge.
Sahlberg, P. (2016). The global educational reform movement and its impact on schooling. In K. Mundy, A. Green, B.
Lingard, & A. Verger (Eds.), The handbook of global education policy (pp. 128–144). Wiley-Blackwell.
|
