23. Policy Studies and Politics of Education
Paper
On the Use and Interpretation of Interview Data in Research for Educational Policy and Practice
Stephen Parker1, Elizabeth Knight2
1University of Glasgow, United Kingdom; 2Victoria University, Australia
Presenting Author: Parker, Stephen;
Knight, Elizabeth
This paper speaks to long standing debates in social science research relating to how data are represented and interpreted in educational research. It does this in two ways. Firstly, it is an exploration of the issues faced research teams when attempting to interpret and understand the stories told by interviewees in relation to students’ choice of institution. Secondly, the paper uses the entry to higher education research context to explore the broader methodological dilemmas to provide an account of these students’ choices that goes beyond existing accounts of student choice and navigational capacity (e.g. Gale & Parker 2014).
Student choice of higher education institution has been considered using a variety of research methods and due to the large sets of data available it is as frequent to see quantitative (e.g. Anders 2012) as well as qualitative (Donnelly & Gamsu 2020) work that explores the issues around student choice. However, in qualitative methods there has been a significant reliance on interviews with individuals either pre-entry, during their studies or afterwards, or in some cases all three (e.g. Bathmaker et al. 2016). Investigating intent of interview respondents is a bedevilled activity; however, presenting text without critical analysis is equally problematic.
During a research project that has been detailed extensively elsewhere (Webb et al., 2020; Sinclair & Webb, 2021; Hodge et al., 2022; Gale, 2022) we generated interview data from enrolled students and recent graduates outlining their decision to choose a college-based higher education. A significant number of students in our research articulated that their choice was preferable to university-based alternatives. The explanation of the students’ choices took multiple forms and were based upon perceived benefits of college-based HE, namely a) the perceived distinctive pedagogy and assessment (Gale, 2022), and b) claims to a connection with industry (Sinclair & Webb, 2021) which meant that students would be taught skills more in line with what is needed for employment compared with their university-based peers. In terms of a), students identified the apparent practicality and non-theoretical aspects, as well as small class sizes.
Methodology, Methods, Research Instruments or Sources UsedaThis paper is derived from several data sources:
1) Reflection on previously published interview and survey data from student studying bachelor degrees in vocational institutions in Australia, generated as part of a broader Australian Research Council project (which involved the authors of this paper);
2) A range of published literature that employs qualitative data and data analysis;
3) Literature that engages with the onto-epistemological issues pertaining to interview data generation, analysis, and interpretation.
Conclusions, Expected Outcomes or FindingsThe paper reflects the apparent fixation and fetishisation in research outputs of data, the reification of first-person accounts, and a marginalisation of the importance of interpretation (McCulloch, 2004). There has been significant reflection on the techniques, purposes and value of interviewing as a qualitative research form (e.g. Burgess 1984; Fawcett & Hearn 2004; Hammersley 2008); how they might be refined to produce more ‘authentic’ responses (e.g. Gale et al. 2020; Mobley et al. 2019); how data are analysed, interpreted and represented (e.g. St Pierre 2013); as well as the variety of onto-epistemological stances that can inform the use of interview data (from grounded theory with a heavy emphasis on coding procedures (e.g. Deterding & Waters (2021)) to post-structural approaches (e.g. Lather 2004)) While we uphold the importance of interviews for producing first-person accounts and engaging with those with lived experience in research to support educational policy and practice, we note that there are disadvantages such as re-traumatisation to insisting of first-person accounts.
Instead, we argue for (re-)recognising interview data as text, with multiple meanings, constituted by and constitutive of discourse (Fairclough, 1995). Further, we argue that interpretation of such texts requires an appreciation of the role and positionality of the researcher in relation to the interview data rather than simplistic presentations of ‘findings’ of ‘themes’ that ‘emerge’.
ReferencesAnders, J. (2012). The Link between Household Income, University Applications and University Attendance. Fiscal Studies, 33(2), 185-210.
Bathmaker, A.-M., Ingram, N., Abrahams, J., Hoare, A., Waller, R., & Bradley, H. (2016). Higher Education, Social Class and Social Mobility: The Degree Generation. London: Palgrave Macmillan
Burgess, R.G. (1984). In the Field: An Introduction to Field Research. London: Allen & Unwin.
Deterding, N. M., & Waters, M. C. (2021). Flexible Coding of In-depth Interviews: A Twenty-first-century Approach. Sociological Methods & Research, 50(2), 708-739.
Donnelly, M., & Gamsu, S. (2020). Spatial structures of student mobility: Social, economic and ethnic ‘geometries of power’. Population, Space and Place, 26(3), e2293.
Elster, J. (1983). Sour Grapes: Studies in the subversion of rationality. Cambridge: Cambridge University Press.
Fairclough, N. (1995). Critical discourse analysis. The critical study of language. London: Longman.
Fawcett, B., & Hearn, J. (2004). Researching others: epistemology, experience, standpoints and participation. International Journal of Social Research Methodology, 7(3), 201-218.
Gale, T. (2022). Higher Vocational Education as a Work of Art. In E. Knight, A.-M. Bathmaker, G. Moodie, K. Orr, S. Webb, & L. Wheelahan (Eds.), Equity and Access to High Skills through Higher Vocational Education (pp. 291-317). Cham: Springer International Publishing.
Gale, T., Cross, R., & Mills, C. (2020). Researching Teacher Practice: Social justice dispositions revealed in activity. In J. Lynch, J. Rowlands, T. Gale, & S. Parker (Eds.), Practice Methodologies in Education Research (pp. 48-62). London: Routledge.
Gale, T., & Parker, S. (2015). To aspire: a systematic reflection on understanding aspirations in higher education. The Australian Educational Researcher, 42(2), 139-153.
Hammersley, M. (2008). Questioning Qualitative Inquiry: Critical Essays. London: Sage.
McCulloch, G. (2004). Documentary Research: In Education, History and the Social Sciences. London: Routledge.
Mobley, C., Brawner, C. E., Lord, S. M., Main, J. B., & Camacho, M. M. (2019). Digging deeper: qualitative research methods for eliciting narratives and counter-narratives from student veterans. International Journal of Qualitative Studies in Education, 32(10), 1210-1228.
St. Pierre, E. A. (2013). The Appearance of Data. Cultural Studies ↔ Critical Methodologies, 13(4), 223-227.
Taylor, C. (1985). Philosophy and the Human Sciences: Philosophical Papers 2. Cambridge: Cambridge University Press.
Wang, G., & Doyle, L. (2020). Constructing false consciousness: vocational college students’ aspirations and agency in China. Journal of Vocational Education & Training, 1-18.
23. Policy Studies and Politics of Education
Paper
Tensions and Embedded Stratification in UK Social Science Doctoral Provision
Richard Budd
Lancaster University, United Kingdom
Presenting Author: Budd, Richard
Scholars assert that, worldwide as well as in Europe, doctoral provision is increasingly characterised by accelerated scales of production, competitive research grants, centralised administration, and interdisciplinary, cohort-based training (Bao et al. 2018, Nerad 2020). This is associated with increased state interest and policy interventions that seek to heighten the contribution of doctorates within the knowledge economy, as well as concerns within higher education institutions (HEIs) around efficiency, cost savings, and differential access to funding (Thune et al. 2012). The situation in the UK appears to mirror this picture on the whole (McGloin and Wynn 2015, Harrison et al. 2016), but scholars have long noted that national settings mediate the forms that broader trends take, due in part to the differing degrees of organisational autonomy and status hierarchies that prevail within countries (Hüther and Krücken 2016, Powell et al. 2016). Both of these are relatively pronounced in the UK’s university sector (Evans et al. 2019), and this paper specifically examines the provision of doctorates in the social sciences within that.
Doctorates in this area have – as other fields – experienced significant growth and a range of policy changes (McGloin and Wynn 2015). Some doctoral-related policies are directed towards UK higher education as a whole, but one – the Economic and Social Research Council’s (ESRC) Doctoral Training Centres/Partnerships (DTCs/DTPs) implemented from 2010 – targets the social sciences in particular. The most striking feature of a DTC/DTP is the creation of an interdepartmental and often inter-institutional unit to coordinate ESRC-funded doctorates and training. These are based on a model that initially emerged to support equipment sharing in science and engineering disciplines, but it has since been transferred to other areas and now proliferates across UK higher education (Lunt et al. 2014). A secondary impact of the ESRC policy was an instant halving of the number of HEIs receiving state funding for social science PhDs; the group of recipients was expanded in 2016 when the policy was renewed, but not to pre-2010 levels. There is some research which documents the not entirely easy experiences of those who retained ESRC patronage (Lunt et al. 2014, Deem et al. 2015), but there is little work analysing how other kinds of universities fared where doctoral numbers have continued to rise.
To unpick this further, we invoked the notion of institutional isomorphism (DiMaggio and Powell 1983), a conceptual framework which asserts how convergence in organisational sectors can be driven by different factors, namely coercion (policy), mimesis (imitation), normativity (taken for granted activity) and rationalisation (calculated optimisation). Isomorphism has been used fruitfully in higher education by a number of scholars interested in convergence and organisational behaviour (McQuarrie and Kondra 2016, Meyer and Powell 2020, Shin and Chung 2020). Applying this to how different universities operated in the social science doctoral space – and, crucially, why – allows us to take a closer look at the extent to which convergence is actually taking place, but also with a view to understanding any potential diversity. Given the manifestly uneven impact of the DTC/DTP policy, and also the markedly hierarchical nature of UK higher education, it might be expected that isomorphism will only be partial. Examining this more closely not only extends the literature in this specific area, but also provides insights into how organisations’ individual positioning and history may have implications for how they are able to behave in policy contexts.
To be precise, our research question here is as follows:
To what extent is convergence occurring in UK social science doctoral provision, and what isomorphic processes promote or hinder it?
Methodology, Methods, Research Instruments or Sources UsedWe recruited a non-representative but purposive sample of 32 senior academics and research directors who had responsibilities associated with their social science doctorates. Seeking to ensure a varied sample, based on the literature-informed expectation that their histories, experiences, and approaches to doctoral provision might differ, they were drawn from a range of HEIs from all four countries of the UK. Although the ESRC is not the only source of social science state funding in the UK, it is the main actor in this specific space, and were therefore used eligibility for its funding over time as an indicative proxy of the profile (i.e. size/history) of an HEI’s doctoral provision. This allowed us to observe that most of the 120 UK HEIs who offered postgraduate research in the social sciences could be divided into three distinct groups:
- Insiders: older, higher status HEIs who retained access to ESRC funding
- Leavers: Mid-range research-intensive HEIs or former polytechnics who lost ESRC funding
- Outsiders: Newer, more teaching-oriented HEIs who never had access to ESRC funding
Following ethical approval, data was collected through semi-structured discussions that ranged across participants’ views of the broader policy context around doctoral provision and their institution’s specific actions and rationales related to this. The data was coded and analysed, using NviVo software, according to a coding scheme which was constructed in part a priori according to the doctoral activities undertaken (growth, organisation, cohort models) and their underpinning rationales – i.e. the isomorphic processes – with emergent subcategories. This allowed us to examine the within- and between-group differences around how and why HEIs were active around their social science research degrees.
Conclusions, Expected Outcomes or FindingsAcross all three groups we saw evidence of considerable dynamism – and anxiety – around their social science doctorates. Participants invariably described their HEI as actively implementing significant changes, but this was most pronounced in the Leaver group, those who had at least temporarily lost ESRC patronage. There were similarities between the groups, but also important differences in how they sought to – or were able to – achieve their goals. This was to a certain extent driven by the influence and interaction of the isomorphic processes, some of which exerted the same kind of pressure across all three groups, while others were more specific to particular kinds of institutions. What mattered overall was not only which processes were at play, but also their strength and whether they operated in concert or in tension and this in particular differed between the groups. So while there was a degree of convergence, a variety of factors also impeded uniformity. In other words, starting points matter greatly because policies here operate in such a way that those in the lead maintain their advantage and those furthest back are the most impeded from catching up.
This work reiterates that attention needs to be paid to the national idiosyncrasies of higher education policy spaces in order to establish if – or how – broader trends are replicated or refracted and what mediates that. What we have shown here is that, in a sector where status differences are pronounced and HEI profiles varied, policies interact with a range of factors that to some extent encourage isomorphism but at the same time can reinforce heterogeneity when they penalise ‘weaker’ players. In the interests of equity, policy models should ameliorate these detrimental effects and support all universities in their development rather than reinforcing entrenched sectoral hierarchies.
ReferencesBao, Y., Kehm, B.M., and Ma, Y., 2018. From product to process. The reform of doctoral education in Europe and China. Studies in Higher Education, 43 (3), 524–541.
Deem, R., Barnes, S., and Clarke, G., 2015. SOCIAL SCIENCE DOCTORAL TRAINING POLICIES AND INSTITUTIONAL RESPONSES : THREE NARRATIVE PERSPECTIVES ON RECENT DEVELOPMENTS IN AND CONSEQUENCES OF THE UK TRANSITION TO COLLABORATIVE DOCTORAL TRAINING ’. In: E. Reale and E. Primeri, eds. Universities in Transition: Shifting institutional and organisational boundaries. Rotterdam: Sense, 137–162.
DiMaggio, P.J. and Powell, W.W., 1983. The Iron Cage Revisited: Institutional Ismorphism and Colective Rationality in Organizational Fields. American Sociological Review, 48 (2), 147–160.
Evans, C., Rees, G., Taylor, C., and Wright, C., 2019. ‘Widening Access’ to higher education: the reproduction of university hierarchies through policy enactment. Journal of Education Policy, 34 (1), 101–116.
Harrison, J., Smith, D.P., and Kinton, C., 2016. New institutional geographies of higher education: The rise of transregional university alliances. Environment and Planning A, 48 (5), 910–936.
Hüther, O. and Krücken, G., 2016. Nested Organizational Fields: Isomorphism and Differentiation among European Universities. Research on the Sociology of Organisations, 46, 53–83.
Lunt, I., McAlpine, L., and Mills, D., 2014. Lively bureaucracy? The ESRC’s Doctoral Training Centres and UK universities. Oxford Review of Education, 40 (2), 151–169.
McGloin, R.S. and Wynn, C., 2015. Structural Changes in Doctoral Education in the UK A Review of Graduate Schools and the Development of Doctoral Colleges.
McQuarrie, F.A.E. and Kondra, A.Z., 2016. Exploring the Process of Institutional Isomorphism in Patchy Organizational Fields. Academy of Management Proceedings, 2016 (1), 15086.
23. Policy Studies and Politics of Education
Paper
Depending on the Kindness of Strangers: The Affective Dimension of Inspection Visits to Low-Performing Schools in Chile
Álvaro González1, Rocío Fernández Ugalde2
1Universidad Católica Silva Henríquez, Chile; 2University of Cambridge, UK
Presenting Author: González, Álvaro
Many countries around the world have implemented Performance-Based Accountability (PBA) policies in education through standardised assessments and quality assurance instruments or mechanisms (Verger and Parcerisa 2017). One such instrument or mechanism corresponds to school inspections, which generally consist of visits from external actors (inspectors) to collect and produce information to evaluate schools’ performance and deliver guidelines for improvement based on standardised quality criteria (Ehren et al. 2015). According to Munoz-Chereau and Ehren (2021, 10) inspections “are performance systems conceived as a key accountability mechanism to govern education” and they “occupy the middle ground between policy and practice”.
As a PBA policy instrument, school inspections usually attempt to prompt change and improvement by setting expectations based on norms and standards, providing performance feedback through evaluation, recommended actions and potential consequences, and enabling stakeholders’ pressure by making the information publicly available. However, inspection models have shown mixed results in terms of promoting improvement, in addition to several negative consequences over curriculum, teaching and learning, practitioners’ professionalism and schools’ culture (de Wolf and Janssens 2007; Penninckx et al. 2014).
In Chile, a country well-known for its PBA policies, inspections take a prominent part in the National System of Education Quality Assurance (SAC). This system, implemented in 2011 (Law 20.529), establishes the basis for evaluating schools’ effectiveness and is considered the core of the country’s PBA policies (Falabella 2021; Parcerisa 2021). Depending on the outcomes of a series of standardised academic and non-academic metrics, schools are ranked and ordered from highest to lowest performance in four categories: High, Middle, Middle-Low and Insufficient. Inspection visits are carried out by a panel of Quality Agency inspectors in schools considered low performing (i.e., Insufficient) with the purpose of guiding their improvement (Munoz-Chereau, González, and Meyers 2022).
Evidence about SAC’s effect on schools has mostly concentrated on results measured by schools’ outcomes in standardised evaluations (e.g., SIMCE tests) or the (expected or unexpected) consequences of performance categories in schools’ practices, however, evidence specific about inspection visits in Chile and their consequences is still scarce (Bravo Cuevas 2019). Moreover, international research about PBA policies focuses mostly on their effects on practices and results, but rarely on what scholarly literature identifies as the affective dimension. The affective dimension can help understanding actors’ level of acceptance of the inspection process in general, and their capacity to act on the feedback provided to improve individual and school practices (Quintelier, De Maeyer, and Vanhoof 2020). This dimension follows what Grek, Lindgren, and Clarke (2014, 117) describe as affective governing, which “not only relate to the rise of feelings of anxiety or stress that school inspections are associated with” but also consider the interactive or relational aspect of inspection “where inspectors and inspectees have to meet face to face and negotiate differences of position, authority and interest”.
To understand how school actors make sense of the performance feedback from the inspection visits through the incorporation of the affective dimension, this paper examines this phenomenon from a policy enactment perspective. Policy enactment offers a critical perspective of how policies are recreated and produced (Ball, Maguire, and Braun 2012), by considering local contingencies and the agency of actors (Parcerisa 2021). Thus, this paper explores to what extent both the emotions of and relationship between Agency inspectors and school leaders in the context of inspection visits encourage school actors to make sense of the performance feedback resulting from the inspection visit, as a way of doing policy work.
Methodology, Methods, Research Instruments or Sources UsedThis paper draws from a three-year exploratory multiple case study that investigates the influence of different instruments of SAC on the improvement of public primary schools, located in disadvantaged areas, and that were classified as Insufficient in 2016 by the Quality Agency. Since then, these schools have had different improvement trajectories according to the Agency’s yearly performance evaluation, until 2019: sustained improvement, irregular improvement, and no improvement. Three schools, each representing one of these trajectories, were chosen for this paper.
School A represents the No Improvement trajectory, as its insufficient category has not changed since 2016. It is located in a densely populated urban area and serves students from low-income families in the surrounding neighbourhoods. The school was inspected in 2017 and a follow up visit was carried out in 2019 by the same panel of inspectors.
School B represents the Irregular Improvement trajectory since it has advanced and regressed in its performance category between 2016 and 2019 and is currently classified as Medium-Low. The school is in an urban area where most of their students live, but also serves students from nearby rural areas, all of them with low income or minimal levels of education. The school was inspected in 2017 by a panel of three inspectors, and two of them returned for a follow up visit in 2019.
School C represents the Sustained Improvement trajectory as it has systematically progressed until reaching the classification Medium. The school is in a middle-class urban area, but a significant number of students come from low-income families living in rural areas, so the school offers free transportation. The school was inspected in 2017 by a panel of three inspectors, with no follow up.
Individual semi-structured interviews were conducted with the principal and curriculum coordinator from each school (n=6), as well as the inspectors that visited them (n=9). Interviews were conducted between 2020 and 2021 through video call, had an average length of 60 minutes and the audio was transcribed for analysis. Data were analysed through a qualitative content analysis strategy (Schreier 2014), for the identification of emergent themes based on a coding framework developed according to the study purpose of understanding how school actors make sense of the performance feedback of the inspection through the incorporation of the affective dimension.
Conclusions, Expected Outcomes or FindingsInspection visits are viewed by inspectors and school leaders as a critical instrument for school improvement. Inspectors generally observe schools from an outsider position, they carefully collect and systematise evidence about instructional and management standards and employ it to provide feedback to school actors about their performance, whilst also attempting to translate quality assurance criteria for schools. School leaders initially try to perform what they believe is expected from them by inspectors, as inspection visits follow a strong normative pattern, framed by the affective forces on which the quality assurance system is built upon (Matus 2017; Falabella 2021). However, the cases also show that some leaders assumed a more dialogical position to better understand the feedback offered to them, which seems to be shaped mainly by the unique and agential affective forces from bodies in place at the moment of the visit. This resembles a form of affective governing that arises from the direct encounter of inspectors and school leaders which enriches an understanding of sensemaking in the inspection process (Grek, Lindgren, and Clarke 2014). Although all three schools were weary of the inspection visit on the back of their emotional response to the performance category, those reporting mutual understanding and kindness seem to have taken active advantage of the performance feedback, which in turn regulates the possibilities for making sense of recommendations for school change. The issue of kindness becomes a surprising and even counterintuitive finding, as low performing schools become scrutinised by the labelling of the quality assurance system, which is represented by these strangers -the inspectors- that show up at the school to offer recommendations. Thus, by turning to the affective dimension of inspections, we shed light on the unplanned but everyday events of this PBA policy instrument, which is key for understanding policy enactment and implementation.
ReferencesBall, Stephen J, Meg Maguire, and Annette Braun. 2012. How Schools Do Policy. Policy Enactments in Secondary School. London: Routledge.
Bravo Cuevas, Sergio. 2019. “Visitas de Orientación y Evaluación Realizadas Por La Agencia de Calidad de La Educación En Chile: Significados Otorgados Por Directivos de Escuelas Públicas.” Temps d’Educació 57 (57): 267–282.
de Wolf, Inge F., and Frans J.G. G Janssens. 2007. “Effects and Side Effects of Inspections and Accountability in Education: An Overview of Empirical Studies.” Oxford Review of Education 33 (3): 379–396.
Ehren, Melanie C.M., J.E. E. Gustafsson, H. Altrichter, G. Skedsmo, D. Kemethofer, and Stefan G. H. Huber. 2015. “Comparing Effects and Side Effects of Different School Inspection Systems across Europe.” Comparative Education 51 (3): 375–400.
Falabella, Alejandra. 2021. “The Seduction of Hyper-Surveillance : Standards, Testing, and Accountability.” Educational Administration Quarterly 57 (1): 113–142.
Grek, Sotiria, Joakim Lindgren, and John Clarke. 2014. “Inspection and Emotion: The Role of Affective Governing.” In Governing by Inspection, edited by Sotiria Grek and Joakim Lindgren, 116–136. New York: Routledge.
Matus, Claudia. 2017. “The Uses of Affect in Education: Chilean Government Policies.” Discourse 38 (2): 235–248.
Munoz-Chereau, Bernardita, and Melanie Ehren. 2021. Inspection Across the UK: How the Four Nations Intend to Contribute to School Improvement. Final Report. London: Edge Foundation.
Munoz-Chereau, Bernardita, Álvaro González, and Coby V. Meyers. 2022. “How Are the ‘Losers’ of the School Accountability System Constructed in Chile, the USA and England?” Compare: A Journal of Comparative and International Education 52 (7): 1125–1144.
Parcerisa, Lluís. 2021. “To Align or Not to Align: The Enactment of Accountability and Data-Use in Disadvantaged School Contexts.” Educational Assessment, Evaluation and Accountability 33 (3): 455–482.
Penninckx, Maarten, Jan Vanhoof, Sven De Maeyer, and Peter Van Petegem. 2014. “Exploring and Explaining the Effects of Being Inspected.” Educational Studies 40 (4): 456–472.
Quintelier, Amy, Sven De Maeyer, and Jan Vanhoof. 2020. “Determinants of Teachers’ Feedback Acceptance during a School Inspection Visit.” School Effectiveness and School Improvement 31 (4): 529–547.
Schreier, Margrit. 2014. “Qualitative Content Analysis.” In The SAGE Handbook of Qualitative Data Analysis, edited by Uwe Flick, 170–183. London: SAGE.
Verger, Antoni, and Lluis Parcerisa. 2017. “La Globalización de La Rendición de Cuentas En El Ámbito Educativo: Una Revisión de Factores y Actores de Difusión de Políticas.” Revista Brasileira de Política e Administração Da Educação 33 (3): 663–684.
|
