09. Assessment, Evaluation, Testing and Measurement
Paper
Competence Assessment Interviewer Effects in a Large-scale Educational Survey: a Replication Using NEPS Data
Andre Pirralha, Laura Löwe
LIfBi, Germany
Presenting Author: Pirralha, Andre;
Löwe, Laura
Large-scale educational studies are an important resource to inform policymakers and the general public about the reach and effectiveness of diverse aspects of educational systems in several countries. Competence assessment in institutional settings (e.g. schools) has been an essential factor to collect valid measurements of cognitive abilities or motivations, for example. In order to conduct the assessment sessions, a significant number of test administrators (TAs) are necessary to supervise and coordinate test groups in the participating schools. The TAs undergo specific training and follow a strict protocol to ensure that competence assessment sessions are standardized and comparable so that student achievement data can be meaningfully collected. The TA characteristics can affect the quality of assessment scores and survey data. Differences in their behavior can originate interviewer effects, systematically impacting the validity and comparability of competence assessment tests. While there has been a recent effort to change competence assessment testing to computer-assisted modes of data collection, there is very little research aimed to uncover whether the training sessions and protocols are effectively delivering the goal of preventing TA effects in the first place.
In this paper, we explore the presence and magnitude of interviewer effects on paper-and-pencil competence assessments for mathematics abilities and survey questions in a German nationally representative longitudinal educational survey (National Educational Panel Study - NEPS). For this purpose, we will replicate the Lüdtke et al. (2007) paper, to date the only empirical investigation of TAs interviewer effects we are aware of. Multilevel analyses for cross-classified data are taken to effect to decompose the variance associated with differences between schools and the variance associated with TAs. The results are of use to improve competence assessment testing procedures, particularly by unveiling whether interviewer training and protocols should be improved and to assess the existence and magnitude of interviewer effects in test assessment sessions under paper and pencil-based modes of data collection.
Methodology, Methods, Research Instruments or Sources UsedTo effectively study test administrator effects in educational assessments, it is necessary to have a cross-classified data structure. If one test administrator conducts the assessment in each school and does not conduct assessments in any other schools, it is not possible to distinguish test administrator effects from school effects – they are inseparably confounded. Therefore, a prerequisite for separating test administrator effects from school effects is having at least two test administrators administering the assessment to separate groups of students in each school, with the students being randomly assigned to these groups. There is even greater potential to disentangle test administrator and school effects when test administrators conduct assessments in different schools. We follow Lüdtke et al. (2007) statistical procedure. We estimate a cross-classified multi-level model with Markov Chain Monte Carlo (MCMC) estimators.
Conclusions, Expected Outcomes or FindingsOverall, like the original Lüdtke et al. (2007) paper we are replicating, the analysis found that a significant proportion of the variance in mathematics achievement and response behavior was at the school level, but much of this variance was explained by the type of school. In contrast, there were no differences in mathematics achievement or response behavior at the test administrator level. The results of the present study suggest that the procedures used to train test administrators and standardize test administration, which are largely the same procedures used in other large-scale assessment studies (e.g. PISA), were successful in ensuring that the tests were administered consistently to all student groups. This is a reassuring finding given the importance often placed on the outcomes of these kinds of assessments.
ReferencesBlossfeld, H.-P. & Roßbach, H.-G. (Eds.). (2019). Education as a lifelong process: The German National Educational Panel Study (NEPS). Edition ZfE (2nd ed.). Springer VS.
Lüdtke, O., Robitzsch, A., Trautwein, U., Kreuter, F., & Ihme, J. M. (2007). Are there test administrator effects in large-scale educational assessments? Using cross-classified multilevel analysis to probe for effects on mathematics achievement and sample attrition. Methodology, 3(4), 149–159. https://doi.org/10.1027/1614-2241.3.4.149
PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematic, Financial Literacy and Collaborative Problem Solving | en | OECD. (n.d.). Retrieved December 20, 2022, from https://www.oecd.org/education/pisa-2015-assessment-and-analytical-framework-9789264281820-en.htm
09. Assessment, Evaluation, Testing and Measurement
Paper
A Semiparametric Regression Model Applicable to Causal Inference in Various Educational Research Data: Extension of Identification via Heteroskedasticity
Akihiro Hashino
The University of Tokyo, Japan
Presenting Author: Hashino, Akihiro
Causal inference is a crucial topic in empirical education research, as well as in other social sciences (Murnane & Willet 2011). In particular, addressing endogeneity (selection bias caused by unobserved confounders) is arguably the most important issue. However, existing methods in applied research have significant limitations in terms of applicability and policy implications.
First, despite the development of causal inference methods such as panel fixed effects model, difference-in-differences, regression discontinuity design, and instrumental variable regression, the data available for their application are limited. Omitted variable bias is a common problem in observational data, and methods that address this issue have high data requirements. While large-scale survey data used in educational research, such as PISA, TALIS, can provide valuable information, the applicability of causal inference methods is limited or non-existent.
Second, even if these methods could be used, many applied studies are limited to those that assume a linear model or a dichotomous treatment variable. If the true relationship between the outcome variable and treatment variable is nonlinear, the policy implications of the analysis by existing methods are limited or misleading. This is especially relevant in the field of education, where there are many continuous or multi-value discrete treatment variables with nonlinear effects. Class size, school size, years of teacher experience, and teachers’ working hours are typical examples(Jerrim & Sims, 2021; Kraft & Papay, 2014). As the vast amount of past empirical research and accompanying discussion on the educational production function has shown, empirical findings on the nonlinear effects of class size and years of teacher experience will have direct implications for the financial resources available to implement educational policy.
The question is, how can we address challenges like these that we often face?
It is necessary to develop a realistic identification strategy that can address endogeneity and nonlinearity. In this paper, I extend a model-based approach that uses identification via conditional heteroskedasticity (Klein & Vella, 2010) to address the above limitations on causal inference in education research.
Methods using conditional heteroskedasticity are not commonly addressed in applied research, but have been discussed in theoretical literatures. This method models the structure of error terms of equations, and differs from those based on usual design-based identification strategies, but has the significant advantage of having relatively realistic side information requirements for identification. Additionally, this approach can be easily combined with various types of existing regression models, providing more options for empirical research using observational data. I extend the linear model with the novel identification strategy to a semiparametric model (partial linear model) within Bayesian framework and demonstrate the effectiveness of the proposed model using simulated and real data.
Methodology, Methods, Research Instruments or Sources Used We propose a model that extends the control function approach discussed in Klein & Vella (2010) to a semiparametric regression model within Bayesian framework. After discussing the model and its estimation using MCMC methods, we evaluate its performance by using simulated data. The simulation considers both cases where the effects of endogenous treatment variables are linear and nonlinear.
In addition to these simulated data, we also demonstrate the usefulness of the model in application to real data. Using real data from the Teaching and Learning International Survey (TALIS) 2018, an international survey on teachers’ working environments, we analyze the impact of teachers' long working hours on well-being, job satisfaction, and efficacy by the proposed model. Although TALIS provides useful information for the policy regarding teachers, it is difficult to apply the usual identification strategies of causal inference. Empirical research on teachers' subjective well-being and working environment has been conducted in several academic disciplines, including psychology, education, and epidemiology, but existing studies are highly flawed in terms of causal inferences. Specifically, workload is assumed to be one of the important factors when job satisfaction, sense of efficacy, and other well-being index are used as outcome variables, but the possibility that workload is an endogenous variable and correlated with unobserved confounding factors has been rarely considered. As to the nonlinearity, the question of what range of working hours has a greater impact on welfare has direct implications for the regulation of working hours and other issues. In particular, the detection of nonlinear effects of working hours (e.g., the impact increases rapidly above a certain threshold) is very important. Using the proposed model, we will analyze the effect of working hours on teachers' well-being, taking into account endogeneity and nonlinear effects.
Conclusions, Expected Outcomes or Findings Our proposed semiparametric model, which uses identification strategies based on conditional heteroskedasticity, offers several advantages over existing standardized causal inference methods. This approach is less limited in terms of the range of data it can be applied to, and has the ability to detect nonlinear effects of treatment variables. The results from both simulated and real data have demonstrated its ability to successfully contribute to research on policy-relevant questions. In particular, the analysis of TALIS data applying the proposed model revealed that existing studies underestimate the impact of teachers’ long working hours on well-being and overlook nonlinear effects.
Furthermore, the proposed model is more flexible due to the adoption of Bayesian modeling. An example is the random effects model (hierarchical model) used in the real data analysis in this paper.
In future research, we may consider relaxing various restrictions and extending the model to a heterogeneous treatment effects model, which would allow for the treatment effect to vary among individuals. In addition, applying this model to various other research topics is also an important avenue for future research.
ReferencesJerrim, J. and Sims, S. (2021). When is high workload bad for teacher wellbeing? Accounting for the non-linear contribution of specific teaching tasks, Teaching and Teacher Education,105:103395.
Klein, R., and Vella, F. (2010). Estimating a class of triangular simultaneous equations models without exclusion restrictions. Journal of Econometrics, 154(2), 154-164.
Kraft, M. A., & Papay, J. P. (2014). Can Professional Environments in Schools Promote Teacher Development? Explaining Heterogeneity in Returns to Teaching Experience. Educational Evaluation and Policy Analaysis, 36(4), 476-500.
Murnane, R. J., and Willett, J. B. (2011). Methods Matter: Improving causal inference in educational and social science research. Oxford; New York: Oxford University Press.
|
