Conference Agenda

Social media has become core infrastructure for accessing information, communicating, and participating in civic life. European policymakers have called for stronger media literacy to counter disinformation and support informed decision-making. Frameworks like DigComp 2.2 describe what competence looks like. But these frameworks cannot function as psychometrically validated measurement instruments—they describe competence but cannot locate individuals on a calibrated scale (Vuorikari et al., 2022). The lack of validated measurement creates difficulties. Evidence-informed curriculum development, assessment design, and cross-context proficiency comparison all require instruments that can establish where individuals stand.

Substantial work on social media literacy (SML) has accumulated, yet the field remains conceptually fragmented (Cho et al., 2024; Valle et al., 2025). Most instruments rely on self-report measures. These capture what users perceive they can do, rather than what they actually demonstrate (Tandoc et al., 2021). Some newer performance measures move beyond self-report using IRT-based approaches. This is progress. But these tools typically target specific populations and do not address standards-referenced interpretation across age groups (Drake et al., 2023). Making defensible proficiency judgements across educational systems is difficult.

We conducted a systematic scoping review and thematic synthesis of SML literature from 2015 to 2025. From this work, we propose a measurement framework applicable across ages and contexts. The framework operationalises SML through three components: Platform Awareness, Critical Evaluation, and Responsible Participation. These consolidate recurring domains in the literature. We describe proficiency using four levels. The four-level structure reflects theoretically distinct transitions in metacognitive regulation. L1 (Unaware) represents minimal recognition. L2 (Retrospective) involves post-hoc awareness—individuals identify relevant influences after engagement (Flavell, 1979). L3 (Real-time) marks a shift to in-process monitoring and deliberate adjustment during engagement (Schraw & Dennison, 1994). L4 (Habitual–Reflexive) describes internalised monitoring across contexts, including low-stakes use, where strategies transfer beyond single situations (Mezirow, 1997). Self-awareness runs through all levels as a difficulty factor rather than a separate component.

Three questions guide the study: What are the core components and indicators of SML for standards-referenced measurement? How do we operationalise and design a rubric-based framework for Rasch calibration? What validity evidence is needed to support the intended uses of the measures?

Our goal is translating competence language into a calibratable rubric that supports defensible proficiency judgements. The conference paper presents the three-component construct map, candidate indicators, draft performance level descriptors, and the validation workflow. We plan to validate the framework across cultural and educational contexts. Cross-cultural comparability will be addressed in later stages. The study contributes a validation logic for standards-referenced interpretation, supporting European efforts to assess SML comparably across populations and systems.

The Successful School Leadership framework (SSL) is a model of best practices for school leadership, as validated through multiple studies over a twenty-year period (Leithwood et al., 2010; Leithwood & Jantzi, 2005; Sun & Leithwood, 2015). The Successful School Leadership survey was created as a measure of perceived school leadership quality under this framework. This survey is a 22-item Likert scale with 5-point response options ranging from “not at all confident” to “very confident”. This scale intends to measure four domains of leadership: (1) setting directions, (2) building relationships and developing people, (3) (re)designing the organization to support desired practices, and (4) improving the instructional program.

Leithwood et al. (2023) provided strong validity evidence for this scale, reporting high reliability, measurement invariance for various person characteristics, and good person and item fit. This study also confirmed the four-factor structure and showed predictive validity evidence of high leadership scores for student achievement, even after controlling for relevant confounding variables.

Leithwood et al. (2023) showed evidence of measurement invariance for gender and education level by using the Many Facet Rasch Model (MFRM) to measure the effect of these covariates at the group level, confirming that gender and education were not predictive of scale scores. However, this method of investigating measurement invariance does not consider differences in subgroup effects at the item level. This difference in item parameters across different groups of people when holding person abilities constant is known as differential item functioning (DIF) and is a common threat to measurement invariance and fairness in scale validation (Ackerman & Evans, 1994; Roussos & Stout, 1996).

This study investigates the presence of DIF in the SSLS items for three covariates: gender, experience, and education level. We use traditional DIF detection methods evaluate these three covariates separately. However, recent interest in intersectional approaches to quantitative research (e.g., Albano et al., 2024; Bauer et al., 2021) underscores the importance of not just treating covariates as independent factors, but as more complex interdependent person characteristics. In response to this recent rise in interest surrounding intersectional approaches to quantitative research, we also use some newer, more exploratory DIF detection methods to investigate intersectional DIF in the SSLS items.

The data used in this study were collected as part of the research conducted in Leithwood et al. (2023). All participants in this survey were teachers in two different neighboring school districts in the southeastern United States. After filtering participants based on missing data and willingness to participate, this dataset had a total of 136 respondents. Most of the sample population (83%) identified as women, and the rest of the population identified as men. 58% of the sample population held graduate degrees, and the rest held bachelor’s degrees. 38 participants had 1-3 years of experience, 29 participants had 4-10 years of experience, 20 had 11-15 years of experience, and 48 had 15 or more years of experience.