Variance in Student Growth, Intervention Effects, and Achievement GapsJob talk for Virginia TechDaniel AndersonMarch 25, 20191 / 33

My background

Behavioral Research and Teaching

Research Assistant to Research Associate to Research Assistant Professor
Grant funded research shop at UO that mostly focuses on measurement
- Curriculum Based Measurement (e.g., easyCBM)
  - Project Manager, 4-year IES award on the development of a middle school math CBM
- Statewide Alternate Assessment
  - Lead psychometrician since 2011
  - Lead development of a new vertical scale in 2015

2 / 33

My background

Project NCAASE

National Center on Assessment and Accountability in Special Education

Large inter-state collaborative focused on the measurement of schools
Lead numerous studies on between-school differences in achievement (and the implications for accountability models)
First foray into very large scale data

3 / 33

The focus of my talk todayThree stories of scholarship4 / 33

The focus of my talk todayThree stories of scholarshipStudy 1: Variance in students' within-year growthAverage differences between teachers and schools
Variance in summer lags (out-of-school opportunities)
4 / 33

The focus of my talk todayThree stories of scholarshipStudy 1: Variance in students' within-year growthAverage differences between teachers and schools
Variance in summer lags (out-of-school opportunities)
Study 2: Variance in intervention effectsRegression Discontinuity Design
Cluster-level design; treatment delivered at the school level
Evaluations of functional form are critical
4 / 33

The focus of my talk todayThree stories of scholarshipStudy 1: Variance in students' within-year growthAverage differences between teachers and schools
Variance in summer lags (out-of-school opportunities)
Study 2: Variance in intervention effectsRegression Discontinuity Design
Cluster-level design; treatment delivered at the school level
Evaluations of functional form are critical
In-Progress Research: Computational methodsVariance in achievement gaps
Open data, open science, and reproducible research
4 / 33

Study 1: Variance in students' within-year growth

Exploring Teacher and School Variance in Students’ Within-Year Reading and Mathematics Growth.

Anderson, D. (conditional acceptance). Exploring Teacher and School Variance in Students’ Within-Year Reading and Mathematics Growth. School Effectiveness and School Improvement

5 / 33

The fundamental questionWe know there is considerable heterogeneity in the rate at which students 
learn.
6 / 33

The fundamental question

We know there is considerable heterogeneity in the rate at which students learn.

Why?

6 / 33

The fundamental question

We know there is considerable heterogeneity in the rate at which students learn.

Why?

Lots of evidence that teachers contribute to learning

6 / 33

The fundamental question

We know there is considerable heterogeneity in the rate at which students learn.

Why?

Lots of evidence that teachers contribute to learning
Lots of evidence that schools contribute to learning

6 / 33

The fundamental question

We know there is considerable heterogeneity in the rate at which students learn.

Why?

Lots of evidence that teachers contribute to learning
Lots of evidence that schools contribute to learning

How much does student learning depend on the set of teachers they are "assigned" to, versus schools?

6 / 33

The fundamental question

We know there is considerable heterogeneity in the rate at which students learn.

Why?

Lots of evidence that teachers contribute to learning
Lots of evidence that schools contribute to learning

How much does student learning depend on the set of teachers they are "assigned" to, versus schools?

Secondary questions

Is evidence of teacher "sorting" between schools present?
How variable is the "summer slide"?

6 / 33

Data

3 Cohorts of students in one school district in the Southwestern United States, progressing from Grades 3-5
- 2007-08 to 2009-10, 2008-09 to 2010-11, or 2009-10 to 2011-12

7 / 33

Data

3 Cohorts of students in one school district in the Southwestern United States, progressing from Grades 3-5
- 2007-08 to 2009-10, 2008-09 to 2010-11, or 2009-10 to 2011-12
Three time points within each year (collected fall, winter, spring)

7 / 33

Data

3 Cohorts of students in one school district in the Southwestern United States, progressing from Grades 3-5
- 2007-08 to 2009-10, 2008-09 to 2010-11, or 2009-10 to 2011-12
Three time points within each year (collected fall, winter, spring)
Variance components estimated for teachers in each grade, necessitating the removal of any student with incomplete teacher records.
- 2,909 students out 5,311 had complete teacher records

7 / 33

Data

3 Cohorts of students in one school district in the Southwestern United States, progressing from Grades 3-5
- 2007-08 to 2009-10, 2008-09 to 2010-11, or 2009-10 to 2011-12
Three time points within each year (collected fall, winter, spring)
Variance components estimated for teachers in each grade, necessitating the removal of any student with incomplete teacher records.
- 2,909 students out 5,311 had complete teacher records
Between 106-119 teachers, depending on the grade, nested in 18 schools

7 / 33

Data

3 Cohorts of students in one school district in the Southwestern United States, progressing from Grades 3-5
- 2007-08 to 2009-10, 2008-09 to 2010-11, or 2009-10 to 2011-12
Three time points within each year (collected fall, winter, spring)
Variance components estimated for teachers in each grade, necessitating the removal of any student with incomplete teacher records.
- 2,909 students out 5,311 had complete teacher records
Between 106-119 teachers, depending on the grade, nested in 18 schools
Approximately 54% of students were coded as Hispanic, 24% White, and 74% were eligible for free or reduced price lunch

7 / 33

Measures

Measures of Academic Progress, developed by the Northwest Evaluation Association (NWEA)
Computer adaptive
- High conditional reliability across a broad ability range
Vertical scale
- Growth within and between grades directly comparable

8 / 33

Piecewise growth model

Slopes

$g 3_{s l p} = 0, 1, 2 | 2, 2, 2 | 2, 2, 2 g 4_{s l p} = 0, 0, 0 | 0, 1, 2 | 2, 2, 2 g 5_{s l p} = 0, 0, 0 | 0, 0, 0 | 0, 1, 2$

9 / 33

Piecewise growth model

Slopes

$g 3_{s l p} = 0, 1, 2 | 2, 2, 2 | 2, 2, 2 g 4_{s l p} = 0, 0, 0 | 0, 1, 2 | 2, 2, 2 g 5_{s l p} = 0, 0, 0 | 0, 0, 0 | 0, 1, 2$

Grade 4 & 5 Intercepts

$g 4 = 0, 0, 0 | 1, 1, 1 | 1, 1, 1 g 5 = 0, 0, 0 | 0, 0, 0 | 1, 1, 1$

9 / 33

Piecewise growth model

Slopes

$g 3_{s l p} = 0, 1, 2 | 2, 2, 2 | 2, 2, 2 g 4_{s l p} = 0, 0, 0 | 0, 1, 2 | 2, 2, 2 g 5_{s l p} = 0, 0, 0 | 0, 0, 0 | 0, 1, 2$

Grade 4 & 5 Intercepts

$g 4 = 0, 0, 0 | 1, 1, 1 | 1, 1, 1 g 5 = 0, 0, 0 | 0, 0, 0 | 1, 1, 1$

Fixed effects

$y_{t i j k} = β_{0} + β_{1} (g 3_{s l p}) + β_{2} (g 4) + β_{3} (g 4_{s l p}) + β_{4} (g 5) + β_{5} (g 5_{s l p})$

9 / 33

Random effects

Student level (nested)

$(\begin{matrix} r_{0_{i j k}} + r_{1_{i j k}} (g 3_{s l p}) + \\ r_{2_{i j k}} (g 4) + r_{3_{i j k}} (g 4_{s l p}) + \\ r_{4_{i j k}} (g 5) + r_{5_{i j k}} (g 5_{s l p}) \end{matrix})$

Teacher level (crossed)

$(\begin{matrix} u_{0_{j (3) k}}^{3} + u_{1_{j (3) k}}^{3} (g 3_{s l p}) \end{matrix})$

$(\begin{matrix} u_{2_{j (4) k}}^{4} + u_{3_{j (4) k}}^{4} (g 4_{s l p}) \end{matrix})$

$(\begin{matrix} u_{4_{j (5) k}}^{4} + u_{5_{j (5) k}}^{4} (g 5_{s l p}) \end{matrix})$

10 / 33

Random effects

School level (nested)

$(\begin{matrix} v_{0_{k}} + v_{1_{k}} (g 3_{s l p}) + \\ v_{2_{k}} (g 4) + v_{3_{k}} (g 4_{s l p}) + \\ v_{4_{k}} (g 5) + v_{5_{k}} (g 5_{s l p}) \end{matrix})$

11 / 33

Random effects

School level (nested)

$(\begin{matrix} v_{0_{k}} + v_{1_{k}} (g 3_{s l p}) + \\ v_{2_{k}} (g 4) + v_{3_{k}} (g 4_{s l p}) + \\ v_{4_{k}} (g 5) + v_{5_{k}} (g 5_{s l p}) \end{matrix})$

Residual error

$e$

11 / 33

Random effects

School level (nested)

$(\begin{matrix} v_{0_{k}} + v_{1_{k}} (g 3_{s l p}) + \\ v_{2_{k}} (g 4) + v_{3_{k}} (g 4_{s l p}) + \\ v_{4_{k}} (g 5) + v_{5_{k}} (g 5_{s l p}) \end{matrix})$

Residual error

$e$

All random effects were assumed to follow a multivariate normal distribution and were estimated with an unstructured variance-covariance matrix

For reading, the variance-covariance matrix at the school level was moderately simplified to help the model converge. Specifically, the school-level intercept and all slope terms were allowed to correlate, but the correlation between these terms and the summer drops were fixed at zero.

11 / 33

Results12 / 33

13 / 33

14 / 33

15 / 33

ConclusionsConsiderable variability in students' growth was between both teachers
and schools
16 / 33

Conclusions

Considerable variability in students' growth was between both teachers and schools
Teacher/School effects may compound, or compensate

16 / 33

Conclusions

Considerable variability in students' growth was between both teachers and schools
Teacher/School effects may compound, or compensate
Generally a mix of high/low growth teachers within each school

16 / 33

Conclusions

Considerable variability in students' growth was between both teachers and schools
Teacher/School effects may compound, or compensate
Generally a mix of high/low growth teachers within each school
Several limitations should be kept in mind
- Small number of schools for the complexity of the model
- Students had to have at least one data point within each school year to be included (mobility is linked with achievement and SES)

16 / 33

Study 2: Evaluating School-Provided Interventions

Examining the Impact and School-Level Predictors of Impact Variability of an 8th Grade Reading Intervention on At-Risk Students’ Reading Achievement

Fien, H., Anderson, D., Nelson, N. J., Baker, S. K., & Kennedy, P. (2018). Examining the Impact and School-Level Predictors of Impact Variability of an 8th Grade Reading Intervention on At-Risk Students’ Reading Achievement. Learning Disabilities Research & Practice, 33, 37-50. doi: 10.1111/ldrp.12161

17 / 33

Background

Middle School Intervention Project

Oregon Department of Education launched Effective Behavioral and Instructional Support System initiative
- MSIP aimed at evaluating its effect
Multi-tiered systems of support

18 / 33

Background

Middle School Intervention Project

Oregon Department of Education launched Effective Behavioral and Instructional Support System initiative
- MSIP aimed at evaluating its effect
Multi-tiered systems of support

Do district-adopted and -implemented interventions have their desired effect on student reading outcomes?

18 / 33

Design

Regression discontinuity (RD)

Students scoring below a school-defined threshold on a reading composite measure were targeted for intervention
Fuzzy design by design
- Up to 5% of students could be exempted on either side of the cut

19 / 33

Design

Regression discontinuity (RD)

Students scoring below a school-defined threshold on a reading composite measure were targeted for intervention
Fuzzy design by design
- Up to 5% of students could be exempted on either side of the cut

Note: The paper had some planned follow-up post-hoc analyses of between school variability, which I will not discuss in depth here

19 / 33

Impact Model

Multilevel Generalized Additive Model

Level 1

$y_{i j} = β_{0 j} + β_{1 j} (L E C_{i j}) + s_{1} (L E C \times a s s i g n V a r_{i j}) + s_{2} (A C \times a s s i g n V a r_{i j}) + e_{i j}$

20 / 33

Impact Model

Multilevel Generalized Additive Model

Level 1

$y_{i j} = β_{0 j} + β_{1 j} (L E C_{i j}) + s_{1} (L E C \times a s s i g n V a r_{i j}) + s_{2} (A C \times a s s i g n V a r_{i j}) + e_{i j}$

Level 2

$β_{0 j} = γ_{00} + γ_{01} (c u t_{j}) + u_{0 j} β_{1 j} = γ_{10} + u_{1 j}$

20 / 33

Impact Model

Multilevel Generalized Additive Model

Level 1

$y_{i j} = β_{0 j} + β_{1 j} (L E C_{i j}) + s_{1} (L E C \times a s s i g n V a r_{i j}) + s_{2} (A C \times a s s i g n V a r_{i j}) + e_{i j}$

Level 2

$β_{0 j} = γ_{00} + γ_{01} (c u t_{j}) + u_{0 j} β_{1 j} = γ_{10} + u_{1 j}$

$s_{p} =$ thin-plate spline smooths
- Degree of smoothing determined via generalized cross-validation

20 / 33

Impact Model

Multilevel Generalized Additive Model

Level 1

$y_{i j} = β_{0 j} + β_{1 j} (L E C_{i j}) + s_{1} (L E C \times a s s i g n V a r_{i j}) + s_{2} (A C \times a s s i g n V a r_{i j}) + e_{i j}$

Level 2

$β_{0 j} = γ_{00} + γ_{01} (c u t_{j}) + u_{0 j} β_{1 j} = γ_{10} + u_{1 j}$

$s_{p} =$ thin-plate spline smooths
- Degree of smoothing determined via generalized cross-validation
$γ_{10} =$ average treatment effect (assuming a sharp design)

20 / 33

Impact Model

Multilevel Generalized Additive Model

Level 1

$y_{i j} = β_{0 j} + β_{1 j} (L E C_{i j}) + s_{1} (L E C \times a s s i g n V a r_{i j}) + s_{2} (A C \times a s s i g n V a r_{i j}) + e_{i j}$

Level 2

$β_{0 j} = γ_{00} + γ_{01} (c u t_{j}) + u_{0 j} β_{1 j} = γ_{10} + u_{1 j}$

$s_{p} =$ thin-plate spline smooths
- Degree of smoothing determined via generalized cross-validation
$γ_{10} =$ average treatment effect (assuming a sharp design)
$u_{1 j} =$ between school variation in the average treatment effect

20 / 33

Accounting for fuzziness

9% crossovers, 18% no-shows

Two step process to estimate the fuzzy RD gap

21 / 33

Accounting for fuzziness

9% crossovers, 18% no-shows

Two step process to estimate the fuzzy RD gap

Model probability gap (of treatment receipt)
- Models equivalent to previous slide, but using multilevel logistic regression

21 / 33

Accounting for fuzziness

9% crossovers, 18% no-shows

Two step process to estimate the fuzzy RD gap

Model probability gap (of treatment receipt)
- Models equivalent to previous slide, but using multilevel logistic regression
Divide sharp RD impact estimate, $γ_{10}$ , by estimated probability gap

(standard errors can be similarly transformed)

21 / 33

RD on State Test

$γ_{10} = - 0.06$ ; $γ_{10_{f}} = - 0.12, S E_{f} = 0.72, z_{f} = - 0.16, p_{f} = 0.87$

22 / 33

By school

23 / 33

Variability

24 / 33

Conclusions

No significant effect of intervention found
Small variability in the null effect between schools

25 / 33

Conclusions

No significant effect of intervention found
Small variability in the null effect between schools
Three possible sources of null effect (Seftor, 2017)
- Methodological failure
- Implementation failure
- Theory failure

25 / 33

Quickly:In-Progress Research: Computational methodsLinking large-scale data sourcesMachine learning approaches 

Open data, open science, and reproducible research
26 / 33

Open science

Much recent focus on open data in research generally
Open data tend to be rare in educational research
- Privacy concerns

27 / 33

Open science

Much recent focus on open data in research generally
Open data tend to be rare in educational research
- Privacy concerns

NCLB Required Publicly Available Data

27 / 33

Open science

Much recent focus on open data in research generally
Open data tend to be rare in educational research
- Privacy concerns

NCLB Required Publicly Available Data

School-level data
Percent proficient in each of at least four proficiency categories
Disaggregated by student subgroups

27 / 33

Reardon & Ho methodCalculate the empirical CDF of each distribution
Pair the ECDFs
Calculate the area under the paired curve
Transform it to an effect-size measure (standard deviation units)
28 / 33

Reardon & Ho method

Calculate the empirical CDF of each distribution
Pair the ECDFs
Calculate the area under the paired curve
Transform it to an effect-size measure (standard deviation units)

28 / 33

Reardon & Ho method

Calculate the empirical CDF of each distribution
Pair the ECDFs
Calculate the area under the paired curve
Transform it to an effect-size measure (standard deviation units)

28 / 33

Transformation to effect size

$V = \sqrt{2} Φ^{- 1} (A U C)$

Why does this all matter?

29 / 33

Achievement gap distributions

Reminder: School-level Distributions

30 / 33

Alameda countynn Income/Poverty Ratio > 2.031 / 33

Wrapping up

Geographic achievement gap variance work presented here was mostly exploratory/visual
- Can we actually model the data with machine learning methods?
IES grant application currently (still) under review under the Statistical and Research Methodology Early Career RFA

32 / 33

Wrapping up

Geographic achievement gap variance work presented here was mostly exploratory/visual
- Can we actually model the data with machine learning methods?
IES grant application currently (still) under review under the Statistical and Research Methodology Early Career RFA

Reproducibility & transparency

I'm leading a training on reproducible research at AERA this year
Embedded within all my teaching
Deeply committed to open and transparent research

32 / 33

Thanks!

Questions?

Slides available at
http://www.datalorax.com/talks/vatech/

33 / 33

↑, ←, Pg Up, k	Go to previous slide
↓, →, Pg Dn, Space, j	Go to next slide
Home	Go to first slide
End	Go to last slide
Number + Return	Go to specific slide
b / m / f	Toggle blackout / mirrored / fullscreen mode
c	Clone slideshow
p	Toggle presenter mode
t	Restart the presentation timer
?, h	Toggle this help