Exploring and Visualizing School Achievement and School EffectsDaniel Anderson 
 Joseph Stevens04/16/181 / 26

Introduction

Much recent focus on open data in research generally
Open data tend to be rare in educational research
- Privacy concerns

Slides available at dandersondata.com/talks/ncme18

2 / 26

Introduction

Much recent focus on open data in research generally
Open data tend to be rare in educational research
- Privacy concerns

Slides available at dandersondata.com/talks/ncme18 NCLB Required Publicly Available Data

2 / 26

Introduction

Much recent focus on open data in research generally
Open data tend to be rare in educational research
- Privacy concerns

Slides available at dandersondata.com/talks/ncme18 NCLB Required Publicly Available Data

School-level data
Percent proficent in each of at least four proficiency categories
Disaggregated by student subgroups

2 / 26

Making ComparisonsWant
Compare differences in achievement between student groups(i.e., evaluate acheivement gaps)

Understand overall differences between groups

3 / 26

Making Comparisons

Want

Compare differences in achievement between student groups
- (i.e., evaluate acheivement gaps)
Understand overall differences between groups

Problem

Data have been "coarsened"
Percent proficient comparisons do not work well⁽¹⁾

(1) Ho, A. D. (2008). The problem with “proficiency”: Limitations of statistics and policy under No Child Left Behind. Educational Researcher, 37, 351-360.

3 / 26

Working with Coarsened Data

Slides available at dandersondata.com/talks/ncme18

Simulate some data from two distributions
- Group 1: $n = 200$ , $μ = 200$ , $σ = 10$
- Group 2: $n = 500$ , $μ = 210$ , $σ = 8$
Cut Scores: 190, 205, 215 (totally made up)

4 / 26

Reardon & Ho Method⁽²⁾

Calculate the empirical CDF of each distribution
Pair the ECDFs
Calculate the area under the paired curve
Transform it to an effect-size measure (standard deviation units)

(2) Reardon, S. F., & Ho, A. D. (2015). Practical issues in estimating achievement gaps from coarsened data. Journal of Educational and Behavioral Statistics, 40, 158–189.

5 / 26

Reardon & Ho Method⁽²⁾

Calculate the empirical CDF of each distribution
Pair the ECDFs
Calculate the area under the paired curve
Transform it to an effect-size measure (standard deviation units)

(2) Reardon, S. F., & Ho, A. D. (2015). Practical issues in estimating achievement gaps from coarsened data. Journal of Educational and Behavioral Statistics, 40, 158–189.

5 / 26

Reardon & Ho Method⁽²⁾

Calculate the empirical CDF of each distribution
Pair the ECDFs
Calculate the area under the paired curve
Transform it to an effect-size measure (standard deviation units)

(2) Reardon, S. F., & Ho, A. D. (2015). Practical issues in estimating achievement gaps from coarsened data. Journal of Educational and Behavioral Statistics, 40, 158–189.

5 / 26

Transformation to effect size

Slides available at dandersondata.com/talks/ncme18

$V = \sqrt{2} Φ^{- 1} (A U C)$

$V$ = Cohen's $d$ under the assumption of respective normality
In this case, Cohen's $d$ = -1.08 and $V$ = -1.07.

6 / 26

Transformation to effect size

Slides available at dandersondata.com/talks/ncme18

$V = \sqrt{2} Φ^{- 1} (A U C)$

$V$ = Cohen's $d$ under the assumption of respective normality
In this case, Cohen's $d$ = -1.08 and $V$ = -1.07.

Why does this all matter?

6 / 26

Transformation to effect size

Slides available at dandersondata.com/talks/ncme18

$V = \sqrt{2} Φ^{- 1} (A U C)$

$V$ = Cohen's $d$ under the assumption of respective normality
In this case, Cohen's $d$ = -1.08 and $V$ = -1.07.

Why does this all matter?

If we know the proportion of students scoring at multiple points in the scale, we can approximate the ECDF.
We can use the approximated ECDFs to estimate $V$

6 / 26

Simulated example again

Slides available at dandersondata.com/talks/ncme18

proportions

7 / 26

Purpose8 / 26

PurposeEvaluate VV with empirical data
8 / 26

Purpose

Evaluate $V$ with empirical data
- Compare estimates from full data and manually coarsened data. Compare results to Cohen'd $d$ estimated with full data.

8 / 26

Purpose

Evaluate $V$ with empirical data
- Compare estimates from full data and manually coarsened data. Compare results to Cohen'd $d$ estimated with full data.
Apply these methods to publicly available data to investigate between-school differences in achievement gaps

8 / 26

Purpose

Evaluate $V$ with empirical data
- Compare estimates from full data and manually coarsened data. Compare results to Cohen'd $d$ estimated with full data.
Apply these methods to publicly available data to investigate between-school differences in achievement gaps
- Specifically using geo-spatial mapping, overlaying data about the surrounding area

8 / 26

Data Sources9 / 26

Student-level

Slides available at dandersondata.com/talks/ncme18

National Center on Assessment and Accountability in Special Education (NCAASE)
- Large inter-state collaborative
Secure data
Data for this study included all student records in reading and mathematics across Grades 3-8 in the 2012-13 school years
- ~37,000 students
- ~62% White, 25% Hispanic/Latino, 5% Multiethnic

10 / 26

Publicly available data

Slides available at dandersondata.com/talks/ncme18

Oregon and California

Percentage scoring in each proficiency category by school
California

Available from two different websites (see here and here)

Oregon

Available from statewide website (see here)

11 / 26

Census Data

Slides available at dandersondata.com/talks/ncme18

Geographic coordinates of census tracts for Alameda county
- Areas with between approximately 1,200 to 8,000 people, with an optimum size of 4,000 people
American Community Survey: 2016
- Median Housing Cost
- Number of individuals identifying as Black
- Number of individuals with income to poverty ration > 2.0

12 / 26

Procedures

Slides available at dandersondata.com/talks/ncme18

Comparing Effect Sizes

Estimate V by school in Oregon
- Full data and manually coarsened data
Estimate Cohen's $d$ by school in Oregon
Compare all estimates

13 / 26

Procedures

Slides available at dandersondata.com/talks/ncme18

Comparing Effect Sizes

Estimate V by school in Oregon
- Full data and manually coarsened data
Estimate Cohen's $d$ by school in Oregon
Compare all estimates

$V_{c}$ = $V$ continuous data estimate

$V_{d}$ = $V$ discrete data estimate

13 / 26

Evaluating Achievement Gaps

Slides available at dandersondata.com/talks/ncme18

Use publicly available data
Evaluate distribution of school-level achievement gaps
- Black-White achievement gap in California
- Hispanic-White achievement gap in Oregon
Follow-up with geographic investigations of school-level achievment gaps for Alameda county
- Overlay census tract information to visually examine geo-spatial relations

14 / 26

Results15 / 26

Comparing $V_{c}$ and $d$

Slides available at dandersondata.com/talks/ncme18

Both continuous: $r = 0.87 / 0.86$ .

16 / 26

Slides available at dandersondata.com/talks/ncme18

Comparing $V_{d}$ and $d$

$r = 0.73 / 0.72$

Comparing $V_{c}$ and $V_{d}$

$r = 0.83 / 0.89$

17 / 26

Distribution of differences: $V_{c}$ and $d$

Slides available at dandersondata.com/talks/ncme18

$μ = - 0.12, σ = 0.15$

$μ = - 0.16, σ = 0.16$

18 / 26

Slides available at dandersondata.com/talks/ncme18

Comparing $V_{d}$ and $d$

$μ = - 0.10 / - 0.13, σ = 0.21 / 0.23$

Comparing $V_{d}$ and $V_{c}$

$μ = 0.02 / 0.03, σ = 0.17 / 0.15$

19 / 26

Substantive Investigations20 / 26

Achievement Gap Distributions

Slides available at dandersondata.com/talks/ncme18

Reminder: School-level Distributions

21 / 26

Alameda County

Median Housing Cost

Slides available at dandersondata.com/talks/ncme18

22 / 26

Alameda County

$n$ Identifying as Black

Slides available at dandersondata.com/talks/ncme18

23 / 26

Alameda County

$n$ Income/Poverty Ratio > 2.0

Slides available at dandersondata.com/talks/ncme18

24 / 26

Conclusions

Effect size appeared well estimated from coarsened data
- $V$ was similar to Cohen's $d$ with these, empirical data
The vast majority, but not all, schools had sizeable estimated achievement gaps
Clear geographic clustering of achievement gaps was evident
- Did Not appear to depend on the Census data investigated here

↑, ←, Pg Up, k	Go to previous slide
↓, →, Pg Dn, Space, j	Go to next slide
Home	Go to first slide
End	Go to last slide
Number + Return	Go to specific slide
b / m / f	Toggle blackout / mirrored / fullscreen mode
c	Clone slideshow
p	Toggle presenter mode
t	Restart the presentation timer
?, h	Toggle this help

Exploring and Visualizing School Achievement and School Effects

Daniel Anderson Joseph Stevens

04/16/18

Introduction

Introduction

Introduction

Making Comparisons

Want

Making Comparisons

Want

Problem

Working with Coarsened Data

Reardon & Ho Method(2)

Reardon & Ho Method(2)

Reardon & Ho Method(2)

Transformation to effect size

Transformation to effect size

Why does this all matter?

Transformation to effect size

Why does this all matter?

Simulated example again

proportions

Purpose

Purpose

Purpose

Purpose

Purpose

Data Sources

Student-level

Publicly available data

California

Oregon

Census Data

Procedures

Comparing Effect Sizes

Procedures

Comparing Effect Sizes

Evaluating Achievement Gaps

Results

Comparing VcVc and dd

Comparing VdVd and dd

Comparing VcVc and VdVd

Distribution of differences: VcVc and dd

Comparing VdVd and dd

Comparing VdVd and VcVc

Substantive Investigations

Achievement Gap Distributions

Alameda County

Median Housing Cost

Alameda County

nn Identifying as Black

Alameda County

nn Income/Poverty Ratio > 2.0

Conclusions

Thanks so much!

Introduction

Help

Daniel Anderson
Joseph Stevens

Reardon & Ho Method⁽²⁾

Reardon & Ho Method⁽²⁾

Reardon & Ho Method⁽²⁾

Comparing $V_{c}$ and $d$

Comparing $V_{d}$ and $d$

Comparing $V_{c}$ and $V_{d}$

Distribution of differences: $V_{c}$ and $d$

Comparing $V_{d}$ and $d$

Comparing $V_{d}$ and $V_{c}$

$n$ Identifying as Black

$n$ Income/Poverty Ratio > 2.0