Loading [MathJax]/jax/output/CommonHTML/fonts/TeX/fontdata.js
+ - 0:00:00
Notes for current slide
Notes for next slide

Exploring and Visualizing School Achievement and School Effects

Daniel Anderson
Joseph Stevens

04/16/18

1 / 26

Introduction

  • Much recent focus on open data in research generally
  • Open data tend to be rare in educational research
    • Privacy concerns

Slides available at dandersondata.com/talks/ncme18

2 / 26

Introduction

  • Much recent focus on open data in research generally
  • Open data tend to be rare in educational research
    • Privacy concerns

Slides available at dandersondata.com/talks/ncme18 NCLB Required Publicly Available Data

2 / 26

Introduction

  • Much recent focus on open data in research generally
  • Open data tend to be rare in educational research
    • Privacy concerns

Slides available at dandersondata.com/talks/ncme18 NCLB Required Publicly Available Data

  • School-level data
  • Percent proficent in each of at least four proficiency categories
  • Disaggregated by student subgroups
2 / 26

Making Comparisons

Want

  • Compare differences in achievement between student groups
    • (i.e., evaluate acheivement gaps)
  • Understand overall differences between groups
3 / 26

Making Comparisons

Want

  • Compare differences in achievement between student groups
    • (i.e., evaluate acheivement gaps)
  • Understand overall differences between groups

Problem

  • Data have been "coarsened"
  • Percent proficient comparisons do not work well(1)

(1) Ho, A. D. (2008). The problem with “proficiency”: Limitations of statistics and policy under No Child Left Behind. Educational Researcher, 37, 351-360.

3 / 26

Working with Coarsened Data

Slides available at dandersondata.com/talks/ncme18

  • Simulate some data from two distributions

    • Group 1: n=200, μ=200, σ=10
    • Group 2: n=500, μ=210, σ=8

  • Cut Scores: 190, 205, 215 (totally made up)

4 / 26

Reardon & Ho Method(2)

  • Calculate the empirical CDF of each distribution
  • Pair the ECDFs
  • Calculate the area under the paired curve
  • Transform it to an effect-size measure (standard deviation units)

(2) Reardon, S. F., & Ho, A. D. (2015). Practical issues in estimating achievement gaps from coarsened data. Journal of Educational and Behavioral Statistics, 40, 158–189.

5 / 26

Reardon & Ho Method(2)

  • Calculate the empirical CDF of each distribution
  • Pair the ECDFs
  • Calculate the area under the paired curve
  • Transform it to an effect-size measure (standard deviation units)

(2) Reardon, S. F., & Ho, A. D. (2015). Practical issues in estimating achievement gaps from coarsened data. Journal of Educational and Behavioral Statistics, 40, 158–189.

5 / 26

Reardon & Ho Method(2)

  • Calculate the empirical CDF of each distribution
  • Pair the ECDFs
  • Calculate the area under the paired curve
  • Transform it to an effect-size measure (standard deviation units)

(2) Reardon, S. F., & Ho, A. D. (2015). Practical issues in estimating achievement gaps from coarsened data. Journal of Educational and Behavioral Statistics, 40, 158–189.

5 / 26

Transformation to effect size

Slides available at dandersondata.com/talks/ncme18

V=√2Φ−1(AUC)

  • V = Cohen's d under the assumption of respective normality
  • In this case, Cohen's d = -1.08 and V = -1.07.
6 / 26

Transformation to effect size

Slides available at dandersondata.com/talks/ncme18

V=√2Φ−1(AUC)

  • V = Cohen's d under the assumption of respective normality
  • In this case, Cohen's d = -1.08 and V = -1.07.

Why does this all matter?

6 / 26

Transformation to effect size

Slides available at dandersondata.com/talks/ncme18

V=√2Φ−1(AUC)

  • V = Cohen's d under the assumption of respective normality
  • In this case, Cohen's d = -1.08 and V = -1.07.

Why does this all matter?

  • If we know the proportion of students scoring at multiple points in the scale, we can approximate the ECDF.
  • We can use the approximated ECDFs to estimate V
6 / 26

Simulated example again

Slides available at dandersondata.com/talks/ncme18

proportions

7 / 26

Purpose

8 / 26

Purpose

  • Evaluate V with empirical data
8 / 26

Purpose

  • Evaluate V with empirical data

    • Compare estimates from full data and manually coarsened data. Compare results to Cohen'd d estimated with full data.
8 / 26

Purpose

  • Evaluate V with empirical data

    • Compare estimates from full data and manually coarsened data. Compare results to Cohen'd d estimated with full data.

  • Apply these methods to publicly available data to investigate between-school differences in achievement gaps

8 / 26

Purpose

  • Evaluate V with empirical data

    • Compare estimates from full data and manually coarsened data. Compare results to Cohen'd d estimated with full data.

  • Apply these methods to publicly available data to investigate between-school differences in achievement gaps

    • Specifically using geo-spatial mapping, overlaying data about the surrounding area
8 / 26

Data Sources

9 / 26

Student-level

Slides available at dandersondata.com/talks/ncme18

  • National Center on Assessment and Accountability in Special Education (NCAASE)
    • Large inter-state collaborative
  • Secure data
  • Data for this study included all student records in reading and mathematics across Grades 3-8 in the 2012-13 school years
    • ~37,000 students
    • ~62% White, 25% Hispanic/Latino, 5% Multiethnic
10 / 26

Publicly available data

Slides available at dandersondata.com/talks/ncme18

Oregon and California

  • Percentage scoring in each proficiency category by school

    California

    Available from two different websites (see here and here)

    Oregon

    Available from statewide website (see here)

11 / 26

Census Data

Slides available at dandersondata.com/talks/ncme18

  • Geographic coordinates of census tracts for Alameda county
    • Areas with between approximately 1,200 to 8,000 people, with an optimum size of 4,000 people
  • American Community Survey: 2016
    • Median Housing Cost
    • Number of individuals identifying as Black
    • Number of individuals with income to poverty ration > 2.0
12 / 26

Procedures

Slides available at dandersondata.com/talks/ncme18

Comparing Effect Sizes

  • Estimate V by school in Oregon
    • Full data and manually coarsened data
  • Estimate Cohen's d by school in Oregon
  • Compare all estimates
13 / 26

Procedures

Slides available at dandersondata.com/talks/ncme18

Comparing Effect Sizes

  • Estimate V by school in Oregon
    • Full data and manually coarsened data
  • Estimate Cohen's d by school in Oregon
  • Compare all estimates

Vc = V continuous data estimate

Vd = V discrete data estimate

13 / 26

Evaluating Achievement Gaps

Slides available at dandersondata.com/talks/ncme18

  • Use publicly available data
  • Evaluate distribution of school-level achievement gaps
    • Black-White achievement gap in California
    • Hispanic-White achievement gap in Oregon
  • Follow-up with geographic investigations of school-level achievment gaps for Alameda county
    • Overlay census tract information to visually examine geo-spatial relations
14 / 26

Results

15 / 26

Comparing Vc and d

Slides available at dandersondata.com/talks/ncme18

Both continuous: r=0.87/0.86.

16 / 26

Slides available at dandersondata.com/talks/ncme18

Comparing Vd and d

r=0.73/0.72

Comparing Vc and Vd

r=0.83/0.89

17 / 26

Distribution of differences: Vc and d

Slides available at dandersondata.com/talks/ncme18

μ=−0.12,σ=0.15

μ=−0.16,σ=0.16

18 / 26

Slides available at dandersondata.com/talks/ncme18

Comparing Vd and d

μ=−0.10/−0.13,σ=0.21/0.23

Comparing Vd and Vc

μ=0.02/0.03,σ=0.17/0.15

19 / 26

Substantive Investigations

20 / 26

Achievement Gap Distributions

Slides available at dandersondata.com/talks/ncme18

Reminder: School-level Distributions

21 / 26

Alameda County

Median Housing Cost

Slides available at dandersondata.com/talks/ncme18

22 / 26

Alameda County

n Identifying as Black

Slides available at dandersondata.com/talks/ncme18

23 / 26

Alameda County

n Income/Poverty Ratio > 2.0

Slides available at dandersondata.com/talks/ncme18

24 / 26

Conclusions

  • Effect size appeared well estimated from coarsened data
    • V was similar to Cohen's d with these, empirical data
  • The vast majority, but not all, schools had sizeable estimated achievement gaps
  • Clear geographic clustering of achievement gaps was evident
    • Did Not appear to depend on the Census data investigated here

Slides available at dandersondata.com/talks/ncme18

25 / 26

Thanks so much!

Slides available at dandersondata.com/talks/ncme18

26 / 26

Introduction

  • Much recent focus on open data in research generally
  • Open data tend to be rare in educational research
    • Privacy concerns

Slides available at dandersondata.com/talks/ncme18

2 / 26
Paused

Help

Keyboard shortcuts

↑, ←, Pg Up, k Go to previous slide
↓, →, Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
t Restart the presentation timer
?, h Toggle this help
Esc Back to slideshow