Where the data comes from

Every source file's exact URL, version, and SHA-256 checksum is recorded in the manifests, and the pipeline rebuilds everything from scratch with two commands.

Sources

• CAASPP research files (California's Smarter Balanced results, 2014–15 onward, every school × grade × subject × 58 student groups) — caaspp-elpac.ets.org. 36 million rows across four incompatible file formats, normalized by the pipeline.
• CDE downloadable data files (enrollment, free/reduced-price meals, English learners, staffing and credentials, chronic absenteeism, discipline, graduation, per-pupil spending) — cde.ca.gov. The same sources that feed DataQuest and Ed-Data. Integration in progress.
• California School Dashboard indicator and growth-model files — used to cross-validate our growth estimates against the state's own model.

Data problems we found (and published)

Official data files have real problems. Our automated checks found — and our published data-quality report documents — among others:

• In the 2015 file, every student group carries the same enrollment number (the whole-school count), statewide. CDE confirmed the definition changed in 2016 and the old file was never fixed. We therefore never use 2015 subgroup enrollment.
• Statewide test participation collapsed to 23.7% in 2021 — that year is excluded from all trend analysis.
• Since 2021, male + female counts no longer add up to all students (nonbinary markers exist in the source systems but aren't published as a group), so gender can no longer be treated as a complete partition.

Each issue lives as a structured entry in known_issues/ with evidence and handling rules the pipeline enforces.

Reproduce it

git clone https://github.com/ssorkin/schoolfactors
cd schoolfactors && uv sync
uv run sf acquire   # download sources (~1 GB), record checksums
uv run sf ingest    # normalize to Parquet + DuckDB
uv run sf check     # regenerate the data-quality report
uv run sf analyze   # fit the models, render the figures