One of the most policy relevant charter school studies is an
observational study conducted by the Center for Research on Education
Outcomes (CREDO) at Stanford University. This study has high external
validity. (Information on their methodology can be found at http://credo.stanford.edu/documents/NCSS2013_Technical%20Appendix.pdf, pages 5-11, which also discusses the internal vs. external validity concerns discussed above. The main report (http://credo.stanford.edu/documents/NCSS%202013%20Final%20Draft.pdf) also discusses the methodology on pages 8-11, but in less detail.)
CREDO matches each charter school student to public school
students from “Charter Feeder Schools” that have similar baseline test
scores and the same race, gender, special education status, limited
English proficiency status and free lunch status. They then compare how
the charter school students perform on low-stakes tests relative to
public school students.
Overall, I was very impressed with the amount of data they were able to collect and process, although when analyzing that many different sections and doing so many statistical analyses, they're bound to find some significant results. They do seem aware of this, mentioning it in the issues associated with repeated tests - although also mentioning that they chose not to reduce the significance level, even with so many tests. Given this, we have to look a bit more carefully what is 'significant', especially since so many of the results they found were within .01 standard deviations, at least when aggregating all of the data nationally. Also when matching the VCRs, they did allow variation in the TPS students' starting scores of plus or minus 0.1 SDs - which is often more than the level of difference they're often finding. They do talk that not having exactly equal starting scores, and seem to feel like it's not an issue once averaged out - but to a less statistically trained eye, I wonder.
And while talking about the VCR matching, I'm also wondering if the charter school students are going to show more variation than the VCRs, because each charter student is matched with an averaged VCR including up to 7 TPS students. The score of one is going to show more variance than an averaged score. Again the study does mention this criticism, as put forth by Hoxby, but says that they did more statistical analysis to find that it wasn't a valid criticism. Another interesting point to me was that the matched TPS students had a lower than average change, meaning that the virtual twins did not progress as much as an average student.
On the issue of confounding variables - Because this study wasn't looking at lottery 'winners' and 'losers' who had all tried to get into the charter schools, the main confounder that stuck out to me was around parent involvement and school choice. The charter school students were matched to TPS students on the basis of race, gender, free or reduced price lunch status, special education status, and English language learner status. But what about the parents? The VCRs came from feeder schools where presumably all of the parents knew about the charter schools, and yet only the parents of the charter school students actually enrolled them in a charter school. The analysis states that the VCR method produces results that are not significantly different from experimental methods, and it also points out some issues with using lottery as an experiment (not totally random, not enough students, etc), but I still felt like the difference between who chooses to attend charter schools versus who doesn't would be a large confounding variable. Not sure which way it would affect the results though. Perhaps parents who are more involved would choose charter schools, and their children might do better because their parents are involved and supportive of their education? Or perhaps parents whose children aren't doing well in TPS are more likely to move to a charter school? (The study did mention some evidence of this.) Or perhaps parents who have a higher education level are more likely to research and enroll their children in a charter school, and parental education level tends to correlate with student achievement?
Other possible confounders... Do charter schools attract better (or worse, or newer, or...) teachers? Perhaps any effects are from differences in teacher quality. Do charter schools have different class sizes, or school sizes? I think the study mentioned that charter schools were often smaller schools than the TPS (although I can't find the reference now that I'm looking again). Do charter schools have more instructional time? Some charter schools have longer school days or longer school years - maybe those '7 extra days in reading' results actually came from having seven extra days of instructional time. Do charter schools have a stronger school community? By choosing to go to a school, students might be more engaged with the school community, which may affect math & reading scores. There are a lot of potential differences between charter school students and TPS students that were not accounted for in the VCR method. Thus the results probably cannot be externally valid for all students.
Another big issue that stood out for me was the wide variation in the state scores - and most likely in individual school scores. Charter schools are not a homogenous group, and so to make judgements based on aggregate might not be very helpful. Just because charter schools as a whole are showing basically the same (or possibly a little bit better) results than TPS, tells you nothing about the charter school down the street from you, or even the charter schools in your state. The study does delve a bit into the state differences, but I would have liked to have seen more. I care less about the overall state of charter schools and more about which ones are working and which ones aren't.
No comments:
Post a Comment