The What Works Clearinghouse, which identifies studies that provide credible and reliable evidence of intervention effectiveness, gives its highest rating of confidence only to randomized controlled trials (RCTs). But is an RCT the best way to provide useful information to consumers about complex interventions like curricula?
In this American Enterprise Institute report, Alan Ginsburg and Carnegie Foundation Senior Fellow Marshall (Mike) Smith analyze 27 RCT mathematics curriculum studies contained in the What Works Clearinghouse, and they find serious threats to the usefulness of 26. They suggest a more nuanced approach to evaluating curricula that is more timely and takes into consideration the complex context in which curricula are implemented.
Some of the threats of evaluating curricula with RTC’s are as follows:
- Authors associated with curriculum developer
- Curriculum intervention not well-implemented
- Unknown comparison curricula
- Instructional time greater for treatment than for control group
- Limited grade coverage
- Assessment favors content of the treatment
- Outdated curricula
The authors recommend that the Institute of Education Sciences (IES) review the RCTs in the What Works Clearinghouse and remove those that, in its view, do not provide useful information. The IES should make its judgments and rationale public. In addition, IES should create an internal expert panel of evaluators, curriculum experts, and users (for example, teachers and administrators) to consider how, in the short term, to improve the current WWC criteria and standards for reviewing RCTs in education.
For more detailed data and recommendations, read the full report here.
