Writing for Fordham’s Flypaper, Amber Northern recently reviewed a new initiative that uses transcript analysis to promote teacher professional learning. Excerpts of the piece appear below:
Traditional classroom observations are time and labor intensive, as they are meant to capture the many nuances of student-teacher interactions and thereby inform future practice. A recent paper from Annenberg, termed a proof-of-concept study by its authors, explores how audio recordings and automated analyses might supplement (or even replace) traditional in-person observation conducted by a principal or outside evaluator. If successful, this innovation could increase the number of evaluations that are possible, reduce the time and effort involved, eliminate possible evaluator bias, and expand the scope of items reviewed.
The paper’s authors, Jing Liu and Julie Cohen, are alumni of Fordham’s Emerging Education Policy Scholars program and teach at the Universities of Maryland and Virginia, respectively. They utilize transcribed videos of fourth- and fifth-grade English language arts classrooms collected as part of the Measures of Effective Teaching (MET) project, to date the largest research project in the United States on K–12 teacher effectiveness. The MET project’s sample was composed mainly of high-poverty, urban schools. Liu and Cohen focused on the first thirty minutes of nearly 1,000 videos (four per teacher)—amounting to 30,000 minutes of ELA teaching—from the first year of MET. A professional transcription company did a word-for-word transcription with time stamps attached to the beginning and end of each speaker’s turn. It also labeled different students speaking, as much as the audio quality allowed. The analysts also have value-added scores from state achievement tests and the SAT-9 and observational data from three of the most popular observation instruments.
They determine that teachers spent 85 percent of the observation time talking to their students, with classrooms varying considerably in the prevalence of back-and-forth conversation, although the average was 4.5 turns per minute. Audio coded as teacher-centered instruction is a consistent negative predictor of value-added scores, while interactive instruction predicts positive value-added scores (the classroom-management construct has negative correlations relative to value added, but they are not significant, possibly due to lack of power).
Lastly, the analysts did a back-of-the-envelope cost calculation that indicates the minimum cost savings is 54 percent from the transcription approach as compared to the traditional human-observer approach. AI voice technology is already out there with a “pedagogical fitbit” that analyzes classroom discourse patterns—in person or virtually—and sends the teacher feedback. What we need now is willingness to try new available technologies like these to improve teaching.
For the study, see: Measuring Teaching Practices at Scale: A Novel Application of Text-as-Data Methods
For more commentary, see: https://fordhaminstitute.org/national/commentary/using-transcription-analysis-replace-traditional-classroom-observations
