Caption Viewer Survey: Error Ranking of Real Time Captions in Live Television News
The WGBH National Center for Accessible Media (NCAM) reported in 2010 that it was conducting the Caption Accuracy Metrics Project (funded by the U.S. Department of Education, National Institute on Disability and Rehabilitation Research) to explore using language-processing tools to develop a prototype automated caption accuracy assessment system for real-time captions in live TV news programming.
Captioning mandates designed to provide equal access to television have resulted in more accessible programming, but a shortage of skilled professionals and the downward pressure on rates by program providers has created the need for a common and automated method of measuring accuracy and quality of real-time captions.
Consumers were presented with error samples. Then, they were asked to indicate their overall opinion of caption quality and to select one of a number of statements that best reflected their general opinion about caption errors in live newscasts. Over 50% indicated that many real-time caption errors are minor but 42% indicated that caption errors negatively impacted their ability to understand what was spoken. Only 6% of respondents felt that real-time captions were generally accurate.