Multimodal AI

In October 2024 the Associated Press reported that Whisper, OpenAI's open speech-recognition model, sometimes writes sentences nobody said. The finding mattered because of where Whisper had gone. The AP counted more than 30,000 clinicians and 40 health systems using Nabla, a clinical note-taking tool built on Whisper, and Nabla estimated that it had transcribed about seven million medical visits. Researchers had already measured the problem: a study presented at the ACM FAccT conference in June 2024 found invented phrases in about 1% of 13,140 transcribed audio clips, and judged 38% of those inventions harmful, from violent language to invented associations and false claims of authority. One design decision made the errors impossible to check. Nabla's tool erased the original audio "for data safety reasons", so no clinician could play back the recording behind a sentence that looked wrong.