Technology · Agentic → Voice Agents · Open Source · wiki:deep
Whisper
Whisper (OpenAI open release) is a large-scale speech recognition model/codebase widely used as the STT backbone under voice agents. It is infrastructure, not a product assistant. Use as the listen half of voice ingress; pair with policy-gated action paths so spoken requests become cases, not unsupervised tool storms. For Whisper, treat autonomy claims as contingent on tool reliability, memory, and human gates; measure loop cost and recovery before promoting it from demo to production path.
In the blog
This pack is on a post’s technology stack. The same words on this page link to those posts when the title appears.
Why it matters here
In this catalog it is the Voice Agents option when you need to turn speech into structured agent turns and back. Keep Whisper on the observe/propose side of the loop: useful signals and drafts, with human or policy authority still required before anything irreversible runs.
How it works
Whisper is an STT model/codebase — infrastructure under voice agents, not an assistant product.
- Load a Whisper model (or whisper.cpp).
- Feed audio segments/files.
- Decode to text (optional translate).
- Pass transcripts to agents or indexes.
Related: pipecat · livekit-agents · llama-cpp · topics/03-channels
When to reach for it
- Use when: Whisper (OpenAI open release) is a large-scale speech recognition model/codebase widely used as the STT backbone under voice agents matches the bottleneck and you specifically need to turn speech into structured agent turns and back.
- Skip when: another pack already covers the need with less ops surface, or Whisper would become a second place people retype truth.
- Prefer instead: compare Pipecat, LiveKit Agents, llama.cpp before locking the cell.
Limits
- Ops: confirm license, hosting, and release cadence for Whisper (openai/whisper) before relying on it in a live corridor.
- Integration: bad credentials, scopes, or schema drift produce empty retrievals or runaway tool/model spend without an obvious UI alarm.
- Scope: Voice Agents components support agents; they are not the company ledger or the approval authority.
What we checked
Claims below are backed by science sources on disk.
Primary / product role
Robust Speech Recognition via Large-Scale Weak Supervision · MODERATE
“When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning.”
Docs or contrast
openai/whisper · MODERATE
“Robust Speech Recognition via Large-Scale Weak Supervision”
Additional source
Zero-shot evaluation protocol (same paper) · MODERATE
“we evaluate Whisper in a zero-shot setting without using any of the training data for each of these datasets so that we are measuring broad generalization.”
Research inventory
9 tags · 8 out · 9 in · 3 artifacts · 1 gaps · 12 corpus docs
Catalog tags
- landscape.layer
- Agentic
- landscape.subcategory
- Voice Agents
- license_tag
- Open Source
- maps.dm
- absent
- maps.features
- 4
- one_liner
- Voice Agents
- review.depth
- science
- slug
- whisper
- title
- Whisper
Artifacts
- dm_map · absent
- features_map · present · technologies/whisper/features.md
- readme · present · technologies/whisper/README.md
Out · alternative_to
- livekit-agentstechnology
- pipecattechnology
- pipertechnology
- vocodetechnology
Out · maps_to
- perception-multimodalfeature
- r2-03-channelsfeature
- triggers-channel-coveragefeature
- voice-agentsfeature
In · alternative_to
- livekit-agentstechnology
- pipecattechnology
- pipertechnology
- vocodetechnology
In · in_stack
- agentic-voice-agentsstack · alternative_slot
In · maps_to
- perception-multimodalfeature
- r2-03-channelsfeature
- triggers-channel-coveragefeature
- voice-agentsfeature
Corpus tags
- category
- Agentic → Voice Agents
- dedication
- open-source
- feature
- perception-multimodalr2-03-channelstriggers-channel-coveragevoice-agents
- kind
- map_edgetech_featurestech_quotetech_readme
- needs_deepen
- false
- quality
- ok
- slug
- whisper
- source_id
- arxiv-2212-04356whisper-githubwhisper-zero-shot
- technology
- whisper
Corpus documents (12)
map_edge · 4
- whisper → perception-multimodal
- whisper → r2-03-channels
- whisper → triggers-channel-coverage
- whisper → voice-agents
tech_features · 1
- Whisper · features
tech_quote · 6
- Whisper · arxiv-2212-04356
- Whisper · arxiv-2212-04356
- Whisper · arxiv-2212-04356
- Whisper · whisper-github
- Whisper · whisper-github
- Whisper · whisper-zero-shot
tech_readme · 1
- Whisper