By whispertranscribe.ai Editorial Team · Last updated: November 2026 · 13 min read
What is Whisper AI? In most searches, it means OpenAI Whisper, an open-source speech recognition model OpenAI published in September 2022. It turns recorded speech into text, identifies the spoken language, and can translate other languages into English. OpenAI trained it on 680,000 hours of web audio and released the code and weights under the MIT License.
A quick disclosure: we make whispertranscribe.ai. whispertranscribe.ai is a web-based AI transcription tool that turns audio and video into text in 130+ languages. This guide keeps the model and the products built around it clearly apart, and every number about the model comes from OpenAI's own paper, repository, or model card.
Quick answer: what is Whisper AI?
Whisper is a speech-to-text model, not an app or a company. You can download it from OpenAI's GitHub repository and run it on your own computer, or use a service that runs it for you.
- Made by: OpenAI. The research paper, Robust Speech Recognition via Large-Scale Weak Supervision, was posted to arXiv on December 6, 2022.
- What it does: transcribes speech, detects the language, translates speech into English, and adds segment timestamps.
- What it doesn't do out of the box: label who is speaking, or transcribe live. OpenAI's model card says it "cannot be used for real-time transcription out of the box."
- Sizes: six, from tiny (39 million parameters) to large (1,550 million).
- Cost: free to download and run. You supply the hardware.
"Whisper AI" can mean three different things
The name gets used loosely, and these three meanings are easy to mix up.
| What people call "Whisper AI" | What it is | Who makes it | How you use it |
|---|---|---|---|
| OpenAI Whisper | An open-source speech recognition model | OpenAI | Download the code and weights, run them yourself |
| whisper-1 | OpenAI's hosted API model for transcription | OpenAI | Send audio files to OpenAI's API and pay per use |
| Apps with "Whisper" in the name | Transcription products, such as whisperai.com and whispertranscribe.ai (this site) | Separate companies | Upload or record audio in their apps |
Products in the third row aren't the model itself or interchangeable. Each company sets its own features, limits, prices, and privacy terms. In this article, "Whisper" means OpenAI's model.
How does Whisper work?
Whisper is an encoder-decoder Transformer. The encoder turns 30 seconds of audio into a numeric representation, and the decoder writes the matching text one token at a time, steered by special tokens that tell it which task to do.
The Whisper paper describes this pipeline:
- Resample. Audio is converted to 16,000 Hz.
- Cut into 30-second windows. The model always looks at 30 seconds at a time. Shorter clips are padded.
- Make a spectrogram. Each window becomes an 80-channel log-Mel spectrogram, a picture of how sound energy changes across frequencies, computed on 25-millisecond frames every 10 milliseconds. (Large-v3 switched to 128 channels.)
- Encode. Two convolution layers and a stack of Transformer blocks process the spectrogram.
- Decode. The decoder first predicts the language, or a "no speech" token. Next comes the task (transcribe or translate), then whether to include timestamps, and finally the text itself.
Timestamps are predicted as tokens too, rounded to the nearest 20 milliseconds. That's how one model handles language detection, transcription, translation, and timing.
What was Whisper trained on?
The original models learned from 680,000 hours of internet audio with matching transcripts. OpenAI calls this "weak supervision" because the transcripts weren't checked by hand the way benchmark datasets are.
| Share of training audio | Hours | Content |
|---|---|---|
| About 65% | 438,000 | English audio with English transcripts |
| About 18% | 126,000 | Non-English audio with English transcripts (translation data) |
| About 17% | 117,000 | Non-English audio with transcripts in the same language |
Source: OpenAI's Whisper model card. The paper rounds the translation share to 125,000 hours.
Two details explain a lot of Whisper's behavior. OpenAI filtered out transcripts that looked machine-made, such as all-caps text, so the model learned to write punctuated sentences. And early models tended to guess speaker names, because many web transcripts included them. OpenAI fine-tuned that habit away, but it shows how closely the model mirrors its training text.
Large-v3 was trained differently, on about 5 million hours: 1 million with weak web labels and 4 million labeled by large-v2, according to its Hugging Face model card.
Whisper model sizes compared
Whisper comes in six sizes. Smaller models run faster on modest hardware; larger ones make fewer errors but need more video memory (VRAM). The figures below are copied from OpenAI's README.
| Size | Parameters | English-only version | Multilingual version | Required VRAM | Speed relative to large |
|---|---|---|---|---|---|
| tiny | 39 M | tiny.en | tiny | ~1 GB | ~10x |
| base | 74 M | base.en | base | ~1 GB | ~7x |
| small | 244 M | small.en | small | ~2 GB | ~4x |
| medium | 769 M | medium.en | medium | ~5 GB | ~2x |
| large | 1550 M | None | large | ~10 GB | 1x |
| turbo | 809 M | None | turbo | ~6 GB | ~8x |
OpenAI measured these speeds on English speech with an A100 data-center GPU and says real-world speed "may vary significantly." A laptop will be slower.
Which Whisper model should you pick?
Start from the memory you have and what you need the transcript for:
- Testing on a basic machine: tiny or base, around 1 GB of VRAM. Expect more mistakes.
- English only: use a
.enmodel. OpenAI says these do better, especially at the tiny and base sizes. - Good quality at reasonable speed: turbo, around 6 GB. It's the command-line default.
- Translating speech into English: medium or large. Turbo is not trained for translation and returns the original language even if you ask it to translate.
- Hardest audio, best possible result: large, around 10 GB of VRAM.
Whisper versions: a short timeline
"Large" has covered three releases, and turbo is a slimmed-down large-v3. The version matters, because results differ by checkpoint.
| Version | Released | What changed |
|---|---|---|
| Original models (tiny to large) | September of 2022 | First open-source release with the paper |
| large-v2 | December of 2022 | Same size, trained 2.5 times longer with extra regularization |
| large-v3 | November of 2023 | 128 Mel channels, a new Cantonese token, 10% to 20% fewer errors than v2 across many languages |
| turbo (large-v3-turbo) | September of 2024 | Decoder layers cut from 32 to 4 for speed; no translation |
Sources: the Whisper model card, the paper's footnote on large-v2, and the Hugging Face cards for large-v3 and large-v3-turbo.
How many languages does Whisper support?
Whisper recognizes about 99 languages. The paper's language detector used 99 language tokens, and the current tokenizer lists 100 language codes after large-v3 added Cantonese. Support is not the same as quality, though: the model card says Whisper shows strong results in "~10 languages."
Quality tracks training data. The paper found that performance in a language is "very well predicted" by how many hours of it the model saw, and that most languages had less than 1,000 hours. English, Spanish, or German will usually come out cleaner than a low-resource language. The README links a per-language error chart if you need to check yours.
Don't confuse this with product counts. whispertranscribe.ai supports 130+ languages. That figure is our product's language support (the pricing page says "130+ languages & accents"), not a property of the open-source Whisper model.
How accurate is Whisper?
There's no single accuracy number for Whisper, because it depends on the model size, the language, and the recording. On clean read English, large-v2 scored a 2.7% word error rate (WER). Averaged over 14 English test sets, most of them harder, it was 12.8%.
WER is the share of words a system gets wrong, counting words it swaps, drops, or adds, measured against a human reference transcript. Lower is better. The figures below come from Table 2 of the paper, comparing Whisper large-v2 with a wav2vec 2.0 model trained on the LibriSpeech audiobook dataset:
| Test set | wav2vec 2.0 WER | Whisper large-v2 WER |
|---|---|---|
| LibriSpeech clean (read audiobooks) | 2.7% | 2.7% |
| Common Voice (crowd-recorded speech) | 29.9% | 9.0% |
| CHiME-6 (noisy dinner-party recordings) | 65.8% | 25.5% |
| Average across all 14 test sets in the table | 29.3% | 12.8% |
The pattern is the point. Both models tie on the audiobook test, but Whisper holds up much better on everything else, making 55.2% fewer errors on average. OpenAI's announcement rounds this to "50% fewer errors." In the paper's noise tests, models trained on LibriSpeech beat Whisper on nearly clean audio but fell behind once pub-style background noise got loud, below a 10 dB signal-to-noise ratio.
These are research benchmarks, not a promise for your files, and product claims are separate. whispertranscribe.ai, for example, reaches up to 99% accuracy on clear audio, and background noise, crosstalk, and strong accents lower it.
What are Whisper's limitations?
Whisper's main weakness is that it can write fluent text that nobody said. OpenAI's model card calls this hallucination and links it to the weakly supervised training. It also lists repetition, uneven accuracy across languages and accents, and no built-in speaker labels.
- Hallucinated text. In a 2024 study, Careless Whisper, researchers found that roughly 1% of the transcriptions they checked included whole phrases or sentences absent from the audio. Hallucinations happened more often with long pauses in speech.
- Repetition loops. The model can get stuck repeating a phrase. Beam search and temperature fallback reduce this "to some degree," the model card says, but not fully.
- Missing first or last words. The paper lists skipping the first or last few words of a segment as a known failure mode.
- Long recordings need windowing. Whisper only sees 30 seconds at a time. For longer files, the transcribe tool slides that window along using predicted timestamps, so a mistake in one window can throw off the next.
- No speaker labels. Diarization, working out who spoke when, wasn't robustly evaluated, according to the model card. You need a separate tool for it.
- Not real-time by default. It was built for recorded files. Live captioning needs extra engineering around it.
- Uneven by language and accent. The model card notes higher error rates for some accents, dialects, and demographic groups.
OpenAI also warns against transcribing people without their consent and against high-risk decision-making uses.
Where official sources disagree
Some figures differ between OpenAI's own documents, mostly from rounding or version changes. Check this before you quote a number:
| Topic | One official source says | Another says | What to use |
|---|---|---|---|
| Error reduction vs LibriSpeech models | Paper: 55.2% fewer errors on average | OpenAI announcement: 50% fewer errors | Quote the paper and name the comparison |
| Languages in training data besides English | Paper: 96 other languages | Model card: 98 languages | "About 99 languages" for recognition |
| Translation training hours | Model card: 126,000 | Paper: 125,000 | Either, labeled as approximate |
| Turbo parameter count | README and Hugging Face: 809 M | Model card table: 798 M | 809 M, the figure most sources use |
| Training data size | Original paper: 680,000 hours | Large-v3 card: 5 million hours | Name the version you mean |
| License | GitHub: MIT for code and weights | Hugging Face large-v3 metadata: Apache-2.0 | Check the license file of the copy you download |
How to use Whisper: run it yourself, call an API, or use an app
You have three practical routes. Running Whisper yourself is free and keeps audio on your machine. OpenAI's API removes the setup but caps file size. A hosted transcription app adds editing, speaker labels, and exports, but it's a third-party service with its own terms.
| Run open-source Whisper | OpenAI transcription API | Hosted transcription app | |
|---|---|---|---|
| Setup | Python, PyTorch, and FFmpeg | Code and an API key | A browser |
| Cost | Free; your hardware | Pay per use | Free tier or subscription, varies by app |
| File limits | Your hardware | 25 MB per file | Set by each app |
| Where audio goes | Stays on your computer | Sent to OpenAI | Sent to the app's servers |
| Speaker labels | Not built in | Not in whisper-1 | Depends on the app |
| Editing and exports | TXT, SRT, VTT, TSV, JSON from the command line | Returned to your code; you build the rest | Usually an editor plus document and subtitle exports |
Run Whisper on your own computer
OpenAI's README gives the install command and a basic run:
pip install -U openai-whisper
whisper interview.mp3 --model turbo
You also need FFmpeg installed, and the README says the code should run on Python 3.8 to 3.11. Add --language to name the spoken language, or --task translate with a multilingual model to get English. The model weights download on first use.
Use OpenAI's API
OpenAI's speech-to-text guide caps uploads at 25 MB per file, so compress or split anything bigger. As checked in November 2026, the guide marks whisper-1 as deprecated and lists a shutdown date of February 26, 2027, pointing developers to newer transcription models. Those newer models are OpenAI speech models, not new releases of the open-source Whisper family.
Use a hosted transcription app
If you'd rather not install anything, a hosted app processes files on its servers. whispertranscribe.ai is one option, and its free audio to text converter is a quick way to try it. Our homepage describes whispertranscribe.ai as "powered by Whisper and next-generation AI." Here's what it offers, from our own product pages:
- Upload audio or video files up to 5GB and up to 10 hours long, or paste a YouTube, Vimeo, Google Drive, or Dropbox link into Transcribe from URL.
- 130+ languages, with speaker detection you can switch on.
- Live transcription from your microphone, plus record and transcribe in the browser.
- Exports in DOCX, PDF, SRT, VTT, TXT, and CSV, including subtitles from the AI subtitle generator.
- Translate audio to English for non-English recordings.
The trade-offs are real. Your file goes to a cloud service rather than staying on your computer. We don't offer an API, custom vocabulary, or a meeting bot. Paid plans start from $9.99/month billed annually (pricing), and you can start for free with no account required. See the Whisper AI transcription tool page or all features for details.
Is Whisper open source, and what does the license allow?
Yes. OpenAI's repository states that "Whisper's code and model weights are released under the MIT License." MIT is a permissive license. In general, it lets you use, copy, modify, and sell software built on the code, as long as you keep the copyright and license notice.
Two cautions. Copies on other sites can carry different license metadata; Hugging Face's large-v3 page lists Apache-2.0. Read the license that comes with the copy you use. And a license covers the software, not the recordings you feed it. Consent and privacy rules still apply to the audio. This is general information, not legal advice; the MIT license text is the official source.
Who is Whisper for?
Whisper suits developers and anyone comfortable with a terminal who wants free, private, unlimited transcription. If you want a finished transcript with an editor and exports, a hosted app is usually quicker.
| You are… | Good fit | Why |
|---|---|---|
| A developer building a product | Open-source Whisper or an API | Full control over the pipeline |
| A researcher with sensitive interviews | Open-source Whisper, run locally | Audio never leaves your machine |
| A creator who needs subtitles this week | A hosted app with SRT or VTT export | No setup; timed files ready to upload |
| Someone on a laptop without a GPU | A small model locally, or a hosted app | Large models need about 10 GB of VRAM |
Before you trust a Whisper transcript: a checklist
We built this checklist from the failure modes in OpenAI's paper and model card:
- Spot-check pauses and silent stretches for sentences nobody said.
- Scan for the same phrase repeated several times in a row.
- Check the first and last lines of the file against the audio.
- Verify names, numbers, and technical terms.
- Confirm the language was detected correctly, or set it by hand.
- For translation, make sure you didn't use turbo.
- Add speaker labels with a separate tool if you need them.
- Get consent before recording or transcribing other people.
FAQs
Is Whisper AI free?
Yes, OpenAI Whisper is free to download and run. OpenAI licenses the code and model weights under MIT, so you can use them without paying OpenAI. You do supply the computer, and the large model wants roughly 10 GB of GPU memory. Hosted options cost money: OpenAI's API charges per use, and transcription apps set their own plans. "Free" also doesn't mean easy; you'll need Python, FFmpeg, and some comfort with the command line.
Can Whisper identify different speakers?
No, not on its own. Whisper writes what was said but doesn't label who said it. OpenAI's model card says capabilities such as speaker diarization were not robustly evaluated. To get labels, people usually pair Whisper with a separate diarization tool, or use a transcription app that adds speaker detection. whispertranscribe.ai, for example, has a speaker detection option you can turn on before you transcribe. Always check the labels on recordings where voices sound alike.
Can Whisper transcribe in real time?
Not out of the box. Whisper was designed for recorded audio and processes it in 30-second windows, and OpenAI's model card says it can't be used for real-time transcription without extra work. Developers have built near-real-time tools around it by feeding in short chunks as they arrive. If you need live captions, look for a tool with a dedicated live mode, and expect live output to need more cleanup than a transcript made from the finished recording.
How many languages does Whisper support?
Whisper recognizes about 99 languages, and its current tokenizer lists 100 language codes after Cantonese was added in large-v3. Quality varies a lot. OpenAI's model card reports strong results in about 10 languages, and the paper found most languages had under 1,000 hours of training audio. Before you rely on a less common language, test a short clip. Product figures, like whispertranscribe.ai's 130+ languages, describe each product's own support, not the open-source model.
Is whispertranscribe.ai the same as OpenAI Whisper?
No. OpenAI Whisper is an open-source model that you download and run yourself. whispertranscribe.ai is a separate, browser-based transcription service; its homepage describes it as powered by Whisper and next-generation AI. It adds things the raw model doesn't include, such as uploads up to 5GB, link import from YouTube or Google Drive, speaker detection, and DOCX, PDF, SRT, and VTT exports. Other apps with "Whisper" in their names are separate products from separate companies.
Why does Whisper sometimes write words nobody said?
Because it predicts text the way a language model does. OpenAI's model card says that training on large, noisy web data leads the model to sometimes produce words not in the audio. OpenAI's explanation is that it blends predicting the next likely word with transcribing what it hears. A 2024 study found whole invented phrases in roughly 1% of the transcriptions it checked, more often around long pauses. Review pauses, quiet stretches, and anything important against the audio.
Want a Whisper transcript without the setup? Upload a file or paste a link into whispertranscribe.ai and get text in 130+ languages, with exports in DOCX, PDF, SRT, and VTT.
References
- OpenAI: Introducing Whisper (September 21, 2022)
- Radford et al.: Robust Speech Recognition via Large-Scale Weak Supervision (arXiv 2212.04356)
- OpenAI: Whisper GitHub repository and README, model card, and MIT license
- Hugging Face: openai/whisper-large-v3 model card
- OpenAI API docs: Speech to text guide (checked November 2026)
- Koenecke et al.: Careless Whisper: Speech-to-Text Hallucination Harms (arXiv 2402.08021)
