Skip to content
Limited-time offer. Save up to 50% on Speechly Premium. See pricing

How Do AI Voices Work? Neural Text to Speech Explained

How do AI voices work? Neural text to speech explained in plain English: the 3 steps inside a voice, how it's trained, and why it still slips up.

Updated 9 min readBy the Speechly team
A network of white dots and threads on a black background gathering into one glowing lime-green sound wave.
Quick answer

AI voices are neural networks trained on many hours of recorded human speech, and they generate new audio from your text. One model turns the text into a spectrogram, a sound plan that sets timing, pitch and tone; a second model, the vocoder, turns that plan into audio. The newest voices predict audio tokens, the way a chatbot predicts words.

Key takeaways
  • An AI voice doesn't play back recordings. It generates new audio from patterns it learned from many hours of human speech paired with transcripts.

  • Most neural voices use three parts: a text front end that works out pronunciation, an acoustic model that plans timing and pitch as a mel spectrogram, and a vocoder that turns that plan into sound.

  • Fast neural vocoders such as HiFi-GAN (2020) generate audio many times faster than real time, even on an ordinary processor, which helped bring natural voices to everyday devices.

  • The newest generative voices work like language models, predicting audio tokens. They are the most expressive, but they can occasionally skip, repeat or invent words.

  • Clean source text still matters most: names, tables and abbreviations trip up even the best voice.

How do AI voices work? Neural text to speech explained in one paragraph: an AI voice is a neural network that learned how people speak by studying many hours of recorded speech paired with transcripts. When you press play, it reads your text, predicts how each sentence should sound, and generates brand-new audio instead of stitching together saved recordings.

That is what neural text to speech means: speech generated by a neural network rather than assembled from recorded fragments or hand-written rules. It explains most of what you hear, from voices that breathe like a narrator to voices that still trip over surnames.

How do AI voices work? The 3-step pipeline

Most neural voices split the job into three parts. Microsoft describes its custom voice service as a text analyzer, a neural acoustic model and a neural vocoder. Amazon Polly's neural engine works the same way: phonemes go into a network that predicts spectrograms, and a neural vocoder turns those into audio.

Step Its job What comes out Examples
1. Text front end Expands numbers and abbreviations, works out pronunciation Phonemes (the individual sounds of speech) Pronunciation dictionaries, letter-to-sound models
2. Acoustic model Plans timing, pitch, stress and tone A mel spectrogram, a "picture" of the sound Tacotron 2 (2017), FastSpeech (2019)
3. Vocoder Renders that picture as sound An audio waveform you can play WaveNet (2016), HiFi-GAN (2020)

Step 1: The front end works out what to say

First the system rewrites your text into something speakable: "Dr." becomes "Doctor" or "Drive", "$5" becomes "five dollars", and each word becomes phonemes. Our guide to what text to speech is and how it works walks through this stage with a full example.

Errors here are permanent. If the front end decides "lead" is the metal rather than the verb, even the most natural voice will say the wrong word, beautifully.

Step 2: The acoustic model plans the performance

The acoustic model is where most of the "AI" happens. This neural network takes the phonemes and predicts a mel spectrogram: a grid showing how much energy each frequency band has at each moment, with frequencies spaced the way human hearing perceives them. Think of it as sheet music for one specific performance.

Google's Tacotron 2 paper shows the level of detail. Each frame describes 50 milliseconds of sound with 80 numbers, and a new frame starts every 12.5 milliseconds, so one second of speech becomes 80 frames: 6,400 numbers instead of the 24,000 audio samples its vocoder produces. The authors chose this format because it emphasizes the lower frequencies that matter most for understanding speech.

Because the network sees the whole sentence, it learns prosody (rhythm, stress and melody) from examples rather than hand-written rules. It picks up that many questions rise at the end, that commas and list items bring short pauses, and that a statement settles down in pitch as it finishes.

Early acoustic models such as Tacotron 2 wrote the spectrogram one frame at a time, each frame depending on the last. That was slow, and the model could lose its place and skip or repeat words. FastSpeech fixed both problems by predicting how long each sound lasts up front and generating all the frames at once. Its authors reported 270 times faster spectrogram generation and almost no skipped or repeated words in their hardest test cases.

Step 3: The vocoder turns the plan into sound

A spectrogram is not audio yet. It leaves out fine detail, and the vocoder has to fill that in at enormous resolution: digital speech typically needs 16,000 or more samples every second.

DeepMind's WaveNet (2016) produced those samples one at a time with a neural network: far more natural than anything before it, but very slow. HiFi-GAN (2020) pits a generator against networks trained to tell its output from real recordings, and it produces whole stretches of audio in parallel. Its authors reported speech 167.9 times faster than real time on a data-center GPU, and 13.4 times faster on an ordinary processor with a smaller version. Fast vocoders like this helped move natural voices out of research labs and onto everyday devices.

What do the newest AI voices do differently?

Two bigger shifts have changed how the best voices are built since 2021.

One model from text to waveform

VITS (2021) trains the acoustic model and vocoder together as a single network. On the public LJ Speech dataset, listeners rated it about as natural as the original human recordings. It also builds in some randomness, because the same sentence can be spoken with many rhythms.

Voices that work like chatbots

The newest generation borrows the idea behind large language models. A neural audio codec compresses speech into a stream of discrete "audio tokens", and a large transformer predicts the next token the way a chatbot predicts the next word.

Microsoft Research's VALL-E (2023) is the best-known example. It turns each second of audio into 600 codes, was trained on 60,000 hours of English speech from about 7,000 speakers, and can imitate a new speaker from a 3-second recording, including their emotion and even the room's acoustics. The researchers warn that it could be misused to impersonate a specific speaker.

Commercial services use this approach too. Amazon describes Polly's generative engine as a billion-parameter transformer that turns text into speech codes, followed by a decoder that turns those codes into audio as a stream. Google says its Chirp 3: HD voices are powered by its latest generation of generative models.

The result is the most expressive, conversational speech yet, with one new weakness: like chatbots, these models can hallucinate. The VALL-E authors note that some words may come out unclear, missed or duplicated. Amazon says hallucinations tend to happen at the end of a sentence, and that the emergency stop it built to catch them can occasionally cut a word in half.

How are AI voices trained?

Every AI voice starts with real people speaking, but the amount varies enormously:

Voice or dataset Speech used Speakers
LJ Speech (public dataset) About 24 hours in 13,100 clips 1
Tacotron 2 (Google, 2017) 24.6 hours 1 professional speaker
LibriTTS (public dataset) 585 hours 2,456
VALL-E (Microsoft, 2023) 60,000 hours About 7,000

Training follows the same basic loop:

  1. Record. A voice actor reads a long script of statements, questions and exclamations, ideally in a studio. Microsoft requires consistent volume, speaking rate and pitch for its professional voices, plus a recorded consent statement from the speaker.
  2. Pair. Each clip is matched with its exact transcript.
  3. Learn. The model predicts audio for each sentence, compares it with the real recording, and adjusts millions of internal settings to close the gap, over and over.
  4. Add voices. Multi-speaker models learn a short list of numbers that captures each speaker's timbre, often called a speaker embedding. Changing those numbers changes the voice, which is how one model can offer many voices. Microsoft's custom voices are built on a multilingual, multi-speaker model, so a voice can even be adapted to other languages.

You can try a small version yourself: Apple's Personal Voice asks you to read 10 randomized phrases, then builds the voice on your device. On iPhone it's under Settings > Accessibility > Personal Voice > Create a Personal Voice, on iPhone 12 or later, according to Apple Support.

Neural vs standard text to speech: what's the difference?

Neural voices learn to generate speech; standard voices assemble it from recorded fragments or calculate it with signal processing. Cloud services still sell both:

Standard voices Neural voices Generative voices
How they make sound Stitch recorded speech fragments (Amazon Polly Standard) or use parametric signal processing (Google Cloud Standard) Acoustic model plus neural vocoder Transformer predicting audio tokens, plus a decoder
How they sound Clear but flat; stitched voices can have audible seams Smooth and natural The most expressive and conversational
Typical weakness Monotone, robotic rhythm Odd emphasis in long or unusual sentences Occasional skipped, repeated or invented words

Google's voice types page draws the same line between its parametric Standard voices and WaveNet voices trained on recordings of real people. If a voice still sounds mechanical, here's why some text-to-speech voices sound robotic.

Why do AI voices sound so human now?

Because every detail of the performance is learned from real speech. Neural models pick up the breaths, micro-pauses and pitch movement that rule-based systems never captured, and neural vocoders render them without the old buzz.

Listening tests show the size of the jump. In the Tacotron 2 study, listeners rated speech on a 1–5 naturalness scale:

System Naturalness score (1–5)
Older parametric voice 3.49
Concatenative (stitched) voice 4.17
Tacotron 2 (neural) 4.53
Real human recordings 4.58

Where do AI voices still go wrong?

Even the best voices have predictable blind spots:

  • Names and jargon. Unfamiliar words are pronounced from their spelling, so surnames, drug names and brands are the most common errors. See how accurate text-to-speech pronunciation is for examples.
  • Context beyond the sentence. A voice may miss sarcasm or stress the wrong word when the meaning depends on earlier paragraphs.
  • Hallucinations. Token-based generative voices can occasionally skip, repeat or add words.
  • Layout. Tables, footnotes and page headers have no natural spoken form, so they can be read in the wrong order or repeated.
  • Drift. Amazon notes that updates to its generative model can slightly change how a voice sounds, which matters for a podcast series recorded over months.

Developers correct many of these with SSML, a markup language for pronunciations, pauses and emphasis.

What does this mean when you listen?

  • Test voices on your hardest text. A one-line sample hides weak spots; try a paragraph with names, numbers and a list.
  • Know where the voice runs. On-device voices keep your text on your device; cloud voices send it to a server to be turned into audio. Check what an app does with your files before uploading anything sensitive.
  • Expect a short pause before playback. Audio has to be generated before it can play, though streaming voices start speaking before the whole passage is done.
  • Follow along when it helps. Many engines report when each word is spoken, which is what makes word-by-word highlighting possible. Not all do yet: Amazon Polly's generative voices don't return these word timings.

Hear natural AI voices in Speechly

Speechly is a text-to-speech reader for iPhone, iPad, Android, Mac and Windows that reads PDFs, Word documents, EPUB books, web pages and photos of printed pages with natural AI voices.

  • 25+ voices in eight languages, which you can preview on the Speechly voices page and switch anytime.
  • Word-by-word highlighting, so you can follow along, and speed from 0.5× to 2×.
  • On a computer, select text in any app and press ⌘E on Mac or Alt+R on Windows to hear it in a small floating player.

Speechly is free to download, and the free plan includes two files with premium voices, no card required. Premium costs $14.99/month or $139/year for unlimited listening on every device.

Frequently asked questions

Are AI voices real people?

No, but they start with real people. Voices are trained on recordings of voice actors or other speakers, and the model then generates new speech that sounds like them. Responsible providers require the speaker's consent; Microsoft, for example, asks for a recorded consent statement before it trains a custom voice.

How long does it take to train an AI voice?

It depends on the system. Classic single-voice models learned from about 24 hours of one person's speech, while large generative models learn from tens of thousands of hours and can then imitate a new voice from a short sample. Apple's Personal Voice asks you to read 10 phrases and builds the voice on your device.

Is neural text to speech the same as voice cloning?

No. Neural text to speech is the underlying technology; voice cloning is one use of it, where a model is adapted to sound like one specific person from a sample of their speech. Reading apps use ready-made voices instead.

Can AI voices show emotion?

Yes, to a degree. Neural voices pick up tone from punctuation and context, and newer generative models can sound excited, calm or conversational. They still misjudge emotion when the meaning depends on context outside the sentence they are reading.

Do AI voices need an internet connection?

Only if the voice runs in the cloud. Smaller neural voices can run on your phone or computer without a connection, while many of the largest, most realistic voices are generated on remote servers and stream the audio back to you.

Why do AI voices mispronounce names?

Because names often aren't in the pronunciation dictionary, so the model guesses from the spelling. Unusual surnames, brands, places and technical terms are the most common errors, even with an otherwise excellent voice.

Speechly

Listen with Speechly

Natural AI voices on iPhone, iPad, Android, Mac and Windows, with word highlighting and your place saved on every device. Free to download.

Prices and features of other products were checked on November 18, 2026 and can change. Speechly is not affiliated with the other companies named here.