Highlighting what the voice is saying — and the dead ends on the way

When a voice explains a page, your eyes want to know where. Read-along highlighting — marking the exact words on screen as they’re spoken — is the feature that makes eye and ear agree. It looks like a flourish. Underneath it’s two hard problems: which words to mark, and when to light them.

This is also a post about subtraction. The earlier posts were about building; this one is mostly about what we tore out, because the removals were the progress.

Dead end #1: let the model paint

The first instinct was to give the model a brush — a separate channel where it could describe what to highlight on the page while narrating. It was clever, and it was a whole layer of machinery sitting between the voice and the screen, with its own ways to be wrong, late, or out of sync. We pulled it out and kept a simpler premise: the highlighter should follow the words the voice is already saying.

Dead end #2: the mark that killed the audio

Then came the subtle one — the kind of bug that teaches you how an API really works. We let the narrator call a tool to mark each phrase as it spoke it, once per sentence. Reasonable on paper.

In practice, narration came out shredded — choppy bursts instead of smooth speech. The cause was buried in the model’s behavior: a tool call ends the current audio response. Forcing a mark every sentence meant ending and restarting the audio response every sentence. We were highlighting each phrase by chopping the voice off at the end of it.

The backbone: match the transcript, locally

Every audio response streams a transcript of exactly what’s being spoken. That transcript is free, already arriving, and perfectly aligned with the voice — because it is the voice. So highlighting becomes a matching problem, solved on-device, with no model round-trip:

• Build candidates from what’s visible. From the captured viewport we generate highlight candidates at several granularities: whole short lines, sliding multi-word phrases inside longer lines, and — for code — individual distinctive symbols like class and function names. Each candidate carries an estimated rectangle built from line and character geometry, so we know precisely where it sits on screen.
• Match exact word runs, not vibes. A local matcher keeps a short rolling window of the spoken words and lights a candidate only when a contiguous run of its visible words actually appears in that window. Paraphrases don’t match; only words truly on the page and truly spoken do. It’s deterministic, fast, and testable — and it fails toward nothing, which is the safe direction.
• Anywhere in view, not just the read lane. Narration might focus on one column, but the narrator can mention a heading or number elsewhere on the page. So candidates come from the whole visible capture, and a phrase resolves wherever it actually is.

There’s a quiet contract that makes this reliable: the narrator is instructed to quote the exact on-screen wording for a few words before explaining a heading, number, claim, or symbol. That’s not cosmetic. It’s what gives the local matcher exact text to lock onto — the model “chooses” what to highlight by choosing to quote it, with no separate channel that can desync or stall the audio.

The hard part is timing

Here’s the subtlety that ate the most engineering: the transcript arrives faster than you hear the audio. The model can have “said” a sentence in text while the corresponding sound is still queued in your speaker. Feed the matcher the raw transcript and the page lights up ahead of the voice — which reads as broken.

So reveal is paced by audio actually played, not by text received. Using the same playback ledger that powers barge-in, we estimate what fraction of the whole narration you’ve truly heard, and only release transcript up to that point into the matcher. A mark can briefly wait because the words are ahead of the sound — and that waiting is intentional. Two details made this robust:

• Pace against the whole narration, not each chunk. Long explanations stream in several segments. Measuring “fraction heard” per segment makes the reveal lurch at every boundary; measuring it cumulatively across the entire narration keeps it smooth.
• Never match half a word. Releasing audio-synced text in fixed-size slices can cut a word in two, and half a word matches nothing. A small gate holds the trailing partial word until a space arrives, so only whole words ever reach the matcher — and the last word gets flushed when the narration ends.

What happened to the model’s brush

It didn’t vanish — it got demoted to an optional layer on top of the deterministic one. The narrator can still mark a phrase explicitly, but a marked phrase isn’t drawn when the model emits it; it’s resolved to a rectangle and queued at the point in the transcript where it will be heard, then drawn when the audio reaches it. If the page has scrolled or the phrase can’t be found, it quietly resolves to nothing rather than painting a stale box. The deterministic matcher remains the timing layer and the fallback; the model’s marks are a bonus, never load-bearing.

Coordinates expire the moment you scroll

Every rectangle belongs to one captured viewport. The instant you scroll, those coordinates are lies. So scrolling clears generated highlights immediately, and even a small movement marks the current episode invalid so pending transcript can’t later paint marks in the wrong place. The next settled view earns a fresh snapshot and fresh timing. The marks themselves are drawn in a separate click-through overlay that floats above every app, maps global coordinates into the window system, and animates a quick wipe-in and fade so a highlight feels like it’s being spoken, not stamped.

The best version of the feature did less. It stopped trying to be clever and started reading what was already there.

Fewer knobs, on purpose

The same week, we made a product-shaped subtraction too. Scroll had been carrying a second cloud backend option; we hid it from the UI and coerced existing selections to a single, well-tuned default, so the engine choice stopped being something users had to reason about. We also gave asking out loud one clear gesture — tap the Right Command key — with a HUD indicator for whether the mic is listening. Every removed choice is one less way for the product to feel uncertain about itself.

What shipped

Transcript-matched, line-level dynamic highlights went out across a run of releases and became the signature visual on the site: the page lighting up a half-step behind the voice, then clearing the moment you scroll. It also set up the realization behind the next post — because once highlighting was paced by played audio and matched locally, it no longer needed anything special from the model. And if it needs nothing special from the expensive model, maybe narration doesn’t need the expensive model at all.