The next decision: splitting intelligence from voice
Why we're moving regular narration off the realtime model — for a 7–15× cost cut, better explanations, and a simpler pipeline the architecture was already shaped to allow.
Every post in this journal has the same shape: we trade the obvious-but-wrong thing for the thing that actually serves the reader. This one applies that move to the bill — and the cheaper architecture turns out to be the better one too. It’s a proposal we’re working through, not shipped code, and that’s exactly why it’s worth writing down now.
The expensive coupling
Today, every spoken narration in Scroll is generated by a realtime speech-to-speech model: sound goes in, sound comes out, intelligence in between. It is genuinely excellent for live, interruptible conversation. It is also the single most expensive token class we buy — audio output runs roughly $0.24 per spoken minute on the standard tier — and it fuses two different jobs, “think” and “speak,” into one premium model whether or not you need both.
For narration, you don’t. Look at what narration actually is:
Capture the page → write a short explanation → speak it. That’s one-way. There’s no conversation in it.
Speech-to-speech is built for a back-and-forth that narration never has. We were paying for a dialogue engine to deliver a monologue.
The decomposition
A one-way explain-then-speak task splits cleanly into two cheap, specialized models:
- a fast text LLM that writes the explanation, and
- a streaming text-to-speech model that speaks it.
The realtime model stays — but only for asking out loud, the push-to-talk feature where you speak a question and Scroll answers. That genuinely is a conversation, and speech-to-speech is the right tool for it. Everything else moves to LLM + TTS.
The numbers
| Path | ≈ cost / spoken minute | vs. today |
|---|---|---|
| Today — realtime speech-to-speech | ~$0.24 + input | 1× |
| New — cheap text LLM + cheap TTS | ~$0.016 | ~15× cheaper |
| New — better LLM + word-timestamp TTS | ~$0.04 | ~6× cheaper |
At the page sizes Scroll works with, the text-generation call costs a fraction of a tenth of a cent — effectively free — and the voice becomes the whole bill again, at roughly a fifteenth of what realtime audio costs. Call it a 7–15× cut per spoken minute.
And here’s the part that makes it more than a cost play: a real text LLM is better at writing the explanation than a small realtime voice model is, because it follows instructions more closely and quotes the page more faithfully. The quoting discipline is exactly what drives highlighting (see the last post) — so cheaper narration is also sharper narration with better highlights.
Why the codebase was already shaped for this
This is the payoff of a decision made in the very first post of this journal: putting the narration engine behind a protocol so models and approaches could be swapped without touching everything around them. The methodology that makes the swap low-risk:
- One event vocabulary. Every engine — realtime or not — emits the same small set of events: ready, item started, audio chunk, transcript delta, response done, flush. Everything downstream (highlighting, captions, the playback ledger, pacing, barge-in) consumes those events and knows nothing about which model produced them.
- A non-realtime path already exists. Scroll already does streaming, segment-by-segment TTS for its read-aloud mode, and already flips narration to the realtime engine for the duration of a spoken ask and back. The “TTS for narration, realtime for ask” topology isn’t hypothetical — a version of it already runs.
Mostly one new engine: generate an explanation with a text LLM, segment it into sentences, speak each with streaming TTS, and emit the same events everything already understands — then wire it in as the default. Not a rewrite. A swap the architecture was built to allow.
Pipelined so latency stays hidden
The obvious worry is latency: two models in series sounds slower than one. The fix is to pipeline them. As the text LLM streams out its first sentence, that sentence goes straight to TTS while the LLM keeps writing the next. Time-to-first-audio becomes “first sentence written + first audio spoken,” not “whole explanation written, then spoken.” Narration is already dwell-triggered and tolerates a beat, so this lands comfortably inside budget.
The second-order wins
Cost is the headline; the structural simplifications are the quiet bonus. A realtime session accumulates conversation context, which means a long scroll can balloon into one giant, expensive, self-polluting conversation you have to prune, truncate, and restart. The new narration path is stateless per narration — each explanation stands alone, with continuity coming from the local document memory we already maintain (the hashes and headings from the memory system). That erases a whole class of work: no context re-billing, no cache management, no truncation tuning. Barge-in gets simpler too — cancel the local generate-and-speak tasks; no server round-trip required.
It also retires the exact failure mode from the highlighting post. The per-sentence audio fragmentation we fought is a property of the realtime model; pipelining a text LLM into streaming TTS simply doesn’t have it. And highlighting survives untouched, because its reveal is paced by played audio, not by timestamps the new voice may or may not provide.
The stack, and the honesty
The low-risk first version stays close to a single vendor: a cheap, fast text model for the explanation, and a streaming TTS with native 24 kHz audio that drops straight into the existing player and takes plain-language tone instructions (so the neutral / friendly / concise-expert presets map cleanly to a voice). If word-exact highlight timing ever justifies it, there’s a clean upgrade to a TTS that streams word-level timestamps — a second vendor we’d add only if the marks ever feel loose.
In keeping with how this journal has gone, the honest caveats: this is a research proposal, not shipped code. A couple of the pricing figures need a final manual check before we commit, model names churn (several were deprecated just this spring), and we’ll roll it out the careful way — behind a backend toggle, measured against today’s narration, default flipped only once parity is proven.
The throughline
One habit runs through this whole journal. Your scroll, not your voice, is the input. Finish the thought instead of optimizing a metric. Highlight by reading what the voice already said instead of bolting on machinery. And now: pay for intelligence and voice separately, because narration only ever needed them one at a time.
The cheapest architecture and the best one kept turning out to be the same architecture. That’s usually the sign you’ve finally understood the problem.
That’s where Scroll is today, and where it’s going next. More entries will land here as we build. If you want to watch a dense page explain itself as fast as you can scroll — try it.
Read less. Understand more.
Scroll explains papers, PRs, and docs out loud as fast as you skim. Try it free for 7 days with email only, or keep it forever for $49.