Locking a voice to your eyes

The pitch is one sentence — “it reads your screen as fast as you scroll.” The engineering underneath it is mostly about timing: what to read, how to know where you’re looking, when to speak, and how to stop the instant your eyes move on.

In the first post we set the law: stale narration is worse than sparse narration. This post is how the loop is built to honor it. We’ll walk it in order — capture, attention, pacing, voice, and the ledger — and explain the methodology at each stage. We’ll keep the exact tuning to ourselves; the approach is the interesting part anyway.

Stage 1 — Read the screen, not the pixels

The first real decision was to extract text on-device and never stream video frames to the model. Feeding a screenshot on every scroll would add latency, cost more, and — worst of all — be unreliable exactly where it matters: dense legal text and code. It would also push images off your machine continuously, which is a privacy non-starter for the documents Scroll is meant to read.

So capture is a router with two providers and a referee between them:

• Accessibility first. macOS apps can expose their content as a structured tree — headings, paragraphs, list items, table cells, code — each with its on-screen frame. When that tree is healthy we read it directly. It’s instant, semantically rich, and knows where every line sits. For Chromium- and Electron-style apps we nudge the accessibility interface awake first, because those trees are often dormant until something asks for them.
• On-device vision as fallback. When an app exposes a hollow or useless tree — canvas-rendered apps, image-only PDFs — we capture the window’s pixels and run Apple’s on-device text recognition, with line boxes, entirely on your Mac. It’s slower and less structured, so it’s a coverage net, not the default.
• A confidence-scored referee. Every capture is scored — how much text, how much of the viewport it covers, how trustworthy the bounds are — and the router picks the source that earns it. Accessibility wins when it’s solid; vision wins when it’s clearly more trustworthy. Either way, the result carries its own diagnostics, so everything downstream knows how much to trust what it’s about to narrate.

Finding the one column worth reading

A real page is mostly chrome: nav bars, sidebars, comment rails, related links. Reading all of it aloud would be worse than useless. So once we have the visible blocks, a selector scores each one for how “body text” it looks — length, prose-like sentence shape, whether it’s a heading, how central it is in the viewport — and penalizes the narrow, tiny, top-of-window fragments that tend to be toolbars. The highest-scoring block anchors a reading lane, and we pull in the blocks that share its column. That one selector is the difference between “Scroll read me the article” and “Scroll read me the sidebar.”

One normalized unit, hashed for memory

Everything past capture speaks one vocabulary: a normalized content block with a type, its text, its vertical position, optional per-line frames, and a content hash. That hash — computed after normalizing whitespace and case — is quietly load-bearing. It’s how Scroll dedupes blocks across captures, remembers what it has already read so it doesn’t repeat itself when you scroll back, and even resumes a document across sessions without re-reading the parts you already heard. A small idea doing a lot of work.

Stage 2 — Model the attention, not just the scroll

“Where are your eyes” is unknowable, so we estimate it from the things we can see: a reading line, your dwell, and your pointer.

• A focus line, not the top of the page. By default the anchor is a horizontal reading line slightly above the vertical center of the viewport — roughly 40% down — because that’s where eyes sit when skimming, not at the very top edge.
• Dwell accumulates intent. An attention tracker adds up how long each block lingers in the reading zone. A block that sits under your eyes for a beat earns weight; a block that flickers past doesn’t. That dwell signal is what lets “stopping” mean “go deeper here.”
• Pointer intent, debounced. If you move the mouse to a spot and pause, that’s a strong hint about what you care about. But re-anchoring on every twitch would make the product nervous, so pointer intent only re-anchors when the cursor moves meaningfully and settles. Small moves nudge; only a real move-and-pause resets the lane.

Together these turn a raw scroll offset into something closer to a guess about what you’re actually trying to read — which is what the next stage needs.

Stage 3 — Decide when to talk (the pacing controller)

This is the part that earns the product, and the part the original blueprint flatly labeled “the secret sauce,” so we’ll describe the method and keep the dials in the drawer. The pacing controller is a small, pure, heavily-tested state machine. It watches a smoothed estimate of your scroll velocity and direction and classifies the moment:

Three principles keep that from feeling laggy or chatty:

• Narrate only on proven intent. Content scrolling into view is never, by itself, a reason to speak. A settle, a sustained steady pass, a dwell — those are. This is the rule that protects both the experience and the bill, since generated speech is the dominant cost.
• Read-ahead hides the model’s think time. The realtime model reasons before it speaks — first audio lands around a second in, not instantly — so the controller works slightly ahead of your scroll position, preparing the next block before it reaches the focus line. Done right, the audio is already arriving when your eyes get there.
• Backpressure coarsens; it never queues. If you out-run the narrator, the answer is not to build a backlog you’ll be hearing thirty seconds from now. It’s to drop to a coarser summary and stay current. The ear must never lag the eye, so when in doubt the controller says less, sooner.

The net effect is intentional: Scroll is not a continuous reader. It waits for meaningful moments, uses the audio still in your speaker as backpressure so it never talks over itself, and would rather say nothing than say something stale.

Stage 4 — Speak it, over a raw socket

For narration we drive a realtime speech model directly. The transport choice is worth a paragraph because it looks wrong until you see why.

The API can be driven over WebRTC or a plain WebSocket. WebRTC is the easy default for voice apps because it solves microphone capture — echo cancellation, jitter buffering, packet loss. But narration has no microphone input: it’s text in, audio out. WebRTC’s benefits buy us nothing and cost us a heavy native dependency.

That choice comes with sharp edges worth documenting. The realtime session is configured audio-only — asking for text and audio together disables the audio — so captions are read from the audio transcript the model streams alongside the speech, not from a second channel. Output is raw PCM audio at 24 kHz, which we schedule into our own playback queue. Get one of those details wrong and you get silence with no error, which is exactly the kind of thing you only learn by shipping.

Stage 5 — The ledger that makes interruption honest

Here’s the piece that never demos but decides everything. When you scroll away and the voice has to cut off, we have to tell the model exactly how much it actually spoke — not how much it generated. Those differ, because audio lives in a local queue between “generated by the model” and “heard by you.” Truncate at the wrong number and the model’s memory of what it said drifts from reality, and the next thing it says makes no sense.

So Scroll keeps an audio playback ledger: per response, it tracks what was generated, what has actually played, and where the local queue sits. A clean barge-in then becomes a precise sequence: invalidate the stale audio, cancel the in-flight response, ask the server to truncate the conversation to the milliseconds you actually heard, flush the local queue ourselves, and return the HUD to listening. (Clearing the local queue is on us — the server-side clear is a WebRTC-only call that does nothing over a socket, another lesson best learned once.)

The metric that actually decides the product

It’s tempting to optimize first-audio latency. But that’s a means, not the end. The number that decides whether Scroll feels magic or broken is stale-audio duration: how many milliseconds of narration keep playing after you’ve visually moved on. We built a meter for it — it counts exactly that, off the ledger — and we watch it like a north star.

First-audio latency is what you measure. Stale-audio is what you feel.

With read-ahead and a model tuned for low think-time, continuous scrolling felt attached to the eye. The plan, at this point, was to drive stale-audio to zero by interrupting the voice the instant the page moved. That plan did not survive contact with real use — which is the next post.