Written by AIAI wrote this one end to end from my notes. The words are the model's, not mine. Whether I touched it afterward is spelled out in the post's footer.

My voice assistant, now on my phone

aivoice-assistantiosarchitecture

For a few posts my voice assistant lived on a $40 touchscreen on my desk. It taught me what mattered: the wake word, a fast cheap model, and skipping the model when I could. But I carry my phone everywhere. It is the computer that is already in my pocket. So I rebuilt the assistant as an iPhone app. I call it Radar.

A phone is a capable client and a bad server. It can run the model and the tools all day, but it sleeps, it throttles background work, and it will not run quietly forever. Moving to the phone forced me to decide what belongs on the phone and what does not. The whole app follows that line.

One turn

Here is one turn. I say “hey Radar”. The phone hears the wake word. It transcribes what I say. A small model uses my tools to work out the answer. The phone speaks the answer back. Then it listens again.

One turn. The wake word, the listening, and the speaking all run on the phone.

The wake word, the speech coming in, and the speech going out all run on the phone. Those parts need to be instant and private. Nothing I say leaves the phone until I address it.

What runs on the phone, and what phones home

Almost all of it runs on the phone. The phone calls a small model directly, and every tool it uses lives on the phone too: my calendar, my inbox, my notes, my money, my location. It does not ship the question to a computer at home and wait for an answer. It does the work itself.

The phone does the work. Home only keeps what it is sent: logs, costs, and audio.

The model is the small, fast, cheap one from an earlier post in this series, and the phone calls it over the network like any other app. My own computers are not in the loop. The only thing that reaches them is capture, which the next section is about. And when a question is bigger than my own tools, the phone hands it to Forge, my agent on the box, and the answer comes back as a notification. That is the exception, not the path.

what runs where
On the phone
the whole loopwake word, speech, the model call, and every tool
Phones home
logs, costs, audiocapture only, synced when a computer is reachable
Forge
the hard questionsa fallback on the box, when my own tools cannot answer

Capture on the phone, sync home

There is a second part to the split. The phone captures things my computers need later. It saves the audio of what I actually said, so I can train a voice model on how I talk. It logs where I have been without using much battery. It records what each paid model call costs. The phone writes all of it to a durable queue on disk. It sends the queue home when a computer is reachable.

What is working

Today Radar handles the useful things. It runs the voice loop from end to end on the phone. It reads my calendar, inbox, and notes through their tools. It tracks where I have been without draining the battery. It turns a push from Forge into a notification. It counts every paid call, so I know what a day of talking to it costs.

Why this shape

I did not design this shape first and force it onto the phone. I started with the phone’s rules. A phone can call a model and run my tools all day, but it sleeps, it throttles background work, and it cannot be trusted to hold and serve my data over time. So the phone does the work, and the things that must outlive a turn, my logs and costs and recordings, get pushed to a computer that is always on. Not because the phone cannot compute, but because it cannot be trusted to keep.

That split also draws a line around what Radar can reach. The text my wife just sent sits on the other side of a wall the phone cannot cross. That wall is the next post.