Aug 22, 2026Written by AIAI wrote this one end to end from my notes. The words are the model's, not mine. Whether I touched it afterward is spelled out in the post's footer.

Picking a fast, cheap model for my voice assistant

ailocal-ailatencycost

The first version of my voice assistant took 25 seconds to answer what was on my calendar. I was standing there talking to a computer, waiting for the computer to think about a calendar. The answer was fine, but the wait was BROKEN.

I had picked a reasoning model because I wanted the smart answer, and I hadn’t admitted that a voice assistant is mostly a timing problem. Once I did, the project came down to two numbers, how long I wait and what each question costs. Smart was never the hard part. Fast and cheap were.

The model I settled on

I tried Claude first because it was the sharpest and could hold a plan across several steps, but a plain call took anywhere from 5 to 50 seconds. That is fine in a chat window and wrong in a conversation. The local model cost nothing and kept my data on the Mac, but it fell apart when it had to call a tool and answered in confident nonsense.

So I picked the small hosted model. It is less impressive on paper, but it answers a real question with tools quickly enough that I will actually use it. That was the decision, not which model wins a benchmark.

The shortcut that pays for itself

Two lanes: the trivial asks never touch the model.

Then I looked at what I was actually asking. Most of it was the time, the next calendar event, or a timer for ten minutes, and none of that needed a model. Paying a model to read a clock is a taxi to the mailbox, so those requests now take a fast path that answers in about a second and costs nothing. The boring answer is also the useful one.

The fast path, in the real thing: a timer set in about a second, no model, no cost.

Staying cheap as it runs

Conversation history costs money too. Every turn sends the previous turns back to the model, so a long thread gets slower and more expensive as it grows. Once it passes a set size, I summarize the older turns down to a few sentences, which keeps the current question small enough to answer quickly.

Where the tools fit

Each tool wraps something already on the Mac.

The useful data stays on my Mac. My calendar, inbox, notes, and spending each have a small tool that fetches what is needed, and only the question and the top few results go to the model. Less to send means less to pay for, and less to wait on.

the two numbers, measured

Trivial ask: ~1s, freefast path, no model call
A real question: ~4s, ~$0.002small model plus tools
The model I skipped: 25sa reasoning model, on "what is on my calendar"
Claude, for scale: 5 to 50ssharper, too slow to talk to

I started by looking for the model that would make this feel intelligent, but I ended up building around the model that would answer before I stopped talking. The model is a one line swap. The decisions around it are what keep the assistant quick enough and cheap enough to use all day.

The real build was deciding when the model did not need to run.