I turned a $40 touchscreen into a voice assistant

For the last few days a 1.8 inch touchscreen has been my assistant. It cost about forty dollars. I say hey jarvis, ask what’s on my calendar or set a timer or check what I spent at Home Depot, and it answers out loud.

The important part is that the device does none of the thinking. It’s a mic, a speaker, a screen, and a radio. The wake word, the speech to text, and the reply all run on my Mac across the room.

It’s like a security camera, which captures the room and sends it somewhere else to be recognized. Once I stopped trying to make the board smart, the whole thing took an afternoon.

The split

There are 3 one way channels between them, all UDP. Mic audio and taps go to port 3333, replies come back on 3334, screen state and text on 3336. That’s the whole contract, so I can swap in a bigger model, or the work version that talks to our CRM, and the firmware never changes.

One detail cost me an afternoon. Any UDP packet over about 1024 bytes vanished, bigger than the wifi frame, and the ESP32 dropped the fragments silently. Smaller packets fixed it.

Getting code onto it

The board runs one Arduino sketch, a single .ino file with setup() and loop(), written in C++ and flashed from the command line with arduino-cli, over the same USB cable that powers it.

The part that fights you is the board definition, one long string naming every hardware quirk: PSRAM mode, flash size, native USB, the partition layout. Get one flag wrong and it still compiles, but the board won’t boot, and nothing on the screen tells you why.

FQBN="esp32:esp32:esp32s3:PSRAM=opi,FlashSize=16M,CDCOnBoot=cdc,USBMode=hwcdc,PartitionScheme=app3M_fat9M_16MB"

arduino-cli compile --upload -b "$FQBN" \
  --libraries waveshare-repo/examples/arduino-v2/libraries \
  -p /dev/cu.usbmodem1101 .

Waveshare ships the screen, audio, and touch drivers as a bundle you point the build at. After that it’s a loop: edit, flash, read the serial log, about a minute each.

One turn

A turn is a straight line. The board streams the mic, the Mac hears the wake word, records until I stop, runs Whisper, and hands the text to the model. Simple asks never reach it. A fast path answers time, calendar, and timer questions in about 1 second, and the rest goes to a small model with tools and comes back in about 4.

The fast path is where I embarrassed myself. My clock shortcut matched “what time is it,” and it also matched “what time is THAT,” so when I asked what time a movie started it gave me the current time. I liked the clever version too much to notice it had become a different question.

Restart is not the same as works

The worst one came at the end. I moved the brain to a background service so it would restart after a crash, and I tested it by killing the process. It came back every time, so I called it done.

It was not done. The service started with an empty PATH, so it couldn’t find the speech to text binary, and every request crashed transcription before the service restarted the same broken process. I had proven it RESTARTS. I had not proven it WORKS, and those are 2 different tests.

The board has no intelligence in it, and that’s the point. It’s a screen and a radio, and the assistant is a program on a Mac that happens to be on. The hard part wasn’t putting a model on the board, it was deciding I didn’t need to. The rest is me pointing it at my own life and asking out loud.