Personal project

Personal AI Agent Infrastructure

An assistant I run myself — my hardware, my tools, my rules. It started in August 2026 as a wake-word experiment and became the thing I actually use: voice at the desk, a watch on my wrist, a root server that holds the tools. Everything on this page is in daily operation, not a slide.

I build it the way I build at work: run it in production, write down why, and keep the failure modes where the next person can read them. What follows includes the decisions that turned out wrong.

Two machines, one assistant

   Nyx — desktop, home LAN          Button — root server, always on
 +-------------------------------+   +-------------------------------+
 | wake word + VAD (on device)   |   | HTTPS gateway (watch, phone)  |
 | speech in / speech out, HUD   |   | headless agent sessions       |
 | speakers, scanner, documents  |<=>| 21 MCP servers                |
 | 4 MCP servers                 |ssh| dashboard collector, services |
 +-------------------------------+   +-------------------------------+

The split is not symmetry, it is reachability. Tools that need the home network — the speakers, the scanner, anything else with a local address — only work from the desktop. Everything else lives on the server, because the server is the one that is always on. A session hook tells every agent session which of the two machines it woke up on, since the answer changes what it is allowed to do.

Requests are routed between them by keyword match, not by a model call. Asking a model where to run the request would have cost an extra round trip per turn and eaten exactly the savings that moving to the local CLI had produced.

Ways in

Voice Wake word and voice-activity detection run on the device — audio only leaves the machine after the wake word. Barge-in is not a feature, it falls out of the thread design: the main thread never blocks, so it keeps listening while the assistant is still speaking.
Smartwatch A Wear OS app posts to an HTTPS endpoint on the server. It gets 202 in ~15 ms instead of holding the connection for the ~25 s a full turn takes — a request left open that long survives neither Android's Doze nor a change of network on mobile.
The way back The answer returns as a push data message, deliberately not a notification: a notification would be drawn by Android without ever waking the app, and the app is almost always gone by then.
Phone An Android app of its own, same path into the gateway.
Desk The CLI directly, plus a small always-on window that shows whether the assistant is idle, listening, thinking or speaking.
Music Takes a fast path around the model entirely: 0.03–0.26 s instead of 7–33 s, acknowledged with two short tones instead of a spoken sentence — you can hear the result anyway.

The tool layer

Everything the assistant can actually do sits behind the Model Context Protocol: 21 servers on the root server, 4 on the desktop. Messaging, tasks and notes, media and home devices, health and training data, documents, a browser, a literature search.

Five of them I wrote from scratch, because no usable server existed — among them the speakers, a chat platform, and two device integrations whose vendors offer no public API at all. Three more are upstream projects carrying local patches. Both live in their own repository, separate from where they run: the running installs hold tokens and session data and are deliberately not synced, which made them the only copy of work that could not be reinstalled.

That inventory is also why the personal dashboard collects through the same MCP servers rather than calling a dozen vendor APIs itself. The logins, token refresh and local repairs already exist once. Doing it twice means breaking twice.

One chain, end to end

 watch --> phone (Health Connect) --> self-written Android app
       --> gateway --> SQLite --> MCP server --> agent / dashboard

The clearest example of how far it goes. The vendor offers no consumer cloud API, so the missing link got written: an Android app that reads the phone's health store and pushes to the server every 30 minutes. From there the data is a normal MCP source — the assistant can answer from it, and the dashboard renders it, without either of them knowing where it came from.

Memory and configuration

Three configs One private, two for work, on one desktop, selected by environment variable. Anything shared — hooks, status line, memory — lives in a directory outside all three and is referenced by absolute path. No config may point into another one; that rule is what keeps an edit from silently changing nothing.
Persistent memory One file per fact, with a short index loaded into every session. The stores are symlinked across all three configs, so what one setup learns, the others know.
Guardrails as hooks Not as good intentions. A pre-tool hook refuses a git push to a branch whose pull request is already merged or closed — it checks the state, which no linter does.
Written-down decisions Every project carries a file for the agent that records decisions and dead ends rather than usage — the assistant's has grown to 120 KB. It is the reason a session six weeks later does not re-litigate a choice that was already made and measured.
A worktree per task Several agent sessions work in the same repository at once, some of them for a second person on Windows who does not use git and gets a stricter pull-request rule because of it. One session landing in another's branch is a collision that is hard to untangle — so no session ever branches in the main checkout.

What I got wrong

"Cheap model first, escalate when unsure" is the wrong default. Triage sent every turn to the small model first and escalated whenever the answer did not match the expected format. That threw away a correct cheap answer and replaced it with a fabricated expensive one. It now escalates only on an explicit marker. Escalating on ambiguity quietly prefers a possibly-wrong expensive answer over a possibly-right cheap one.
Cost against latency, decided consciously rather than tuned. Running turns through the local CLI instead of the API took every answer from ~3 s to ~7–10 s. Accepted on purpose. The gap is covered by a spoken acknowledgement and soft beeps, and those phrases are cached as audio files keyed by a hash of the voice config — so changing the voice invalidates the cache by itself, instead of leaving the old voice to leak out at startup.
Local-first is not automatically better. Fully local speech synthesis was built, measured and then dropped: cheap, and it sounded bad. Wake word and detection stayed on the device, because there the local option is also the better one. "Runs locally" is an argument about privacy and cost, not about quality, and mixing those up costs an evening.
Check what already exists before building it. A startup health check was about to be written from scratch. The dashboard had been publishing an ok field every 30 minutes for weeks. Reading it costs 0.4 s and nothing to maintain.
A pipeline of replaceable stages beats one integrated API. A realtime speech API would be a second or two faster per turn. But when an answer is bad I can see in the log whether the transcription or the model was at fault, every stage can be swapped for one new file and one line, and a provider change is not a rewrite. That trade is the same one I argue for at work.

What it runs

A personal dashboard whose collectors read through the MCP layer; the web server computes nothing, so one dead source never empties a page
This site, built end to end through the agent itself
A tool ecosystem for an ongoing tabletop campaign — bots, recaps, an image and audio pipeline
Community servers and containers that have been running since 2011 and 2020
A spare-parts overview for a family member, produced by driving a browser through the agent and published behind basic auth

None of this is a product evaluation. It is the same habit I bring to engineering: build it, run it in production, measure it, and write down why — including the parts that did not work.