Concept

Why

The number of seniors is growing, the number of caregivers isn't, and the state wants to shift care into people's homes (the National Strategy for Social Services Development 2026–2030). But there, people are often alone.

I want to build an assistant that helps the people who struggle most with technology. That's why the first version focuses on seniors.

Goals

1. Understanding intent, not commands. Lexie has no fixed set of commands — she has to understand what the person means.
2. Data stays home. Everything about the user is processed and stored locally; nothing goes to the cloud.
3. Internet only when needed. Outbound access opens only for as long as strictly necessary, under fixed rules.
4. Training cognitive ability. Games and activities to maintain memory and attention.
5. Help when something happens. Alerts to family or a dispatch service.
6. A safe link to the physical world. Lexie isn't just a chat — she talks to hardware and controls the home so it stays safe for the resident.

The main problem: understanding free speech

The biggest challenge of the whole project is understanding free-form speech and the imperfections of how this group of people speaks. Everything else (the 3D world, voice, hardware) is craft. This is the open problem I'm working on most right now.

Seniors don't speak in commands. They won't say "turn on the radio" — they'll say "I'm feeling a bit sad today, could you play me something?" They speak more slowly, colloquially, sometimes indistinctly or in dialect. So Lexie has to:

• recognize when a transcription isn't reliable, and ask again rather than guess,
• tell intent apart from chatting — whether the person wants something, or is just talking,
• handle Czech (inflection, colloquial speech) on a small local model that fits limited hardware.

Speech transcription

• Silero VAD ahead of Whisper, so noise, breathing, or mumbling never reach the transcription step — material Whisper can otherwise "invent" a sentence from.
• More signals than just confidence: no-speech probability, average token logprob, and repetition detection, since slow speech with pauses tends to loop.
• Whisper without grammar. whisper.cpp can constrain transcription to an allowed-phrase grammar, but for seniors' free speech that would be dangerous: a sentence outside the grammar wouldn't transcribe as unintelligible — it would get forced into the nearest allowed command instead. Whisper should only be an ear; understanding belongs to the intent model.

Two kinds of uncertainty

I distinguish them because a person should respond to each differently:

• Transcription uncertainty is resolved before the intent model ever sees it. Lexie says "Sorry, I didn't catch that" and the resident repeats the sentence.
• Intent uncertainty (UNCERTAIN) comes from the intent model. The words are clear, but the intent isn't. Nothing runs, and Lexie asks: "Should I play the radio, or did you just want to chat about it?"

Intent vs. chatting

The intent model's structured output goes through a GBNF grammar (llama.cpp) — the model returns only valid JSON, an array of intents typed as COMMAND, CHAT, or UNCERTAIN. The prompt decides meaning; the grammar decides shape. Multiple intents in one sentence ("turn on the lights and close the windows") is then just a one-line grammar change — C++ validates each intent separately, executes left to right, and confirms risky actions (windows, doors, heating).

When a sentence combines emotion and a request ("I'm sad, play me something"), Lexie responds to the emotion first, then the action. Keyword matching in C++ is only a fast shortcut for unambiguous cases, never a substitute for understanding.

Czech, memory, and latency

I'm comparing how various small quantized models (Llama 3.2, Qwen 2.5, Granite) handle colloquial Czech and inflection, with an eye on commercial-use licensing. Because loading a model from NVMe takes seconds — too long between "hey Lexie" and a reply — models stay resident in memory at all times. On the 8GB tier that means a single language model for both intent and conversation: the weights load once, and two contexts run on top of them — a short one for grammar-constrained intent, a longer one for conversation.

How it works

The whole chain runs locally, written in C++ and Qt:

• Everything stays resident in memory — no reloading models.
• Speaker recognition: ECAPA-TDNN, so Lexie knows who she's talking to.
• Games: a deterministic engine drives state and rules; the LLM only generates commentary and reactions — keeping hallucination on a short leash.
• Data: SQLite, local, encrypted.

More than chat: the physical world

Most digital companions for seniors can talk, remind, and call. Lexie also talks to the physical world: controlling lights, heating, windows, doors, and appliances, and watching to keep the home safe. For a senior, the home is where things most often go wrong: a forgotten stove, a door left open at night, a room gone cold.

This is where "the model proposes, C++ decides" matters most:

• The model never talks to hardware directly. Every intent passes through a C++ handler with a fixed list of allowed actions and ranges.
• Risky actions get confirmed. Windows, doors, heating, and appliances are always double-checked with a question first.
• Safety doesn't depend on the model. Safety rules are deterministic and hold even if the language model fails or isn't running — the same logic as railway signaling systems: an unsafe state must never be reachable, no matter where the command comes from.
• Device connectivity via an open Gate API with drivers for KNX, Zigbee, and Shelly. The core that decides stays closed and under control.

Lexie's memory

Lexie needs to remember two different things: the conversation (so "turn it down" refers to the radio she started a minute ago), and the person (gradually learning who their family is, what they like, their habits).

"Learning" here doesn't mean fine-tuning the model — on-device fine-tuning is impractical and, worse, irreversible: whatever the model learns wrong stays wrong. Learning means extracting facts into a database and recalling them at the right moment: overnight, the model proposes fact candidates from the day's conversations, and C++ validates and writes them. Memory has three layers: raw transcripts (kept briefly), daily summaries, and a fact profile. Only what the person actually said gets stored — never the model's assumptions.

• Contradictions and dementia. A resident might say they have two sons one day, three the next. Lexie must never correct them — facts are stored with a date, and older versions are never deleted.
• Sensitive facts (a death in the family, an illness) are never raised by Lexie on her own — she only responds if the person brings it up.

Interface

The interface is built in QML with a C++ backend, on a large screen (42" and up) with an IR touch frame. There's no classic button-based menu.

The central character is Lexie, a black Friesian unicorn mare, who guides the resident through the whole system. She lives in her own 3D world, and that world is tied to the real one: it changes with the weather, time of day, and season.

The world also works as a state machine. Wherever Lexie happens to be in the scene tells a person whether she's resting, thinking, or listening — without reading anything. Seniors read spatial context instinctively. The world also holds memories (photos, family), which don't open from a menu — they simply appear in the environment, echoing the principles of reminiscence therapy.

Privacy and outbound access

• Internet: access is governed by a modified router with rules managed by the C++ layer, not the language model. A connection opens only for as long as necessary, then closes.
• Camera: evaluated locally on-device only, no recording, no cloud. On an event, a local recording is saved, and a stream opens on request — existing only for as long as someone is actually watching it.
• Conversations stay between the resident and Lexie. Neither family nor a dispatch service can access them — dispatch only sees alerts. If the resident says "forget that," it's erased from every layer of memory.
• Family only gets access if the resident grants it themselves.

Hardware

The demo will run on a MacBook M3 with a monitor and an IR touch frame for now — I want to validate interest before investing in final hardware. Longer term, for care homes with multiple rooms, I'm planning a central server (two Jetson AGX Orin units in active/standby) with thin panels for wake word, video, and audio.

Media

• internet radio
• videos and streams
• movies (Premium tier)
• photo gallery and memories

Status & roadmap

The system is under development. The individual pieces (wake word, speech transcription, the LLM, voice) each work on their own, and I'm now putting them together.

Why I'm doing this

I've been programming for over 20 years, from assembler through C++ to C#. I worked on systems for Správa železnic (the Czech Railway Administration), where technology controls an entire station and must not fail. I want to put that same discipline into something that matters.

I want a senior at home not to be alone — and for no stranger to hear what they talk about.