arthur@homelab:~$ cat ~/posts/state-of-self-hosted-ai.md
What Does Self-Hosted AI Actually Mean?
“Self-hosted AI” compresses several very different systems into one phrase. Running a chat interface at home is not the same as running inference at home, and neither is the same as training a model from scratch.
I find it more useful to separate four layers.
1. The interface
A project such as Open WebUI can run in the homelab while talking to local models, cloud APIs, or both. Self-hosting the interface gives me control over access, provider configuration, backups, and the primary conversation database.
It does not mean every prompt remains local. The selected provider still determines where inference happens.
2. Retrieval and personal context
A locally operated knowledge base can index notes, documentation, and other material I am allowed to use. Retrieval adds relevant passages to a prompt; it does not retrain the underlying model.
For personal use, this is often more valuable than training. Good retrieval lets a general model reason over the specific names, decisions, and worklogs in my second brain while keeping the source material under my control.
3. Inference
Local inference means model weights and prompt processing run on hardware I operate. That provides a stronger privacy boundary and works without a cloud provider, but model size, memory bandwidth, power, latency, and software support become my problem.
Evaluation should use the work I actually do, not just a public leaderboard. A useful test set might include:
- answering questions from my homelab documentation;
- editing a configuration without losing important constraints;
- summarizing an incident accurately;
- producing code that passes an existing test suite;
- refusing to invent details missing from the notes.
4. Training
Training a blank general-purpose model on Wikipedia, books, blogs, and video transcripts is a research-scale data and compute project. It also raises licensing, provenance, filtering, and evaluation questions long before the first useful checkpoint appears.
Fine-tuning a smaller existing model for a narrow behavior is more accessible, but it is still not a substitute for retrieval when the goal is current factual memory.
The useful question
Instead of asking “Is this self-hosted?”, I ask:
Where is the UI?
Where is the system of record?
Where do prompts travel?
Where does inference run?
Who controls the model weights?
What stops working without the internet?
That vocabulary makes hybrid systems easier to describe honestly. I can value a local interface and database without claiming cloud inference is private. I can use a local model for sensitive or offline tasks and a hosted model when capability matters more.
The state of self-hosted AI is not one winner replacing the cloud. It is a toolbox of boundaries, and the right architecture depends on which boundary I am actually trying to control.