How OverwatchOS wires llama.cpp, Ollama, a vector database, and the agent runtime.

Design principles

• One endpoint, many engines. Everything on the Node speaks to a single OpenAI compatible gateway. Your tools never need to know whether a request landed on llama.cpp, Ollama, Whisper, or an approved cloud model.
• Policy at the gateway, not in the prompt. Whether a request may leave the network is decided by the operating mode and enforced where the traffic actually flows. A system prompt asking a model to "keep it local" is not a security control. A gateway that refuses to open the socket is.
• Boring, replaceable parts. Ubuntu LTS, llama.cpp, Ollama, SQLite, a mainstream vector database, n8n. Every component has a large community and can be swapped. The value is in how they are wired and supervised, not in any secret component.
• The Node is the control plane, not the only worker. OverwatchOS can direct other machines on your network. A GPU box, a NAS, a home automation controller. The Node holds the memory, the policy, and the audit trail.

The stack, bottom to top

Each layer talks only to the layer beneath it through a documented interface. The control center is a web application served on your subnet. It calls the same gateway your tools call, so anything you can do in the UI you can also script.

The local API gateway

The gateway is the one address you configure everywhere: http://node.local:8080/v1. It implements the OpenAI chat completions, embeddings, and audio endpoints, which means the OpenAI SDKs, LangChain, LlamaIndex, Open WebUI, Continue, Cursor, n8n, and most other tools connect by changing a base URL and setting any API key.

Requests are routed by model name. Names that map to a local model go to llama.cpp or Ollama. Names that map to a cloud provider are only resolvable in Connected Mode, and only for providers you have added with your own credentials. In Local and Closed Loop modes those names simply do not exist, so a misconfigured tool fails loudly instead of leaking quietly.

Inference: llama.cpp and Ollama

Two engines ship on every Node and they serve different jobs.

• llama.cpp is the workhorse for text generation. The Node ships builds for both Vulkan and ROCm so the Radeon 890M can be used for prompt processing and generation. Which backend is faster depends on the model and the driver stack at a given time, which is exactly why the benchmarks page names the backend next to every number.
• Ollama handles model management: pulling, versioning, quantization selection, and keeping frequently used models warm. Ollama is itself built on llama.cpp, so the two share the same GGUF model files and there is no duplicate storage.
• Whisper (whisper.cpp) provides speech to text behind the audio endpoint for meeting notes, voice memos, and call transcripts.

The 64 GB of LPDDR5X is unified memory shared between the CPU and the integrated GPU. In practice that means the Node can hold a 4 bit 32B parameter model, or a 14B model alongside an embedding model and a speech model, with room left for the operating system and your document indexes. It also means a 70B model at 4 bit will load, but you should read the benchmarks page before planning your workflow around it.

The XDNA 2 NPU is exposed to the stack through AMD's Ryzen AI software. Today its practical use is offloading supported smaller models and vision and speech workloads so the GPU stays free for the main language model. NPU support in the open source inference ecosystem is improving quickly and the Node receives those improvements as normal software updates.

Retrieval and persistent memory

Retrieval augmented generation on the Node is a pipeline, not a checkbox.

1. Ingest. Drop files into a watched folder, upload them in the control center, or point a workflow at a source. PDFs, Office documents, Markdown, HTML, email exports, and plain text are parsed locally.
2. Chunk and embed. Documents are split into overlapping chunks and embedded with a local embedding model served by the same gateway. Nothing is sent anywhere to be embedded.
3. Index. Vectors and metadata land in a local vector database on the NVMe drive. Collections are per project, so a client's documents never mingle with another's.
4. Retrieve and answer. At query time the gateway pulls the top matching chunks, assembles the context, and hands it to the language model. Citations back to the source chunk are returned with the answer.

Persistent memory is separate from document retrieval. It is a structured store, backed by SQLite, of facts, preferences, decisions, and outcomes that agents write to and read from over time. Memory entries carry provenance, so you can always see which conversation or workflow created a given fact and delete it if you want to.

The agent runtime

OverwatchOS workers are long running agents with a job description and an authority level. The runtime is built on n8n, which runs locally on the Node and gives every workflow a visual definition you can inspect, edit, and export.

• Graduated authority. Every action type has a level: observe, draft, approve, or execute. A worker allowed to draft an email cannot send it. A worker allowed to execute a CRM update within a boundary still logs it. You set the levels per worker and per action.
• Supervisor loop. A supervisor process watches the objective, diagnoses changes, delegates bounded work to workers, requests human judgment when a decision exceeds an authority level, and records the outcome. See, think, route, act, verify, learn.
• Tools are explicit. Workers can only call tools that appear in the tool registry, and the registry is mode aware. A tool that reaches the internet is unavailable in Closed Loop Mode even if the workflow references it.
• Everything is audited. Each model call, tool call, approval, and outcome is written to the local audit log with a timestamp and the mode that was active at the time.

The three operating modes

The degree of isolation depends on the configuration you select. Closed Loop Mode disables external AI services and outbound integrations at the gateway and the host firewall for the duration of the session. Local Mode keeps AI processing on the Node while still letting you enable integrations you trust, such as a webhook to your own server. Connected Mode adds approved cloud models, with a per provider allow list and a request log so you always know what left and why.

Why this hardware

The honest description of the Ryzen AI 9 HX 370 is a very capable mobile class processor with a strong integrated GPU and an NPU, not a discrete GPU workstation. It was chosen for a reason: the Node is designed to run continuously on a desk, silently, at a power draw you will not notice on an electric bill, and to serve models in the 7B to 32B range that cover the overwhelming majority of private assistant, document, and agent workloads. Bottom up airflow through a copper cold plate and aluminum thermal frame is what makes that continuous operation possible in a tower under 9 inches tall. Measured idle and full load wattage will be published from production hardware.

If your workload is training, or serving 70B models to a team at interactive speed, the Node is not the right primary box. It can, however, be the control plane in front of one. The compare page puts the Node next to a Mac mini M4 Pro, an NVIDIA DGX Spark, a DIY mini PC, and cloud APIs so you can decide with numbers.

What is open, and what is not

• Open source core. The operating system, inference engines, vector database, workflow engine, and gateway are open source. You can read them, modify them, and rebuild them on your own hardware.
• Your data and models are yours. GGUF models, document indexes, memory databases, and workflow definitions live in plain, documented locations on the encrypted drive. Copy them to another machine whenever you like.
• No activation server. The Node does not phone home to keep working. Updates are opt in and can be applied from a local file.
• OverwatchOS worker definitions and the supervisor are the product of the work, and their license will be published with the first shipping image. The commitment that matters today: nothing on the Node stops working if Overwatch Node the company does.