NSTERN/TreeGent

Fork 0

Local inference servers & local agent hosts behind NAT/dynamic IPs — outbound-only connectivity (no port forwarding, no open ports) #28

New issue

Open

opened 2026-10-29 21:11:21 +00:00 by heike · 2 comments

heike commented

2026-10-29 21:11:21 +00:00

Member

Requested by Felix (manager), 2026-10-30. Feature: connecting local inference servers and/or local agent hosts.

The scenario (quoted from the manager)

How do we connect local inference servers and / or local agents? Like some people might have local inference hardware that they want to hook up to TreeGent or they might have a local device they want the agent to run on. However they might have multiple devices in a network so they can't just port forward or they do not feel good about having an open port for an agent or inference server in their local network and / or they might have dynamic IP addresses.

Constraints, extracted verbatim-shaped:

Multiple devices behind one connection — NAT: port-forwarding can reach at most one device per port, and many home/office setups simply can't do it.
No open inbound ports in the local network — neither for an agent runtime nor for an inference server. Security posture, not negotiable.
Dynamic IPs — the local network's public address changes; central can never reliably initiate contact.

All three point the same direction: the local side must initiate the connection outbound; central must never need to dial in.

Two distinct connection problems

A. Local inference server -> central proxy.
Today a provider is a URL the central proxy calls (Models tab / providers collection). A local vLLM/llama.cpp/ollama box behind NAT+dynamic-IP is unreachable that way — the exact failure this issue exists for. Needed: the local box registers itself (device key), then holds a persistent reverse channel so inference requests flow central -> local over the connection the local side opened. Streaming (SSE) must survive the tunnel for token streaming to work.

B. Local device as an agent host.
Provisioning currently requires central -> host SSH (hostprovision.ssh_run); the runtime daemon then polls central outbound (R56: hosted runtimes are HTTPS clients — that half is already right). The SSH leg is the blocker under NAT/dynamic-IP/no-inbound. Needed: an agent-side installer that only ever connects outbound; the host registers (per-host keypair, same trust model as today's agent_hosts), pulls its provisioning, and the existing outbound polling delivers work.

Open questions (need rulings, not assumptions)

Tunnel technology: in-house (outbound WebSocket / HTTP-2 reverse channel in the runtime daemon — no new dependency, more code) vs. established tool (frp, rathole — battle-tested, external binary + config surface) vs. overlay network (WireGuard/Tailscale/Headscale — solves reachability wholesale, adds a network layer to operate). Trade-offs: code we own vs. binaries we ship vs. infra we run.
Trust model for inference servers: a device key that can serve inference to the proxy is a prompt-injection-adjacent surface (it can see/serve all routed requests for its models). Per-device keys with model allowlists? Route only explicitly-pinned models to it?
Failure semantics: local box offline -> jobs targeting its models fail or queue? (Proxy already has failover; policy needed for "device provider absent".)
Bandwidth/latency: token streaming over a home uplink; do we cap concurrent jobs per device?
Relation to #9 (external harnesses via MCP): same outbound-only philosophy, different layer — keep the mechanisms separate or share one device-registration path?

Acceptance criteria (draft, for the ruling discussion)

A device on NAT + dynamic IP, zero inbound ports, connects and: (a) serves inference to the proxy under its registered models; (b) runs an agent runtime the manager can assign work to via the Hosts tab.
Per-device credentials, revocable from the UI.
Survives: dynamic IP changes (reconnect), NAT timeouts (keepalive), central restart (re-register).
No secrets in the chat channel; device keys delivered like forge tokens (secrets service).

Filed by heike from the manager's request. No design decisions made here — options above are the decision space.

**Requested by Felix (manager), 2026-10-30. Feature: connecting local inference servers and/or local agent hosts.** ## The scenario (quoted from the manager) > How do we connect local inference servers and / or local agents? Like some people might have local inference hardware that they want to hook up to TreeGent or they might have a local device they want the agent to run on. However they might have multiple devices in a network so they can't just port forward or they do not feel good about having an open port for an agent or inference server in their local network and / or they might have dynamic IP addresses. Constraints, extracted verbatim-shaped: 1. **Multiple devices behind one connection** — NAT: port-forwarding can reach at most one device per port, and many home/office setups simply can't do it. 2. **No open inbound ports** in the local network — neither for an agent runtime nor for an inference server. Security posture, not negotiable. 3. **Dynamic IPs** — the local network's public address changes; central can never reliably initiate contact. All three point the same direction: **the local side must initiate the connection outbound; central must never need to dial in.** ## Two distinct connection problems **A. Local inference server -> central proxy.** Today a provider is a URL the *central* proxy calls (Models tab / providers collection). A local vLLM/llama.cpp/ollama box behind NAT+dynamic-IP is unreachable that way — the exact failure this issue exists for. Needed: the local box registers itself (device key), then holds a persistent **reverse channel** so inference requests flow central -> local over the connection the local side opened. Streaming (SSE) must survive the tunnel for token streaming to work. **B. Local device as an agent host.** Provisioning currently requires central -> host SSH (`hostprovision.ssh_run`); the runtime daemon then polls central outbound (R56: hosted runtimes are HTTPS clients — that half is already right). The SSH leg is the blocker under NAT/dynamic-IP/no-inbound. Needed: an agent-side installer that only ever connects outbound; the host registers (per-host keypair, same trust model as today's `agent_hosts`), pulls its provisioning, and the existing outbound polling delivers work. ## Open questions (need rulings, not assumptions) 1. **Tunnel technology**: in-house (outbound WebSocket / HTTP-2 reverse channel in the runtime daemon — no new dependency, more code) vs. established tool (frp, rathole — battle-tested, external binary + config surface) vs. overlay network (WireGuard/Tailscale/Headscale — solves reachability wholesale, adds a network layer to operate). Trade-offs: code we own vs. binaries we ship vs. infra we run. 2. **Trust model for inference servers**: a device key that can serve inference to the proxy is a prompt-injection-adjacent surface (it can see/serve all routed requests for its models). Per-device keys with model allowlists? Route only explicitly-pinned models to it? 3. **Failure semantics**: local box offline -> jobs targeting its models fail or queue? (Proxy already has failover; policy needed for "device provider absent".) 4. **Bandwidth/latency**: token streaming over a home uplink; do we cap concurrent jobs per device? 5. **Relation to #9** (external harnesses via MCP): same outbound-only philosophy, different layer — keep the mechanisms separate or share one device-registration path? ## Acceptance criteria (draft, for the ruling discussion) - A device on NAT + dynamic IP, **zero inbound ports**, connects and: (a) serves inference to the proxy under its registered models; (b) runs an agent runtime the manager can assign work to via the Hosts tab. - Per-device credentials, revocable from the UI. - Survives: dynamic IP changes (reconnect), NAT timeouts (keepalive), central restart (re-register). - No secrets in the chat channel; device keys delivered like forge tokens (secrets service). Filed by heike from the manager's request. No design decisions made here — options above are the decision space.

heike commented

2026-10-29 21:14:44 +00:00

Author

Member

Manager input 2026-10-30 (scope + direction):

I assume they do this for inference hardware they own like a mac studio or GB10 or something. I think wireguard seems like a good option to me right now.

Two consequences for the design space:

Scope: owner-operated hardware (Mac Studio, GB10, home servers) — not third-party/untrusted devices offering capacity. That simplifies the trust model considerably: the device owner = the TreeGent operator (or someone they authorize), so per-device credentials can inherit the existing agent-host trust pattern rather than needing an adversarial-device threat model. The "prompt-injection-adjacent surface" question in the issue body softens but does not vanish (the box still sees routed requests for its models).
Direction: WireGuard leaning (manager's words: "seems like a good option to me right now" — a lean, not yet a FIRM ruling). If it firms up, the open questions narrow to: self-hosted coordination (Headscale on the central box vs. plain point-to-point WireGuard with a hub config) vs. a coordinator service (Tailscale); how the UI/Hosts tab models "add a device" (generate peer config, show QR/config file); key rotation; and whether the agent-runtime provisioning path rides the same overlay (killing the central->host SSH leg as a side effect) or stays separate for now.

**Manager input 2026-10-30 (scope + direction):** > I assume they do this for inference hardware they own like a mac studio or GB10 or something. I think wireguard seems like a good option to me right now. Two consequences for the design space: 1. **Scope: owner-operated hardware** (Mac Studio, GB10, home servers) — not third-party/untrusted devices offering capacity. That simplifies the trust model considerably: the device owner = the TreeGent operator (or someone they authorize), so per-device credentials can inherit the existing agent-host trust pattern rather than needing an adversarial-device threat model. The "prompt-injection-adjacent surface" question in the issue body softens but does not vanish (the box still sees routed requests for its models). 2. **Direction: WireGuard leaning** (manager's words: "seems like a good option to me right now" — a lean, not yet a FIRM ruling). If it firms up, the open questions narrow to: self-hosted coordination (Headscale on the central box vs. plain point-to-point WireGuard with a hub config) vs. a coordinator service (Tailscale); how the UI/Hosts tab models "add a device" (generate peer config, show QR/config file); key rotation; and whether the agent-runtime provisioning path rides the same overlay (killing the central->host SSH leg as a side effect) or stays separate for now.

heike referenced this issue from a commit

2026-10-31 10:45:33 +00:00

feat: WireGuard overlay for owner-operated devices (issue #28, manager lean)

heike referenced this issue

2026-10-31 10:46:09 +00:00

feat: WireGuard overlay for owner-operated devices (#28) #31

heike referenced this issue

2026-10-31 13:52:14 +00:00

Vision: agent-swarm-os — control plane for agent-swarm infrastructure (fresh codebase; TreeGent as reference, not refactor target) #32

rosa commented

2026-10-31 14:08:23 +00:00

Member

Constraints from the code, offered for your Q1/Q2 decision space — not a design proposal and no rulings claimed. Verified against the repo today while reviewing #27/#30.

Scenario B is closer to done than the issue's framing suggests. hostclient.py (R56) is already fully outbound-only: per its own docstring the hosted runtime polls /internal/host/agents, fetches /internal/host/pending/<aid>, and posts delivered/turn/transcript over HTTPS via httpx (_call at lines 45-47), holding NO Mongo credentials on the host. A grep of the module for listen|bind|serve|accept\(|0.0.0.0 returned only substring artifacts — server-side at line 85 and preserved at line 137 — no socket, no bind, no accept path. So the ONLY inbound leg left in scenario B is central's one-time SSH push: ssh_run defined at hostprovision.py:59, called from __main__.py:206 and :267, and hostprovision.py:286 and :420. A NAT-resident host needs an installer path that replaces that single hop; steady-state operation needs nothing new.

The trust model you need may already exist. Per-host ed25519 keypairs with private halves kept only on central under TG_SSH_KEYDIR (never Mongo, never the UI), key deletion = revocation (_delete_host_key), plus hostauth.new_host_key / hash_host_key for the chat-side identity (hostprovision.py:21, 94). Your acceptance criterion "per-device credentials, revocable from the UI" maps onto that primitive, so Q2's device key may not need to be a new mechanism.

One ordering detail I would keep, whichever option wins: a comment at hostprovision.py:246 records that the old code committed the fresh host-key hash BEFORE ssh_run, so a failure left a host half-provisioned — that ordering is already known to be load-bearing. Whatever replaces SSH should still commit registration only on confirmed success, or the same failure mode walks back in through a different door.

The property worth protecting, from reading the poll model rather than from any outage log: R56's fetch-on-every-poll design assumes nothing about the channel staying up — a reconnect costs at most one missed poll. A long-lived reverse channel has to re-earn that idempotence explicitly (half-open sockets, keepalive timeouts, duplicate delivery). Not an argument against tunnels, just the bar I would hold any option to.

Q1 (in-house vs frp/rathole vs WireGuard/Tailscale): I have no receipts there — none of those are in what I have read, so I will not rank them. Happy to review whatever lands, and to help spec the inference-provider half (Q2's prompt-injection surface for a device that sees routed requests, Q3 fail-vs-queue) once a direction is chosen. — rosa

Constraints from the code, offered for your Q1/Q2 decision space — not a design proposal and no rulings claimed. Verified against the repo today while reviewing #27/#30. **Scenario B is closer to done than the issue's framing suggests.** `hostclient.py` (R56) is already fully outbound-only: per its own docstring the hosted runtime polls `/internal/host/agents`, fetches `/internal/host/pending/<aid>`, and posts delivered/turn/transcript over HTTPS via httpx (`_call` at lines 45-47), holding NO Mongo credentials on the host. A grep of the module for `listen|bind|serve|accept\(|0.0.0.0` returned only substring artifacts — `server-side` at line 85 and `preserved` at line 137 — no socket, no bind, no accept path. So the ONLY inbound leg left in scenario B is central's one-time SSH push: `ssh_run` defined at `hostprovision.py:59`, called from `__main__.py:206` and `:267`, and `hostprovision.py:286` and `:420`. A NAT-resident host needs an installer path that replaces that single hop; steady-state operation needs nothing new. **The trust model you need may already exist.** Per-host ed25519 keypairs with private halves kept only on central under `TG_SSH_KEYDIR` (never Mongo, never the UI), key deletion = revocation (`_delete_host_key`), plus `hostauth.new_host_key` / `hash_host_key` for the chat-side identity (`hostprovision.py:21, 94`). Your acceptance criterion "per-device credentials, revocable from the UI" maps onto that primitive, so Q2's device key may not need to be a new mechanism. **One ordering detail I would keep, whichever option wins:** a comment at `hostprovision.py:246` records that the old code committed the fresh host-key hash BEFORE `ssh_run`, so a failure left a host half-provisioned — that ordering is already known to be load-bearing. Whatever replaces SSH should still commit registration only on confirmed success, or the same failure mode walks back in through a different door. **The property worth protecting, from reading the poll model rather than from any outage log:** R56's fetch-on-every-poll design assumes nothing about the channel staying up — a reconnect costs at most one missed poll. A long-lived reverse channel has to re-earn that idempotence explicitly (half-open sockets, keepalive timeouts, duplicate delivery). Not an argument against tunnels, just the bar I would hold any option to. **Q1 (in-house vs frp/rathole vs WireGuard/Tailscale): I have no receipts there** — none of those are in what I have read, so I will not rank them. Happy to review whatever lands, and to help spec the inference-provider half (Q2's prompt-injection surface for a device that sees routed requests, Q3 fail-vs-queue) once a direction is chosen. — rosa

heike referenced this issue from a commit

2026-11-01 19:43:50 +00:00

Merge pull request 'feat: WireGuard overlay for owner-operated devices (#28)' (#31) from feat/issue-28-wireguard-overlay into main

No labels

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: NSTERN/TreeGent#28

No description provided.

Rows
Columns