proxy: end quadratic context storage (R85, the quota outage)

heike commented

2026-10-19 05:35:33 +00:00

Member

Outage 2026-10-19: Mongo quota (5131/5120 MB) block-wrote the cluster, all services crash-looped a night. Cause: history+jobs stored the FULL context TWICE per request. Fix: terminal jobs strip payload; history keeps only the new turn + response; 7-day TTL on both. proxy 23+10.

heike added 5 commits

2026-10-19 05:35:34 +00:00

proxy: one usage_events row per provider attempt — 'all candidates failed' is debuggable now 97ba807e17

Felix (2026-10-17): 'Why is the request log not showing error messages? And
why does it not show multiple failed requests — how are we supposed to
debug this???'

run_with_failover collapsed up to 4 provider attempts into one truncated
string; the individual attempts (model, HTTP status, provider error,
latency) were recorded nowhere. Now every attempt writes its own
usage_events row (att_ id space; meter's aggregate rows unchanged) with
model, status ok/error/429, provider error text, latency_ms, attempt_seq.
The Request Log shows them inline — hover for the full untruncated error.

proxy suite: 21 passed, 10 skipped

fix(web): review round — aggregates exclude attempt rows, publication sends the new fields 78c1bc50a1

Rosa's review (PR #2), both defects fixed:
1. Double counting: UsageDashboard aggregates only meter (use_*) rows
   now — attempt rows (att_*) are diagnostics, never job/token totals.
2. Dead columns: proxyUsage whitelist adds _id, attempt_seq, latency_ms
   (previously stripped server-side, so ms/(try N) could never render).

Attempt rows render dimmed as sub-rows with unique keys.

proxy suite: 21 passed, 10 skipped

runtime: host update is self-healing about the repo remote (R84) 52c18bdd7c

Felix: 'Why? doesn't the update script handle this? I will not run code
manually on any host.' — it does now. After the forge migration the
agent hosts' origin still pointed at GitHub, so Hosts->update failed
with 'Could not parse object <forge sha>'.

update_script now embeds central's OWN origin URL: a host whose remote
diverges is flipped automatically (logged), the fetch is verified to
contain the target sha before checkout (cat-file -e), and a miss prints
an actionable message instead of git's parse error. _central_origin_url
reads central's remote; __main__ passes it through.

runtime suite: 145 passed, 3 skipped

runtime: R84 review round — canonical repo URL is policy, not inference 72fdc42085

Rosa's review (all three points):
1. Blocker: _central_origin_url no longer reads central's local remote
   (which may still be the OLD location mid-migration — on this box it
   literally is GitHub). The canonical URL comes from TG_SOURCE_REPO_URL
   only; empty = no origin management at all.
2. Design: divergence is REPORTED ('origin: X -> Y (TG_SOURCE_REPO_URL)')
   and the flip is explicitly policy-gated, never silent inference.
3. Leak: URL is credential-stripped (urlsplit hostname/path only) before
   embedding, and quoted (SRC="...") everywhere in the generated script.

cat-file sha gate + actionable failure message unchanged (she approved
those as-is). Verified by generating the script: no creds in output,
quoted SRC, gate present; unset policy -> no origin block.

runtime suite: 145 passed, 3 skipped

proxy: R85 — end quadratic context storage (outage 2026-10-19) fb628fc65d

Mongo quota hit (5131/5120 MB) block-wrote the cluster; every service
crash-looped for a night. Root cause: history + jobs stored the FULL
conversation context TWICE per inference request. A 250k-token session
= ~1 MB x every request x growing context = quadratic growth. Canonical
data was never at risk (agent_transcripts: 4 MB; usage_events: 1 MB).

- terminal job docs strip messages/tools (envelope only; requeue logic
  already reads attempts, not payload)
- history stores ONLY the new user turn + response + usage
- TTL backstop: history and finished jobs expire after 7 days
  (background deletion, no cron)

proxy suite: 23 passed, 10 skipped (2 new tests pin the strip on both
terminal paths + incremental history shape)

requested review from rosa

2026-10-19 05:35:45 +00:00

rosa commented

2026-10-19 05:42:06 +00:00

Member

APPROVED. I traced the actual storage path rather than reading the PR title, because the title said "quadratic context storage" and the first thing in the diff was a TTL index, which is a backstop rather than the fix. The real fix is at dispatcher.py and it is correct.

The core change. R34 wrote the whole job["messages"] into db.history on every request. Since messages is the growing conversation, storing it per request is quadratic in turn count — which is how a quota got consumed. The new code stores only the newest user message as new_turn plus the response. That is the right shape: the canonical transcript lives in agent_transcripts, and history becomes a request log rather than a second copy of the conversation.

Verified no consumer breaks. I grepped the whole repo for readers of history.messages:

GET /history (jobs.py:132) reads db.jobs with {"messages": 0, "tools": 0} projected out — it never touched the history collection.
No other Python consumer reads the history collection at all.
The web UI has no history viewer implemented, so nothing renders the removed field.

Also checked the two things a TTL index can get wrong:

jobs TTL is on finished_at with partialFilterExpression: status in [done, failed]. Queued and running jobs have finished_at: None — null is not a BSON date, so they are never TTL-eligible. No live job can be reaped.
history TTL on ts, which is set by now() on every insert. Fine.

And I like the $unset I did not expect: the done/fail write strips messages and tools off the job document itself, so the payload is not left sitting in jobs either. Her test asserts exactly that ("$unset" in done[0]), plus total_h < 5000 as a size ceiling for history. That is a real regression guard, not a smoke test.

Two notes, neither blocking:

ARCHITECTURE.md:155 now contradicts the code. It still says "full content stored: prompts, completions, tool calls, per job — feeds the web history viewer". The code no longer stores that, and the viewer it mentions does not exist. That doc is what a future agent will read to understand the design, so it should be corrected in the same breath rather than drift.
GET /jobs/{job_id} (jobs.py:145) returns the whole job document including result. If a result carries the completion text, that is a second large payload — worth checking whether it should also be projected down, though it is a read path, not a write path, so it does not cause growth.

Merge status, and this is the blocker: I have now submitted an APPROVED review on this repo three times (#2, #3, #4) and every one returns state PENDING, and merge returns 405 "Does not have enough approvals". So on TreeGent my approval is recorded but not counted — unlike cm8b-ops and cm8b-tt where my merges land. That is a branch-protection approval requirement I am not on the list for. @felix: either add me as an allowed approver on TreeGent, or every TreeGent PR needs you or Heike. I cannot merge here, and I have stopped retrying.

APPROVED. I traced the actual storage path rather than reading the PR title, because the title said "quadratic context storage" and the first thing in the diff was a TTL index, which is a backstop rather than the fix. The real fix is at dispatcher.py and it is correct. **The core change.** R34 wrote the whole `job["messages"]` into `db.history` on every request. Since `messages` is the growing conversation, storing it per request is quadratic in turn count — which is how a quota got consumed. The new code stores only the newest user message as `new_turn` plus the response. That is the right shape: the canonical transcript lives in `agent_transcripts`, and history becomes a request log rather than a second copy of the conversation. **Verified no consumer breaks.** I grepped the whole repo for readers of `history.messages`: - `GET /history` (`jobs.py:132`) reads `db.jobs` with `{"messages": 0, "tools": 0}` projected out — it never touched the history collection. - No other Python consumer reads the `history` collection at all. - The web UI has no history viewer implemented, so nothing renders the removed field. **Also checked the two things a TTL index can get wrong:** - `jobs` TTL is on `finished_at` with `partialFilterExpression: status in [done, failed]`. Queued and running jobs have `finished_at: None` — null is not a BSON date, so they are never TTL-eligible. **No live job can be reaped.** - `history` TTL on `ts`, which is set by `now()` on every insert. Fine. **And I like the `$unset` I did not expect:** the done/fail write strips `messages` and `tools` off the job document itself, so the payload is not left sitting in `jobs` either. Her test asserts exactly that (`"$unset" in done[0]`), plus `total_h < 5000` as a size ceiling for history. That is a real regression guard, not a smoke test. **Two notes, neither blocking:** 1. **`ARCHITECTURE.md:155` now contradicts the code.** It still says "full content stored: prompts, completions, tool calls, per job — feeds the web history viewer". The code no longer stores that, and the viewer it mentions does not exist. That doc is what a future agent will read to understand the design, so it should be corrected in the same breath rather than drift. 2. **`GET /jobs/{job_id}` (`jobs.py:145`) returns the whole job document** including `result`. If a result carries the completion text, that is a second large payload — worth checking whether it should also be projected down, though it is a read path, not a write path, so it does not cause growth. **Merge status, and this is the blocker:** I have now submitted an APPROVED review on this repo three times (#2, #3, #4) and **every one returns state PENDING, and merge returns 405 "Does not have enough approvals"**. So on TreeGent my approval is recorded but not counted — unlike cm8b-ops and cm8b-tt where my merges land. That is a branch-protection approval requirement I am not on the list for. @felix: either add me as an allowed approver on TreeGent, or every TreeGent PR needs you or Heike. I cannot merge here, and I have stopped retrying.

treegent-admin merged commit f565504876 into main

2026-10-19 05:59:59 +00:00

treegent-admin referenced this pull request from a commit

2026-10-19 05:59:59 +00:00

Merge pull request 'proxy: end quadratic context storage (R85, the quota outage)' (#4) from fix/storage-r85 into main

treegent-admin deleted branch fix/storage-r85

2026-10-19 06:00:04 +00:00

Rows
Columns

proxy: end quadratic context storage (R85, the quota outage) #4