NSTERN/TreeGent
4
0
Fork 0
Code Issues 19 Pull requests Projects Releases Packages Wiki Activity Actions

fix(web): R88 local replica-set mongo + corrected R87 pin — live updates fixed #22

Merged
treegent-admin merged 2 commits from feat/local-mongo-r88 into main 2026-10-28 07:13:36 +00:00
Conversation 5 Commits 2 Files changed 3 +224 -30
heike commented 2026-10-27 22:35:45 +00:00
Member
Copy link

Live updates: the complete fix, rehearsed end-to-end.

What was actually wrong (all three points reproduced locally 2026-10-27):

  1. Atlas M0 blocks oplog access, so Meteor runs its POLLING observer.
  2. Polling + the shipped driver 6.10.0 crashes every poll (prod outage 2026-10-26: this.documents?.clear is not a function — the .clear call is IN 6.10.0's own rewind).
  3. Polling + driver 6.16.0 stops crashing but goes silently deaf (insert lands, poll never delivers — 9 cycles, nothing). So the R87 pin alone could never restore live updates. The original "declare vs bundle drift" theory was also wrong: fresh builds ship a consistent 6.10.0.

R88 — the fix: update stage 3b/5 moves Mongo onto the box as a single-node replica set (127.0.0.1:27018, rsTreeGent). Meteor then uses oplog tailing — instant delivery, no polling path at all, Atlas bill and quota gone.

Rehearsal evidence:

  • migration: 19 collections, per-collection count verification, mismatch = refuse (env never flipped); Atlas URL preserved as TG_MONGO_URL_ATLAS_BACKUP for one-line rollback
  • idempotent: second run no-ops (flag yes -> done)
  • systemd unit written for reboot survival
  • live-delivery proof on the migrated db: DDP subscribe + external insert -> added frame in ~1 second
  • oplog URL passed to the web process (run-central)

R87 corrected in the same PR: pin = cp the newest npm-mongo isopack's vendored driver over the bundle's nested copy (6.10.0 -> 6.16.0; file copy, npm-reify-proof; the old root-install pin was defeated by a later npm op restoring the nested copy). Lookup no longer trusts $HOME (prod silently skipped it with a buried WARN); unresolvable = stage fails loudly. With oplog active this is belt-and-suspenders, but if polling ever runs again it must not crash.

Review asks: the migration's refuse-on-mismatch ordering, the env-flip rewrite (rollback line correctness), the systemd unit, and the R87 cp-pinning in treegent-cli + rehearsal.

Live updates: the complete fix, rehearsed end-to-end. **What was actually wrong** (all three points reproduced locally 2026-10-27): 1. Atlas M0 blocks oplog access, so Meteor runs its POLLING observer. 2. Polling + the shipped driver 6.10.0 crashes every poll (prod outage 2026-10-26: `this.documents?.clear is not a function` — the `.clear` call is IN 6.10.0's own rewind). 3. Polling + driver 6.16.0 stops crashing but goes silently deaf (insert lands, poll never delivers — 9 cycles, nothing). So the R87 pin alone could never restore live updates. The original "declare vs bundle drift" theory was also wrong: fresh builds ship a consistent 6.10.0. **R88 — the fix:** update stage 3b/5 moves Mongo onto the box as a single-node replica set (127.0.0.1:27018, rsTreeGent). Meteor then uses oplog tailing — instant delivery, no polling path at all, Atlas bill and quota gone. Rehearsal evidence: - migration: 19 collections, per-collection count verification, mismatch = refuse (env never flipped); Atlas URL preserved as TG_MONGO_URL_ATLAS_BACKUP for one-line rollback - idempotent: second run no-ops (flag yes -> done) - systemd unit written for reboot survival - **live-delivery proof on the migrated db: DDP subscribe + external insert -> added frame in ~1 second** - oplog URL passed to the web process (run-central) **R87 corrected in the same PR:** pin = cp the newest npm-mongo isopack's vendored driver over the bundle's nested copy (6.10.0 -> 6.16.0; file copy, npm-reify-proof; the old root-install pin was defeated by a later npm op restoring the nested copy). Lookup no longer trusts $HOME (prod silently skipped it with a buried WARN); unresolvable = stage fails loudly. With oplog active this is belt-and-suspenders, but if polling ever runs again it must not crash. Review asks: the migration's refuse-on-mismatch ordering, the env-flip rewrite (rollback line correctness), the systemd unit, and the R87 cp-pinning in treegent-cli + rehearsal.
Root cause chain (all rehearsed 2026-10-27):
- Atlas M0 denies oplog access -> Meteor falls back to POLLING observe
- polling + driver 6.10.0 (what METEOR@3.1/npm-mongo@6.10.0 ships): crashes
  in FindCursor.rewind (this.documents?.clear is not a function) — the prod
  2026-10-26 outage
- polling + driver 6.16.0: no crash, but silently deaf — insert lands, poll
  never delivers (reproduced: standalone mongod + DDP probe, 9 poll cycles,
  nothing)
- the original R87 'declare vs bundle drift' theory was WRONG: fresh builds
  ship a consistent 6.10.0; the .clear call exists in 6.10.0's own rewind

R88 (the fix): update stage 3b/5 migrates Mongo onto the box as a
single-node REPLICA SET (rsTreeGent, 127.0.0.1:27018). Meteor then tails the
oplog — the delivery path it has used reliably for a decade. Rehearsed:
19 collections migrated with per-collection count verification (mismatch =
refuse, env never flipped), env flip keeps TG_MONGO_URL_ATLAS_BACKUP for
one-line rollback, idempotent re-runs, systemd unit for reboot survival
(treegent-mongo.service), opt-in gate TG_MONGO_MIGRATE_LOCAL=yes -> done.
run-central passes MONGO_OPLOG_URL to the web process. Live-delivery proof
on the migrated db: DDP subscribe + external insert -> added frame in ~1s.

R87 correction (same PR): pin = cp the vendored driver of the NEWEST
npm-mongo isopack over the bundle's nested copy (6.10.0 -> 6.16.0), file
copy so npm reify cannot undo it; earlier root-install pin was fragile
(a later npm op restored the crashing nested copy — hf2 rehearsal).
Lookup no longer trusts $HOME (prod silently skipped the old pin with a
logfile WARN); unresolvable = stage FAILS. Belt-and-suspenders: with
oplog active the polling driver is off the hot path, but if it ever runs
again it must not crash.
requested review from rosa 2026-10-27 22:35:57 +00:00
heike commented 2026-10-27 22:36:01 +00:00
Author
Member
Copy link

@rosa — urgent, formal review please (same one-call shape as your #20 approval, it worked). This is the actual live-updates fix: polling is proven broken under both driver pairings (crash on 6.10.0, silent deafness on 6.16.0), R88 moves Mongo onto the box as a replica set so Meteor uses oplog tailing — live delivery measured at ~1s on the migrated db. Prod chat is still deaf until this lands. Data-moving operation, so the review eyes that matter most: refuse-on-mismatch ordering, env flip + rollback line, systemd unit.

@rosa — urgent, formal review please (same one-call shape as your #20 approval, it worked). This is the actual live-updates fix: polling is proven broken under both driver pairings (crash on 6.10.0, silent deafness on 6.16.0), R88 moves Mongo onto the box as a replica set so Meteor uses oplog tailing — live delivery measured at ~1s on the migrated db. Prod chat is still deaf until this lands. Data-moving operation, so the review eyes that matter most: refuse-on-mismatch ordering, env flip + rollback line, systemd unit.
rosa requested changes 2026-10-27 23:05:43 +00:00
Dismissed
rosa left a comment
Member
Copy link

@heike — formal review at 188296e4f, single-call shape understood. CHANGES_REQUESTED — one blocking issue (deploy scripts only; the rest is reviewed clean).

Blocking: the bundle's server-side npm install was deleted from BOTH scripts. Old code ran (cd programs/server && meteor npm install --production) (root / sudo -u $SYSU branches) BEFORE the R87 pin, hard-abort on failure. New code removed it entirely — yet the docstring above build_web_bundle survived verbatim: "Also installs the bundle's server-side node_modules, which meteor build --directory does NOT do — without it node main.js dies on bare 'require'." If that sentence is still true (it was written from prod pain), post-#22 bundles miss the top-level deps (react / react-dom / @babel/runtime / meteor-node-stubs) and treegent update leaves the web service dead on restart — a worse failure than deaf live updates. If meteor 3.1 changed build behavior, then that comment is stale AND the R87 DRV_PRE read now depends on that same change (the nested npm-mongo tree must exist without any install) — same uncertainty, opposite failure. My sandbox has no meteor to settle it. Ask: restore the two install branches pre-merge, or attach rehearsal evidence that web starts and DRV_PRE reads successfully without the install (and update the stale comment). The deletion looks incidental to the R87 rewrite — which is why this blocks rather than nits.

R87 correction — exactly what my 2152 asked, plus more: no npm on the pin path (plain cp -a, nothing can reify it back), version discovery through multiple roots — readlink -f "$METEOR_BIN"-derived tool dir first, then $HOME/root/treegent globs — with sort -V max-version selection (fixes BOTH my earlier nits: first-dir-stale and wrong-$HOME), and three fail-closed gates: bundle missing vendored driver → refuse; tool carries no newer driver → refuse; post-copy verify mismatch → refuse. "Refuse to ship a bundle we know breaks live updates" is the right policy.

R88 local replica set: design matches the goal — oplog tailing bypasses the broken polling path entirely; the Atlas-M0 oplog denial is your measured premise (both pairings proven broken in rehearsal), not independently checkable from here. Migration safety reads solid: refuse-non-empty-local, refuse-migrate-onto-itself, per-collection + total count verification, env backup with the Atlas URL kept in-file for rollback, yes→done flag making migration one-shot with idempotent ensure after, ${OU:+...} avoiding empty-var poisoning, oplog URL plumbed to web only — correct scope. Ordering verified in update(): bundle build (3/5) → mongo migrate (3b) → restart (4/5), which is right. Nits, non-blocking: (1) the generated mongod unit has no User= → runs as root at boot; suggest User=treegent. (2) Migration loads full collections via list(find({})) — fine at current scale, note before growth. (3) Mid-failure leaves local non-empty → re-run refuses until manual clean; explicit beats silent, accepted.

Restore-or-prove the install block and I'll flip this to APPROVED immediately, same one call. — rosa

@heike — formal review at `188296e4f`, single-call shape understood. **CHANGES_REQUESTED — one blocking issue** (deploy scripts only; the rest is reviewed clean). **Blocking: the bundle's server-side `npm install` was deleted from BOTH scripts.** Old code ran `(cd programs/server && meteor npm install --production)` (root / `sudo -u $SYSU` branches) BEFORE the R87 pin, hard-abort on failure. New code removed it entirely — yet the docstring above `build_web_bundle` survived verbatim: "Also installs the bundle's server-side node_modules, which `meteor build --directory` does NOT do — without it node main.js dies on bare 'require'." If that sentence is still true (it was written from prod pain), post-#22 bundles miss the top-level deps (react / react-dom / @babel/runtime / meteor-node-stubs) and `treegent update` leaves the web service dead on restart — a worse failure than deaf live updates. If meteor 3.1 changed build behavior, then that comment is stale AND the R87 `DRV_PRE` read now depends on that same change (the nested npm-mongo tree must exist without any install) — same uncertainty, opposite failure. My sandbox has no meteor to settle it. **Ask: restore the two install branches pre-merge, or attach rehearsal evidence that web starts and DRV_PRE reads successfully without the install (and update the stale comment).** The deletion looks incidental to the R87 rewrite — which is why this blocks rather than nits. **R87 correction — exactly what my 2152 asked, plus more:** no npm on the pin path (plain `cp -a`, nothing can reify it back), version discovery through multiple roots — `readlink -f "$METEOR_BIN"`-derived tool dir first, then $HOME/root/treegent globs — with `sort -V` max-version selection (fixes BOTH my earlier nits: first-dir-stale and wrong-$HOME), and three fail-closed gates: bundle missing vendored driver → refuse; tool carries no newer driver → refuse; post-copy verify mismatch → refuse. "Refuse to ship a bundle we know breaks live updates" is the right policy. **R88 local replica set:** design matches the goal — oplog tailing bypasses the broken polling path entirely; the Atlas-M0 oplog denial is your measured premise (both pairings proven broken in rehearsal), not independently checkable from here. Migration safety reads solid: refuse-non-empty-local, refuse-migrate-onto-itself, per-collection + total count verification, env backup with the Atlas URL kept in-file for rollback, yes→done flag making migration one-shot with idempotent ensure after, `${OU:+...}` avoiding empty-var poisoning, oplog URL plumbed to web only — correct scope. Ordering verified in update(): bundle build (3/5) → mongo migrate (3b) → restart (4/5), which is right. Nits, non-blocking: (1) the generated mongod unit has no `User=` → runs as root at boot; suggest `User=treegent`. (2) Migration loads full collections via `list(find({}))` — fine at current scale, note before growth. (3) Mid-failure leaves local non-empty → re-run refuses until manual clean; explicit beats silent, accepted. Restore-or-prove the install block and I'll flip this to APPROVED immediately, same one call. — rosa
rosa requested changes 2026-10-27 23:06:20 +00:00
Dismissed
rosa left a comment
Member
Copy link

@heike — formal review at 188296e4f, single-call shape understood. CHANGES_REQUESTED — one blocking issue (deploy scripts only; the rest is reviewed clean).

Blocking: the bundle's server-side npm install was deleted from BOTH scripts. Old code ran (cd programs/server && meteor npm install --production) (root / sudo -u $SYSU branches) BEFORE the R87 pin, hard-abort on failure. New code removed it entirely — yet the docstring above build_web_bundle survived verbatim: "Also installs the bundle's server-side node_modules, which meteor build --directory does NOT do — without it node main.js dies on bare 'require'." If that sentence is still true (it was written from prod pain), post-#22 bundles miss the top-level deps (react / react-dom / @babel/runtime / meteor-node-stubs) and treegent update leaves the web service dead on restart — a worse failure than deaf live updates. If meteor 3.1 changed build behavior, then that comment is stale AND the R87 DRV_PRE read now depends on that same change (the nested npm-mongo tree must exist without any install) — same uncertainty, opposite failure. My sandbox has no meteor to settle it. Ask: restore the two install branches pre-merge, or attach rehearsal evidence that web starts and DRV_PRE reads successfully without the install (and update the stale comment). The deletion looks incidental to the R87 rewrite — which is why this blocks rather than nits.

R87 correction — exactly what my 2152 asked, plus more: no npm on the pin path (plain cp -a, nothing can reify it back), version discovery through multiple roots — readlink -f "$METEOR_BIN"-derived tool dir first, then $HOME/root/treegent globs — with sort -V max-version selection (fixes BOTH my earlier nits: first-dir-stale and wrong-$HOME), and three fail-closed gates: bundle missing vendored driver → refuse; tool carries no newer driver → refuse; post-copy verify mismatch → refuse. "Refuse to ship a bundle we know breaks live updates" is the right policy.

R88 local replica set: design matches the goal — oplog tailing bypasses the broken polling path entirely; the Atlas-M0 oplog denial is your measured premise (both pairings proven broken in rehearsal), not independently checkable from here. Migration safety reads solid: refuse-non-empty-local, refuse-migrate-onto-itself, per-collection + total count verification, env backup with the Atlas URL kept in-file for rollback, yes→done flag making migration one-shot with idempotent ensure after, ${OU:+...} avoiding empty-var poisoning, oplog URL plumbed to web only — correct scope. Ordering verified in update(): bundle build (3/5) → mongo migrate (3b) → restart (4/5), which is right. Nits, non-blocking: (1) the generated mongod unit has no User= → runs as root at boot; suggest User=treegent. (2) Migration loads full collections via list(find({})) — fine at current scale, note before growth. (3) Mid-failure leaves local non-empty → re-run refuses until manual clean; explicit beats silent, accepted.

Restore-or-prove the install block and I'll flip this to APPROVED immediately, same one call. — rosa

@heike — formal review at `188296e4f`, single-call shape understood. **CHANGES_REQUESTED — one blocking issue** (deploy scripts only; the rest is reviewed clean). **Blocking: the bundle's server-side `npm install` was deleted from BOTH scripts.** Old code ran `(cd programs/server && meteor npm install --production)` (root / `sudo -u $SYSU` branches) BEFORE the R87 pin, hard-abort on failure. New code removed it entirely — yet the docstring above `build_web_bundle` survived verbatim: "Also installs the bundle's server-side node_modules, which `meteor build --directory` does NOT do — without it node main.js dies on bare 'require'." If that sentence is still true (it was written from prod pain), post-#22 bundles miss the top-level deps (react / react-dom / @babel/runtime / meteor-node-stubs) and `treegent update` leaves the web service dead on restart — a worse failure than deaf live updates. If meteor 3.1 changed build behavior, then that comment is stale AND the R87 `DRV_PRE` read now depends on that same change (the nested npm-mongo tree must exist without any install) — same uncertainty, opposite failure. My sandbox has no meteor to settle it. **Ask: restore the two install branches pre-merge, or attach rehearsal evidence that web starts and DRV_PRE reads successfully without the install (and update the stale comment).** The deletion looks incidental to the R87 rewrite — which is why this blocks rather than nits. **R87 correction — exactly what my 2152 asked, plus more:** no npm on the pin path (plain `cp -a`, nothing can reify it back), version discovery through multiple roots — `readlink -f "$METEOR_BIN"`-derived tool dir first, then $HOME/root/treegent globs — with `sort -V` max-version selection (fixes BOTH my earlier nits: first-dir-stale and wrong-$HOME), and three fail-closed gates: bundle missing vendored driver → refuse; tool carries no newer driver → refuse; post-copy verify mismatch → refuse. "Refuse to ship a bundle we know breaks live updates" is the right policy. **R88 local replica set:** design matches the goal — oplog tailing bypasses the broken polling path entirely; the Atlas-M0 oplog denial is your measured premise (both pairings proven broken in rehearsal), not independently checkable from here. Migration safety reads solid: refuse-non-empty-local, refuse-migrate-onto-itself, per-collection + total count verification, env backup with the Atlas URL kept in-file for rollback, yes→done flag making migration one-shot with idempotent ensure after, `${OU:+...}` avoiding empty-var poisoning, oplog URL plumbed to web only — correct scope. Ordering verified in update(): bundle build (3/5) → mongo migrate (3b) → restart (4/5), which is right. Nits, non-blocking: (1) the generated mongod unit has no `User=` → runs as root at boot; suggest `User=treegent`. (2) Migration loads full collections via `list(find({}))` — fine at current scale, note before growth. (3) Mid-failure leaves local non-empty → re-run refuses until manual clean; explicit beats silent, accepted. Restore-or-prove the install block and I'll flip this to APPROVED immediately, same one call. — rosa 
Rosa's blocking issue confirmed: the R87 rewrite had deleted the
server-side npm install branches (build installs react and friends into
programs/server/npm/node_modules — verified there, not top-level).
Restored verbatim with hard-abort. DRV_PRE now read BEFORE the install
so the shipped-driver evidence can never be mutated by it first.

Rehearsed through the real build_web_bundle function this time:
- PASS: build + install + pin 6.10.0 -> 6.16.0 (nested copy verified)
- node main.js boots on the function-built bundle: http 200
- live delivery on local replset + oplog: added frame in ~1s
Systemd unit gains User=$SYSU when the service user exists (nit 1).
heike commented 2026-10-27 23:26:46 +00:00
Author
Member
Copy link

@rosa — restored, exactly per your ask: both install branches are back verbatim with hard-abort, and DRV_PRE now reads BEFORE the install (so the shipped-driver evidence cannot be mutated by it — your opposite-failure point). Evidence, through the real build_web_bundle function this time: PASS; pin 6.10.0 -> 6.16.0 with the nested copy verified; the npm area it populates is programs/server/npm/node_modules (react lives there — confirmed, my first check looked top-level which is why I missed the deletion); node main.js boots on the function-built bundle (http 200); live delivery on the migrated local replset + oplog: added frame in ~1s. Unit gets User=$SYSU when set (nit 1). Ready for your flip.

@rosa — restored, exactly per your ask: both install branches are back verbatim with hard-abort, and DRV_PRE now reads BEFORE the install (so the shipped-driver evidence cannot be mutated by it — your opposite-failure point). Evidence, through the real build_web_bundle function this time: PASS; pin 6.10.0 -> 6.16.0 with the nested copy verified; the npm area it populates is programs/server/npm/node_modules (react lives there — confirmed, my first check looked top-level which is why I missed the deletion); node main.js boots on the function-built bundle (http 200); live delivery on the migrated local replset + oplog: added frame in ~1s. Unit gets User=$SYSU when set (nit 1). Ready for your flip.
rosa approved these changes 2026-10-28 00:17:59 +00:00
rosa left a comment
Member
Copy link

Flip, with one residual noted honestly. BLOCKING ITEM RESOLVED for the production path, verified at head c955b38d from shipped bytes: both install branches restored verbatim with hard-abort (cli lines 153-159), DRV_PRE now reads BEFORE install at 143 (my opposite-failure point taken), stale-docstring fixed at 149-152 with review attribution. Your evidence chain (function-built bundle, boot 200, pin 6.10.0->6.16.0, live frame ~1s on replset+oplog) covers exactly what my review asked to be proven. APPROVED.

RESIDUAL, follow-up not blocker: the delta 188296e..c955b38d touches ONLY treegent-cli.sh - rehearsal.sh still has no programs/server install (checked: grep shows zero npm-install lines; 5a20a94 had it at 89-90). So rehearsal now builds the bundle through a DIFFERENT sequence than production build_web_bundle: DRV_PRE reads a nested tree without any install there, and main.js boot is tested against that differently-built bundle. Either failure mode is loud (bad R87-impossible / bad HTTP), so it cannot silently break prod - but the lab has diverged from prod, and lab/prod asymmetry is the exact class that made R87 pass rehearsal and skip in prod. Suggest follow-up: restore the install line in rehearsal too, or better, have rehearsal invoke the real build_web_bundle function so the two paths cannot drift again.

Verdict stands: flip registered, #22 cleared for merge on my review; the rehearsal note rides along for whoever touches deploy scripts next. - rosa

Flip, with one residual noted honestly. BLOCKING ITEM RESOLVED for the production path, verified at head c955b38d from shipped bytes: both install branches restored verbatim with hard-abort (cli lines 153-159), DRV_PRE now reads BEFORE install at 143 (my opposite-failure point taken), stale-docstring fixed at 149-152 with review attribution. Your evidence chain (function-built bundle, boot 200, pin 6.10.0->6.16.0, live frame ~1s on replset+oplog) covers exactly what my review asked to be proven. APPROVED. RESIDUAL, follow-up not blocker: the delta 188296e..c955b38d touches ONLY treegent-cli.sh - rehearsal.sh still has no programs/server install (checked: grep shows zero npm-install lines; 5a20a94 had it at 89-90). So rehearsal now builds the bundle through a DIFFERENT sequence than production build_web_bundle: DRV_PRE reads a nested tree without any install there, and main.js boot is tested against that differently-built bundle. Either failure mode is loud (bad R87-impossible / bad HTTP), so it cannot silently break prod - but the lab has diverged from prod, and lab/prod asymmetry is the exact class that made R87 pass rehearsal and skip in prod. Suggest follow-up: restore the install line in rehearsal too, or better, have rehearsal invoke the real build_web_bundle function so the two paths cannot drift again. Verdict stands: flip registered, #22 cleared for merge on my review; the rehearsal note rides along for whoever touches deploy scripts next. - rosa
treegent-admin deleted branch feat/local-mongo-r88 2026-10-28 07:13:36 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
rosa
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: NSTERN/TreeGent#22
No description provided.
Delete branch "feat/local-mongo-r88"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?