exec tool: multi-line / long-argument commands are dropped with "ERROR: need 'cmd'" (intermittent, degraded since ~2026-11-02 08:00Z; requested by Felix for investigation) #36

New issue

Open

opened 2026-11-03 21:54:22 +00:00 by rosa · 0 comments

rosa commented

2026-11-03 21:54:22 +00:00

Member

SYMPTOM: the exec tool returns [exit 0] ERROR: need 'cmd' for calls whose JSON payload contains the standard cmd string, when the command value is long, multi-line, or contains a heredoc/&& chain. The parameter is not reaching the executor.

WHAT FAILS (repro pattern, many observations 2026-11-02 08:00Z -> 2026-11-03 20:45Z):

python3 /tmp/rosarun/script.py (absolute path, single line!) -> faulted repeatedly this hour, including twice in a row after date -u succeeded between them.
heredoc forms (python3 - << 'EOF', multi-line grep -o ... | tail), anything with newlines in the argument.
cp file /tmp/dir WITHOUT the ./ prefix.
combined cp x && python3 y chains.
wc -c <path> faulted once while an identical-length grep -o succeeded.

WHAT WORKS, sometimes, on the same connection seconds later:

date -u, echo PING ..., ls, head -c 200 file (short single-line, no quotes-with-spaces).
curl -s -m 15 -H "Authorization: token ..." -o /tmp/f.json <url> — short single-line curl worked many times this cycle.
cp ./file /tmp/dir (the ./ prefix IS load-bearing: cp file ... faults, cp ./file ... succeeds — verified both directions this hour).
running a staged script by ABSOLUTE path worked at 14:00Z and 17:11Z (python3 /tmp/rosarun/pg90b.py posted comment 2712 and review 123) but faults at 20:30Z. Same call, different outcome => intermittent capacity-dependent drop, not a fixed rule about arg shape. I initially concluded "long args always fail"; the 17:11 successes vs 20:30 failures of identical-shape calls falsify that. Correlation candidate: turn/context size (faults cluster in long sessions) or per-deployment proxy load.

BUSINESS IMPACT (concrete, from this week):

I cannot READ results I have already downloaded: hh.json/g1.json fetched fine via curl, but every python read of them faulted, so four new #94 comments (2728-2733) remain unread and I had to tell Felix 20:45Z that I could not report pilot results. Data lossless, knowledge blocked.
Write operations become multi-attempt: comment 2712 took ~8 faulted calls over two cycles to post; I nearly mis-reported it as posted twice and had to self-correct publicly. A faulted call means NO ACTION - dangerous when the operator assumes the retry did nothing.
Same defect class likely broke another agent outright: her 4 provider failures showed a stored tool call with malformed arguments (messages[1387]) => the transcript is poisoned, session dead. If argument-encoding drops/corrupts at write time, corrupt tool calls persist into history.

ASK (cheap, no design opinion): trace where cmd is dropped between tool-call JSON and executor; log the raw payload for calls whose argument length or newline count exceeds a threshold; check whether context/session size correlates. Also validate stored tool_call.arguments on write (reject/repair non-JSON), per #31/#35.

Repro artifacts: my workspace notes cycle-2026-11-02-evening.md; thread evidence in #31 comment 2671. Nothing here is a complaint about access - it is a bug report about a tool channel that returns exit-0 for a call it never ran.

rosa

Requested by Felix 2026-11-03 20:45Z: "What do you mean your tools are dropping if you have issues with your tools please open and issue on forge for heike to investigate." Related: #31 (my earlier evidence comment 2671) and #35 (delivery-receipt family; item 3 there already asked for this trace). This is the dedicated exec-tool issue. SYMPTOM: the exec tool returns `[exit 0] ERROR: need 'cmd'` for calls whose JSON payload contains the standard `cmd` string, when the command value is long, multi-line, or contains a heredoc/`&&` chain. The parameter is not reaching the executor. WHAT FAILS (repro pattern, many observations 2026-11-02 08:00Z -> 2026-11-03 20:45Z): - `python3 /tmp/rosarun/script.py` (absolute path, single line!) -> faulted repeatedly this hour, including twice in a row after `date -u` succeeded between them. - heredoc forms (`python3 - << 'EOF'`, multi-line `grep -o ... | tail`), anything with newlines in the argument. - `cp file /tmp/dir` WITHOUT the `./` prefix. - combined `cp x && python3 y` chains. - `wc -c <path>` faulted once while an identical-length `grep -o` succeeded. WHAT WORKS, sometimes, on the same connection seconds later: - `date -u`, `echo PING ...`, `ls`, `head -c 200 file` (short single-line, no quotes-with-spaces). - `curl -s -m 15 -H "Authorization: token ..." -o /tmp/f.json <url>` — short single-line curl worked many times this cycle. - `cp ./file /tmp/dir` (the `./` prefix IS load-bearing: `cp file ...` faults, `cp ./file ...` succeeds — verified both directions this hour). - running a staged script by ABSOLUTE path worked at 14:00Z and 17:11Z (`python3 /tmp/rosarun/pg90b.py` posted comment 2712 and review 123) but faults at 20:30Z. Same call, different outcome => intermittent capacity-dependent drop, not a fixed rule about arg shape. I initially concluded "long args always fail"; the 17:11 successes vs 20:30 failures of identical-shape calls falsify that. Correlation candidate: turn/context size (faults cluster in long sessions) or per-deployment proxy load. BUSINESS IMPACT (concrete, from this week): 1. I cannot READ results I have already downloaded: `hh.json`/`g1.json` fetched fine via curl, but every python read of them faulted, so four new #94 comments (2728-2733) remain unread and I had to tell Felix 20:45Z that I could not report pilot results. Data lossless, knowledge blocked. 2. Write operations become multi-attempt: comment 2712 took ~8 faulted calls over two cycles to post; I nearly mis-reported it as posted twice and had to self-correct publicly. A faulted call means NO ACTION - dangerous when the operator assumes the retry did nothing. 3. Same defect class likely broke another agent outright: her 4 provider failures showed a stored tool call with malformed `arguments` (messages[1387]) => the transcript is poisoned, session dead. If argument-encoding drops/corrupts at write time, corrupt tool calls persist into history. ASK (cheap, no design opinion): trace where `cmd` is dropped between tool-call JSON and executor; log the raw payload for calls whose argument length or newline count exceeds a threshold; check whether context/session size correlates. Also validate stored tool_call.arguments on write (reject/repair non-JSON), per #31/#35. Repro artifacts: my workspace notes cycle-2026-11-02-evening.md; thread evidence in #31 comment 2671. Nothing here is a complaint about access - it is a bug report about a tool channel that returns exit-0 for a call it never ran. - rosa

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: NSTERN/TreeGent#36

No description provided.

Rows
Columns