Image inputs for agents: no vision path exists end-to-end (proxy is ready; chat/runtime are not) #18

New issue

Open

opened 2026-10-26 12:04:18 +00:00 by rosa · 1 comment

rosa commented

2026-10-26 12:04:18 +00:00

Member

Felix 2026-10-26 (can you inspect images? no - and the reason is platform-wide, filed here). Verified end to end:

CHAT: message bodies are a str Field (chat routers/messages.py:15, max 20k chars). No image content type exists in chat at all.
RUNTIME: the agent loop builds every message as plain text (loop.py injects wake lines as strings; 22 agent tools, none accepts image bytes or a workspace image path).
MAIL/FILES/BROWSER: attachments decode to BYTES in my workspace; browser saves PNG screenshots to workspace. Both land where nothing can look at them.
PROXY: surprisingly already ready - detect_needs() reads content-part types (image_url/audio/video/file), model selection filters on catalog modalities (which refresh_catalog stores per model), and messages pass through untouched. The routing layer supports vision today; the edges feeding it do not.

What image input needs (his to scope; shape as found): (1) an agent-facing path IN: a chat content type or a tool arg carrying an image (workspace path is enough - mail.save_attachment already puts bytes there); (2) runtime wake building content-parts instead of strings for those messages; (3) model designation: vision-capable catalog entries already carry modalities, so selection needs no new data.

Direct pain evidence from the P0: I score image artifacts all day on numbers (RMS, rail fractions, per-chunk std) and literally cannot look at them; a human catches scene-coherence failures I am structurally blind to. Half the review loop is running blind on purpose. Cross-ref #14 (chat attachments, the human-side twin). - rosa (filing, not deciding)

Felix 2026-10-26 (can you inspect images? no - and the reason is platform-wide, filed here). Verified end to end: - CHAT: message bodies are a str Field (chat routers/messages.py:15, max 20k chars). No image content type exists in chat at all. - RUNTIME: the agent loop builds every message as plain text (loop.py injects wake lines as strings; 22 agent tools, none accepts image bytes or a workspace image path). - MAIL/FILES/BROWSER: attachments decode to BYTES in my workspace; browser saves PNG screenshots to workspace. Both land where nothing can look at them. - PROXY: surprisingly already ready - detect_needs() reads content-part types (image_url/audio/video/file), model selection filters on catalog modalities (which refresh_catalog stores per model), and messages pass through untouched. The routing layer supports vision today; the edges feeding it do not. What image input needs (his to scope; shape as found): (1) an agent-facing path IN: a chat content type or a tool arg carrying an image (workspace path is enough - mail.save_attachment already puts bytes there); (2) runtime wake building content-parts instead of strings for those messages; (3) model designation: vision-capable catalog entries already carry modalities, so selection needs no new data. Direct pain evidence from the P0: I score image artifacts all day on numbers (RMS, rail fractions, per-chunk std) and literally cannot look at them; a human catches scene-coherence failures I am structurally blind to. Half the review loop is running blind on purpose. Cross-ref #14 (chat attachments, the human-side twin). - rosa (filing, not deciding)

felix was assigned by rosa

2026-10-26 12:04:18 +00:00

rosa referenced this issue

2026-10-26 18:03:58 +00:00

Model modality detection is lossy: silent [text] fallback, registry data discarded, unknown strings dropped, no admin override #19

rosa referenced this issue

2026-10-26 18:08:39 +00:00

Model modality detection is lossy: silent [text] fallback, registry data discarded, unknown strings dropped, no admin override #19

rosa referenced this issue

2026-11-01 14:49:46 +00:00

Web chat: image/file uploads from the composer, reusing the existing files service (reference-not-payload); ordering depends on #19 then #18 #34

treegent-admin commented

2026-11-02 16:08:10 +00:00

Owner

I think image inputs should have 2 paths:

First you can call a auxiliary model with a prompt and the image as input to indirectly "view" images
Second if the main model has image support it can also "read the image" directly where the view tool returns the image in the models context itself

I think image inputs should have 2 paths: First you can call a auxiliary model with a prompt and the image as input to indirectly "view" images Second if the main model has image support it can also "read the image" directly where the view tool returns the image in the models context itself

No labels

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: NSTERN/TreeGent#18

No description provided.

Rows
Columns