Model modality detection is lossy: silent [text] fallback, registry data discarded, unknown strings dropped, no admin override #19

New issue

Open

opened 2026-10-26 18:03:58 +00:00 by rosa · 1 comment

rosa commented

2026-10-26 18:03:58 +00:00

Member

Felix 2026-10-26 (chat: something is wrong with how we detect model modalities). His hunch is right — the detection is verified-broken in four places (live main 5a50635: dispatcher.refresh_catalog -> select_candidates -> admin.py). Facts first, from shipped code:

SILENT TEXT FALLBACK: a model with no architecture.input_modalities is stored as [text] (dispatcher.py:738-739) and stays text-only. Metadata-poor gateways (opencode et al., named in the #69 comment directly below) can therefore NEVER be selected for image/audio work, even when the underlying model supports it.
THE REGISTRY FALLBACK EXISTS BUT ISN'T USED FOR MODALITIES: models.dev carries per-model input modalities, and #69 already established the exact pattern for context windows — gateway value wins, registry fills holes (_resolved_ctx). But _parse_models_dev reduces the dump to context ints ONLY; modality data is fetched-and-discarded. Defect 1 is fixable by mirroring the ctx pattern — no new infrastructure.
UNKNOWN MODALITY STRINGS VANISH QUIETLY: the whitelist (x in text/image/audio/video/file) drops anything else with no log line. A gateway reporting pdf or vision-vocabulary produces a text-only catalog row nobody can notice.
NOT CORRECTABLE: ModelPolicyIn (admin.py:132-155) takes designations/rank/excluded/context_window — no modalities. And refresh $sets modalities unconditionally, so even a manual DB fix is erased at the next refresh. Wrong detection + no override + overwrite = admins cannot fix a bad row through any supported path.

Forward-looking landmine: select_candidates filters request needs (image/video/audio/file) with $all against stored modalities, while unknown-vocabulary models lost their claims at write time. The moment #18 (image inputs) lands real multipart traffic, misclassified models will produce all-candidates-failed proxy errors that LOOK like provider outages but are catalog defects. Fix classification before opening the input path.

Suggested scope (his to decide): registry modality fallback mirroring ctx; loud log for dropped modality strings; a modalities_override policy field honored across refreshes (like ctx: provider value wins unless an explicit override exists); a doc note on vocabulary mapping per gateway. — rosa (filing, not deciding)

Felix 2026-10-26 (chat: something is wrong with how we detect model modalities). His hunch is right — the detection is verified-broken in four places (live main 5a50635: dispatcher.refresh_catalog -> select_candidates -> admin.py). Facts first, from shipped code: 1. **SILENT TEXT FALLBACK:** a model with no `architecture.input_modalities` is stored as `[text]` (dispatcher.py:738-739) and stays text-only. Metadata-poor gateways (opencode et al., named in the #69 comment directly below) can therefore NEVER be selected for image/audio work, even when the underlying model supports it. 2. **THE REGISTRY FALLBACK EXISTS BUT ISN'T USED FOR MODALITIES:** models.dev carries per-model input modalities, and #69 already established the exact pattern for context windows — gateway value wins, registry fills holes (`_resolved_ctx`). But `_parse_models_dev` reduces the dump to context ints ONLY; modality data is fetched-and-discarded. Defect 1 is fixable by mirroring the ctx pattern — no new infrastructure. 3. **UNKNOWN MODALITY STRINGS VANISH QUIETLY:** the whitelist (`x in text/image/audio/video/file`) drops anything else with no log line. A gateway reporting `pdf` or vision-vocabulary produces a text-only catalog row nobody can notice. 4. **NOT CORRECTABLE:** `ModelPolicyIn` (admin.py:132-155) takes designations/rank/excluded/context_window — no modalities. And refresh `$set`s modalities unconditionally, so even a manual DB fix is erased at the next refresh. Wrong detection + no override + overwrite = admins cannot fix a bad row through any supported path. Forward-looking landmine: `select_candidates` filters request needs (image/video/audio/file) with `$all` against stored modalities, while unknown-vocabulary models lost their claims at write time. The moment #18 (image inputs) lands real multipart traffic, misclassified models will produce all-candidates-failed proxy errors that LOOK like provider outages but are catalog defects. Fix classification before opening the input path. Suggested scope (his to decide): registry modality fallback mirroring ctx; loud log for dropped modality strings; a `modalities_override` policy field honored across refreshes (like ctx: provider value wins unless an explicit override exists); a doc note on vocabulary mapping per gateway. — rosa (filing, not deciding)

felix was assigned by rosa

2026-10-26 18:03:58 +00:00

rosa commented

2026-10-26 18:08:39 +00:00

Author

Member

CONFIRMED with live data (Felix 17:59: opencode and qwen coding plan all text-only in our list). Just fetched https://models.dev/api.json with 232 providers and traced the why:

The catalog modality column comes ONLY from the gateway /models field architecture.input_modalities in refresh_catalog. These gateways never send it, so every opencode row hits the 738-739 fallback and stores text-only. 130/130 rows: that is the whole symptom, at scale.
The registry HAS the real data and we throw it away. Non-text signals in models.dev: opencode 87 of 130 (gpt-6.1-sol text+image+pdf, qwen3.8-max text+image); opencode-go, the qwen coding plan, 26 of 37 (qwen3.8-max text+image+video, gemini-4-argon text+image+video+audio+pdf); alibaba/dashscope 28-30 per endpoint. _parse_models_dev at 563-591 extracts ONLY limit.context ints; modalities.input never survives parsing. The fix for this exact complaint is the ctx fallback pattern already shipped in #69 - gateway wins, registry fills holes - extended to modalities.

Two facts the fix must respect, found in the same dump:
(a) Three-way vocabulary mismatch: the registry says pdf, our whitelist drops it (defect 3 now has a live example), and detect_needs maps attachments to file. Fallback without pdf/file normalization would still mis-select; normalization lands in the same change.
(b) Host collision: opencode and opencode-go share api host opencode.ai - the docstring already notes they disagree on three ids for ctx, first-writer-wins. A host-keyed modality slot would silently mix the two plans claims: a false capability claim, worse than the mild ctx case. Key modality slots by plan path segment, e.g. /inference/go/, not bare host.

Impact restated once: about 70 percent of the opencode catalog, including our free-tier fallback models, is mislabeled text-only today. Harmless only because nothing can send images yet - decisive when #18 lands, actively misleading during any multipart debugging. The fix is small: parse modalities in _parse_models_dev, registry fallback in refresh, pdf/file normalization, path-keyed slots - all testable OFFLINE against the same api.json dump. Priority argument for the board. - rosa

**CONFIRMED with live data** (Felix 17:59: opencode and qwen coding plan all text-only in our list). Just fetched https://models.dev/api.json with 232 providers and traced the why: - The catalog modality column comes ONLY from the gateway /models field architecture.input_modalities in refresh_catalog. These gateways never send it, so every opencode row hits the 738-739 fallback and stores text-only. 130/130 rows: that is the whole symptom, at scale. - The registry HAS the real data and we throw it away. Non-text signals in models.dev: opencode 87 of 130 (gpt-6.1-sol text+image+pdf, qwen3.8-max text+image); opencode-go, the qwen coding plan, 26 of 37 (qwen3.8-max text+image+video, gemini-4-argon text+image+video+audio+pdf); alibaba/dashscope 28-30 per endpoint. _parse_models_dev at 563-591 extracts ONLY limit.context ints; modalities.input never survives parsing. The fix for this exact complaint is the ctx fallback pattern already shipped in #69 - gateway wins, registry fills holes - extended to modalities. Two facts the fix must respect, found in the same dump: (a) Three-way vocabulary mismatch: the registry says pdf, our whitelist drops it (defect 3 now has a live example), and detect_needs maps attachments to file. Fallback without pdf/file normalization would still mis-select; normalization lands in the same change. (b) Host collision: opencode and opencode-go share api host opencode.ai - the docstring already notes they disagree on three ids for ctx, first-writer-wins. A host-keyed modality slot would silently mix the two plans claims: a false capability claim, worse than the mild ctx case. Key modality slots by plan path segment, e.g. /inference/go/, not bare host. Impact restated once: about 70 percent of the opencode catalog, including our free-tier fallback models, is mislabeled text-only today. Harmless only because nothing can send images yet - decisive when #18 lands, actively misleading during any multipart debugging. The fix is small: parse modalities in _parse_models_dev, registry fallback in refresh, pdf/file normalization, path-keyed slots - all testable OFFLINE against the same api.json dump. Priority argument for the board. - rosa