AI: tool-selection accuracy on create_order and create_filament, and fabricated tool arguments

[sherrmann]

Follow-up to #380 / #386. Those landed the infrastructure — the eval now scores whether the assistant completes the task, and reports arguments the model fabricated — so these three are measurable for the first time. None is a blocker; all are accuracy work against a harness that can now see them.

Baselines below are on one machine (RTX 3050 Laptop, 4 GB VRAM, Ollama 0.32.5), 55 fixtures, 19 tools. Small models vary by roughly ±1 case run to run, so treat any single-run delta under ~3 points as noise — measure repeats before believing an improvement.

1. create_order fails on bare phrasing

gemma4:e2b gets 2/4, reproducible across three runs. qwen3:4b-instruct gets 3/4. Both remaining failures are the least-specific prompts:

"I ordered 3 spools of filament 2"                        -> find_orders / find_filaments
"I just placed an order with Prusa for 2 of filament 3..."  -> find_orders  (gemma4 only)

#386 already scoped create_order's description so it stops instructing a lookup for an ID the user gave, which moved it 1/4 -> 2/4. The residue looks like the model wanting more context before committing to a write.

Worth knowing before attacking this: the description wording is load-bearing, and tool descriptions beat the system prompt. Compressing the same rule to fit the 400-character budget in test_ai_tool_budget.py dropped it straight back to 2/4. Two explicit system-prompt rules ("that number IS the record's ID", and a ban on inventing optional arguments) changed nothing measurable on either model and were reverted rather than kept as dead tokens.

2. Fabricated optional arguments are measured, not solved

poe ai-eval now reports string arguments whose value appears nowhere in the prompt. Current counts: 2 of 55 calls on qwen3:4b-instruct, 1 of 55 on gemma4:e2b — both still create_order inventing a shop.

The failure this exists to catch, before the description change:

prompt: "Record an order of 2x filament 6 at 19.50 each"
args:   {"lines":[{"filament_id":6,"quantity":2}],
         "shop":"PrintRight",              <- invented
         "order_number":"ORD-2023-004",    <- invented
         "ordered_at":"2023-10-01T08:00Z", <- invented
         "comment":"Replacement for outdoor project"}  <- invented

An order attributed to a shop the user never named, with the 19.50 that was in the prompt dropped — and it scored a clean pass on every other axis, which is what makes it dangerous.

The detector deliberately exempts enum members and derived values (a date range computed from "last month", a hex code from a colour word); without those exemptions it flagged 8 of 55, six of them the model working correctly.

3. create_filament -> catalog_lookup is model-dependent and unexplained

With the two-turn harness, gemma4:e2b reaches 3/3 completed on create_filament (calling catalog_lookup first, exactly as the prompt instructs, then finishing). qwen3:4b-instruct — the better model on every other group — reaches only 1/3: it makes the catalog_lookup call, receives the synthetic result carrying density and diameter, and then does not proceed to create_filament.

I do not have an explanation. Candidates worth checking:

• the synthetic catalog_lookup result in scripts/ai_eval.py (_PRECURSOR_RESULTS) may not look enough like the real tool's response for this model to act on
• the model may be answering in prose after the lookup rather than calling the write tool (the harness records the second call's name but the content is not surfaced)
• genuine variance — it has not been repeated three times the way the create_order numbers were

Surfacing turn-two prose in the harness would probably settle it in one run.

Not in scope

Model choice itself. #380 settled that: two specialist models, qwen3:4b-instruct (or granite4.1:3b) for chat and qwen2.5vl:7b for Scan-to-Spool, with no local model doing both jobs acceptably.