Error message improvements

[odow]

Background

@mlubin and I have been talking about improving JuMP for coding agents. One area in which we can do a lot better is to provide more informative error messages, so that when things go wrong agents (and users) have more guidance on what to do next, or where to look in the documentation to understand why they hit an error.

As a recent example, see @DimitriAlston's recent issue #4171 and fix in #4172.

See also this talk: https://www.youtube.com/watch?v=bRDtplyhRYw

I don't know that there's a perfect template that we should apply.

One approach would be to follow commercial solvers and have a short unique identifier with a longer description in the docs about when and why it happens.

Another is to add more content to the error message.

At minimum, I think we need two-part error messages: the first says what (and perhaps why) the error happened. The second should prompt some sort of action item as the next step for how to fix the error.

I would have thought agents liked more verbose, well formatted error messages, but Claude is trying to tell me not. I don't know I believe it. Here's part of our conversation:

❯ Are longer, more descriptive error messages helpful for coding agents?

Yes, with some nuance.

Coding agents benefit from context that a human would infer from experience. A human seeing unsupported operator mentally fills in "oh, I probably wrote = instead of ==." An agent has to reason that out from the message alone, so a hint like "did you mean ==?" is more valuable to an agent than a human.

However, the Markdown and verbosity problems still apply — arguably more so. An agent reads the full message into its context window, so padding (circular explanations, redundant examples) wastes tokens and can dilute the signal.

The practical sweet spot for a library like JuMP, which is increasingly used in agentic workflows:

- Keep: the bad value, the exhaustive list of valid alternatives, and the fix hint (operator_to_set for custom sets)
- Add: a direct "did you mean X?" suggestion when the bad value is close to a valid one (e.g., = → ==)
- Drop: prose explanation of what the error is, redundant examples, section headers

So the right response to "should I write longer messages for agents" is: more precise and complete, not more verbose. An agent benefits from a message that removes ambiguity about the fix, not one that has more words.

Action items

So, action items:

• Do you have examples of packages/libraries/languages with very nice error messages? If so, please share them below.
• Do other projects have style guides for how to write nice error messages? Should JuMP adopt such a style guide?
• Crank through the 167 error messages in JuMP/src and improve them...

[odow]

❯ What are some examples of Python packages with great error messages?

A few standouts:

Pydantic — validation errors name the field path, the value, and the exact constraint violated, all in a structured format that's readable as text and parseable programmatically. Nested model errors show the full dotted path to the offending field.

Rust's compiler (rustc) — not Python, but it set the bar that Python packages now aim for: points to the exact source span, explains why it's wrong, and often suggests the fix inline.

Hypothesis — when a test fails, it prints the minimal shrunk example that reproduces the failure, explains what strategy generated it, and tells you how to replay it with a fixed seed. The message does the debugging for you.

Rich / Typer (Textualize) — Typer's CLI argument errors name the bad argument, show the valid choices, and print a usage snippet. Rich's own errors include a traceback renderer that highlights the relevant line with syntax coloring.

Polars — relatively new but deliberately designed with agents/users in mind. Type mismatch errors name the operation, both the expected and actual dtype, and often suggest a cast. Schema mismatch errors list which columns are missing or wrong.

SQLAlchemy — when a query fails due to a missing relationship or wrong column name, the error names the model class, the attribute that was accessed, and what attributes are available.

The common thread: all of them include the bad value, the constraint, and a concrete fix — the same principles that apply to Julia.

✻ Churned for 22s

❯ Can you show me some examples from each package?

Here are concrete examples for each:

---
Pydantic — nested field paths and structured output:

from pydantic import BaseModel

class Address(BaseModel):
    zip_code: str
    country: str

class User(BaseModel):
    name: str
    age: int
    address: Address

User(name="Alice", age="old", address={"zip_code": 123, "country": "US"})
2 validation errors for User
age
  Input should be a valid integer, unable to parse string as an integer
  [type=int_parsing, input_value='old']
address.zip_code
  Input should be a valid string [type=string_type, input_value=123]

---
Polars — names the dtypes and suggests the fix:

import polars as pl

df = pl.DataFrame({"a": [1, 2, 3], "b": ["x", "y", "z"]})
df.select(pl.col("a") + pl.col("b"))
InvalidOperationError: arithmetic on Series of dtypes `Int64` and `String`
is not supported. Consider casting to a numeric type first.

Column name typos:
df.select("c")
ColumnNotFoundError: unable to find column "c"; valid columns: ["a", "b"]

---
Hypothesis — shrinks to the minimal failing case:

from hypothesis import given, strategies as st

@given(st.lists(st.integers(), min_size=1))
def test_first_element_positive(xs):
    assert xs[0] > 0
Falsifying example: test_first_element_positive(xs=[0])

You can reproduce this example by temporarily adding
@reproduce_failure('6.112.1', b'AAEAAaA=') as a decorator on your test case

Without shrinking, xs might have been [743, -12, 0, 88, ...]. Hypothesis reduces it to [0] — the smallest input that triggers the failure.

---
SQLAlchemy — links to dedicated error docs:

user = session.query(User).first()
session.close()
print(user.posts)  # access lazy relationship after session closed
DetachedInstanceError: Instance <User at 0x7f...> is not bound to a Session;
attribute refresh operation cannot proceed.

Background on this error at: https://sqlalche.me/e/14/bhk3

The sqlalche.me short links go to a page that explains the error in depth — a nice pattern for errors that are inherently complex to explain inline.

---
Typer — shows valid choices and a help hint:

import typer

app = typer.Typer()

@app.command()
def main(fmt: str = typer.Option(..., "--format", "-f")):
    pass
$ myapp --format xml

Error: Invalid value for '--format' / '-f': 'xml' is not one of 'json', 'csv', 'text'.

Try 'myapp --help' for help.

---
The SQLAlchemy pattern of linking to external docs is worth noting for JuMP — for complex errors like the nonlinear skip_missing case, a link to the relevant manual section can carry explanation that would be too long to embed in the message itself.

✻ Cogitated for 36s

❯ What about the Rust compiler?

The Rust compiler errors are best shown visually — let me generate one.

  Ran 1 shell command

Rust isn't installed here, so I'll show representative examples from memory.

---
Borrow checker — explains the conflict and points to both locations:

fn main() {
    let mut v = vec![1, 2, 3];
    let first = &v[0];
    v.push(4);
    println!("{}", first);
}
error[E0502]: cannot borrow `v` as mutable because it is also borrowed as immutable
 --> src/main.rs:4:5
  |
3 |     let first = &v[0];
  |                  ^ immutable borrow occurs here
4 |     v.push(4);
  |     ^^^^^^^^^ mutable borrow occurs here
5 |     println!("{}", first);
  |                    ----- immutable borrow later used here

It points to three lines: where the immutable borrow started, where the conflicting mutation happens, and where the borrow is last used — enough to understand and fix the problem without reading any documentation.

---
Type mismatch — suggests a concrete fix:

fn add(x: i32, y: i32) -> i32 {
    x + y;
}
error[E0308]: mismatched types
 --> src/main.rs:2:5
  |
1 | fn add(x: i32, y: i32) -> i32 {
  |                            --- expected `i32` because of return type
2 |     x + y;
  |     ^^^^^^ expected `i32`, found `()`
  |
help: consider removing this semicolon
  |
2 -     x + y;
2 +     x + y

The help: line shows the exact edit — remove the semicolon — as a diff.

---
Unknown field — lists valid options:

struct Point { x: f64, y: f64 }
let p = Point { x: 1.0, z: 2.0 };
error[E0560]: struct `Point` has no field named `z`
 --> src/main.rs:2:25
  |
2 |     let p = Point { x: 1.0, z: 2.0 };
  |                             ^ help: a field with a similar name exists: `y`

---
The key things rustc does that most tools don't:

- Source spans — arrows pointing to the exact token, not just a line number
- Multi-location errors — shows all the relevant sites in one message
- Inline diffs — the help: blocks show - old / + new rather than describing the change in prose
- Error codes — every error has an E0xxx code; rustc --explain E0502 gives a full tutorial on that class of error

That last pattern — the error code linking to deeper explanation — is what SQLAlchemy's sqlalche.me links borrow from.

[odow]

This course is quite helpful: https://developers.google.com/tech-writing/error-messages

[odow]

I need to go through #4180 in detail, but there are some easy wins:

I think my style guide rule is:

error(
    """
    A sentence describing the error.

    A paragraph providing more detail, if needed.

    An actionable next step.
    """,
)

The last part is the most important. We should never have an error that doesn't tell the user something to do next. That might be to look at their code for a mistake, to read some portion of the documentation, or to change their code in some way.

In the example above we change "we can't do that" into "we can't do that; make sure you do this first and here's how to check"