Increase robustness (implicit retries)

[ceskyDJ]

There are at least some cases when Neuron doesn't handle implicit retries in case the LLM request tool use with an invalid request body (I personally ran into situation with missing required parameters, but it may not do the recovery in more cases, like bad parameter types, invalid array items – less items, more items, bad type of some item, …).

Also, a similar problem comes with MALFORMED_FUNCTION_CALL finish reason sent by the Gemini provider—the app crashes, which isn't acceptable.

I think this is one of the core features of a decent-quality AI library. Every crash could cause data loss, bad customer experience, and other similar problems. As LLMs are stochastic and sometimes hallucinate, they generate invalid JSON schemas, don't respect property definitions, etc. There should be a recovery mechanism that implicitly retries the request, so developers don't have to handle this every time they write an agent that has some tools.

Also, I hope an implicit retry mechanism is implemented for structured outputs or casual API errors (the API is unavailable for a few seconds, it crashed during request processing, etc.). The reasons are pretty much the same as for tool use.

[ceskyDJ]

Also, I hope an implicit retry mechanism is implemented for structured outputs or casual API errors (the API is unavailable for a few seconds, it crashed during request processing, etc.). The reasons are pretty much the same as for tool use.

I just got a crash due to an empty finish reason caused by a network failure between OpenRouter and Amazon Bedrock, providing Opus 4.6. So, this isn't handled either.

[ilvalerione]

Structured output generation already has a robust retry mechanism in place. It's also able to collect the violations of the data generated by the LLM and retry informing the model about what was wrong in the previous response.

To leverage this feature you need to attach validation rules to structured class properties: https://docs.neuron-ai.dev/neuron-v3/agent/structured-output#validation-rules

You can decide how many times you want to retry passing the third argument in the structured method: https://github.com/neuron-core/neuron-ai/blob/3.x/src%2FAgent%2FAgent.php#L156

Tool exceptions are not cached by the retry loop and they are forwarded back to the application. This is on purpose. Tools always involve third party APIs so we want to inform the application about thair failure and let developers decide what to do.

[ceskyDJ]

I'm aware of the validation rules for the structured output, but we're speaking of something else here.

Tool exceptions are not cached by the retry loop and they are forwarded back to the application. This is on purpose.

I think I can understand the idea. The problem is that the errors I mentioned are just a problem with generated code (practically the same situation as with the structured output), so your argument isn't valid here. I see this as a pretty similar problem to bad output in case of structured outputs—it's just another JSON structure violation. And since tool use implicitly defines "structured output", I think you should use it not just for validation, but also for retry, as you do for the original structured outputs defined by validation rules.

A similar situation is with the MALFORMED_FUNCTION_CALL. This is another problem on the LLM provider's side and has little in common with an app's logic, so handling it in the app isn't very practical – it's more like a redundant step I have to do the same way every time. I looked at some issues of the official Python library from Google—googleapis/python-genai#430, googleapis/python-genai#1789. There is a comment with possible solutions to this problem—googleapis/python-genai#430 (comment). But yeah, it looks like a problem of older models (according to googleapis/python-genai#1789 (comment)), so you may not waste your time with it.

I thought you had already had the rety mechanism for tools before I started digging into it. You have the number of retries per tool, and you limit it there. I recently figured out that it's not the number of retries but the number of tool usages, but it's still confusing. When you have more tools called on many places, it's inconvenient to handle the bad RPC assignment JSON every time. Would you happen to have any tips for doing it efficiently outside the library?

[ilvalerione]

The problem is that the errors I mentioned are just a problem with generated code (practically the same situation as with the structured output), so your argument isn't valid here.

What do you mean with "generated code"? Can you provide an example of what you are trying to accomplish?

When you have more tools called on many places, it's inconvenient to handle the bad RPC assignment JSON every time. Would you happen to have any tips for doing it efficiently outside the library?

Tools called in many places? bad RPC assignement JSON? I'm sorry but it's not easy to me to get your internal implementation from this statements. Can you provide code examples of your use case?

[ceskyDJ]

Oh, I'm sorry. I have something else to do, but I'll provide you with some clarification in the evening.

[ceskyDJ]

Ok, so please let me explain it in detail. I'm sorry for being very brief last time. I generally try to be short, as people often say I'm too long and give too much information. I try to find some balance for each situation, but usually fail.

I was pointing to the key similarity between tool uses/calls (de facto implementation of RPC, or Remote Procedure Call; not exact, but the concept is almost the same) and structured output. In both cases, the LLM needs to generate precise output (typically JSON or a token-optimized alternative like TOON) to succeed (produce a valid response). For structured output, the developer models the validation rules using an annotated PHP data class. For tool use, the validation rules are defined via tool properties. Both of these procedures basically lead to a JSON schema (of course, it's only implicit, as you validate the JSON via PHP checks) that validates any output the LLM generates and its provider sends to your app.

In both cases, the LLM could hallucinate, the context window could "overflow" (I mean more tokens can't fit the context window, or you reach a provider's synthetic limit), or the maximum number of generated tokens (set by you in a parameter sent along with the request to LLM or the provider's hard limit) could be reached. Also, some (mostly smaller or specialized) models aren't trained well to generate structured outputs and/or use tools, so they can more easily mess up parameter types, forget some parameters, produce invalid JSON, etc.

Now, in case of structured output, when you find some schema-related problem (so during lexical and syntactic checks), you request a fix from the LLM. However, when there is a schema-related problem with tool use (missing parameter, bad type of parameter, invalid JSON—basically the same issues with a different meaning, as a parameter can be interpreted as an attribute, too), you don't request the fix from the LLM, but only throw an exception and let the developer handle it. What could the developer do in this situation? I think the only reasonable option is to retry the tool call or specifically tell the LLM that the tool was misused and add the concrete problem (I don't know how you handle structured outputs; I haven't checked this part of Neuron yet).

There are other cases where tool misuse should really lead to throwing an exception and propagating it to the app so developers can handle it as needed. The perfect example here is exceeding the tool's maximum use/call limit, as the library can't know how the developer wants to handle this situation. A similar situation is exceeding the retry policy (the tool was called wrong, as I described above, so invalid JSON, missing required parameters, etc.) set for the tool or agent. I think you can find more like these, but this shouldn't be a complete list. I want to give you the idea and provide some examples for better understanding.

I'm sorry but it's not easy for me to get your internal implementation from these statements. Can you provide code examples of your use case?

This is a conceptual issue. You can use the simplest tool you can write (like get_current_time – no properties, just a tool to use that returns the current time in a string form), as well as a very complex tool with many non-trivial properties (array, objects, etc.). Of course, the simple tool will probably be used correctly (as there are no properties, so the LLM can basically only add some in advance; or the JSON could be invalid due to missing }, e.g.), but the problem is the same—tool use can fail in a way, the LLM could fix if the library requests another usage (or with possible error details if available—but this is often not neccesary).

[ceskyDJ]

I hope it's better now. In other case, feel free to ask. Also, I'd like to know your arguments if you disagree. I may look at it from a bad perspective, or I may miss something.

[kristianbinau]

I agree with @ceskyDJ that this could really increase LLM robustness. I'm not sure if there's a clean way of handling tool parameter errors today (or max tools calls), I haven't tried to find a solution in v3 yet.

If this isn't something you want to improve, or there is already an easy way of handling, then I would expect a section about in the docs, as this is a pretty common scenario.

[ilvalerione]

Thank you for the feedback, I now have a better understanding of what you mean.

First, I changed the name of the maxTries() method in maxRuns() to avoid misunderstandings, as you rightly pointed out. maxTries wasn't removed to guarantee backward compatibility, but it's now marked as @deprecated.

I never took in consideration the possibility of a model generating wrong tool calls. Is it something you experienced or it's forward thinking?

[ceskyDJ]

I never took in consideration the possibility of a model generating wrong tool calls. Is it something you experienced or it's forward thinking?

I only speak from my experience :D. Some models (mostly smaller and cheaper) occasionally have trouble with tool calls. The MALFORMED_FUNCTION_CALL problem occurs even with Gemini 2.5 models, which are still kind of LTS (as 3 and 3.1 are considered experimental as "preview").

[kristianbinau]

We also sometimes experience wrong tool calls. Like trying to call our web_extract tool, but the model calling web_find instead.
Or wrong parameters like query instead of queries. This can be helped, by figuring out what the model was "trained" to understand, or providing better descriptions for the tool or parameter.

I think a solution like providing the option of specifying aliases for tool and parameter names, might help with that part. Not sure how other libraries handle this case?

[ceskyDJ]

I think a solution like providing the option of specifying aliases for tool and parameter names, might help with that part. Not sure how other libraries handle this case?

I'd just catch the exception and generate a fix request for the LLM with a brief description of the issue: "The tool web_find is not available." or something like that. Or just a very easy approach—always send universal feedback like "The tool call of {$toolName} was invalid. Try it again, but double-check if you use the correct tool name and you respect the tool definition." I didn't try any of these messages; they may be completely useless to most LLMs, so some prompt engineering is needed. I just wanted to present a conceptual solution.

I don't know how other libraries (especially mainstream ones like Langchain) solve this problem. I think it's worth looking at their implementations.

[ilvalerione]

Part of the challenge is maintaining a consistent conversation structure, because after a tool call the model only accepts a tool call result, not another generic message like we managed to do in the structured output. For tools, the workaround seems quite complicated.

[kristianbinau]

Part of the challenge is maintaining a consistent conversation structure, because after a tool call the model only accepts a tool call result, not another generic message like we managed to do in the structured output. For tools, the workaround seems quite complicated.

That's also why I think a solution like allowing aliases would allow it to be less harsh. Like allowing both "web_extract" and "web_find" as the tool name. And same solution for parameters.

Then we won't need to handle errors, as Neuron would simply correct the models mistake via aliases.