Feature Request: Enhance observability and tracing capabilities

[anzax]

Currently, txtai lacks built-in observability and tracing capabilities that would help track:

• Embeddings generation and indexing processes
• LLM prompts/generations
• Pipeline execution flows
• Agent activities
• Workflow execution metrics

Potential solution: Integration with MLflow's LLM tracking (https://mlflow.org/docs/latest/llms/tracing/index.html) as an optional component.

This would enable:

• Log and trace LLM interactions
• Track embeddings operations
• Monitor system performance
• Visualize execution flows

[davidmezzetti]

Thank you for the suggestion! I'll take a look.

It seems like ultimately, the best way to do this would be through this: https://mlflow.org/docs/latest/llms/tracing/contribute.html

[davidmezzetti]

Hello - I just put this PR out with the MLflow project: mlflow/mlflow#14697

Assuming it's merged, this issue will be closed in favor of that.