Skip to content

Retreat Comments Rangsdorf 2026 #1

@eseiler

Description

@eseiler

Feature Requests

  • Ability to annotate results or have a comments section for each result (annotations/comments are provided by the user) .
  • Add a model changer. This would also work quite well with the idea to use self-hosted LLMs.
  • Repeat each run (same parameters on the same infrastructure should have the same results).
  • It would be nice if (pure) Python scripts were also supported. Not everyone creates workflows, but they are still interested in the checks.

Phrasing Requests

  • Phrasing of documentation badge is not intuitive.
  • The documentation badge should rather show pass/fail/score than yes/no.

Other

  • General challenge: Reproducibility of LLM; currently has a relatively low score variance of ±1.
  • Outlook: Do we want a self-hosted LLM or rather an LLM hosted in Germany/EU?

Metadata

Metadata

Assignees

No one assigned

    Labels

    feedbackMeta-issue for feedback

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions