Feat: Cleanup stale assets in graphs and configs

[joshuaunity]

Description

• Ignore stale asset references in sensors_to_show when resolving chart inputs, so chart and chart_data do not fail if a referenced asset was deleted
• Make sensors_to_show flattening resilient by skipping invalid asset plot references instead of raising validation errors
• Clean stale asset references from other assets’ sensors_to_show during asset deletion, with audit-log entries for traceability
• Harden graph modal rendering so unavailable asset plots are shown as Not available instead of breaking the full plot list
• Improve asset fetch error handling in frontend utils so missing assets fail predictably and can be handled gracefully by callers
• Add regression tests for:
  • missing asset references in chart data generation
  • stale asset-reference cleanup on asset delete
• Added changelog item in documentation/changelog.rst

Look & Feel

• API behavior improvement:
  • Before: chart endpoints could return UNPROCESSABLE_ENTITY when sensors_to_show referenced a deleted asset
  • After: valid plots still render; stale asset references are ignored
• Graph modal behavior improvement:
  • Before: one failing asset lookup could stop rendering of plot cards
  • After: available plots are still shown, stale ones are marked as Not available with a warning banner
• UI helper behavior:
  • getAsset in flexmeasures/ui/static/js/ui-utils.js now throws descriptive errors on non-2xx responses, enabling robust fallback rendering

How to test

1. Run focused regression tests:
   • .venv/bin/python -m pytest flexmeasures/api/v3_0/tests/test_assets_api.py -k delete_asset_cleans_stale_asset_references_in_sensors_to_show
2. Manual API/UI scenario:
   • Create an asset graph config where sensors_to_show contains an asset plot reference
   • Delete the referenced asset
   • Open the graph page and confirm:
     • chart still loads for remaining valid plots
     • stale plot is shown as Not available in graph modal
     • warning banner appears about unavailable saved plots
3. Verify cleanup on delete:
   • Delete an asset referenced by another asset’s sensors_to_show
   • Confirm reference is removed from persisted sensors_to_show
   • Confirm audit log entry notes stale asset reference removal

Further Improvements

• Add a dedicated UI integration test for graph modal unavailable-plot rendering path
• Add optional auto-repair action in graph modal to remove all unavailable references in one click
• Consider a lightweight migration/check command to proactively detect stale asset references across existing data

Related Items

• Closes Graph more stable against deleted assets #2132

---

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

[nhoening]

I re-did the initial setup, and the deleted asset reference was removed befire the graph page could even get in trouble, nice.

I need some examples in two utils, as they are not easy to grasp without. Then I know better what I approve.

Also, I ran into a separate but related problem: When a config setting is removed from its asset after it was added to some sensor_to_show, we also get an error when that graph dialogue is opened. Is that better to do in a follow-up PR?

[nhoening]

Thanks for the examples.

Please check failing tests, it might just have to do with quotes:

Expected:
    [{"title": "Power", "plots": [{"sensor": 1}]}, {"title": "Price", "plots": [{"sensor": 3}]}]
Got:
    [{'title': 'Power', 'plots': [{'sensor': 1}]}, {'title': 'Price', 'plots': [{'sensor': 3}]}]

And do check the diff - do you know why we are deleting a lot of code in flexmeasures/data/tests/test_belief_charts.py? That seems unrelated to this PR.

[nhoening]

This looks okay to me, I just have questions.
We can probably merge this today :)

[nhoening]

This is great and can be merged :)

All the JSON structures & dialogues are getting much safer against deletions.

Maybe the only thing left is when I delete an asset directly in the DB, then the flex-context dialogue which references a sensor of the asset will still have that reference, and it can from then on not be saved with new changes anymore.
Let's create a new issue for that, and I'm curious how you would solve that - probably we should do a similar fix in the flex-context and flex-model dialogues, only for direct sensor references?

[joshuaunity]

ing left is when I delete an asset directly in the DB, then the flex-context dialogue which references a sensor of the asset will still have that reference, and it can from then on not be saved with new changes anymore.
Let's create a new issue for that, and I'm curious how you would solve that - probably we should do a similar fix in the flex-context and flex-model dialogues, only for direct sensor references?

ok, i can make the issue, or would you want to writ eit up? Also ill test myself to understand it better, casue if a asset is mising the UI should still save...

[nhoening]

ing left is when I delete an asset directly in the DB, then the flex-context dialogue which references a sensor of the asset will still have that reference, and it can from then on not be saved with new changes anymore.
Let's create a new issue for that, and I'm curious how you would solve that - probably we should do a similar fix in the flex-context and flex-model dialogues, only for direct sensor references?

ok, i can make the issue, or would you want to writ eit up? Also ill test myself to understand it better, casue if a asset is mising the UI should still save...

The UI opened nicely, it also saved, but only when no real change was necessary. When I made some other change, the API complained that the referenced sensor doesn't exist.