Skip to content

Backport #2072 and #2309 to the 0.33 line, for a v0.34.0rc1 pre-release#2317

Closed
Flix6x wants to merge 3 commits into
0.33.xfrom
backport/v0.34.0rc1
Closed

Backport #2072 and #2309 to the 0.33 line, for a v0.34.0rc1 pre-release#2317
Flix6x wants to merge 3 commits into
0.33.xfrom
backport/v0.34.0rc1

Conversation

@Flix6x

@Flix6x Flix6x commented Jul 14, 2026

Copy link
Copy Markdown
Member

Cuts a pre-release on the 0.33 maintenance line so a downstream plugin can pick up two v1-track features without taking the whole v1.0.0 development line (51 commits, 3 extra migrations).

Base is v0.33.1. On top of it:

⚠️ Neither PR cherry-picked cleanly — please review the adaptations

Both were written against main commits that are not on this branch, so each needed the main-only parts stripped. The adaptations are all of the same kind: removing support for features the 0.33 line does not have. No behaviour was invented.

#2072 — dropped its dependency on #2209 (source-filtered sensor references)

SensorReference is introduced by #2209, which is not on this line. #2072 does not use it for anything new — it only inherits it from surrounding main code. Three edits in storage.py:

main (#2072) this branch
from ...schemas.sensors import SensorReference, VariableQuantityField dropped (neither symbol exists here; VariableQuantityField was unused anyway)
if isinstance(state_of_charge_sensor, SensorReference): state_of_charge_sensor = ....sensor dropped — a no-op here, since state_of_charge is always a plain Sensor on 0.33
isinstance(soc_max, (Sensor, SensorReference)) isinstance(soc_max, Sensor) — which is exactly what v0.33.1 already had

Everything else from #2072 (SchedulingJobResult, _build_soc_schedule, _compute_unresolved_targets, the jobs-API result field) is byte-identical to main.

#2309 — took only the Alembic-head half, dropped the template-provisioning half

#2309 does two things. The second depends on #2268 (starter templates), which is not on this line:

  • ✅ kept: the head check — data/utils.py (get_database_schema_revision_status), data/__init__.py (app.database_schema_is_migrated_to_head + the warning), and its 5 tests.
  • ❌ dropped: provision_default_template_assets_on_startup in app_utils.py and its call in app.py. It calls provision_default_template_assets from Three initial starter templates & improved copy-asset UX #2268 — dead code here. app.py is therefore unchanged from v0.33.1.

Also dropped a stray context line importing from flexmeasures/utils/secrets_utils.py (a module #2236 introduced; on this line those symbols still live in app_utils).

The changelog entry for #2309 is worded to match what actually landed (the warning only).

Also included

  • The docker-publish.yml :latest guard from fix(ci): never claim :latest on a manual docker-publish run #2316, so a manual image build from this ref cannot clobber lfenergy/flexmeasures:latest.
  • openapi-specs.json's info.version stamped to 0.34.0rc1, matching what a build at that tag reports (".".join(__version__.split(".")[:3])).

Verification

Run locally against the adapted code:

Why v0.34.0rc1

#2072 adds API surface, so it is a minor bump, not a patch. It is deliberately not a 1.0.0.* version: hatch-vcs already mints 1.0.0.devN for every commit on main, and an image carrying the 0.33 Alembic head under a v1 label would be a downgrade trap — which #2309's own new check would then hard-refuse to boot on.

Ordering: 0.33.1 < 0.34.0rc1 < 1.0.0.devN < 1.0.0.

Once merged, the tag v0.34.0rc1 is cut here and docker-publish.yml is dispatched with latest unticked, producing lfenergy/flexmeasures:v0.34.0rc1 only.

🤖 Generated with Claude Code

Copilot AI and others added 3 commits July 14, 2026 18:01
@Flix6x
@nhoening @claude
Add soft constraint analysis (unresolved/resolved SoC constraints) to…
013b7aa 
… scheduling job results (#2072)

* Initial plan

* feat: compute first unmet soc-minima/soc-maxima targets in storage scheduler

- Add SchedulingJobResult dataclass (JSON-serializable) to store job results
- Modify _build_soc_schedule to also return per-device MWh SoC schedules,
  including for devices with soc-minima/soc-maxima constraints but no SoC sensor
- Add _compute_unresolved_targets to find the first violated soc-minima/soc-maxima
- StorageScheduler.compute() now includes scheduling_result in return_multiple output
- make_schedule() stores SchedulingJobResult in rq_job.meta["scheduling_result"]
- get_schedule API endpoint returns scheduling_result next to scheduler_info
- Document that soc-targets are hard constraints (not reported in unresolved_targets)
- Add tests: test_unresolved_targets_soc_minima and test_unresolved_targets_none_when_met
- Add changelog entry for PR #2072

Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/710e6bc9-87d9-4238-9c3f-c79a445aff3e

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* refactor: address code review comments on unresolved targets feature

- Add docstring to SchedulingJobResult.unresolved_targets documenting the dict structure
- Rename has_soc_constraints to has_soc_minima_maxima for clarity
- Move SchedulingJobResult import to module level in storage.py
- Use self.round_to_decimals for delta precision (defaults to 6)
- Return None (not {}) from get_schedule when scheduling_result not in job meta
- Mark scheduling_result as nullable in OpenAPI schema
- Improve test docstring to clarify efficiency assumptions

Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/710e6bc9-87d9-4238-9c3f-c79a445aff3e

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* fix: address coordinator findings in unresolved targets feature

- Fix multi-device ordering bug: now finds earliest violation across ALL devices
  instead of stopping at first device with a violation
- Normalize violation datetime to UTC in isoformat() output
- Round soc_schedule_mwh to round_to_decimals precision before comparison
  to avoid epsilon false positives from floating-point arithmetic
- Add test_unresolved_targets_soc_maxima to cover the soc-maxima violation path

Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/710e6bc9-87d9-4238-9c3f-c79a445aff3e

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* refactor: introduce SCHEDULING_RESULT_KEY constant and clean up precision comment

- Define SCHEDULING_RESULT_KEY constant in storage.py to avoid magic strings
- Use the constant in compute(), make_schedule(), and get_schedule API
- Add explanatory comment for round_to_decimals fallback precision

Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/710e6bc9-87d9-4238-9c3f-c79a445aff3e

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* scheduling: track unmet targets per sensor with positive kWh deltas

Context:
- Review feedback on the 'compute first unmet targets' feature requested
  per-sensor tracking, always-positive deltas with units, and omitting
  scheduling_result from the API response for legacy jobs.

Change:
- SchedulingJobResult.unresolved_targets is now keyed by sensor ID string
  (SoC sensor if available, else power sensor), with per-device constraint
  violations; an empty dict means all targets were met.
- _compute_unresolved_targets returns per-device violations only (no cross-
  device earliest logic); delta is always positive in kWh as a string.
- sensors.py omits scheduling_result from the response entirely for legacy
  jobs (was returning null); OpenAPI description updated accordingly.
- Tests updated to assert the new structure and exact delta values.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* scheduling: guard against missing power sensor in _compute_unresolved_targets

Change:
- Skip devices where neither SoC sensor nor power sensor is available,
  rather than crashing with AttributeError on None.id.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* scheduling: add resolved_targets and rename delta→unmet in scheduling result

Context:
- Review feedback on the "compute first unmet targets" feature
- unresolved_targets previously used "delta" key and fell back to power
  sensor when no SoC sensor was set

Change:
- Rename "delta" → "unmet" in unresolved_targets entries for clarity
- Add resolved_targets field: tracks soft constraints that WERE met,
  reporting the tightest (smallest margin) slot per sensor
- Only use state-of-charge sensors as keys; skip devices without one
- _compute_unresolved_targets now returns (unresolved, resolved) tuple
- Update to_dict/from_dict to include resolved_targets
- Update OpenAPI docstring in sensors.py for both fields
- Update tests: add SoC sensor fixtures with unique names, update
  assertions to use "unmet" key and check resolved_targets
- Add "The schedule" section to scheduling.rst documenting both fields

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* scheduling: clarify margin invariant comments and improve SoC docs note

Context:
- Code review flagged potential ambiguity about margin sign in resolved_targets
- Docs note lacked guidance on how to configure the state-of-charge sensor

Change:
- Add inline comments explaining that violations.empty guarantees margins >= 0
  for both soc-minima and soc-maxima resolved branches
- Expand the note in scheduling.rst to mention the flex model field syntax

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* docs: update changelog entry to reflect current scheduling_result format (unmet/margin in kWh)

Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/34e9c4eb-65c2-45d1-8a93-a6f159c4d0a3

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* AGENTS.md: learned to verify merge status with git log --left-right before claiming conflicts resolved

Signed-off-by: F.N. Claessen <claessen@seita.nl>

* api: create jobs endpoint for scheduling result details

Add new GET /api/v3_0/jobs/{uuid} endpoint that retrieves detailed
constraint analysis from scheduling jobs. The endpoint returns
unmet and resolved soft constraints (soc-minima and soc-maxima)
organized by asset, with timestamps and magnitude/margin values.

Includes comprehensive OpenAPI docstring with multiple examples:
- All constraints met with no violations
- Some constraints unmet during optimization
- No constraints defined

This endpoint provides an alternative to retrieving results embedded
in the sensor schedule endpoint, and is useful for dashboards,
monitoring, and fleet management systems that need constraint
analysis without the full schedule timeseries.

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* api/jobs: add asset-keyed transformation for scheduling results

Context:
- Scheduling results are currently stored with sensor ID as key in job.meta
- API endpoint needs to return results with asset ID as primary key
- Problem statement requests moving results from sensor-centric to asset-centric API

Change:
- Add _transform_sensor_keyed_to_asset_keyed() helper to convert sensor-keyed to asset-keyed format
- Update get_job_result() to transform scheduling_result from job.meta before returning
- Result format now includes both asset and sensor information per the API specification

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* Merge origin/main and implement asset-keyed scheduling results

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* api/jobs: rename JobResultAPI to JobAPI for consistency

Context:
- During merge conflict resolution, JobResultAPI (from copilot branch) was kept
- origin/main expects JobAPI to be imported and registered
- Both classes serve the same endpoint (/api/v3_0/jobs/<uuid>)

Change:
- Rename JobResultAPI to JobAPI to match expected import/registration
- Endpoint functionality remains unchanged: returns scheduling result details

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* AGENTS.md: document merge and implementation session details

Context:
- Completed merge of origin/main into copilot/compute-first-unmet-targets
- Implemented asset-keyed scheduling results transformation

Change:
- Expanded lesson learned section with specific details about the merge
- Documented code changes made to API endpoint
- Recorded patterns discovered during implementation
- Added insights from code review and merge resolution process

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* api/v3_0/sensors: remove scheduling_result from get_schedule endpoint

Context:
- Users should use the jobs endpoint for constraint analysis
- Simplifies the sensor schedule endpoint to focus on schedule values

Change:
- Removed scheduling_result field from response body
- Removed scheduling_result documentation from OpenAPI schema
- Added cross-reference to jobs endpoint in description
- Removed SCHEDULING_RESULT_KEY import (no longer needed)

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* docs: remove scheduling_result from scheduling.rst

Context:
- Users should use the jobs endpoint for constraint analysis
- scheduling_result field removed from sensor schedule endpoint

Change:
- Removed description of scheduling_result field
- Removed note about state-of-charge sensor requirement
- Removed sensor-keyed example from documentation
- Updated "Accessing constraint results" section to focus on jobs endpoint
- Simplified interpretation section to reference only jobs endpoint format

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* docs: update changelog to reflect scheduling_result removal

Context:
- scheduling_result field removed from sensor schedule endpoint
- constraint analysis now only available via jobs endpoint

Change:
- Removed reference to scheduling_result from changelog
- Updated jobs endpoint entry to mention constraint analysis feature

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* api/v3_0/jobs: fix import path for job_status_description

Context:
- Import error prevented module from loading

Change:
- Changed import from api.common.utils.api_utils to data.services.utils
- Matches correct location of job_status_description function

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* api/scheduling: rename key names to unresolved/resolved (listings) and violation/margin (values)

Applies consistent key names across all layers of the scheduling result API:
- Listings: unresolved / resolved (was unresolved_targets/resolved_targets internally, unmet/resolved in API)
- Values: violation / margin (was unmet / margin)

Updated files:
- scheduling_result.py: rename dataclass fields and serialization keys
- storage.py: update SchedulingJobResult constructor and violation dict keys
- jobs.py: update API listing keys, field access, and OpenAPI spec
- test_storage.py: update field access and value key assertions
- scheduling.rst: update docs with correct key names and fix example values

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* docs/changelog: move constraint analysis entry to v1.0.0, fix version and scope

- Revert v0.33.0 jobs endpoint entry to original (status and message only)
- Add v1.0.0 entry for soft constraint analysis via GET /api/v3_0/jobs/<uuid>
  with correct PR #2072 reference

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* docs: fix scheduling.rst field-name errors and add API changelog v3.0-32 entry

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* planning/storage: key unresolved targets by asset id instead of soc sensor id

_compute_unresolved_targets previously skipped devices without a
state_of_charge Sensor and keyed results by str(state_of_charge_sensor.id).

Now: derive device_key from the power sensor's generic_asset.id (preferred)
or fall back to the power sensor's own id. Devices that have soc_minima or
soc_maxima constraints but no SoC sensor are therefore included, consistent
with _build_soc_schedule which already computes soc_schedule_mwh for those
devices. Devices for which no key can be derived (no 'sensor' in flex_model_d)
are skipped as before.

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* docs(scheduling_result): describe unresolved/resolved as keyed by asset ID

Update SchedulingJobResult class and field docstrings to reflect that
the outer dicts are keyed by asset ID string rather than
state-of-charge sensor ID string, consistent with the class-level
description and the asset-keyed API response design.

- 'per-sensor' → 'per-asset' in class summary sentence
- Field summaries: 'per sensor' → 'per asset'
- 'Keyed by state-of-charge sensor ID string (str(sensor.id))' →
  'Keyed by asset ID string (str(asset.id))'
- Empty-dict notes: replace 'no state-of-charge sensor is set' with
  'no asset has state-of-charge constraints defined'
- 'Devices with no …' → 'Assets with no …'

No production code changed.

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* storage: restore main-branch fixes for soc_min/soc_max None handling and power capacity signature

Context:
- Merge conflict resolution on this branch dropped several origin/main fixes in storage planning.

Change:
- Restored None-safe storage constraints, SensorReference handling, and device power-capacity fallback logic while preserving the scheduling_result feature additions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* tests/planning: update unresolved/resolved assertions to asset ID keys

- Replace str(soc_sensor.id) keys with str(battery.generic_asset.id) in
  test_unresolved_targets_soc_minima, test_unresolved_targets_none_when_met,
  and test_unresolved_targets_soc_maxima, matching the production change in
  94a3446 (StorageScheduler now keys results by asset ID).

- Add test_unresolved_targets_no_soc_sensor: regression test for a device
  with soc-minima constraints but NO state-of-charge sensor configured.
  Verifies that unresolved/resolved dicts are still produced and keyed by
  the asset ID (str(battery.generic_asset.id)), not by a SoC sensor ID.

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* tests/planning: update unresolved/resolved assertions to asset ID keys

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* Fix constraint analysis field names in API and documentation

- jobs.py: Rename 'margins' to 'resolved' to match SchedulingJobResult class
- scheduling.rst: Update example JSON to use 'resolved' instead of 'margins'
- scheduling.rst: Fix documentation text references from 'margins' to 'resolved'

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* api/jobs: fix asset-keyed constraint analysis transform

The constraint analysis results from storage are keyed by asset ID (or
sensor ID as fallback), not sensor ID alone. The previous transform
function incorrectly tried to look up sensor IDs that were actually asset
IDs, causing constraint analysis to fail silently.

This fixes the transform to handle asset-keyed data correctly and return
a list format suitable for the API response.

- Rename _transform_sensor_keyed_to_asset_keyed to _transform_asset_keyed_to_list
- Update function to process asset-keyed input directly
- Returns list of constraint entries with asset ID

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* feat: restore missing StorageScheduler.compute() consumption/production handling

Restores code lost during merge with origin/main (commit 7f56bfa).

Changes:
- Adds @staticmethod _build_consumption_production_schedules() method
  - Handles flex-model consumption/production sensor definitions
  - Maintains proper sign conventions (consumption positive, production negative)
  - Clips schedules appropriately when both sensors are defined
  - Applies unit conversion from MW to sensor units

- Integrates method into StorageScheduler.compute():
  - Calls _build_consumption_production_schedules() after SoC schedule
  - Resamples consumption/production schedules when needed
  - Rounds schedules with rounding precision
  - Includes consumption/production schedules in return_multiple result
  - Identifies output sensors correctly (consumption vs. production)

Impact: Fixes all 14 test failures from missing schedule persistence
- 12 API tests that expected 400 responses (schedules never stored)
- 1 test expecting 96 production values
- Resolves merge artifact regression

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* agents/api-backward-compatibility-specialist: learned data format mismatch detection patterns

PR #2072 revealed a critical pattern: data transformations between API layers can
silently use incompatible key types (asset_id vs sensor_id), corrupting responses.

Root cause: transform functions had misleading names and no schema validation to
catch the mismatch. Layer 1 produced asset-keyed data, but Layer 2 expected and
treated keys as sensor IDs, resulting in silent data corruption.

New patterns added:
- Function naming must clearly indicate input/output format
- Add Marshmallow response schemas to validate data flow end-to-end
- Integration tests must verify data semantics, not just null checks
- Document key types in docstrings

This matters for backward compatibility because silent data corruption breaks
client contracts in subtle, hard-to-detect ways.

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* docs(scheduling_result): consolidate docstrings and clarify asset-keyed format

- Enhanced class docstring with explicit 'Core Purpose', 'Backward Compatibility Note', and 'Structure' sections
- Clarified that results are keyed by asset ID (not sensor ID)
- Emphasized that scheduling_result is only available via jobs endpoint, not sensor endpoint
- Streamlined field docstrings to eliminate redundancy while preserving detailed examples
- Added migration guidance for clients moving from sensor endpoint to jobs endpoint

This addresses backward compatibility concerns about the move from sensor endpoint
to jobs endpoint and ensures clear API contract documentation.

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* api/jobs: clarify scheduling_result response format and backward compatibility

- Enhanced transformation function docstring to remove internal storage details
- Clarified that results are asset-keyed in internal format, list format in API response
- Removed misleading mention of 'sensor ID fallback' from public API documentation
- Enriched endpoint OpenAPI docstring with scheduling_result structure details
- Added explicit backward compatibility note about migration from sensor endpoint
- Added inline comments explaining the format transformation logic

This ensures the API contract is clear about:
- Asset-keyed storage format (not sensor-keyed)
- Jobs endpoint as the authoritative source (not sensor endpoint)
- The format transformation from internal to API representation

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* docs: apply PR #2072 documentation review suggestions

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* storage: clarify constraint analysis docstring - asset keying and satisfied constraints

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* agents/test-specialist: learned data format transformation testing patterns from PR #2072

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* agents/documentation-developer-experience-specialist: learned cross-document consistency patterns from PR #2072

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* agents/architecture-domain-specialist: learned asset-keying patterns and prevent format mismatches from PR #2072

- Documented Asset ID Keying Pattern showing how scheduler/constraint analysis shifts from sensor-keyed to asset-keyed organization
- Identified silent data corruption risk when format mismatches cross layer boundaries
- Added guidance on identifying data flow stages (storage → API → client)
- Provided pattern for preventing format mismatches with Marshmallow schemas
- Documented domain invariant: Asset ID is authoritative key, multiple sensors per asset
- Added end-to-end testing pattern to verify transformation correctness
- Captured lessons from PR #2072 review in Lessons Learned section

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* AGENTS.md: learned self-improvement enforcement from PR #2072 review session

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* docs: clarify transformation rationale and fix constraint grammar

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* agents/test-specialist: strengthen type checking in transformation test examples

Add explicit type check before key assertion to catch type mismatches early.
The pattern demonstrates checking isinstance(result, dict) before calling .keys(),
preventing silent failures when result is None, a list, or other unexpected type.

Improves: Data Format Transformation Testing best practices section (lines 617-620)

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* api: clarify endpoint backward compatibility and constraint reporting docstrings

Updates docstrings to remove ambiguity about backward compatibility status:

- Constraint analysis is EXCLUSIVELY available via GET /api/v3_0/jobs/<uuid>
  (not removed, but endpoint changed to not include constraint analysis)
- Sensor schedule endpoint (GET /api/v3_0/sensors/<id>/schedules/<job_id>)
  still exists and returns power values only
- Clarify that soc-targets are not included in SchedulingJobResult because
  they are modelled as hard constraints strictly enforced by the scheduler
  (not a vague 'are not reported here')

Affects:
- flexmeasures/data/services/scheduling_result.py: Backward compatibility note,
  constraint reporting explanation
- flexmeasures/api/v3_0/jobs.py: Constraint analysis availability statement

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* refactor: follow up on prevention advice with a new instruction file; merge&simplify two lessons learned

Signed-off-by: F.N. Claessen <claessen@seita.nl>

* docs: merge and simplify SchedulingJobResult docstrings

Signed-off-by: F.N. Claessen <claessen@seita.nl>

* style: format docstrings

Signed-off-by: F.N. Claessen <claessen@seita.nl>

* feat: add docstring instruction regarding line breaks

Signed-off-by: F.N. Claessen <claessen@seita.nl>

* style: in docstrings, use :returns: instead of :return:

Signed-off-by: F.N. Claessen <claessen@seita.nl>

* Changes before error encountered

Agent-Logs-Url: https://github.com/FlexMeasures/flexmeasures/sessions/59806792-062e-4c1c-ae72-0b749d2bcc07

Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>

* docs: remove irrelevant sensor key from example

Signed-off-by: F.N. Claessen <claessen@seita.nl>

* feat: claude agents reusing copilot instructions

Signed-off-by: F.N. Claessen <claessen@seita.nl>

* tests/planning: update unresolved/resolved test assertions to list format

Context:
- Production code (storage.py, scheduling_result.py) reports unresolved/
  resolved targets as a list of dicts keyed by "asset" (per PR #2072
  review guidance), but 4 tests in test_storage.py still asserted the
  older dict-keyed-by-asset-ID-string format, causing failures.

Change:
- Updated the 4 affected tests to look up entries via
  next(e for e in ... if e["asset"] == asset_id) instead of dict
  indexing, and compare empty results against [] instead of {}.
- Full flexmeasures/data/models/planning/ and flexmeasures/data/services/
  test suites re-run: 166 passed, 3 xfailed, 0 failed.

* api/v3_0/jobs: restore main's job-status endpoint, graft on scheduling_result

Context:
- The crashed Copilot session's diff-minimization attempt (commit
  7ac94d2) rewrote get_job_status() from scratch instead of minimizing
  the diff, introducing an access-control regression: it only resolved
  read-context via job.meta["asset_or_sensor"], dropping main's fallback
  to job.meta["sensor_id"] / job.meta["forecast_kwargs"]["sensor_id"] /
  job.kwargs["sensor_id"]. Forecasting/ingestion jobs (e.g. enqueued via
  api_utils.py's meta={"sensor_id": sensor_id}) would have bypassed
  check_access() entirely, letting any authenticated user read another
  account's job status/result by UUID.
- It also scoped the Redis connection to current_app.queues["scheduling"]
  instead of the generic current_app.redis_connection (this endpoint
  serves all job types), dropped the 404 response's "status" key, and
  stripped almost all OpenAPI response documentation and examples.

Change:
- Restored jobs.py to main's structure (_job_read_context,
  _isoformat_or_none, _job_queue_unavailable_response, full OpenAPI
  schema/examples, response shape).
- Grafted on only what's needed for the constraint-analysis feature: a
  SCHEDULING_RESULT_KEY import from storage.py, a "scheduling_result"
  field in the response (populated from job.meta) when present, its
  OpenAPI schema/example, and a doc note cross-referencing this endpoint
  from the sensor schedule endpoint's description.
- Regenerated openapi-specs.json accordingly (also fixes a stray
  pre-existing "flex_model": {} entry unrelated to this endpoint, which
  a clean regeneration replaces with the correct "flex-model" schema).

* data/models/planning: restore main's storage scheduler, graft on unresolved targets

Context:
- The crashed Copilot session's diff-minimization attempt reintroduced
  ensure_soc_min_max()/get_min_max_soc_from_asset(), which main
  deliberately deleted (commit 1b1d630: "ensure_soc_min_max is
  obsolete; the StorageScheduler runs fine without hard constraints on
  the SoC", part of PR #2221 "soc min does not need to be mandatory").
  It also narrowed build_device_soc_values()'s signature, dropping the
  ur.Quantity and None input handling that main's current version
  requires precisely because of PR #2221 (soc-min/soc-max can now be
  None). And it reworded several unrelated docstrings.

Change:
- Reset storage.py to main's content, then grafted on only what the
  soft constraint analysis feature needs: the SchedulingJobResult
  import, the SCHEDULING_RESULT_KEY constant, _build_soc_schedule()
  returning (soc_schedule, soc_schedule_mwh) and covering devices
  without a state-of-charge sensor, the new _compute_unresolved_targets()
  method, and compute() building/returning the "scheduling_result" entry.
- Full flexmeasures/data/models/planning/ and flexmeasures/data/services/
  test suites re-run: 224 passed, 3 xfailed, 0 failed.

* AGENTS.md: remove redundant self-referential lessons-learned entry

Context:
- PR #2072 review comment flagged this entry as redundant ("Is there a
  better place to implement this rule than in a lessons learned in the
  AGENTS.md?" / "Seems redundant.").
- The "2026-07-01 PR #2072 review comments" entry only restated the
  already-existing "Must Enforce Agent Self-Improvement" and "Session
  Close Checklist" policy elsewhere in this file, without adding a new
  actionable pattern.

Change:
- Removed the entry. Kept the unrelated, unflagged "2026-06 feature
  branch sync" lesson and its "Branch in sync with main" checklist item.

* style: in docstrings, use :returns: instead of :return:

Signed-off-by: F.N. Claessen <claessen@seita.nl>

* style: format docstrings

Signed-off-by: F.N. Claessen <claessen@seita.nl>

* docs(api): document soc-minima/soc-maxima as lists in jobs endpoint

_compute_unresolved_targets now always returns a list of
datetime/violation (or margin) entries per constraint key, covering
every violated or met slot by default, instead of a single dict for
the first violation or tightest margin. Update the OpenAPI docstring
prose and finished-job example on GET /api/v3_0/jobs/<uuid>
accordingly, and regenerate openapi-specs.json to match.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLBuNdhpXdpHgq2ZDLUUq8

* docs(scheduling): describe soc-minima/soc-maxima results as lists

The "Accessing constraint results" section described each
soc-minima/soc-maxima entry as a single datetime/violation (or
margin) value for the tightest or first-violated slot. Update the
field descriptions and JSON example to reflect that these are always
lists, containing one entry per violated or met slot by default, to
match the current jobs.py OpenAPI docs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLBuNdhpXdpHgq2ZDLUUq8

* tests/planning: update unresolved-target assertions for list-wrapped constraints

Context:
- _compute_unresolved_targets now always wraps each soc-minima/soc-maxima
  entry under `unresolved`/`resolved` in a list (supporting the new `all`
  flag), instead of a single dict for the first/tightest slot.

Change:
- Update the four existing unresolved-target tests to index into the
  single-element list (`entry["soc-minima"][0][...]`) and assert the
  list has exactly one element, since each test only defines a single
  constraint datetime.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLBuNdhpXdpHgq2ZDLUUq8

* tests/planning: cover the all=True/False flag of _compute_unresolved_targets

Context:
- _compute_unresolved_targets gained an `all` parameter (default True):
  report every violated/met slot, or (all=False) only the first violation /
  tightest margin. This behavior had no dedicated coverage yet.

Change:
- Add test_unresolved_targets_all_flag_soc_minima_violations: three
  soc-minima checkpoints are all unreachable given a capacity-limited
  battery; asserts the default (all=True) reports all three violated
  slots in chronological order, and that all=False (invoked directly via
  a spy on the private method) reports only the first.
- Add test_unresolved_targets_all_flag_soc_minima_resolved_margins:
  two soc-minima checkpoints (one tight, one slack) are both met with
  different margins; asserts all=True reports both margins and all=False
  reports only the tightest one.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLBuNdhpXdpHgq2ZDLUUq8

* data/models/planning: add all=True/False to _compute_unresolved_targets

Reports every violated/met slot per constraint by default, instead of
only the first violation or tightest margin. Constraint entries are
now always lists, even when all=False keeps just the single entry.

* data/models/planning: rename all to most_relevant_only, default False

Avoids shadowing the all() builtin. Flips the polarity so the
new, more informative multi-slot reporting is the implicit default.

* tests/planning: rename all to most_relevant_only, default False

Follows the production code rename in _compute_unresolved_targets.

* docs: fix field-name and stale-example inconsistencies in job result docs

documentation/api/change_log.rst named the wrong field (result instead
of scheduling_result), and SchedulingJobResult's docstring example
still showed the pre-list-wrapping single-dict-per-constraint shape.

* api/v3_0, data/services: carry scheduling result via job return value, not meta

Context:
- Maintainer decided the soft SoC constraint analysis produced by
  StorageScheduler should be surfaced through the job's actual return
  value (job.return_value()) instead of a bolted-on `scheduling_result`
  field sourced from RQ job.meta.

Change:
- make_schedule() now returns the SchedulingJobResult dict when the
  scheduler produced one, or True otherwise (return type: bool | dict).
  This also fixes direct (non-RQ) make_schedule() calls, which
  previously silently dropped the constraint-analysis data.
- GET /api/v3_0/jobs/<uuid> no longer reads job.meta[SCHEDULING_RESULT_KEY]
  or exposes a separate `scheduling_result` field; the data now arrives
  automatically via the existing `result` field.
- Updated the OpenAPI docstring/schema/examples in jobs.py accordingly
  and regenerated flexmeasures/ui/static/openapi-specs.json.
- Updated the SchedulingJobResult docstring to point at the `result`
  field instead of the now-removed `scheduling_result` field.

Note: this is a breaking change for every finished StorageScheduler job,
not only ones whose flex model defines soc-minima/soc-maxima — see the
API changelog entry for details.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLBuNdhpXdpHgq2ZDLUUq8

* docs: describe result field carrying scheduling constraint analysis

Context:
- Follows the previous commit, which stopped exposing a separate
  `scheduling_result` field on GET /api/v3_0/jobs/<uuid> and instead
  surfaces soft SoC constraint analysis via the existing `result` field.

Change:
- Updated documentation/features/scheduling.rst's constraint-results
  section and worked example to reference `result` instead of
  `scheduling_result`, and to clarify that a finished StorageScheduler
  job always returns the analysis object (with empty arrays when no
  soc-minima/soc-maxima are defined), while other scheduling jobs keep
  returning `true`.
- Marked the v3.0-32 API changelog entry as a breaking change: for
  every finished StorageScheduler job (not only ones with
  soc-minima/soc-maxima defined), `result` is now an object instead of
  the boolean `true` it used to return unconditionally. Flagged the
  risk for external integrators (e.g. flexmeasures-client, or custom
  scripts) that check `result === true`/`result is True` unconditionally.
- Reworded the main changelog's PR #2072 one-liner so it no longer
  reads as purely additive.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLBuNdhpXdpHgq2ZDLUUq8

* api/v3_0/tests: cover result-field shape for StorageScheduler jobs

Context:
- GET /api/v3_0/jobs/<uuid> no longer exposes a separate
  `scheduling_result` field; scheduling constraint analysis now arrives
  via `result` directly (see the preceding scheduling.py/jobs.py commit).

Change:
- Fixed the stale comment on test_get_job_status_finished's assertion
  and tightened it to assert the exact StorageScheduler result shape
  ({"unresolved": [], "resolved": []}), since this test's flex model
  defines no soc-minima/soc-maxima.
- Added test_get_job_status_finished_with_unresolved_soc_minima,
  driving an unreachable soc-minima (soft constraint via a breach
  price) through the real trigger -> work_on_rq -> job-status flow, and
  asserting `result` contains the expected unresolved entry and that
  the old `scheduling_result` field is really gone.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLBuNdhpXdpHgq2ZDLUUq8

* docs: point out that FM Client is unaffected

Signed-off-by: F.N. Claessen <claessen@seita.nl>

* style: more strategic linebreaks

Signed-off-by: F.N. Claessen <claessen@seita.nl>

* api/v3_0, data/services: make scheduling jobs always return a dict

Context:
- Maintainer wants result consistently a dict for every scheduling
  job, not just StorageScheduler jobs with soft SoC constraints, to
  simplify the docs and the UX of GET /api/v3_0/jobs/<uuid>.

Change:
- make_schedule() now always returns a dict: the SchedulingJobResult
  dict when the scheduler produced one, otherwise an empty {} (for
  now, pending a proper result spec for other schedulers).
- Adapted the one caller relying on the old truthy boolean
  (cli/data_add.py's `add schedule` command): an empty dict is falsy,
  so the "New schedule is stored." message no longer depends on the
  return value's truthiness.
- Updated jobs.py's OpenAPI docstring/schema/example accordingly and
  regenerated openapi-specs.json.

* docs: describe result as always a dict for scheduling jobs

Follows the previous commit. Updated scheduling.rst's constraint-
results section, the v3.0-32 API changelog breaking-change note, and
the main changelog's PR #2072 one-liner to state that every finished
scheduling job now returns an object for `result`, not only
StorageScheduler jobs with soc-minima/soc-maxima defined.

* tests: cover always-dict result for non-StorageScheduler jobs too

- Fixed a stale comment in test_get_job_status_finished.
- Added a regression assertion to test_add_process (ProcessScheduler,
  called directly via CLI, not via RQ) that "New schedule is stored."
  still prints now that make_schedule() can return a falsy {} on
  success.

* docs: the FM Client is still unaffected

Signed-off-by: F.N. Claessen <claessen@seita.nl>

* tests: tolerate float rounding in unit-conversion assertions

Python 3.10 resolves an older pint/numpy (0.24.4/2.2.6 vs 0.25.3/2.4.6 on
3.11+, per uv.lock), which can produce sub-1e-12 float noise after unit
conversion. Exact equality then flakes on some runner architectures even
though the values are correct to any sane precision.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* tests: tolerate float noise in quantity string comparisons too

pytest.approx on the magnitude (added previously) doesn't cover the
str(Quantity) comparisons in the same test, which still fail on Python
3.10's older pint (0.24.4) for cases like 914.7 EUR/kWh -> kEUR/MWh,
where the converted magnitude carries ~1e-13 float noise that shows up
in the string repr (e.g. "914.7000000000003" instead of "914.7").

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015oSbjKoErHkpDmeBjkxinY

* data/services: default scheduling result to unresolved/resolved shape

Per review, non-StorageScheduler jobs returned an empty {} result
where StorageScheduler jobs return {"unresolved": [...], "resolved":
[...]}. Default to the same two-key shape for all scheduling jobs, so
callers can rely on one consistent result structure regardless of
scheduler.

Also clarify the scheduling_result.py docstring's cross-reference to
the docs by naming the section explicitly, since "scheduling_constraint_results"
is a Sphinx anchor, not a string a reader would search the docs for.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015oSbjKoErHkpDmeBjkxinY

* docs: address PR #2072 review comments

- Simplify the API/main changelog entries for the scheduling result
  field per review suggestion, dropping the now-inaccurate note that
  non-StorageScheduler jobs return a different (empty-object) shape.
- Restore the missing "Work on other schedulers" section header in
  scheduling.rst (lost in an earlier edit).
- Fix a duplicated "Checking the status via the API" header; the
  second one covers the CLI command, not the API.
- Add "if the battery is in an EV, charge en-route" to the shortfall
  use-case bullets.
- Explicitly state that `violation`/`margin` are always positive
  magnitudes, never negative.
- Fix a stale session-date placeholder in the architecture specialist's
  lessons-learned log.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015oSbjKoErHkpDmeBjkxinY

* tests: fix stale comment about ProcessScheduler's result shape

The comment said make_schedule() returns an empty dict; it now
returns {"unresolved": [], "resolved": []} for every scheduler,
per the previous commit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015oSbjKoErHkpDmeBjkxinY

* docs: move schedule inspection section above constraint results

Per nhoening's review comment, the high-level "Inspecting schedules"
section (errors, job status via API/CLI, asset status page, RQ
dashboard) belongs before the more detailed "Accessing constraint
results" section, not after it — readers should learn how to check
on a job in general before diving into constraint-analysis specifics.

Also cross-reference the constraint-results section from the API
status-check paragraph, since that's precisely where the data lives.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015oSbjKoErHkpDmeBjkxinY

* fix: stop pruning dependencies from externally-installed plugins when updating the docs

Signed-off-by: F.N. Claessen <claessen@seita.nl>

* style: fix spacing after image

Signed-off-by: F.N. Claessen <claessen@seita.nl>

* docs: actually cross-reference the docs section

Signed-off-by: F.N. Claessen <claessen@seita.nl>

---------

Signed-off-by: Nicolas Höning <nicolas@seita.nl>
Signed-off-by: F.N. Claessen <claessen@seita.nl>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Flix6x <30658763+Flix6x@users.noreply.github.com>
Co-authored-by: Nicolas Höning <nicolas@seita.nl>
Co-authored-by: F.N. Claessen <claessen@seita.nl>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
(cherry picked from commit a737210)
Signed-off-by: F.N. Claessen <felix@seita.nl>
@nhoening @Flix6x
Chore: check on startup if the db schema is on alembic head (#2309)
b954d59 
* chore: check on startup if the db schema is on alembic head and print a warning otherwise - useful for hosts. Concrete use case: do not creste templates in this case, so db CLI commands are still free to run.

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* add changelog entry

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* fix missing app context

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* do not show the warning when we already run the upgrade, also show the actual revisions to be more useful

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* upgrade the warning to error status to increase visibility, add the revision identifiers as well, adapt warning if db connectivity is the problem

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* more robust check if we are running db upgrade

Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
Signed-off-by: Nicolas Höning <iam@nicolashoening.de>

* do without the util function database_schema_is_migrated_to_head()

Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
Signed-off-by: Nicolas Höning <iam@nicolashoening.de>

---------

Signed-off-by: Nicolas Höning <nicolas@seita.nl>
Signed-off-by: Nicolas Höning <iam@nicolashoening.de>
Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
(cherry picked from commit 8d775f6)
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x @claude
Prepare v0.34.0rc1 pre-release on the 0.33 line
2a4236f 
Changelog section for v0.34.0rc1, stamp the OpenAPI spec's info.version to
match what a v0.34.0rc1 build reports, and carry over the docker-publish
`:latest` guard (#2316) so a manual image build is safe from this ref too.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: F.N. Claessen <felix@seita.nl>
@read-the-docs-community

read-the-docs-community Bot commented Jul 14, 2026

Copy link
Copy Markdown

Documentation build overview

📚 flexmeasures | 🛠️ Build #33585835 | 📁 Comparing 2a4236f against latest (c35554a)

  🔍 Preview build  

82 files changed · ± 72 modified · - 10 deleted

± Modified

- Deleted

@Flix6x Flix6x mentioned this pull request Jul 14, 2026
Open
@Flix6x

Flix6x commented Jul 14, 2026

Copy link
Copy Markdown
Member Author

Superseded by #2320, which also backports #2209.

With #2209 applied first, #2072's patch lands byte-for-byte identically to main — the SensorReference stripping this PR needed is no longer necessary. Closing in favour of that cleaner history.

@Flix6x Flix6x closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Flix6x @nhoening