Skip to content

Conversation

dominiquekleeven
Copy link
Collaborator

@dominiquekleeven dominiquekleeven commented Sep 22, 2025

Draft PR – Migrating Forecasting Models to Darts

Closes #29
Branched from #43

This PR introduces a major structural change to both the back end and front end by migrating native forecasting model usage and bindings to the Darts framework.

Changes

  • Migrates the existing Prophet model to DartsProphet.
  • Adds a new XGBoost model with its own configuration.
  • Models are now stored in .pkl format, the default serialization used by Darts for trained/fitted models. The model_storage_service now uses Darts bindings to save and load models.
  • Updates terminology for consistency:
    • In general, regressors are now referred to as covariates.
    • In the context of Prophet, the term regressor is still used, as it remains technically accurate.
  • Adds support for both past and future covariates:
    • Future covariates require both historical and predicted data.
    • Past covariates require only historical data.
    • For reference: a Prophet regressor is essentially a future covariate, while XGBoost and other models also support past covariates.
  • Introduces a dedicated model registry on the front end, which registers model configs and makes them available to the editor. Each forecast model now has its own config containing base settings, handlers, and templates.

Status

Work in progress.

commit 3862e03
Author: Dominique Kleeven <[email protected]>
Date:   Thu Sep 18 19:35:06 2025 +0200

    Revert "Merge branch 'main' into enhancement/openremote-client-service-register"

    This reverts commit 95c9591, reversing
    changes made to c4edea6.

commit 95c9591
Merge: c4edea6 dd9f984
Author: Dominique Kleeven <[email protected]>
Date:   Thu Sep 18 19:34:14 2025 +0200

    Merge branch 'main' into enhancement/openremote-client-service-register

commit c4edea6
Author: Dominique Kleeven <[email protected]>
Date:   Thu Sep 18 12:44:42 2025 +0200

    ClientRoles duplicate

commit fe117c8
Author: Dominique Kleeven <[email protected]>
Date:   Tue Sep 16 11:24:20 2025 +0200

    Fix test failure and typing changes for registration

commit 17a36d0
Merge: 78bad1e ff8a0fb
Author: Dominique Kleeven <[email protected]>
Date:   Tue Sep 16 11:09:31 2025 +0200

    Merge branch 'feature/keycloak-middleware-clean' into enhancement/openremote-client-service-register

commit 78bad1e
Author: Dominique Kleeven <[email protected]>
Date:   Tue Sep 16 11:07:09 2025 +0200

    instanceId is now an integer

commit ff8a0fb
Author: Dominique Kleeven <[email protected]>
Date:   Wed Sep 3 15:22:53 2025 +0200

    Update route_exception_handlers.py

commit 880395c
Author: Dominique Kleeven <[email protected]>
Date:   Wed Sep 3 11:10:54 2025 +0200

    Fix failing test and format

commit 22aea98
Author: Dominique Kleeven <[email protected]>
Date:   Wed Sep 3 11:08:46 2025 +0200

    Only include the roles that are actually usable for services

commit 6b4e2fc
Author: Dominique Kleeven <[email protected]>
Date:   Wed Aug 20 19:09:13 2025 +0200

    Unnecessary since these tests were covered by the forecast tests themselves

commit 437a394
Author: Dominique Kleeven <[email protected]>
Date:   Wed Aug 20 19:09:13 2025 +0200

    Unnecessary since these tests were covered by the forecast tests themselves

commit 67d81b9
Author: Dominique Kleeven <[email protected]>
Date:   Sat Aug 16 11:06:26 2025 +0200

    Support additional service fields

commit 21a5484
Author: Dominique Kleeven <[email protected]>
Date:   Fri Aug 15 14:00:18 2025 +0200

    Format

commit e255c5c
Author: Dominique Kleeven <[email protected]>
Date:   Fri Aug 15 13:59:58 2025 +0200

    Format

commit df21082
Merge: a793b27 4a18dfe
Author: Dominique Kleeven <[email protected]>
Date:   Fri Aug 15 13:58:28 2025 +0200

    Merge branch 'feature/keycloak-middleware-clean' into enhancement/openremote-client-service-register

commit 4a18dfe
Author: Dominique Kleeven <[email protected]>
Date:   Fri Aug 15 13:58:11 2025 +0200

    Remove the logging prefix override, and clean up regex for bundle path detection

commit a793b27
Author: Dominique Kleeven <[email protected]>
Date:   Fri Aug 15 13:52:42 2025 +0200

    API changes

commit bb9a48f
Author: Dominique Kleeven <[email protected]>
Date:   Tue Aug 12 18:50:02 2025 +0200

    Update middleware.py

commit f129442
Author: Dominique Kleeven <[email protected]>
Date:   Tue Aug 12 17:06:55 2025 +0200

    Remove unused param

commit 32fd3f9
Author: Dominique Kleeven <[email protected]>
Date:   Tue Aug 12 12:06:55 2025 +0200

    Fix failing tests due to changed service object

commit b80f59c
Author: Dominique Kleeven <[email protected]>
Date:   Tue Aug 12 12:03:07 2025 +0200

    Adjusted registration logic + role changes

commit 155f5c0
Merge: c7ac51b a316e9d
Author: Dominique Kleeven <[email protected]>
Date:   Mon Aug 11 07:59:27 2025 +0200

    Merge branch 'feature/keycloak-middleware-clean' into enhancement/openremote-client-service-register

commit c7ac51b
Author: Dominique Kleeven <[email protected]>
Date:   Mon Aug 11 07:58:24 2025 +0200

    Performance improvement, allow functions to run in parallel rather than in the async event loop

    It is because the underlying calls are synchronous by nature, in FastAPI endpoints marked with async are always ran in the eventloop even when blocking, when non async it attempts to coalasce and run requests in threads

commit a316e9d
Author: Dominique Kleeven <[email protected]>
Date:   Mon Aug 11 07:58:24 2025 +0200

    Performance improvement, allow functions to run in parallel rather than in the async event loop

    It is because the underlying calls are synchronous by nature, in FastAPI endpoints marked with async are always ran in the eventloop even when blocking, when non async it attempts to coalasce and run requests in threads

commit 40ed5be
Author: Dominique Kleeven <[email protected]>
Date:   Fri Aug 8 17:06:25 2025 +0200

    Fallback to relative url for manager config

commit 875811e
Author: Dominique Kleeven <[email protected]>
Date:   Fri Aug 8 17:06:25 2025 +0200

    Fallback to relative url for manager config

commit c21a9b2
Author: Dominique Kleeven <[email protected]>
Date:   Fri Aug 8 16:37:21 2025 +0200

    Revert "Default config change"

    This reverts commit 2f55344.

commit 74f0641
Author: Dominique Kleeven <[email protected]>
Date:   Fri Aug 8 16:37:14 2025 +0200

    Revert "Default config change"

    This reverts commit 611dafc.

commit 2f55344
Author: Dominique Kleeven <[email protected]>
Date:   Fri Aug 8 16:22:10 2025 +0200

    Default config change

commit 611dafc
Author: Dominique Kleeven <[email protected]>
Date:   Fri Aug 8 16:22:10 2025 +0200

    Default config change

commit e366a92
Author: Dominique Kleeven <[email protected]>
Date:   Fri Aug 8 16:11:09 2025 +0200

    Update index.ts

commit 52828e5
Author: Dominique Kleeven <[email protected]>
Date:   Fri Aug 8 16:11:09 2025 +0200

    Update index.ts

commit f9127b1
Author: Dominique Kleeven <[email protected]>
Date:   Thu Aug 7 10:54:22 2025 +0200

    accomodate OpenRemoteClient changes

commit beda4a9
Merge: a7d170e 0f9f434
Author: Dominique Kleeven <[email protected]>
Date:   Thu Aug 7 10:50:54 2025 +0200

    Merge branch 'feature/keycloak-middleware-clean' into enhancement/openremote-client-service-register

commit 0f9f434
Author: Dominique Kleeven <[email protected]>
Date:   Thu Aug 7 10:32:13 2025 +0200

    Enhanced OpenRemoteClient to allow specifying the default realm rather than defaulting to the hardcoded "master"

commit 945b258
Author: Dominique Kleeven <[email protected]>
Date:   Fri Aug 1 10:48:35 2025 +0200

    Remove keycloak-js, since it is no longer needed

commit 1e56000
Author: Dominique Kleeven <[email protected]>
Date:   Fri Aug 1 10:41:39 2025 +0200

    Replace auth boilerplate by re-using manager.init which handles auto login and token refresh

commit a7d170e
Author: Dominique Kleeven <[email protected]>
Date:   Fri Aug 1 09:20:30 2025 +0200

    Attempt re-registration if the service heartbeat could not be sent

commit 73ffb0c
Author: Dominique Kleeven <[email protected]>
Date:   Wed Jul 30 22:54:31 2025 +0200

    Formatting

commit 6086c68
Author: Dominique Kleeven <[email protected]>
Date:   Wed Jul 30 22:51:44 2025 +0200

    Take into account embedded content when displaying service contents

commit 3b1663a
Merge: 3ea1c00 e0c1f74
Author: Dominique Kleeven <[email protected]>
Date:   Wed Jul 30 22:23:08 2025 +0200

    Merge branch 'feature/keycloak-middleware-clean' into enhancement/openremote-client-service-register

commit e0c1f74
Author: Dominique Kleeven <[email protected]>
Date:   Wed Jul 30 22:22:42 2025 +0200

    Update model_scheduler.py

commit 5340f11
Author: Dominique Kleeven <[email protected]>
Date:   Wed Jul 30 21:36:03 2025 +0200

    Increase grace period for job scheduling, job overlap could cause a job to be constantly rescheduled.

commit 3ea1c00
Author: Dominique Kleeven <[email protected]>
Date:   Wed Jul 30 21:15:30 2025 +0200

    Fix test method signatures and resolve linter issues

commit cc07c2e
Author: Dominique Kleeven <[email protected]>
Date:   Wed Jul 30 21:11:30 2025 +0200

    Fix openremote client mocked method signatures and missing types for datapoints in tests

commit b254152
Author: Dominique Kleeven <[email protected]>
Date:   Wed Jul 30 21:04:22 2025 +0200

    Tests for training data request batching and time operations/data preperation

commit 31b1f8e
Author: Dominique Kleeven <[email protected]>
Date:   Wed Jul 30 21:04:22 2025 +0200

    Tests for training data request batching and time operations/data preperation

commit d6358cd
Merge: a318a01 f65ec6e
Author: Dominique Kleeven <[email protected]>
Date:   Wed Jul 30 15:34:34 2025 +0200

    Merge branch 'feature/keycloak-middleware-clean' into enhancement/openremote-client-service-register

commit f65ec6e
Author: Dominique Kleeven <[email protected]>
Date:   Wed Jul 30 15:10:36 2025 +0200

    Resolve potential data cut off in training data consolidation

commit a318a01
Author: Dominique Kleeven <[email protected]>
Date:   Tue Jul 29 07:39:00 2025 +0200

    License header

commit 4d5d40e
Merge: a37667a efcbec6
Author: Dominique Kleeven <[email protected]>
Date:   Tue Jul 29 07:34:25 2025 +0200

    Merge branch 'feature/keycloak-middleware-clean' into enhancement/openremote-client-service-register

commit efcbec6
Author: Dominique Kleeven <[email protected]>
Date:   Tue Jul 29 07:34:06 2025 +0200

    Clearer decorator name for realm access

commit a37667a
Author: Dominique Kleeven <[email protected]>
Date:   Tue Jul 29 07:24:03 2025 +0200

    Resolve incorrect package export, resolve incorrect import path in tests

commit c7193bc
Merge: 0f92205 44bc47f
Author: Dominique Kleeven <[email protected]>
Date:   Tue Jul 29 07:21:28 2025 +0200

    Merge branch 'feature/keycloak-middleware-clean' into enhancement/openremote-client-service-register

commit 44bc47f
Author: Dominique Kleeven <[email protected]>
Date:   Mon Jul 28 22:05:25 2025 +0200

    Mock the issuer provider for the keycloak middleware

commit c6bd020
Author: Dominique Kleeven <[email protected]>
Date:   Mon Jul 28 21:54:33 2025 +0200

    Keycloak middleware refactor and decorator additions

commit 30c9ecb
Author: Dominique Kleeven <[email protected]>
Date:   Mon Jul 28 16:56:04 2025 +0200

    Remove isEmbedded function since it is unused

commit 0f92205
Author: Dominique Kleeven <[email protected]>
Date:   Mon Jul 28 10:27:07 2025 +0200

    Resolve docker image failing after uv workspace addition

commit 38b2db7
Author: Dominique Kleeven <[email protected]>
Date:   Mon Jul 28 03:30:38 2025 +0200

    Clean up and adjustments

commit 352f456
Author: Dominique Kleeven <[email protected]>
Date:   Sun Jul 27 22:32:50 2025 +0200

    Update models.py

commit 45c8346
Author: Dominique Kleeven <[email protected]>
Date:   Sun Jul 27 22:10:41 2025 +0200

    Clean up

commit 8764f57
Author: Dominique Kleeven <[email protected]>
Date:   Sun Jul 27 20:57:54 2025 +0200

    Extended OpenRemoteClient with services resource and registrar scheduler

commit 9096f54
Author: Dominique Kleeven <[email protected]>
Date:   Sun Jul 27 18:01:00 2025 +0200

    Extract OpenRemote Client into package, setup uv workspace and additional testing

commit 6d8a642
Merge: 20416cf fc2332e
Author: Dominique Kleeven <[email protected]>
Date:   Sun Jul 27 16:24:09 2025 +0200

    Merge branch 'feature/keycloak-middleware-clean' into enhancement/openremote-client-service-register

commit fc2332e
Author: Dominique Kleeven <[email protected]>
Date:   Mon Jul 21 12:25:20 2025 +0200

    Minor adjustment to env handling front-end

commit b3497f1
Author: Dominique Kleeven <[email protected]>
Date:   Fri Jul 18 11:55:24 2025 +0200

    Update docker-compose.yml

commit 0dd0e30
Author: Dominique Kleeven <[email protected]>
Date:   Fri Jul 18 11:23:55 2025 +0200

    Accomodate new front-end env variable (ML_OR_URL)

commit 20416cf
Merge: 3072bab dc52829
Author: Dominique Kleeven <[email protected]>
Date:   Wed Jul 16 15:17:32 2025 +0200

    Merge branch 'feature/keycloak-middleware-clean' into enhancement/openremote-client-service-register

commit dc52829
Author: Dominique Kleeven <[email protected]>
Date:   Wed Jul 16 15:11:48 2025 +0200

    Fix front-end linter error

commit 0094bac
Author: Dominique Kleeven <[email protected]>
Date:   Wed Jul 16 15:02:45 2025 +0200

    Directly call manager API in front-end rather than proxying requests via the service backend

    Couple technical notes:
    - Front-end initialises the manager.rest.api in the app layout after authentication has succeeded
    - Auth service also adds the interceptor to Axios so that the manager.rest.api can use the token
    - Removed proxy logic from service backend, so api routes, tests and service methods.

commit 3072bab
Author: Dominique Kleeven <[email protected]>
Date:   Mon Jul 14 16:56:49 2025 +0200

    WIP tbd on registration mechanism location

commit ece85b9
Merge: bd2bb47 3819b3d
Author: Dominique Kleeven <[email protected]>
Date:   Mon Jul 14 15:55:33 2025 +0200

    Merge branch 'feature/keycloak-middleware-clean' into enhancement/openremote-client-service-register

commit 3819b3d
Author: Dominique Kleeven <[email protected]>
Date:   Mon Jul 14 15:06:44 2025 +0200

    Fix prettier formatting issue

commit a8fbd08
Author: Dominique Kleeven <[email protected]>
Date:   Mon Jul 14 15:00:53 2025 +0200

    Use ISO8601 period for training dataset (regressors + target), added chunking for datapoint retrieval

    Deprecated cutoff_timestamp

commit bd2bb47
Author: Dominique Kleeven <[email protected]>
Date:   Fri Jul 4 10:15:36 2025 +0200

    Minor refactor to openremote client before extraction

commit 944cb0d
Merge: e6d4e38 d833a45
Author: Dominique Kleeven <[email protected]>
Date:   Fri Jul 4 09:31:13 2025 +0200

    Merge branch 'main' into feature/keycloak-middleware-clean

commit e6d4e38
Author: Dominique Kleeven <[email protected]>
Date:   Fri Jul 4 09:27:23 2025 +0200

    Address misleading function name, uses a more explicit name now

commit 11be849
Author: Dominique Kleeven <[email protected]>
Date:   Fri Jul 4 09:08:43 2025 +0200

    Update frontend/src/services/api-service.ts

    Co-authored-by: Rich Turner <[email protected]>

commit bb6a3fd
Author: Dominique Kleeven <[email protected]>
Date:   Tue Jun 10 11:50:38 2025 +0200

    Fix exception on retrieving realm config when realm isn't in the manager config

commit fced7b4
Author: Dominique Kleeven <[email protected]>
Date:   Thu May 22 09:40:13 2025 +0200

    Minor refactor to OR client to properly support realm users

commit a3c95dc
Author: Dominique Kleeven <[email protected]>
Date:   Wed May 21 12:41:10 2025 +0200

    Update openremote_proxy_client.py

commit 365f31b
Author: Dominique Kleeven <[email protected]>
Date:   Wed May 21 12:38:44 2025 +0200

    Fix timeout on proxy client

commit a2e0086
Author: Dominique Kleeven <[email protected]>
Date:   Wed May 21 12:14:42 2025 +0200

    Increase timeout for HTTPX to accomodate slower requests

commit 8f075b0
Author: Dominique Kleeven <[email protected]>
Date:   Wed May 21 11:25:50 2025 +0200

    Update pages-config-editor.ts

commit 4ccfd65
Author: Dominique Kleeven <[email protected]>
Date:   Wed May 21 11:15:56 2025 +0200

    Don't expect an optional property in the token payload

commit 66fb3d1
Author: Dominique Kleeven <[email protected]>
Date:   Tue May 20 19:22:09 2025 +0200

    Use vaadin router commands for redirect rather than using the router directly to prevent inconsistent routing behaviour

commit a923798
Author: Dominique Kleeven <[email protected]>
Date:   Mon May 19 17:28:30 2025 +0200

     Proxy service was unnecessary, we can just override the OpenRemoteClient dependency

commit 098f611
Author: Dominique Kleeven <[email protected]>
Date:   Mon May 19 15:53:59 2025 +0200

    Update app-layout.ts

commit ada37e1
Author: Dominique Kleeven <[email protected]>
Date:   Mon May 19 14:57:04 2025 +0200

    Login and logout are promises, should be awaited

commit 2150eec
Author: Dominique Kleeven <[email protected]>
Date:   Thu May 15 11:54:17 2025 +0200

    Fix failing test due to model changes

commit 75c8b76
Author: Dominique Kleeven <[email protected]>
Date:   Thu May 15 11:51:07 2025 +0200

    Additional logging

commit aea9d25
Author: Dominique Kleeven <[email protected]>
Date:   Thu May 15 11:32:51 2025 +0200

    Added TODO comment with issue link for RBAC improvements

commit b4c5c1d
Author: Dominique Kleeven <[email protected]>
Date:   Thu May 15 11:27:47 2025 +0200

    Minor tweaks, use both issuer and kid as cache key for JWKS.

commit 76fc179
Author: Dominique Kleeven <[email protected]>
Date:   Thu May 15 11:13:38 2025 +0200

    Unreachable code

commit 6dfc19d
Author: Dominique Kleeven <[email protected]>
Date:   Thu May 15 10:56:09 2025 +0200

    Increase JWKS cache as its not tied to the expiry interval

commit a4b7b07
Author: Dominique Kleeven <[email protected]>
Date:   Wed May 14 16:29:39 2025 +0200

    Additional comment

commit 128fc3a
Author: Dominique Kleeven <[email protected]>
Date:   Wed May 14 16:24:39 2025 +0200

    Construct valid issuer list based on OR realms rather than pattern matching based on issuer + configured OR URL.

commit 6c122de
Author: Dominique Kleeven <[email protected]>
Date:   Wed May 14 09:42:29 2025 +0200

    Update pyproject.toml

commit 331dfe8
Author: Dominique Kleeven <[email protected]>
Date:   Wed May 14 09:40:05 2025 +0200

    Clean commit history from feature/keycloak-middleware
Model eval, termporal resample and edge filling + additional tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spike: Consider Using Darts for Managing Prophet and Other Forecasting Models

1 participant