Data processing improvements

[dominiquekleeven]

Depends on #42 (base branch)

This work is from my earlier exploration of Darts. I’ve updated it to reflect recent changes, added a performance/metrics test, and replaced the old dataprocessing branch with this one.

Related to #29

Summary

• Improves data processing by resampling all data to a fixed interval based on the forecast frequency.
• Aligns target and regressors (covariates), with covariates resampled to match the target datapoint interval.
• Resampled data dramatically improves training and forecasting times, since the dataset becomes smaller and de-duplicated.
• Adds additional validation for generated forecasts.
• Introduces an evaluate_model function to the model_provider protocol.
• Adds test_prophet_model_performance.py, which uses the evaluation method to validate forecast accuracy and performance.
• The test asserts accuracy and catches regressions when the model changes, ensuring we don’t accidentally cause forecasting to fail or regress in performance.
• evaluate_model also enables us to later display metrics in the Web UI. At that stage, results should be stored alongside the model.

The data processing also prepares for a future migration to Darts by implementing the necessary steps (resampling of target and covariates, handling NaN values, etc.).

---

Evaluation Metrics (NEW) – test_prophet_model_performance.py

Dataset: mock-datapoints-power-grid.json (~130k datapoints)
Using default model settings (no fine tuning)

Training Time: 0.88s  
Forecast Time: 0.04s  
RMSE: 7.0kW (typical forecast error, lower is better)  
MAE: 5.2kW (average absolute error, lower is better)  
MAPE: 15.5% (average percentage error, lower is better)  
MdAPE: 7.6% (median percentage error, lower is better)  
R²: 0.277 (variance explained, higher is better)  

Evaluation Metrics (OLD)

https://github.com/openremote/service-ml-forecast/tree/old-data-processing-metrics
Dataset: mock-datapoints-power-grid.json (~130k datapoints)
Using default model settings (no fine tuning)

Training Time: 522.53s  
Forecast Time: 0.29s  
RMSE: 11.3kW  (typical forecast error, lower is better)  
MAE: 8.6kW (average absolute error, lower is better)  
MAPE: 30.0% (average percentage error, lower is better)  
MdAPE: 17.1% (median percentage error, lower is better)  
R²: 0.113 (variance explained, higher is better)  

---

Side-by-Side Comparison

The metrics do not reflect production performance, they should be interpreted as a comparison between old and new. With old being the baseline.

Metric	Old	New	Improvement
Training Time	522.53s	0.88s	~590× faster
Forecast Time	0.29s	0.04s	~7× faster
RMSE	11.3kW	7.0kW	~38% lower (lower is better)
MAE	8.6kW	5.2kW	~40% lower (lower is better)
MAPE	30.0%	15.5%	~48% lower (lower is better)
MdAPE	17.1%	7.6%	~56% lower (lower is better)
R²	0.113	0.277	~145% higher (higher is better)

---

Conclusion

By de-duplicating datapoints, filling gaps/NaN values, and resampling data to a consistent interval, the dataset size is dramatically reduced and becomes more uniform.
This cleaner dataset makes it easier for Prophet to detect trends, resulting in significantly faster training and forecasting times and improved accuracy across all metrics.

[dominiquekleeven]

I do want to include an additional performance/metrics test that also makes use of a regressor, to ensure they are handled appropriately before the changes are accepted and merged.

[Copilot]

Pull Request Overview

This PR significantly improves data processing for ML forecasting by implementing advanced data resampling, alignment, and validation. Additionally, introduces comprehensive model evaluation capabilities, resulting in substantial performance gains (590× faster training, 7× faster forecasting) and improved accuracy metrics.

• Implements data resampling and alignment to a consistent frequency, dramatically reducing dataset size and improving model performance
• Adds comprehensive model evaluation with cross-validation and multiple accuracy metrics (RMSE, MAE, MAPE, MdAPE, R²)

Reviewed Changes

Copilot reviewed 39 out of 43 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
src/service_ml_forecast/ml/prophet_model_provider.py	Implements new data processing pipeline with resampling, evaluation metrics, and forecast validation
src/service_ml_forecast/ml/data_processing.py	New module providing data resampling, interpolation, and alignment utilities
src/service_ml_forecast/ml/evaluation_metrics.py	New dataclass defining comprehensive model evaluation metrics
packages/openremote_client/	New dedicated package for OpenRemote API interactions with service registration
tests/ml/test_prophet_model_performance.py	New performance test validating model accuracy and speed improvements
pyproject.toml	Updates dependencies and workspace configuration for new package structure

Comments suppressed due to low confidence (1)

src/service_ml_forecast/ml/prophet_model_provider.py:1

• Hard-coded magic numbers for reasonable value ranges make the validation inflexible across different domains. Consider making these configurable parameters in the model config or deriving them from training data statistics.

# Copyright 2025, OpenRemote Inc.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}