Speed up queries via materialized view

[Muhammad-Moiz626]

Description

Our data model with multiple beliefs per event time (multiple horizons as well as sources) makes queries computationally expensive. Most-used are queries where we look only for the most recent beliefs about each event, when given any moment in time as max belief time.

Pre-computation with Postgres' materialized views can make queries much faster. This PR will enable hosts to use this performance enhancement.

• Created migrations file to create materialized view.
• Use the MV in queries.
• Use SQL Alchemy for MV.
• Automate the refreshing of the MV.
• Make MV optional / configurable.
• Document the process

Performance (benchmarked)

Benchmarked with timely-beliefs>=4.0.1 (required by this PR — 4.0.1 pushes event-window filters down into the mview subquery and closes two correctness holes; without it, windowed queries scan the entire view and can even get slower). Setup: one bulk sensor with 10 belief revisions per event, on a deliberately de-clustered heap (per-sensor rows scattered across pages, as they are in practice when many sensors' data interleaves on arrival). Timings are min-of-4 after warm-up, via TimedBelief.search(most_recent_beliefs_only=True).

Query time vs. table size

365-day event window — first line: without mview (fallback), second line: with mview (ms):

xychart-beta
    title "365d window query time (ms) vs timed_belief rows"
    x-axis ["4M", "8M", "16M", "32M"]
    y-axis "ms" 0 --> 1100
    line [1048, 786, 677, 967]
    line [658, 582, 559, 613]

Loading

30-day event window — same series order (ms):

xychart-beta
    title "30d window query time (ms) vs timed_belief rows"
    x-axis ["4M", "8M", "16M", "32M"]
    y-axis "ms" 0 --> 200
    line [107, 119, 135, 170]
    line [122, 112, 151, 160]

Loading

Takeaways:

• Wide windows: the mview wins ~1.2–1.6x, and the advantage holds as the table grows. Much of the remaining end-to-end time is pandas BeliefsDataFrame construction, so the SQL-side gain is larger than these end-to-end factors suggest.
• Narrow windows: parity — the (event_start, sensor_id, source_id) INCLUDE (belief_horizon) covering index already bounds the fallback's work tightly at small windows.
• These factors are conservative: on tables where many sensors interleave (worse per-sensor page locality than the benchmark heap), the fallback degrades further while the mview path doesn't.

One-time migration cost and refresh cost

First line: view + index creation (the migration); second line: non-concurrent REFRESH (including the built-in ANALYZE). Values at 64M and 128M are linear extrapolations (marked *):

xychart-beta
    title "Creation and refresh time (s) vs timed_belief rows"
    x-axis ["4.6M", "8.6M", "16.6M", "32.6M", "64M*", "128M*"]
    y-axis "seconds" 0 --> 550
    line [10, 25, 40, 132, 260, 520]
    line [8, 12, 25, 25, 50, 100]

Loading

• Migration: roughly 1 minute per 15 million rows (so ~4 min at 64M, ~9 min at 128M). The changelog warning states this rate, and the migration logs an estimate for the host's own table size (from pg_class.reltuples) when it starts.
• Refresh: ~25 s at 32M rows, extrapolating to ~1 min at 64M and ~2 min at 128M. Non-concurrent refresh builds a full second copy of the view before swapping — provision disk headroom accordingly (~5 GB per 32M rows), or use --concurrent.
• Both the migration and the refresh command run ANALYZE on the view afterwards; without fresh statistics the planner picks bad plans and the view can lose to the fallback.

...

Further Improvements

• the full queries (used in playbacks, getting fulls set of beliefs for each event) might also enjoy a materialized view, but maybe it is just too much data, then we should just limit playback to shorter time ranges
• give hosts a range of options (e.g. refresh time)
• actually move to TimeScaleDB, which auto-refreshes their materialized views on inserts

Related Items

Closes 108

...

---

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

[Flix6x]

Some early review comments. Nice to see the refresh logic taking shape.

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #33532336 | 📁 Comparing 2a0e359 against latest (922ecf6)

  🔍 Preview build  

13 files changed · + 1 added · ± 12 modified

+ Added

• _autosummary/flexmeasures.data.services.materialized_views.html

± Modified

• changelog.html
• configuration.html
• genindex.html
• py-modindex.html
• _autosummary/flexmeasures.api.dev.sensors.html
• _autosummary/flexmeasures.data.models.forecasting.pipelines.train_predict.html
• _autosummary/flexmeasures.data.models.generic_assets.html
• _autosummary/flexmeasures.data.models.time_series.html
• _autosummary/flexmeasures.data.services.html
• api/dev.html
• and 2 more...

[Flix6x]

Exciting work! Thanks for addressing my earlier comments. Just a short code review for now, so you don't have to wait for me testing. Looking forward to testing tomorrow.

[nhoening]

Also a quick review from me, before having tested myself, mostly about concepts.

I also suggest moving get_timed_belief_min_v() to timely-beliefs and add basic documentation there (not much more than the SQL command so anybody reading only that could in principle use it)

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	timely-beliefs@​3.5.5 ⏵ 4.0.1	+1

View full report