Feat: Check freshness for mixed external & sqlmesh models #5499
Conversation
| # filters: | ||
| # branches: | ||
| # only: | ||
| # - main |
There was a problem hiding this comment.
TODO: Reinstate before merging
| upstream_parent_snapshots = {p for p in parent_snapshots if not p.is_external} | ||
| external_parents = snapshot.node.depends_on - {p.name for p in upstream_parent_snapshots} | ||
|
|
||
| if context.parent_intervals: |
There was a problem hiding this comment.
We should exclude external models from this check. There could be intervals for external models to, for example, run audits, but it doesn't mean that the table is actually fresh.
There was a problem hiding this comment.
Right. I guess I can exclude them at the time of parent_intervals construction, since that's a list so we've lost the snapshot mapping of each Interval.
There was a problem hiding this comment.
Implemented in 9a8af2e, external snapshots are excluded from parent_intervals.
sqlmesh/core/scheduler.py
Outdated
| if parent.snapshot_id not in snapshot_intervals: | ||
| continue | ||
| _, p_intervals = snapshot_intervals[parent.snapshot_id] | ||
| parent_intervals.append(p_intervals) |
There was a problem hiding this comment.
I don't think this is the right place to construct this. The missing intervals will change as we apply signal checks to snapshots. If you set them once here, the parent signals will not be reflected.
There was a problem hiding this comment.
Good call.
I guess our best choice would be to check snapshot_batches then (same loop) which applies the signal to filter out the unready intervals.
So, for each snapshot in the DAG, its ExecutionContext::parent_intervals list will be extended by the snapshot_batches[parent], provided the parent snapshot is NOT external.
There was a problem hiding this comment.
yes, this sounds correct, assuming I understood the idea correctly
There was a problem hiding this comment.
Implemented in 9a8af2e
VaggelisD
force-pushed
the
vaggelisd/mixed_model_freshness
branch
from
46715f5 to
9a8af2e
Compare
2 participants
Follow up PR on model freshness (Previous PR)
This PR enables mixed SQLMesh & external (source) model freshness, e.g:
Model
Cwill be evaluated if its upstream models are fresh, which is determined by:C'slast_altered_tscompared to external model's (A)INFORMATION_SCHEMAmetadataB), signalling if any model will be evaluated as part of this runA new freshness-specific test file is added to break up the original test in various scenarios.