-
Notifications
You must be signed in to change notification settings - Fork 468
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
adapter: Lazily deserialize the audit log #30782
Conversation
2287422
to
7c0fbb5
Compare
On startup the adapter reconciles the contents of most builtin tables via the following procedure: 1. Read in the updates from the relevant durable catalog collection. 2. Generate builtin table updates from the updates. 3. Read in the current contents of the builtin table. 4. Generate builtin table retractions from the current contents. 5. Consolidate the two sets of builtin tables updates. 6. Append the consolidated builtin table updates to the builtin table. The audit log is extremely large and deserializing all updates in step (1) can take an extremely long time. Previously, this deserialization was done in a background thread to try and hide some of the latency. The audit log is append only and un-migratable, so the above process is very wasteful. We know that most of the updates from step (1) will cancel out with all updates from step (3) and we'll only be left with a small amount of additions to the builtin table. In the common happy case, they'll cancel out completely. This commit special cases the reconciliation process for the audit log so that it follows the following optimized procedure: 1. Read in the audit log updates from the durable catalog, but don't deserialize any events. 2. Read in the current contents of mz_audit_events. 3. Sort mz_audit_events by ID and find the largest ID. 4. Only deserialize the audit log updates from the durable catalog, that have an ID larger than the max ID. 5. Generate builtin table updates from the deserialized updates. 6. Append the updates to mz_audit_events. Note: When/if we make the durable catalog shard queryable via SQL, then we can remove the entire reconciliation process (including the contents of this commit), which will be a huge win for startup times. Works towards resolving #MaterializeInc/database-issues/issues/8384
7c0fbb5
to
bc305f7
Compare
In my staging account, this saves about 2 - 2.5 seconds during startup. Specifically my startup times go from roughly 19.8 seconds to 17.45 seconds. |
if self.controller.read_only() { | ||
info!("coordinator init: bootstrap: stashing builtin table updates while in read-only mode"); | ||
|
||
// TODO(jkosh44) Optimize deserializing the audit log in read-only mode. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't contribute to downtime, so it's not that important.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
neat!
On startup the adapter reconciles the contents of most builtin tables
via the following procedure:
table.
The audit log is extremely large and deserializing all updates in step
(1) can take an extremely long time. Previously, this deserialization
was done in a background thread to try and hide some of the latency.
The audit log is append only and un-migratable, so the above process is
very wasteful. We know that most of the updates from step (1) will
cancel out with all updates from step (3) and we'll only be left with a
small amount of additions to the builtin table. In the common happy
case, they'll cancel out completely.
This commit special cases the reconciliation process for the audit log
so that it follows the following optimized procedure:
deserialize any events.
that have an ID larger than the max ID.
Note: When/if we make the durable catalog shard queryable via SQL, then
we can remove the entire reconciliation process (including the contents
of this commit), which will be a huge win for startup times.
Works towards resolving #MaterializeInc/database-issues/issues/8384
Motivation
This PR adds a known-desirable feature.
Checklist
$T ⇔ Proto$T
mapping (possibly in a backwards-incompatible way), then it is tagged with aT-proto
label.