Skip to content

improvement(container-runtime): For resubmit, don't submit ID Allocation op until submitting another op #24545

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

markfields
Copy link
Member

@markfields markfields commented May 8, 2025

Description

tl;dr - Delay submitting ID Allocation Ops until right before submitting other ops. Specifically updating the ID Allocation Op generated before replaying pending state.

Background

ID Allocation ops must be submitted before other ops to ensure compressed IDs contained in those ops can be understood by all clients. This change focuses on a special case before replaying pending state where the ID Compressor needs to re-take all unfinalized creation ranges, since any original ID Allocation ops (if any) won't be resubmitted.

Today, this op is submitted directly to the Outbox under a different codepath than we really expect to be submitting ops. In fact, there's a bug where we don't schedule a flush, so it's possible if the user stops interacting with the document at this moment it will not be submitted at all.

The fix is to still generate the op before replaying pending states (so ID Compressor internal state is correct), but don't actually submit the op until we know we have reason to. (It's ok if we generate it, then drop it, and generate it again)

One other motivation for this change: ID Allocation ops are special because they are not "dirtyable" meaning they alone don't mark the container as dirty. We've found that the dirty/saved logic is a bit inconsistent though (see the struggle in #24646), so it's advantageous to avoid submitting these ops apart from other "dirtyable" ops. In fact we may follow up and stop marking them as "non-dirtyable" after this PR.

Prerequisite bug fix in Pending State Manager

There's a fix in PendingStateManager that became necessary after the main change here, it should have been this way all along. Since ID Allocation ops aren't resubmitted, we don't care to track their batch IDs. This means we shouldn't include batchId when replaying them (they're going to get dropped anyway) - without this fix we were triggering the empty batch logic (since they're not actually submitted on replay) when we shouldn't be resubmitting an empty batch.

@github-actions github-actions bot added area: tests Tests to add, test infrastructure improvements, etc base: main PRs targeted against main branch labels May 8, 2025
@github-actions github-actions bot added the area: runtime Runtime related issues label May 10, 2025
@github-actions github-actions bot removed the area: tests Tests to add, test infrastructure improvements, etc label May 10, 2025
@markfields
Copy link
Member Author

Postponing. This was a precursor to #24510 but we went in a different direction with #24614

@markfields markfields closed this May 16, 2025
@markfields markfields deleted the idc/1 branch May 16, 2025 16:54
@markfields markfields restored the idc/1 branch May 21, 2025 02:58
Take all Current on conflicts in containerRuntime.ts.  Will sort out the merge next.
@markfields markfields reopened this May 21, 2025
@markfields markfields changed the title [Proof-of-Concept] Exploring changes to ID Allocation ops in Container Runtime improvement(container-runtime): For resubmit, don't submit ID Allocation op until submitting another op May 21, 2025
@markfields markfields marked this pull request as ready for review May 21, 2025 03:57
@markfields
Copy link
Member Author

This is not the priority at the moment, but I do hope to merge this to move us towards consolidating logic around op submission/batching.

Maybe after the next release, there is some FUD about unintended side effects, i.e. whether the existing test coverage is sufficient since the behavior is so nuanced.

@markfields markfields marked this pull request as draft May 21, 2025 22:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: runtime Runtime related issues base: main PRs targeted against main branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant