Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

guesstimate raw e2e latency and subtract from target chunk time #124

Open
morgo opened this issue May 6, 2023 · 0 comments
Open

guesstimate raw e2e latency and subtract from target chunk time #124

morgo opened this issue May 6, 2023 · 0 comments
Labels
enhancement New feature or request for-future-development Not currently planned

Comments

@morgo
Copy link
Collaborator

morgo commented May 6, 2023

The main goal of setting a target chunk time is to limit the maximum time row locks are held during chunk copies. Since chunk copies are performed in auto_commit the e2e latency from spirit (the client) never factors.

Because spirit is reasonably network efficient (it performs most of the work on the server - the main exception being reading binary logs) it might be desirable to have a central spirit server which performs migrations for a remote region. i.e. have spirit in us-west-2 run migrations for us-west-2 and us-east-1.

A way to make it require less configuration is to guess the e2e latency by periodically issuing a "select 1" query. This time can then be subtracted from the chunk copy time when coming up with new target times.

@morgo morgo added the enhancement New feature or request label May 6, 2023
@morgo morgo added the for-future-development Not currently planned label Feb 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request for-future-development Not currently planned
Projects
None yet
Development

No branches or pull requests

1 participant