Read windows with lastConsumedChangeId efficiently #68
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
haaawk
reviewed
Feb 28, 2022
|
|
||
| if (task.state.getLastConsumedChangeId().isPresent()) { | ||
| ChangeId lastConsumedChangeId = task.state.getLastConsumedChangeId().get(); | ||
| streams = streams.filter(s -> s.compareTo(lastConsumedChangeId.getStreamId()) > 0); |
There was a problem hiding this comment.
Is compareTo on StreamId doing the right thing? Is it using the same sorting order as a select would use? That means the order of tokens of partition keys.
There was a problem hiding this comment.
Yes, as far as I tested this did the right thing. Also the integration tests verify that assumption.
haaawk
reviewed
Feb 28, 2022
Yes, I'll work on that. |
Read windows with present lastConsumedChangeId more efficiently. Instead of filtering out rows on the client side, perform two queries to Scylla: first - reading a single partial stream, second - reading all consecutive entire streams.
avelanarius
force-pushed
the
partial-windows
branch
from
9d84258 to
30edf92
Compare
|
@haaawk Pushed v2 - reduced duplication of code. |
|
@avelanarius @haaawk |
dkropachev
force-pushed
the
master
branch
4 times, most recently
from
012c0c8 to
c89661c
Compare
dkropachev
force-pushed
the
master
branch
17 times, most recently
from
78c0d0b to
0fc243b
Compare
dkropachev
force-pushed
the
master
branch
12 times, most recently
from
3c4bf55 to
3318b32
Compare
dkropachev
force-pushed
the
master
branch
8 times, most recently
from
5031da9 to
b5730b8
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Read windows with present lastConsumedChangeId more efficiently. Instead of filtering out rows on the client side, perform two queries to Scylla: first - reading a single partial stream, second - reading all consecutive entire streams.
TODO: Two modes of reading - single stream and multiple stream selects.