design: lift max result size limitation #32211
Conversation
| happening in `environmentd`, but could happen on the cluster side. We think it | ||
| is easier to lift result-size limitations for queries that don't require | ||
| post-processing. We should be able to just "stream them through", without | ||
| materializing in `environmentd`. |
There was a problem hiding this comment.
FWIW, I think it wouldn't be super hard to support these cases using the Consolidator machinery we have now. Happy to chat about this if you're interested!
There was a problem hiding this comment.
Nice! I was thinking this might be possible, glad to know you think the same!
| `environmentd`) stream those results from persist back to the client. | ||
|
|
||
| We would use a (configurable) threshold for determining whether to send a | ||
| result inline in the compute protocol or out of band via the blob store. |
There was a problem hiding this comment.
Sounds like this could use the same machinery as @ParkMyCar worked up for large insert / update queries, but streaming the resulting batch into a client instead of linking it into a shard?
There was a problem hiding this comment.
I was thinking the same! I started to build an abstraction for this in #31189, I added a new crate there called row-stash a response type called PeekResponseUnary::Batches, and finally the rows get returned to the user via ExecuteResponse::SendingRowsStreaming.
FWIW some things in that PR are a bit hacked together so the code might not be perfect 🙈
Rendered: https://github.com/aljoscha/materialize/blob/adapter-lift-max-result-size-limitation/doc/developer/design/20250415_lift_max_result_size_limitation.md
Design doc for https://github.com/MaterializeInc/database-issues/issues/9180