Problem
Request::send() currently collects all pages into memory before returning. For large result sets (thousands of records) this:
- holds the entire dataset in memory at once
- cannot be cancelled mid-way through
- provides no progress feedback to the caller
- cannot be consumed lazily by a downstream pipeline
Proposed feature
Add a .stream() method on Request that returns a futures::Stream<Item = Result<T>>:
use futures::StreamExt;
let mut stream = api
.get::<Vec<Resource>>("/resources")?
.pagination(PaginationRule::OneShot)
.stream(); // returns impl Stream<Item = Result<Vec<Resource>>>
while let Some(page) = stream.next().await {
let records = page?;
process(records).await;
}
Or with per-item streaming (flat-mapping pages):
let mut stream = api
.get::<Resource>("/resources")?
.stream_items(); // returns impl Stream<Item = Result<Resource>>
stream.for_each(|item| async move {
println!("{:?}", item?);
}).await;
Design notes
.stream() — yields one page (Vec<T>) per Poll::Ready, fetching the next page lazily on demand
.stream_items() — flat-maps pages into individual items
- Both should respect the rate limiter between pages
- Cancellation is implicit: dropping the stream stops fetching further pages
- The existing
.await (eager) behaviour is preserved unchanged
Dependencies
futures or async-stream crate
- Compatible with all
Pagination strategy types (page/offset/cursor)
Problem
Request::send()currently collects all pages into memory before returning. For large result sets (thousands of records) this:Proposed feature
Add a
.stream()method onRequestthat returns afutures::Stream<Item = Result<T>>:Or with per-item streaming (flat-mapping pages):
Design notes
.stream()— yields one page (Vec<T>) perPoll::Ready, fetching the next page lazily on demand.stream_items()— flat-maps pages into individual items.await(eager) behaviour is preserved unchangedDependencies
futuresorasync-streamcratePaginationstrategy types (page/offset/cursor)