Skip to content

Feature: streaming pagination via Stream<Item = T> instead of eager collection #55

Description

@ZialeHub

Problem

Request::send() currently collects all pages into memory before returning. For large result sets (thousands of records) this:

  • holds the entire dataset in memory at once
  • cannot be cancelled mid-way through
  • provides no progress feedback to the caller
  • cannot be consumed lazily by a downstream pipeline

Proposed feature

Add a .stream() method on Request that returns a futures::Stream<Item = Result<T>>:

use futures::StreamExt;

let mut stream = api
    .get::<Vec<Resource>>("/resources")?
    .pagination(PaginationRule::OneShot)
    .stream(); // returns impl Stream<Item = Result<Vec<Resource>>>

while let Some(page) = stream.next().await {
    let records = page?;
    process(records).await;
}

Or with per-item streaming (flat-mapping pages):

let mut stream = api
    .get::<Resource>("/resources")?
    .stream_items(); // returns impl Stream<Item = Result<Resource>>

stream.for_each(|item| async move {
    println!("{:?}", item?);
}).await;

Design notes

  • .stream() — yields one page (Vec<T>) per Poll::Ready, fetching the next page lazily on demand
  • .stream_items() — flat-maps pages into individual items
  • Both should respect the rate limiter between pages
  • Cancellation is implicit: dropping the stream stops fetching further pages
  • The existing .await (eager) behaviour is preserved unchanged

Dependencies

  • futures or async-stream crate
  • Compatible with all Pagination strategy types (page/offset/cursor)

Metadata

Metadata

Assignees

No one assigned

    Labels

    B-ideaDiscussion; or implementation attempt, to be reviewed before further workrustPull requests that update rust code

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions