Add ObjectStore::list_opts

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

We should add an ObjectStore::list_opts to allow

• Specifying both a start offset and delimiter Expose list_paginated in object_store #291
• Providing Extensions feat: add Extensions to object store PutOptions arrow-rs#7213

Describe the solution you'd like

I think the following API should work and eventually allow providing default implementations of list and list_with_delimiter that call through to list_opts

struct ListOptions {
    pub offset: Option<Path>,
    pub delimiter: bool,
    pub extensions: Extensions
}

fn list_opts(&self, opts: ListOptions) -> BoxStream<'static, Result<ListResult>>

Describe alternatives you've considered

We could expose an API that more explicitly exposes the paginated nature of the underlying APIs, however, such APIs are quite cumbersome to use, and wouldn't obviously translate to LocalFileSystem.

Additional context

[stayrascal]

how about add a option filed max_keys in ListOptions for the case that want to check whether a 'dir' or prefix is empty via list with max_keys=1 instead of listing 1000 objects if there are too much objects under the given prefix.

[stayrascal]

I'm trying to add the list_opts, but meet a problem about Extensions during passing it into a closure, which need the Extensions implemented std::marker::Copy trait, for example, the list_paginated method of ListClientExt will continue calling list_request in a closure to continue pull list result from object store, if we add the Extensions parameter to list_request method of ListClient, which required the passed parameters are copied. So it looks that it's hard to add a Extensions parameter.

BTW, why don't using a Option<HashMap> instead of Extensions during defined the xxxOptions and covert it to Extensions and setting in http client builder, since after add the Extensions into some options, e.g. PutOptions, PutMultipartOpts, we have to re-implementation PartialEq and Eq trait. (I'm a newbie about Rust, didn't get what means ' // That's 3 words. Instead, this is only 1 word.' below)

/// A type map of protocol extensions.
///
/// `Extensions` can be used by `Request` and `Response` to store
/// extra data derived from the underlying protocol.
#[derive(Clone, Default)]
pub struct Extensions {
    // If extensions are never used, no need to carry around an empty HashMap.
    // That's 3 words. Instead, this is only 1 word.
    map: Option<Box<AnyMap>>,
}

[tustvold]

The other APIs use https://docs.rs/http/1.2.0/http/struct.Extensions.html which implements Clone

[tustvold]

Thinking about this a bit more, this API won't work for pagination as the iteration order is not defined and therefore you can't use startAfter. We may need to create a separate trait for this