Enhancement: add telemetry/context propagation across search() -> to_xarray() #27

Problem

We probably want lightweight telemetry around how catalogs are actually used, especially across the path:

search() -> selected result object -> to_xarray()

A slightly awkward wrinkle here is that the same underlying icechunk store can back the datastore both pre- and post-search, even when those show up as different Python objects. That means naive object-level instrumentation is likely to be brittle, and we'll lose lineage unless we propagate some explicit context.

Idea

Add a mechanism to carry telemetry context through catalog operations so we can link:

• the original search/query
• the selected catalog/result object
• the eventual to_xarray() / materialization call

My first thought was something like an accessor mechanism on IcechunkCatalog, but that may not be the best abstraction if the real need is stable context propagation rather than just a nicer API surface.

What we want to observe

At minimum, it would be useful to capture:

Search event

• query / filter parameters
• result count
• timestamp / duration if cheap
• a generated search_id or similar

Result / catalog lineage

• stable store_id for the underlying icechunk store
• optional selection metadata (group/key/path/etc.)
• linkage back to the originating search_id

Load / materialization event

• to_xarray() / to_dataset_dict() invocation
• originating search_id
• store_id
• maybe variable selection / chunking / timing if available and not too invasive

That would let us trace something like:

search performed -> result set created -> result chosen -> to_xarray invoked

Design constraints / considerations

• Do not rely on Python object identity as the stable key.
  • Use something durable instead: store URI, snapshot/version id, catalog entry id, or similar.
• Avoid smearing telemetry code everywhere.
  • Ideally there is one place where context is created and a small number of places where it is propagated/emitted.
• Keep it optional / low-friction.
  • This should not make the happy path ugly or force people into a logging framework.

Possible approaches

1. Wrapper/context objects on catalog/search results

Probably the most explicit option.

• search() returns an object carrying both the catalog/result and telemetry context
• downstream operations propagate that context
• to_xarray() emits an event with the carried lineage

2. Accessor-like mechanism on IcechunkCatalog

Potentially nice ergonomically.

• could expose telemetry state and helper methods cleanly
• may still need explicit context propagation into post-search objects and xarray materialization

3. Shared telemetry context + optional xarray accessor

My current guess for the best overall shape.

• maintain a lightweight TelemetryContext (or similar) on catalog/result objects
• propagate it through search() and selection
• emit on to_xarray() / to_dataset_dict()
• optionally surface the context in xarray-land via attrs/accessor for debugging/inspection

Acceptance criteria

• Decide on the stable identity used for correlating pre/post-search catalog objects.
• Pick an instrumentation shape that survives the search() -> to_xarray() path.
• Add tests showing lineage is preserved across at least one realistic flow.
• Keep telemetry optional and non-invasive for normal users.
• Document the extension points so downstream code can plug in a logger/emitter cleanly.

Related issues

This feels adjacent to:

• Enhancement: bring search() semantics closer to intake-esm #23 (search() semantics/parity)
• Enhancement: expand load API parity for to_dataset_dict() / to_xarray() #24 (to_xarray() / load API parity)

…but it is probably worth tracking separately because this is more about cross-operation observability and lineage than API parity by itself.