tool-router: add reusable search index with invalidation and incremental refresh

[sasaeo]

Summary

Introduce a reusable in-memory search index for tool discovery, with invalidation and refresh behavior that avoids recomputing BM25 state on every query.

Problem

The current search flow rebuilds the document token set, document frequencies, and score inputs on every search call. This is simple, but it does not scale well for repeated searches or larger tool catalogs.

The package should treat indexing as a first-class concern rather than recomputing ranking inputs per request.

Proposed direction

Add a search index abstraction that:

• stores normalized searchable documents for indexed tools
• precomputes ranking state needed for search
• supports cache invalidation when the catalog changes
• supports explicit refresh and future partial refresh workflows

This should make repeated searches cheaper and create a cleaner foundation for future ranking improvements.

Acceptance criteria

• Search no longer rebuilds the full BM25 working set from scratch on every query.
• The index can be invalidated and rebuilt when source catalogs change.
• Router-level refresh operations update the searchable state predictably.
• The index layer is independently unit tested.
• Benchmarks or test evidence show repeated searches are more efficient than the current approach.