Implement cache tagging (RemoveByTagAsync + tagged writes)

[ejsmith]

Implementation Plan: Cache Tagging

Adding cache tagging lets us associate one or more tags with a cache entry and later bulk-invalidate all entries for a tag via a single RemoveByTagAsync call, without iterating the keyspace (which is impractical in Redis clusters).

This decouples invalidation groups from key naming. A single key can belong to multiple tags (e.g. "user:123", "tenant:acme", "reports"), which RemoveByPrefixAsync cannot express.

---

Design: Hybrid Strategy

Different mechanism per layer, same public contract:

• InMemory: index-based physical deletion (immediate reclaim, respects MaxItems/MaxMemorySize)
• Distributed (Redis/Azure/AWS): timestamp-barrier lazy invalidation (O(1) invalidate, no write amplification, cluster-safe)
• HybridCacheClient: both -- physical delete in the local layer + timestamp barrier in distributed + IMessageBus broadcast to other nodes

Why two mechanisms

A reverse index (tag -> keys) is ideal in-process: the index is free, and dead entries must be reclaimed because capacity is bounded. But a server-side index in Redis (SETs) has serious problems: write amplification on every tagged write, unbounded SET growth with stale references as keys expire, SMEMBERS/bulk-DEL blocking on large tags, and cross-slot incompatibility in clusters.

The timestamp barrier avoids all of that for distributed: RemoveByTagAsync writes one small timestamp key; reads compare an entry's creation time against its tags' invalidation timestamps and treat older entries as a miss.

---

ICacheClient Changes

RemoveByTagAsync is added to the interface. Two options for the tag-aware write API are below (Primary vs. Alternative); we should pick one before implementing.

Semantics

• tags = null: no tags. An overwrite clears any previous tags on that key.
• tags = ["a", "b"]: replaces all previous tags on that key.
• RemoveByTagAsync(tag): returns the count of entries invalidated. For InMemory this is exact; for distributed (timestamp barrier) it returns 0, since an exact count is unknowable without scanning. This asymmetry is documented, not "fixed" with an expensive scan.
• Atomic numeric ops (IncrementAsync, SetIfHigherAsync, SetIfLowerAsync) are NOT tagged -- they are counters, not cached objects.
• GetExpirationAsync/SetExpirationAsync are orthogonal to tags -- TTL changes do not affect tag associations.

---

InMemoryCacheClient -- Physical Deletion via Reverse Index

Data structures:

// Reverse index: tag -> keys
private readonly ConcurrentDictionary<string, HashSet<string>> _tagIndex = new();

// CacheEntry addition
internal string[]? Tags { get; set; }

Thread safety: the outer ConcurrentDictionary handles tag-level concurrency; each inner HashSet<string> is mutated under lock(set). (ConcurrentBag is unsuitable because it lacks efficient Remove.)

Write path:

1. If the key already exists with old tags, remove the key from those old tag sets (and drop empty sets).
2. Store the entry with the new tags.
3. Add the key to each new tag's set.

RemoveByTagAsync(tag):

1. _tagIndex.TryRemove(tag, out var keySet); if absent return 0.
2. Snapshot keys under lock(keySet).
3. Physically remove each key from storage (reusing RemoveAllAsync logic).
4. For removed entries carrying OTHER tags, clean those tag sets too.
5. Fire ItemExpired, update memory tracking, return count.

Cleanup must also run on eviction (CompactAsync, LRU) and TTL expiry (RemoveExpiredKey, maintenance) so the index never accumulates references to keys that no longer exist.

GetAsync is unchanged -- zero read-path overhead. InMemory uses eager physical deletion only.

---

Distributed Providers -- Timestamp Barrier

Entries are stored in an envelope that carries the creation timestamp and tags alongside the payload:

[header: version(1B) | creation_ticks(8B) | tag_count(2B) | (len|tag)... ]
[payload: serialized value]

Entries written before this feature (no header) deserialize as untagged with creation_time = 0 (always valid) -- backward compatible.

Tag invalidation timestamp:

Key:   __fc:tag:{tagName}
Value: ticks when RemoveByTagAsync was called
TTL:   configurable; must outlive the longest-lived tagged entry

• Write: serialize into the envelope; standard SET [EX]. No extra commands.
• RemoveByTagAsync: write the tag timestamp key. O(1). Returns 0.
• Read: deserialize envelope; if no tags, return immediately (zero overhead). Otherwise compare creation_ticks against each tag's invalidation timestamp; if older, return a miss. Tag timestamps are cached in-process with a short refresh interval to avoid a round trip per read.

Constraint: with the timestamp barrier, logically-dead entries linger in storage until their own TTL expires. Tagged entries in distributed caches should always have a TTL; entries with no TTL would occupy memory indefinitely after invalidation (invisible to reads). Document as a strong recommendation; optionally warn at write time.

---

HybridCacheClient

RemoveByTagAsync:

1. _distributedCache.RemoveByTagAsync(tag) (writes the barrier)
2. _localCache.RemoveByTagAsync(tag) (physical delete from local InMemory)
3. publish InvalidateCache { CacheId, Tags = [tag] }

Remote handler removes the tag from the local cache for messages originating elsewhere.

When populating the local cache from distributed on a miss, deserialize tags from the envelope, check tag freshness, and only populate locally (with tags) if fresh -- so local RemoveByTagAsync works on those entries.

InvalidateCache gains a string[]? Tags field alongside the existing Keys/FlushAll/Expired.

---

ScopedCacheClient

Tags are scoped exactly like keys: RemoveByTagAsync(tag) => _unscoped.RemoveByTagAsync(prefix + tag). This prevents cross-tenant tag leakage. Needs explicit isolation tests.

---

Tag Validation

• Non-null, non-empty, non-whitespace
• Max length 256; max 16 tags per entry
• Reserved prefix __fc: rejected (collides with internal tag timestamp keys)
• Case-sensitive (consistent with keys)

---

Edge Cases

• No-TTL tagged entry in Redis: dead after invalidation but occupies memory -> document as anti-pattern.
• Tag timestamp TTL < entry TTL: entry could "resurrect" -> tag timestamp TTL must be >= max entry TTL.
• Concurrent Set during RemoveByTag (InMemory): an entry written after invalidation begins may survive -- acceptable, same semantics as RemoveByPrefixAsync today.
• RemoveByTagAsync on unknown tag returns 0.
• Empty tags: [] is treated as null.

---

Interaction Summary

• SetAsync(key, val, tags): InMemory stores entry + updates index; distributed serializes tags + timestamp.
• SetAsync(key, val) (no tags): clears prior tags.
• RemoveByTagAsync(tag): InMemory index lookup + physical delete; distributed writes timestamp barrier.
• RemoveAsync/RemoveByPrefixAsync/RemoveAllAsync: InMemory also cleans the tag index; distributed needs no index cleanup.
• Eviction / TTL expiry: InMemory cleans the index.
• GetAsync/ExistsAsync: InMemory unchanged; distributed checks tag freshness.

---

Implementation Scope

Core Foundatio:

• src/Foundatio/Caching/ICacheClient.cs -- tag-aware write API + RemoveByTagAsync
• src/Foundatio/Caching/InMemoryCacheClient.cs -- tag index, Tags on entry, tag-aware write/remove/eviction
• src/Foundatio/Caching/HybridCacheClient.cs -- InvalidateCache.Tags, RemoveByTagAsync, tag-aware local population
• src/Foundatio/Caching/HybridAwareCacheClient.cs -- RemoveByTagAsync + bus publish
• src/Foundatio/Caching/ScopedCacheClient.cs -- prefix tags with scope
• src/Foundatio/Caching/NullCacheClient.cs -- no-op
• src/Foundatio/Extensions/CacheClientExtensions.cs -- pass tags through overloads
• src/Foundatio.TestHarness/Caching/CacheClientTestsBase.cs -- tag tests
• docs + .agents/skills/foundatio/SKILL.md

Provider repos (follow-up): Foundatio.Redis (envelope, backward-compatible deserialization, timestamp barrier, local tag-timestamp cache), then Azure/AWS.

---

Code Quality Directives

1. Extract a CacheTagIndex helper rather than scattering index bookkeeping across the (already large) InMemoryCacheClient. Each call site becomes one line (Track/Untrack/TakeKeysForTag/Clear); the complexity lives in one focused file.
2. Extract a cache-entry envelope reader/writer for distributed providers. Do NOT inline header parsing through every Get/Set and do NOT modify the shared ISerializer abstraction -- the envelope is cache-specific and wraps the serializer output.
3. Do NOT build a second caching layer inside RedisCacheClient. For HybridCacheClient the local InMemoryCacheClient already serves as the tag-timestamp store; direct Redis users get eventual consistency via a minimal refresh cache.
4. RemoveByTagAsync return-value asymmetry is documented, not papered over with an expensive distributed scan.
5. Scoped tags are a one-line prefix, mirroring scoped keys; add cross-scope isolation tests.

---

API Option A (Primary): tags parameter

Add an optional tags parameter to the write methods (breaking change to existing signatures):

Task<bool> SetAsync<T>(string key, T value, TimeSpan? expiresIn = null, IEnumerable<string>? tags = null);
Task<bool> AddAsync<T>(string key, T value, TimeSpan? expiresIn = null, IEnumerable<string>? tags = null);
Task<int>  SetAllAsync<T>(IDictionary<string, T> values, TimeSpan? expiresIn = null, IEnumerable<string>? tags = null);
Task<bool> ReplaceAsync<T>(string key, T value, TimeSpan? expiresIn = null, IEnumerable<string>? tags = null);
Task<bool> ReplaceIfEqualAsync<T>(string key, T value, T expected, TimeSpan? expiresIn = null, IEnumerable<string>? tags = null);

Task<int> RemoveByTagAsync(string tag);

Simple at the call site, but it grows the parameter list and breaks existing signatures; any future per-entry setting adds another parameter.

---

API Option B (Alternative): fluent cache-entry options

Avoid parameter explosion with a per-entry options object (matching the existing Foundatio options/builder idiom, and mirroring HybridCacheEntryOptions/FusionCacheEntryOptions):

public class CacheEntryOptions
{
    public TimeSpan? ExpiresIn { get; set; }
    public DateTime? ExpiresAtUtc { get; set; }
    public IEnumerable<string>? Tags { get; set; }
}

public class CacheEntryOptionsBuilder
{
    public CacheEntryOptionsBuilder ExpiresIn(TimeSpan expiresIn);
    public CacheEntryOptionsBuilder ExpiresAt(DateTime expiresAtUtc);
    public CacheEntryOptionsBuilder Tags(params string[] tags);
    public CacheEntryOptions Build();
}

Add one additive overload per write method (existing expiresIn overloads stay, so only RemoveByTagAsync is a new interface member):

Task<bool> SetAsync<T>(string key, T value, CacheEntryOptions options);
Task<bool> AddAsync<T>(string key, T value, CacheEntryOptions options);
Task<int>  SetAllAsync<T>(IDictionary<string, T> values, CacheEntryOptions options);
Task<bool> ReplaceAsync<T>(string key, T value, CacheEntryOptions options);
Task<bool> ReplaceIfEqualAsync<T>(string key, T value, T expected, CacheEntryOptions options);

Task<int> RemoveByTagAsync(string tag);

Usage:

await cache.SetAsync("product:42", product, o => o
    .ExpiresIn(TimeSpan.FromMinutes(30))
    .Tags("electronics", "featured"));

await cache.SetAsync("product:42", product, new CacheEntryOptions {
    ExpiresIn = TimeSpan.FromMinutes(30),
    Tags = ["electronics", "featured"]
});

// common no-tags case unchanged
await cache.SetAsync("product:42", product, TimeSpan.FromMinutes(30));

Pros: no parameter explosion, additive (not breaking) for existing expiration usage, future per-entry settings extend CacheEntryOptions instead of method signatures, and it matches the direction taken by Microsoft HybridCache and FusionCache.

Cons: one extra overload per write method, a small per-call allocation, and two ways to express expiration (needs clear guidance).

Recommendation: Option B. It avoids breaking the existing signatures (only RemoveByTagAsync is new on the interface) and scales cleanly as per-entry options grow.

[niemyjski]

+1 This would also help fix issues listed here: exceptionless/Exceptionless#125

[ejsmith]

This would also help with the fact that iterating keys in redis is highly discouraged and will be impossible in a cluster.

[niemyjski]

@ejsmith We'll we already have remove by prefix and set support. Should we close this issue?

[niemyjski]

https://stackify.com/implementing-cache-tagging-redis/

[niemyjski]

@ejsmith let me know what your thoughts are on this plan.