HybridCache L1 cache getting out of sync when using multiple instances

[Rick-Anderson]

Description

moved from dotnet/AspNetCore.Docs#34083, created by @dukesteen

Description

The current documentation for HybridCache does not adequately warn developers about the risk of L1 cache getting out of sync when invalidating cache entries from another instance. Consider the following example:

Instance 1
 - memorycache with tenant id 1 inside
 - redis cache nothing inside
Instance 2
 - memorycache with nothing inside
 - redis cache also nothing inside

Now instance 2 does an update on the tenant and invalidates both the redis and local cache, which gives you the following situation

Instance 1
 - memorycache with tenant id 1 inside
 - redis cache also has tenant id 1 inside
Instance 2
 - memorycache with tenant id 1 inside
 - redis cache also has tenant id 1 inside

Instance 1's layer 1 cache is now out of sync, and can respond to requests with the wrong data. This can be catastrophic if instance 2 removed access for a user while instance 1 is still serving requests for this user as if they were active.

Related issue: dotnet/aspnetcore#55308

Page URL

https://learn.microsoft.com/en-us/aspnet/core/performance/caching/hybrid?view=aspnetcore-9.0

Reproduction Steps

@dukesteen please provide public GitHub link with the simplest possible app that reproduces the problem.

Expected behavior

@dukesteen please provide

Actual behavior

@dukesteen please provide

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

[jose-escalona]

I found this issue while searching for documentation confirming this theoretical problem, is this really happening? Or is this new library implementing a solution to keep L1 cache sync between multiple instances? Still not clear to me.

I think that most of us are running multiple replicas of our services today in production environment, having this mechanism to keep sync between L1 cache for each of our replicas would make a huge difference, and definitely, make this library really valuable for everybody.

[IbrahimMNada]

it has been discussed here a little --> #5517