Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Too Many Store Commands Failing with message Unable to locate node #255

Open
Rezga12 opened this issue Feb 18, 2025 · 0 comments
Open

Too Many Store Commands Failing with message Unable to locate node #255

Rezga12 opened this issue Feb 18, 2025 · 0 comments

Comments

@Rezga12
Copy link

Rezga12 commented Feb 18, 2025

I am using EnyimMemcachedCore version 3.3.0, which is connected to single memcached instance.
memcached version is 1.6.8. it's running on windows machine as a windows service. It's running with configuration: -m 10144 -c 2048 -t 8

My application uses memcached exstensively, I have a lot of reads and writes.
I have about 7-8K Get requests per second and about 400 Store requests per second.
But after application has started and is running for about 15 minutes, Store requests start to get failing, with the error code Unable to Locate Node
I dig though the code and find out that when there is single memcached node library uses SIngleNodeLocator that has just one private field _node and if it's null Operation will fail with such error. I am using PerformStore method and if node locator returns null it will fail with such error:

result.Fail("Unable to locate node");
return result;

I am using single static MemcachedClient instance for both get and store requests.
here's my memcached configuration:

var loggerFactory = new LoggerFactory().AddSerilog();
var memcachedClientOptions = new MemcachedClientOptions()
{
    Protocol = MemcachedProtocol.Binary,
    SocketPool = new SocketPoolOptions()
    {
    },
    Servers = new List<Server>()
    {
        new Server()
        {
            Address = address,
            Port = port
        }
    },
    Transcoder = "MessagePackTranscoder",
};

var memcachedClientConfiguration = new MemcachedClientConfiguration(loggerFactory, memcachedClientOptions)
{
    NodeLocator = typeof(SingleNodeLocator),
};

_memcachedClient = new MemcachedClient(loggerFactory, memcachedClientConfiguration);

Also almost everywhere I have classic caching logic:

  1. Get from Cache
  2. If cache entry is not present Get from DB
  3. Store the result in cache

due to the fact that most Store requests are failing, number of store requests becomes more frequent and goes up to about 3k.

Is there any way I can debug this or some options in configuration I can try to pinpoint the issue? Also if there is specific scenarios when _node field in SingleNode locator can be null that might help me to investigate.
On development environment application has far less get/store requests and there are no issues, but production load is much higher and maybe that is the issue.
Anyways will appreciate any help :)

@Rezga12 Rezga12 changed the title Too Many Store Commands Failing with message Unable to locate node error Too Many Store Commands Failing with message Unable to locate node Feb 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant