Skip to content

Too Many Store Commands Failing with message Unable to locate node #255

@Rezga12

Description

@Rezga12

I am using EnyimMemcachedCore version 3.3.0, which is connected to single memcached instance.
memcached version is 1.6.8. it's running on windows machine as a windows service. It's running with configuration: -m 10144 -c 2048 -t 8

My application uses memcached exstensively, I have a lot of reads and writes.
I have about 7-8K Get requests per second and about 400 Store requests per second.
But after application has started and is running for about 15 minutes, Store requests start to get failing, with the error code Unable to Locate Node
I dig though the code and find out that when there is single memcached node library uses SIngleNodeLocator that has just one private field _node and if it's null Operation will fail with such error. I am using PerformStore method and if node locator returns null it will fail with such error:

result.Fail("Unable to locate node");
return result;

I am using single static MemcachedClient instance for both get and store requests.
here's my memcached configuration:

var loggerFactory = new LoggerFactory().AddSerilog();
var memcachedClientOptions = new MemcachedClientOptions()
{
    Protocol = MemcachedProtocol.Binary,
    SocketPool = new SocketPoolOptions()
    {
    },
    Servers = new List<Server>()
    {
        new Server()
        {
            Address = address,
            Port = port
        }
    },
    Transcoder = "MessagePackTranscoder",
};

var memcachedClientConfiguration = new MemcachedClientConfiguration(loggerFactory, memcachedClientOptions)
{
    NodeLocator = typeof(SingleNodeLocator),
};

_memcachedClient = new MemcachedClient(loggerFactory, memcachedClientConfiguration);

Also almost everywhere I have classic caching logic:

  1. Get from Cache
  2. If cache entry is not present Get from DB
  3. Store the result in cache

due to the fact that most Store requests are failing, number of store requests becomes more frequent and goes up to about 3k.

Is there any way I can debug this or some options in configuration I can try to pinpoint the issue? Also if there is specific scenarios when _node field in SingleNode locator can be null that might help me to investigate.
On development environment application has far less get/store requests and there are no issues, but production load is much higher and maybe that is the issue.
Anyways will appreciate any help :)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions