AWS endpoint with varying IP address

[dwickstrom]

Please use the following questions as a guideline to help me answer
your issue/question without further inquiry. Thank you.

Which version of Elastic are you using?

[x] elastic.v6 (for Elasticsearch 6.x)

Please describe the expected behavior

Hello 👋 We're trying to use this library with an AWS cluster of 3 nodes and specifying the endpoint hostname from AWS as a single entry in the hosts key in the library config file. The ideal situation would be where the client would be able to detect when the IP address changes, re-resolve the hostname and send a retry request, such that during re-provisioning phase no requests are dropped.

Please describe the actual behavior

Requests will fail during the provisioning phase and then, in our case after about 15 minutes, the client will heal itself and requests stop failing.

Because of AWS not exposing the node IPs on the /_nodes endpoint these are my thoughts so far:

With sniffing disabled we see that the single node connection won't be MarkAsDead, due to

elastic/client.go

Lines 1204 to 1209 in 60d62e5

	if !c.snifferEnabled {
	c.errorf("elastic: all %d nodes marked as dead; resurrecting them to prevent deadlock", len(c.conns))
	for _, conn := range c.conns {
	conn.MarkAsAlive()
	}
	}

With sniffing enabled it's not going to work because sniffing can't be done due to AWS only exposing the load balancer IP. The client won't be able to detect any other nodes:

elastic/client.go

Lines 964 to 978 in 60d62e5

	if err := json.NewDecoder(res.Body).Decode(&info); err == nil {
	if len(info.Nodes) > 0 {
	for nodeID, node := range info.Nodes {
	if c.snifferCallback(node) {
	if node.HTTP != nil && len(node.HTTP.PublishAddress) > 0 {
	url := c.extractHostname(c.scheme, node.HTTP.PublishAddress)
	if url != "" {
	nodes = append(nodes, newConn(nodeID, url))
	}
	}
	}
	}
	}
	}
	return nodes

Any steps to reproduce the behavior?

1. instantiate a new client, setting the AWS endpoint as a single host entry in the config
2. Trigger cluster re-provisioning in AWS, described here: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html#es-managedomains-configuration-changes

[olivere]

Hmm... if I understand it correctly, with AWS you should simply disable sniffing and health checks as AWS does load-balancing for you, and you should simply use the hostname provided by AWS as a single endpoint.

What I don't understand is why the *http.Client won't find the new IP address when it changes. It should simply use the hostname, and the resolver should return the new IP. Unless there's some caching going on, that should simply work... as long as the hostname is the same.

I'm sorry if misunderstood—I'm not an AWS customer.

[dwickstrom]

Thank you for responding so quickly. Your suggestion to disable both sniffing and healthchecks sounds good and I'm trying it in a moment.

What I don't understand is why the *http.Client won't find the new IP address when it changes. It should simply use the hostname, and the resolver should return the new IP. Unless there's some caching going on, that should simply work... as long as the hostname is the same.

Yes, and even with healthchecks & sniffing disabled I'm guessing that this problem will appear. I'll post my findings here soon.

[dwickstrom]

Alright, thank you! That seems to have solved the problem 🎈 Not sure why though 🤔

I was hoping to learn something from this so I figure I should probably give some more context as to what I was going through.
They way we are set up is that we have a cluster of 3 master nodes that we connect to through a single endpoint, as you probably know, the one provided through the aws console. Initially I though of that AWS endpoint address as likely pointing to a load balancer, but after a while I realised that this isn't the case. Instead it's going to cycle randomly, resolving to the IP of any of the nodes.

And so the problem that I have been trying to resolve happens when the cluster is re-provisioned. This is what I think happens during that phase:

1. The amount of nodes are doubled
2. The data from the first set of nodes are migrated over to the new cluster
3. Once data is migrated the endpoint starts resolving to the addresses of the new set of cluster nodes
4. When data migration is complete, the old nodes are shut down one by one

Here's what I don't understand: after step 3, in the case where healthchecks are enabled, why would the healthcheck requests start failing - as opposed to when healthchecks are disabled, why would the normal requests not fail?

[olivere]

Hmm... let's see.

First of all, the whole idea of sniffing and health checks is only necessary because in the early days of ES, load-balancing was done client-side. If you have a server-side solution, which I think is the right solution, you shouldn't be able to do any of the things. Just let the server do the right thing and keep the client dumb.

Now, sniffing is the process of initially and periodically finding the list of nodes in the connected cluster. Let's say you initially have a 1 node cluster and use elastic to connect to that cluster with a URL. Then ES will use the URL to find all nodes in the cluster (1 node only) via the Cluster State API. It will then throw away the initial URL and use the IP/hostname reported from the cluster API. Once in a while, this process is re-executed to find new nodes in the cluster that were eventually added by the admin. So, eventually, ES will have a full list of IPs/hostnames to connect to and use them via round-robin. Notice there are a few edge-cases like ensuring this process if we do end up with an empty list of nodes for some reason. But let's try to keep it simple.

Health checks serve another purpose. They periodically check the list of nodes and manage the individual state of those nodes. E.g. if elastic tried to send a request to a node that didn't respond, it is marked as dead and no longer used. However, that could only be a blip in the network, so health check runs periodically to mark them as alive eventually. Again, there are some edge cases.

I currently don't see why one would disable sniffing but keep health checks enabled. So maybe they should be disabled as well, automatically, when sniffing is disabled.

[dwickstrom]

I currently don't see why one would disable sniffing but keep health checks enabled. So maybe they should be disabled as well, automatically, when sniffing is disabled.

That sounds reasonable to me.

In any case it might be a good idea to put it into the AWS section of the wiki, to not use either sniffing or healthchecks.

[olivere]

I changed the docs in the Wiki and advised to disable both sniffing and health checks for AWS Elasticsearch Service.

[dwickstrom]

Great, thanks for helping out 🥇

[iandees]

I'm running into the same problem as David, even with healthcheck and sniff turned off. @dwickstrom do you remember if you changed anything on the underlying HTTP client instance maybe?

[dwickstrom]

Hi @iandees, no I didn't change anything on the HTTP client. Lately however there has been some issue with this, again. Back in May, the way I tested it was by toggling some parameter in the cluster settings, to trigger a cluster "rollover". Recently however, when AWS themselves were triggering an elasticsearch upgrade on their side, that "rollover" did not go well - clients were not able to connect without intervention, just like the incidents I had ~6 months ago.

[olivere]

Maybe there's still a problem. Reopening.

[olivere]

There was a change quite recently that addresses an issue on AWS ES with nodes changing IPs particularly. Don't know if this has anything to do with it. #1125

[g-wilson]

Hi all, resurrecting this thread to shine some more info. We're seeing this issue as well. After some pretty thorough testing I can replicate the issue. I don't think the issue is with this library.

AWS ES uses DNS based load-balancing to resolve the hostname to the ES nodes, it's not an EC2-style load balancer.

If an HTTP client is used which uses keep-alive connections (http.DefaultClient does by default), and your volume of requests is high enough that the idle timeout is never reached, the connection will not be re-established.

This means that when AWS rotates the nodes and changes the DNS records, an application using this library is none the wiser, it won't do another DNS lookup until the connections are left idle and then terminated.

Eventually this library does recognise that requests are failing and resets everything, however this does cause fairly significant interruption of service.

This issue is described well here golang/go#23427

[olivere]

Thanks for reporting your findings, @g-wilson.