TransactionExecutionLimit Timeout: Unable to retrieve routing table (Neo4j connectivity issue)

[riyaoayaw]

Hi, I am encountering an intermittent issue with Neo4j where the driver occasionally fails to retrieve the routing table, resulting in a TransactionExecutionLimit: timeout error.

Updating Neo4j connection due to err : TransactionExecutionLimit: timeout (exceeded max retry time: 30s) after 4 attempts, last error: ConnectivityError: Unable to retrieve routing table from XXX.databases.neo4j.io:7687: context deadline exceeded

The issue occurs randomly and is not consistently reproducible. At times, the service runs without any problems, but occasionally, the connection times out, leading to service disruptions.

We are uncertain about the root cause and would appreciate guidance on resolving this issue or understanding why it might be happening.

• Neo4j Version: Neo4j Aura Professional version 5
• Neo4j Mode: Single instance
• Driver Version: Go language driver v5.27.0
• Memory: 1GB
• Storage: 2GB
• CPU: 1

Steps to reproduce

1. Attempt to connect to the Neo4j instance.
2. Perform transactions or queries on the database.

Expected behavior

Queries and transactions should execute without errors, and the routing table should be retrieved successfully.

Actual behavior

The connection intermittently fails, producing a ConnectivityError, and the query times out with a TransactionExecutionLimit: timeout error.

[StephenCathcart]

Hi @riyaoayaw, sorry you're seeing the above issue. Could you supply more information such as debug/bolt logs of when this error occurs? These can be enabled at the driver or sessions level like so:

neo4j.NewDriverWithContext(...), func(config *config.Config) {
		config.Log = log.ToConsole(log.DEBUG)
	})
// ...
driver.NewSession(ctx, neo4j.SessionConfig{
    BoltLogger: log.BoltToConsole(),
})

Also, to rule out general networking issues, is there any middleware (proxies/load balancers, etc.) between your client and the database that could be causing issues? Have you checked for latency issues perhaps due to the location/region of the Aura database to where you're running the client?

Thank you

[riyaoayaw]

Hi @StephenCathcart ,

I have added the logs as mentioned. I hope these can help. If you need anything else, please do let me know

Jan 20 09:51:17 ip-172-31-18-61 web[976684]: 2025-01-20 09:51:17.499  DEBUG  [session 99] connection acquisition timeout is 5s, resolved deadline is: 2025-01-20 09:50:13.705034527 +0000 UTC m=+248932.771804371
Jan 20 09:51:17 ip-172-31-18-61 web[976684]: 2025-01-20 09:51:17.499   INFO  [router 98] Reading routing table from initial router: XXXXX.databases.neo4j.io:7687
Jan 20 09:51:17 ip-172-31-18-61 web[976684]: 2025-01-20 09:51:17.499  DEBUG  [pool 98] Trying to borrow connection from [XXXXX.databases.neo4j.io:7687]
Jan 20 09:51:17 ip-172-31-18-61 web[976684]: 2025-01-20 09:51:17.499   INFO  [pool 98] Connecting to XXXXX.databases.neo4j.io:7687
Jan 20 09:51:17 ip-172-31-18-61 web[976684]: 2025-01-20 09:51:17.499  DEBUG  [pool 98] Deactivating server XXXXX.databases.neo4j.io:7687
Jan 20 09:51:17 ip-172-31-18-61 web[976684]: 2025-01-20 09:51:17.499   WARN  [pool 98] Failed to connect to XXXXX.databases.neo4j.io:7687: dial tcp: lookup XXXXX.databases.neo4j.io: i/o timeout
Jan 20 09:51:17 ip-172-31-18-61 web[976684]: 2025-01-20 09:51:17.499   WARN  [pool 98] Borrow time-out
Jan 20 09:51:17 ip-172-31-18-61 web[976684]: 2025-01-20 09:51:17.499  ERROR  [router 98] Unable to retrieve routing table from XXXXX.databases.neo4j.io:7687: context deadline exceeded
Jan 20 09:51:17 ip-172-31-18-61 web[976684]: 2025-01-20 09:51:17.499  ERROR  [session 99] TransactionExecutionLimit: timeout (exceeded max retry time: 30s) after 6 attempts, last error: ConnectivityError: Unable to retrieve routing table from XXXXX.databases.neo4j.io:7687: context deadline exceeded
Jan 20 09:51:17 ip-172-31-18-61 web[976684]: 2025-01-20 09:51:17.499  DEBUG  [router 98] Cleaning up
Jan 20 09:51:17 ip-172-31-18-61 web[976684]: 2025-01-20 09:51:17.499  DEBUG  [session 99] Closed
Jan 20 09:51:17 ip-172-31-18-61 web[976684]: 2025/01/20 09:51:17 Updating Neo4j connection due to err : failed to run query to get id Person, err: TransactionExecutionLimit: timeout (exceeded max retry time: 30s) after 6 attempts, last error: ConnectivityError: Unable to retrieve routing table from XXXXX.databases.neo4j.io:7687: context deadline exceeded

As for the issues due to region, my Neo4j database is hosted on AWS in Singapore, while my service is hosted in Paris. Do you think this could lead to any issues, particularly in terms of latency, bandwidth, or overall performance? Any insights or recommendations to mitigate potential challenges would be greatly appreciated.

[riyaoayaw]

Hi @StephenCathcart , I am still facing this issue, is there any update regarding this?