[TT-14479] Potential Data Loss in Tyk Pump Due to Slow Elasticsearch Backend

[masoudhaghbin]

We have multiple backends configured in Tyk Pump, one of which is Elasticsearch (ES). We observed the following error in the logs:

Pump Elasticsearch Pump is taking more time than the value configured of purge_delay.

As a result, we noticed missing access logs not only in Elasticsearch but also in other backends. To mitigate this, we enabled batch insert, which reduced the frequency of errors, but we still want to ensure that access logs are never lost due to this issue.

Expected Behavior:
• Tyk Pump should ensure that access logs are consistently written to all configured backends, even if one of them (e.g., ES) is slow.
• Logs should not be lost if one backend is lagging.

Current Behavior:
• When Elasticsearch takes longer than purge_delay, logs are not recorded properly, causing data loss across multiple backends.
• Enabling batch insert has reduced errors, but the risk of losing logs remains.

Steps to Reproduce:
1. Configure multiple backends in Tyk Pump, including Elasticsearch.(we were using SQS, prometheus, mongo, kafka and ES)
2. Set a purge_delay value that might be exceeded by Elasticsearch under load.
3. Monitor logs and backend data consistency.
4. Observe missing access logs when the delay occurs.

Environment Details:
• Tyk Pump version: v1.9.0