It seems fairly inconsistent on which queries it happens on, but consistently between creative queries that Tier 1 ends up having some connection timeouts. Stack trace:
retriever | 2026-04-06T18:35:25.419+00:00 21 ERROR query timed out: Connection timed out retriever.data_tiers.tier_1.elasticsearch.driver:run():156
retriever | Traceback (most recent call last):
retriever | File "<string>", line 1, in <module>
retriever | File "/usr/local/lib/python3.13/multiprocessing/spawn.py", line 122, in spawn_main
retriever | exitcode = _main(fd, parent_sentinel)
retriever | File "/usr/local/lib/python3.13/multiprocessing/spawn.py", line 135, in _main
retriever | return self._bootstrap(parent_sentinel)
retriever | File "/usr/local/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
retriever | self.run()
retriever | File "/usr/local/lib/python3.13/multiprocessing/process.py", line 108, in run
retriever | self._target(*self._args, **self._kwargs)
retriever | File "/usr/src/app/.venv/lib/python3.13/site-packages/uvicorn/_subprocess.py", line 80, in subprocess_started
retriever | target(sockets=sockets)
retriever | File "/usr/src/app/.venv/lib/python3.13/site-packages/uvicorn/supervisors/multiprocess.py", line 63, in target
retriever | return self.real_target(sockets)
retriever | File "/usr/src/app/.venv/lib/python3.13/site-packages/uvicorn/server.py", line 65, in run
retriever | return asyncio.run(self.serve(sockets=sockets))
retriever | File "/usr/local/lib/python3.13/asyncio/runners.py", line 195, in run
retriever | return runner.run(main)
retriever | File "/usr/local/lib/python3.13/asyncio/runners.py", line 118, in run
retriever | return self._loop.run_until_complete(task)
retriever | File "/usr/src/app/src/retriever/lookup/subquery.py", line 127, in handle_subquery_batch
retriever | list[list[ESEdge]], await query_driver.run_query(payload_batch)
retriever | File "/usr/src/app/.venv/lib/python3.13/site-packages/opentelemetry/util/_decorator.py", line 71, in async_wrapper
retriever | return await func(*args, **kwargs) # type: ignore
retriever | File "/usr/src/app/src/retriever/data_tiers/tier_1/elasticsearch/driver.py", line 191, in run_query
retriever | query_result = await self.run(query, bypass_cache)
retriever | > File "/usr/src/app/src/retriever/data_tiers/tier_1/elasticsearch/driver.py", line 144, in run
retriever | results = await run_batch_query(
retriever | File "/usr/src/app/src/retriever/data_tiers/tier_1/elasticsearch/aggregating_querier.py", line 125, in run_batch_query
retriever | responses = await es_connection.msearch(index=index_name, body=query_body_list)
retriever | File "/usr/src/app/.venv/lib/python3.13/site-packages/elasticsearch/_async/client/__init__.py", line 3476, in msearch
retriever | return await self.perform_request( # type: ignore[return-value]
retriever | File "/usr/src/app/.venv/lib/python3.13/site-packages/elasticsearch/_async/client/_base.py", line 271, in perform_request
retriever | response = await self._perform_request(
retriever | File "/usr/src/app/.venv/lib/python3.13/site-packages/elasticsearch/_async/client/_base.py", line 315, in _perform_request
retriever | meta, resp_body = await self.transport.perform_request(
retriever | File "/usr/src/app/.venv/lib/python3.13/site-packages/elastic_transport/_async_transport.py", line 264, in perform_request
retriever | resp = await node.perform_request(
retriever | File "/usr/src/app/.venv/lib/python3.13/site-packages/elastic_transport/_node/_http_aiohttp.py", line 228, in perform_request
retriever | raise err from None
retriever | elastic_transport.ConnectionTimeout: Connection timed out
It seems fairly inconsistent on which queries it happens on, but consistently between creative queries that Tier 1 ends up having some connection timeouts. Stack trace: