Commit 2f144c6
authored
Do not panic on failure to acquire connections to store events (#4132)
# Description
During a particularly bad period of volatility we failed to acquire a DB
connection on time which lead to a panic
<details><summary>Partial Stacktrace</summary>
<p>
```
1770314825856 2026-02-05T18:07:05.856Z 2026-02-05T18:07:03.033Z ERROR observe::tracing: thread 'tokio-runtime-worker' panicked at /src/crates/autopilot/src/infra/persistence/mod.rs:301:54:
1770314825856 2026-02-05T18:07:05.856Z failed to acquire tx: PoolTimedOut
1770314825856 2026-02-05T18:07:05.856Z stack backtrace:
1770314825856 2026-02-05T18:07:05.856Z 0: observe::tracing::tracing_panic_hook
1770314825856 2026-02-05T18:07:05.856Z at ./src/crates/observe/src/tracing.rs:187:21
1770314825856 2026-02-05T18:07:05.856Z 1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
1770314825856 2026-02-05T18:07:05.856Z at ./rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2220:9
1770314825856 2026-02-05T18:07:05.856Z 2: observe::panic_hook::install::{{closure}}
1770314825856 2026-02-05T18:07:05.856Z at ./src/crates/observe/src/panic_hook.rs:14:9
1770314825856 2026-02-05T18:07:05.856Z 3: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
1770314825856 2026-02-05T18:07:05.856Z at ./rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/alloc/src/boxed.rs:2220:9
1770314825856 2026-02-05T18:07:05.856Z 4: std::panicking::panic_with_hook
1770314825856 2026-02-05T18:07:05.856Z at ./rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:833:13
1770314825856 2026-02-05T18:07:05.856Z 5: std::panicking::panic_handler::{{closure}}
1770314825856 2026-02-05T18:07:05.856Z at ./rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:698:13
1770314825856 2026-02-05T18:07:05.856Z 6: std::sys::backtrace::__rust_end_short_backtrace
1770314825856 2026-02-05T18:07:05.856Z at ./rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/sys/backtrace.rs:176:18
1770314825856 2026-02-05T18:07:05.856Z 7: __rustc::rust_begin_unwind
1770314825856 2026-02-05T18:07:05.856Z at ./rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/std/src/panicking.rs:689:5
1770314825856 2026-02-05T18:07:05.856Z 8: core::panicking::panic_fmt
1770314825856 2026-02-05T18:07:05.856Z at ./rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/panicking.rs:80:14
1770314825856 2026-02-05T18:07:05.856Z 9: core::result::unwrap_failed
1770314825856 2026-02-05T18:07:05.856Z at ./rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/result.rs:1867:5
1770314825856 2026-02-05T18:07:05.856Z 10: core::result::Result<T,E>::expect
1770314825856 2026-02-05T18:07:05.856Z at ./rustc/254b59607d4417e9dffbc307138ae5c86280fe4c/library/core/src/result.rs:1185:23
1770314825856 2026-02-05T18:07:05.856Z 11: autopilot::infra::persistence::Persistence::store_order_events::{{closure}}
1770314825856 2026-02-05T18:07:05.856Z at ./src/crates/autopilot/src/infra/persistence/mod.rs:301:54
```
</p>
</details>
# Changes
- [ ] Remove the panic
- [ ] Log on error and continue
## How to test
NA1 parent 950180e commit 2f144c6
2 files changed
+12
-2
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
298 | 298 | | |
299 | 299 | | |
300 | 300 | | |
301 | | - | |
302 | | - | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
303 | 312 | | |
304 | 313 | | |
305 | 314 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
964 | 964 | | |
965 | 965 | | |
966 | 966 | | |
| 967 | + | |
967 | 968 | | |
968 | 969 | | |
969 | 970 | | |
| |||
0 commit comments