rustic panics on sending message on channel

[simonsan]

Tried the copy operation again this time without "-- init"

rustic copy 4246b963

[00:47:12] copying blobs...               ████████████████████████████████████████ 215.87 GiB            78.29 MiB/s                                                    [ERROR] service=s3 operation=Reader::read path=data/81/815238a856c9dc4f25a08179f4bc7092b99a6f6f5696d31c841ad5fa8c1ae2c6 read=0 -> read failed: Unexpected (temporary) at read, context: { url: https://s3.somecloudservice.com/bucket/data/81/815238a856c9dc4f25a08179f4bc7092b99a6f6f5696d31c841ad5fa8c1ae2c6, called: http_util::Client::send, service: s3, path: data/81/815238a856c9dc4f25a08179f4bc7092b99a6f6f5696d31c841ad5fa8c1ae2c6, range: 2748878-4859002 } => send http request, source: error sending request for url (https://s3.somecloudservice.com/bucket/data/81/815238a856c9dc4f25a08179f4bc7092b99a6f6f5696d31c841ad5fa8c1ae2c6): connection closed before message completed
[00:47:14] copying blobs...               ████████████████████████████████████████ 216.09 GiB            78.45 MiB/s                                                    [WARN] operation=Reader::read path=data/4e/4ec7008bbba0b0246cab82fe468592e562a7088a6822d42294a09801441ab764 -> retry after 4.812975168s: error=Unexpected (temporary) at read, context: { url: https://s3.somecloudservice.com/bucket/data/4e/4ec7008bbba0b0246cab82fe468592e562a7088a6822d42294a09801441ab764, called: http_util::Client::send, service: s3, path: data/4e/4ec7008bbba0b0246cab82fe468592e562a7088a6822d42294a09801441ab764, range: 1540147-3014903 } => send http request, source: error sending request for url (https://s3.somecloudservice.com/bucket/data/4e/4ec7008bbba0b0246cab82fe468592e562a7088a6822d42294a09801441ab764): connection error: unexpected end of file
[00:47:21] copying blobs...               ████████████████████████████████████████ 216.54 GiB            76.19 MiB/s                                                    [WARN] operation=Reader::read path=data/4e/4ec7008bbba0b0246cab82fe468592e562a7088a6822d42294a09801441ab764 -> retry after 8.348090649s: error=Unexpected (temporary) at Reader::read, context: { url: https://s3.somecloudservice.com/bucket/data/4e/4ec7008bbba0b0246cab82fe468592e562a7088a6822d42294a09801441ab764, service: s3, path: data/4e/4ec7008bbba0b0246cab82fe468592e562a7088a6822d42294a09801441ab764, read_buf: 1474757 } => read data from http stream, source: request or response body error: error reading a body from connection: end of file before message length reached
[00:47:30] copying blobs...               ████████████████████████████████████████ 217.32 GiB            81.50 MiB/s                                                    [WARN] operation=Reader::read path=data/4e/4ec7008bbba0b0246cab82fe468592e562a7088a6822d42294a09801441ab764 -> retry after 16.068087101s: error=Unexpected (temporary) at Reader::read, context: { url: https://s3.somecloudservice.com/bucket/data/4e/4ec7008bbba0b0246cab82fe468592e562a7088a6822d42294a09801441ab764, service: s3, path: data/4e/4ec7008bbba0b0246cab82fe468592e562a7088a6822d42294a09801441ab764, read_buf: 1474757 } => read data from http stream, source: request or response body error: error reading a body from connection: end of file before message length reached
[01:28:34] copying blobs...               ████████████████████████████████████████ 383.45 GiB            68.28 MiB/s                                                    [WARN] operation=Reader::read path=data/63/63005e109decac2219c770a18b541217f020c4bc1f1a61b4e4a57d3bc7af46b5 -> retry after 1.8064556120000002s: error=Unexpected (temporary) at read, context: { url: https://s3.somecloudservice.com/bucket/data/63/63005e109decac2219c770a18b541217f020c4bc1f1a61b4e4a57d3bc7af46b5, called: http_util::Client::send, service: s3, path: data/63/63005e109decac2219c770a18b541217f020c4bc1f1a61b4e4a57d3bc7af46b5, range: 0-981761 } => send http request, source: error sending request for url (https://s3.somecloudservice.com/bucket/data/63/63005e109decac2219c770a18b541217f020c4bc1f1a61b4e4a57d3bc7af46b5): connection error: unexpected end of file
[04:12:56] copying blobs...               ████████████████████████████████████████   1.11 TiB            89.06 MiB/s                                                    The application panicked (crashed).
Message:  called `Result::unwrap()` on an `Err` value: "SendError(..)"
Location: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustic_core-0.2.0/src/blob/tree.rs:500

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
The application panicked (crashed).
Message:  called `Result::unwrap()` on an `Err` value: "SendError(..)"
Location: /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustic_core-0.2.0/src/blob/tree.rs:500

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
Aborted (core dumped)

Originally posted by @kalleyne in rustic-rs/rustic#1118

https://github.com/rustic-rs/rustic_core/blob/main/crates/core/src/blob/tree.rs#L500

[simonsan]

This issue is getting quite a ride, and we should investigate with a higher priority.

Related:

• refactor(errors): improve message and add logging when sending tree from backend panics #314
• Revert "refactor(errors): improve message and add logging when sending tree from backend panics" #325
• panic on prune #239
• v0.9.2 breaks prune operations rustic#1313