bzip2 cli app compresses better than bzip-rs with Level::Best

[kamikaze]

I've got a 33% smaller archive (2.7GB instead of 3.6GB) with a regular bzip2 -9 compression comparing to Level::Best of this library. Why?

[kamikaze]

bzip2-rs/src/write.rs

Line 35 in f5f9d09

buf: Vec::with_capacity(32 * 1024),

Please make this buffer size adjustable

[bjorn3]

Vec::with_capacity(n) is equivalent in behavior to Vec::new(). Both create an empty vector. The only difference is that the former can avoid some allocations by allocating the buffer up-front. It won't have any effect on the archive size at all.

[bjorn3]

Are you using bzip2-rs with the C or the Rust backend? Which version of the bzip2 cli are you comparing against?

[kamikaze]

$ bzip2 --version
bzip2, a block-sorting file compressor.  Version 1.0.8, 13-Jul-2019.

Good question about C/Rust - actually I'm a noob in Rust for now. I'm using it via

async-compression = { version = "0.4.22", features = ["tokio", "bzip2"] }

use async_compression::Level;
use async_compression::tokio::write::BzEncoder;

let bz2_encoder = BzEncoder::with_quality(sink, Level::Best);

[folkertdev]

I wonder if async-compression does something that flushes data early, i.e. for compression the rust version only looks at a small section of the input, while the CLI tool can look at much bigger sections, and make better decisions based on that extra knowledge.

That normally should not be the case, because bzip2 does its own buffering as far as I remember, but maybe something weird is happenign.

[NobodyXu]

https://docs.rs/async-compression/latest/src/async_compression/codec/bzip2/encoder.rs.html

IIRC there isn't any early flushing in async-compression

[NobodyXu]

https://docs.rs/async-compression/latest/src/async_compression/codec/bzip2/encoder.rs.html

IIRC there isn't any early flushing in async-compression

[kamikaze]

-rw-rw-r-- 1 kamikaze kamikaze 180M Apr  9 18:36 archive_20250408_210018_1.tar.bz2
-rw-rw-r-- 1 kamikaze kamikaze 102M Apr  9 18:34 archive_20250408_210018_6.tar.bz2
-rw-rw-r-- 1 kamikaze kamikaze  91M Apr  9 15:34 archive_20250408_210018_9.tar.bz2
-rw-rw-r-- 1 kamikaze kamikaze 262M Apr  9 14:49 archive_orig.tar.bz2

archive_orig.tar.bz2 is what I've got generated from async-compression + bzip2

others are corresponding compression levels from bzip2 cli. "-1" is a 100KB block size... and even it is being compressed much better o_O

[kamikaze]

I just switched encoder to XzEncoder (xz2 feature in async-compression), that's the only change. And file looks really compressed

-rw-rw-r-- 1 kamikaze kamikaze 180M Apr  9 18:36 archive_20250408_210018_1.tar.bz2
-rw-rw-r-- 1 kamikaze kamikaze 102M Apr  9 18:34 archive_20250408_210018_6.tar.bz2
-rw-rw-r-- 1 kamikaze kamikaze  91M Apr  9 15:34 archive_20250408_210018_9.tar.bz2
-rw-rw-r-- 1 kamikaze kamikaze  38M Apr  9 21:36 archive_20250409_183059.tar.xz2
-rw-rw-r-- 1 kamikaze kamikaze 262M Apr  9 14:49 archive_orig.tar.bz2

[folkertdev]

can you share the code used for those output numbers in like a gist or something?

[kamikaze]

That will be a good task for me, meanwhile I can show my code in a project: https://github.com/bixority/object-storage-maintenance/blob/main/src/compressor.rs

[NobodyXu]

@kamikaze I think your process_objects can be optimized, it currently downloads and compress sequentially while it can be parallelised.

The code wait for the current download and tarball append to complete, before starting to download another, which is likely why it's slower than using bzip cli with all data already downloaded.

You could parallelise it by:

• use tokio::spawn to spawn the tarball append task compress_object, it will run in a loop, use mpsc to receive objects to be compressed
• in process_objects spawns a task for each process_object call in for object in content, so that the s3 object get request can be sent and processed in parallel, and the process_objects can continue the loop and issue another lsdir request
• in process_object once s3 request get returns, send the object to the compress_object tarball append task via the mpsc

Reason that the mpsc + spawned task for compress_objects is that the tokio_tar::Builder requires mutable reference and using a tokio::sync::Mutex is worse than having a dedicated task + mpsc as with mutex many tasks would be blocked and taking up memory, where as with mpsc + task these tasks can finish quickly and free up resources.

Not sure how much of a speed up it will be, but I think it would be better in that it can fully saturate your network bandwidth with requests

[NobodyXu]

@kamikaze one of the key difference between your demo and bzip2 cli is the I/O part, fetching from network has more latency and less bandwidth than disk I/O, and your code is not optimized to fully take advantage of all network bandwidth to saturate it and make sure there is always some data it can process

[kamikaze]

@kamikaze one of the key difference between your demo and bzip2 cli is the I/O part, fetching from network has more latency and less bandwidth than disk I/O, and your code is not optimized to fully take advantage of all network bandwidth to saturate it and make sure there is always some data it can process

huge thanks for your suggestions. Will adjust my code. (but I guess this is not the reason of poor compression, since I've swapped BzEncoder to XzEncoder and things went waaaay better compressed)

[NobodyXu]

since I've swapped BzEncoder to XzEncoder

@kamikaze it might be that xz creates much smaller archive that takes much faster to upload, since it's likely the network bandwidth is the bottleneck