Pixelator-core

This repository implements the core, compute-heavy functions from pixelator in Rust. Those routines are exposed through Python bindings as well as a CLI and the pixelator-core library so you can run performance-critical steps from either Python or the command line.

Documentation

Building the packages

Build all packages as follows (skip --release if you don't want an optimized build):

cargo build --release

After building the cli will be in target/release/community-detection and can be run with:

./target/release/community-detection --help

Python bindings

The python bindings are in the packages/python_bindings folder.

To build and install the bindings in the local uv environment:

cd packages/python_bindings
uv sync --extra dev --no-install-project
uv run maturin develop
uv sync --refresh --extra dev

You can then use them as follows:

from pixelator_core import run_label_propagation

run_label_propagation(
    parquet_file="./edgelist.parquet",
    epochs=1,
    output="./output.parquet",
)

If you want to build a wheel file, e.g. to install elsewhere, you can do so with:

cd packages/python_bindings
maturin build --release --out dist/

A wheel file will be created in the dist/ folder after building. You can install it in other environments with pip:

pip install path/to/wheel/file.whl

To run Python binding tests (build + pytest, using uv), use:

bash scripts/test-python-bindings.sh

Benchmark tests

We have some basic benchmarking in-place which is useful to get a feel for the performance of different potential choices. To run the benchmarks use:

cargo bench

And you will get some output like:

Timer precision: 10 ns
pixelator_core_benchmarks              fastest       │ slowest       │ median        │ mean          │ samples │ iters
├─ bench_community_detection                         │               │               │               │         │
│  ├─ bench_fast_label_propagation     2.617 ms      │ 7.626 ms      │ 3.233 ms      │ 3.799 ms      │ 100     │ 100
│  ├─ bench_leiden_cpm                 (ignored)     │               │               │               │         │
│  ├─ bench_leiden_modularity          14.41 ms      │ 17.03 ms      │ 14.54 ms      │ 14.59 ms      │ 100     │ 100
│  ╰─ bench_leiden_modularity_medium   14.05 s       │ 14.87 s       │ 14.19 s       │ 14.37 s       │ 3       │ 3
╰─ bench_parquet_io                                  │               │               │               │         │
   ├─ bench_create_graph_from_parquet  1.229 s       │ 1.47 s        │ 1.246 s       │ 1.286 s       │ 10      │ 10
   ├─ bench_parquet_reading            242.5 ms      │ 269.1 ms      │ 250.9 ms      │ 252.5 ms      │ 10      │ 10
   ╰─ bench_parquet_writing            213 ms        │ 251.5 ms      │ 224.4 ms      │ 224.9 ms      │ 10      │ 10

To add more benchmarks see the packages/pixelator_core/benches/main.rs file.

Peak-heap memory for graph construction, fast label propagation, and Leiden is reported by Divan's AllocProfiler as max alloc on the mem_* benches. The same cargo bench run records those values as extra charts at https://pixelgentechnologies.github.io/pixelator-core/dev/bench/index.html on pushes to main and dev. Pull requests still run the benchmarks and comment the deltas, but they do not write the published history.