Skip to content

manpage.md

Latest commit

 

History

History
478 lines (269 loc) · 22.4 KB

manpage.md

File metadata and controls

478 lines (269 loc) · 22.4 KB
Raw

Command-Line Help for git-perf

This document contains the help content for the git-perf command-line program.

Command Overview:

git-perf

Usage: git-perf [OPTIONS] <COMMAND>

Subcommands:
  • measure — Measure the runtime of the supplied command (in nanoseconds)
  • add — Add single measurement
  • import — Import measurements from test runners and benchmarks
  • push — Publish performance results to remote
  • pull — Pull performance results from remote
  • report — Create an HTML performance report
  • audit — For given measurements, check perfomance deviations of the HEAD commit against <n> previous commits. Group previous results and aggregate their results before comparison
  • bump-epoch — Accept HEAD commit's measurement for audit, even if outside of range. This is allows to accept expected performance changes. This is accomplished by starting a new epoch for the given measurement. The epoch is configured in the git perf config file. A change to the epoch therefore has to be committed and will result in a new HEAD for which new measurements have to be taken
  • remove — Remove all performance measurements for commits that have been committed at or before the specified time (inclusive boundary, uses <=)
  • prune — Remove all performance measurements for non-existent/unreachable objects. Will refuse to work if run on a shallow clone
  • status — Show pending measurements that haven't been pushed
  • reset — Drop locally pending measurements that haven't been pushed
  • list-commits — List all commits that have performance measurements
  • size — Estimate storage size of live performance measurements
  • study — Analyze cross-runner variance to recommend .gitperfconfig settings
  • config — Manage git-perf configuration
Options:
  • -v, --verbose — Increase verbosity level (can be specified multiple times.) The first level sets level "info", second sets level "debug", and third sets level "trace" for the logger

git-perf measure

Measure the runtime of the supplied command (in nanoseconds)

Usage: git-perf measure [OPTIONS] --measurement <NAME> -- <COMMAND>...

Arguments:
  • <COMMAND> — Command to measure
Options:

  • -n, --repetitions <REPETITIONS> — Repetitions

    Default value: 1

  • -m, --measurement <NAME> — Name of the measurement

  • -k, --key-value <KEY_VALUE> — Key-value pairs separated by '='

  • --commit <COMMIT> — Target commit for measurement (default: HEAD)

  • --skip-env — Skip reading environment variables from the [environment] config section

    Default value: false

git-perf add

Add single measurement

Usage: git-perf add [OPTIONS] --measurement <NAME> <VALUE>

Arguments:
  • <VALUE> — Measured value to be added
Options:

  • -m, --measurement <NAME> — Name of the measurement

  • -k, --key-value <KEY_VALUE> — Key-value pairs separated by '='

  • --commit <COMMIT> — Target commit for measurement (default: HEAD)

  • --skip-env — Skip reading environment variables from the [environment] config section

    Default value: false

git-perf import

Import measurements from test runners and benchmarks

Parse and store runtime measurements from external tools like cargo-nextest (JUnit XML) and cargo-criterion (JSON). This allows tracking test and benchmark performance over time using git-perf's measurement infrastructure.

Supported Formats

junit - JUnit XML format - Works with: cargo-nextest, pytest, Jest, JUnit, and many other test frameworks - Requires: Configure nextest with JUnit output in .config/nextest.toml - Command: cargo nextest run --profile ci (outputs to target/nextest/ci/junit.xml)

criterion-json - cargo-criterion JSON format - Works with: cargo-criterion benchmarks - Command: cargo criterion --message-format json

Measurement Naming

Tests: test::<test_name> Benchmarks: bench::<benchmark_id>::<statistic> (mean, median, slope, mad)

Examples

# Import from stdin cat junit.xml | git-perf import junit

# Import with metadata git-perf import junit junit.xml --metadata ci=true --metadata branch=main

# Import with filtering (regex) git-perf import junit junit.xml --filter "^integration::"

# Dry run to preview git-perf import junit junit.xml --dry-run --verbose

# Import benchmarks cargo criterion --message-format json > bench.json git-perf import criterion-json bench.json ```

**Usage:** `git-perf import [OPTIONS] <FORMAT> [FILE]`

###### **Arguments:**

* `<FORMAT>` — Format of the input data

  Possible values:
  - `junit`:
    JUnit XML format (nextest, pytest, Jest, etc.)
  - `criterion-json`:
    cargo-criterion JSON format

* `<FILE>` — Input file path (use '-' or omit for stdin)

###### **Options:**

* `--commit <COMMIT>` — Target commit for measurements (default: HEAD)
* `-p`, `--prefix <PREFIX>` — Optional prefix to prepend to measurement names
* `-m`, `--metadata <METADATA>` — Key-value pairs separated by '=' to add as metadata to all measurements
* `-f`, `--filter <FILTER>` — Regex filter to select specific tests/benchmarks
* `--dry-run` — Preview what would be imported without storing
* `-v`, `--verbose` — Show detailed information about imported measurements
* `--skip-env` — Skip reading environment variables from the `[environment]` config section

  Default value: `false`



## `git-perf push`

Publish performance results to remote

**Usage:** `git-perf push [OPTIONS]`

###### **Options:**

* `-r`, `--remote <REMOTE>` — Remote to push to (defaults to git-perf-origin)



## `git-perf pull`

Pull performance results from remote

**Usage:** `git-perf pull`



## `git-perf report`

Create an HTML performance report

**Usage:** `git-perf report [OPTIONS] [COMMIT]`

###### **Arguments:**

* `<COMMIT>` — Target commit to start report from (default: HEAD)

###### **Options:**

* `-o`, `--output <OUTPUT>` — HTML output file

  Default value: `output.html`
* `-n`, `--max-count <MAX_COUNT>` — Limit the number of previous commits considered. HEAD is included in this count

  Default value: `40`
* `--since <SINCE>` — Only include commits more recent than a specific date. Accepts all formats that `git log --since` accepts, e.g. "2025-01-01", "2 weeks ago", "yesterday"
* `--until <UNTIL>` — Only include commits older than a specific date. Accepts all formats that `git log --until` accepts, e.g. "2025-12-31", "yesterday"
* `-m`, `--measurement <MEASUREMENT>` — Select an individual measurements instead of all
* `-k`, `--key-value <KEY_VALUE>` — Key-value pairs separated by '=', select only matching measurements
* `-f`, `--filter <FILTER>` — Filter measurements by regex pattern (can be specified multiple times). If any filter matches, the measurement is included (OR logic). Patterns are unanchored by default. Use ^pattern$ for exact matches. Example: -f "bench.*" -f "test_.*"
* `-s`, `--separate-by <SEPARATE_BY>` — Create individual traces in the graph by grouping with the value of this selector. Can be specified multiple times to split on multiple dimensions (e.g., -s os -s arch). Multiple splits create combined group labels like "ubuntu/x64"
* `-a`, `--aggregate-by <AGGREGATE_BY>` — What to aggregate the measurements in each group with

  Possible values: `min`, `max`, `median`, `mean`

* `-t`, `--template <TEMPLATE>` — Path to custom HTML template file (overrides config)
* `-c`, `--custom-css <CUSTOM_CSS>` — Path to custom CSS file to inject into the template
* `--title <TITLE>` — Custom title for the report (overrides default)
* `--show-epochs` — Show epoch boundary markers in the report (hidden by default, toggleable via legend)
* `--show-changes` — Detect and show change points in the report (hidden by default, toggleable via legend)



## `git-perf audit`

For given measurements, check perfomance deviations of the HEAD commit against `<n>` previous commits. Group previous results and aggregate their results before comparison.

The audit can be configured to ignore statistically significant deviations if they are below a minimum relative deviation threshold. This helps filter out noise while still catching meaningful performance changes.

## Statistical Dispersion Methods

The audit supports two methods for calculating statistical dispersion:

**Standard Deviation (stddev)**: Traditional method that is sensitive to outliers. Use when your performance data is normally distributed and you want to detect all performance changes, including those caused by outliers.

**Median Absolute Deviation (MAD)**: Robust method that is less sensitive to outliers. Use when your performance data has occasional outliers or spikes, or when you want to focus on typical performance changes rather than extreme values.

## Configuration

Configuration is done via the `.gitperfconfig` file:

**Default settings:** - `[measurement].min_relative_deviation = 5.0` - `[measurement].min_absolute_deviation = 0.5` - `[measurement].dispersion_method = "mad"` - `[measurement].min_measurements = 3` - `[measurement].aggregate_by = "median"` - `[measurement].sigma = 3.5`

**Measurement-specific settings (override defaults):** - `[measurement."name"].min_relative_deviation = 10.0` - `[measurement."name"].min_absolute_deviation = 1.0` - `[measurement."name"].dispersion_method = "stddev"` - `[measurement."name"].min_measurements = 5` - `[measurement."name"].aggregate_by = "mean"` - `[measurement."name"].sigma = 4.5`

## Precedence

All audit options follow the same precedence order: 1. CLI option (if specified) - highest priority 2. Measurement-specific config - overrides default 3. Default config - overrides built-in default 4. Built-in default - lowest priority

**Note:** When `--min-measurements` is specified on CLI, it applies to ALL measurements in the audit, overriding any per-measurement config values.

Built-in defaults: - `min_measurements`: 2 - `aggregate_by`: min - `sigma`: 4.0 - `dispersion_method`: stddev

When the relative deviation is below the threshold, the audit passes even if the z-score exceeds the sigma threshold. The relative deviation is calculated as: `|(head_value / tail_median - 1.0) * 100%|` where tail_median is the median of historical measurements (excluding HEAD).

The sparkline visualization shows the range of measurements relative to the tail median (historical measurements only).

**Usage:** `git-perf audit [OPTIONS] [COMMIT]`

###### **Arguments:**

* `<COMMIT>` — Target commit to audit (default: HEAD)

###### **Options:**

* `-m`, `--measurement <MEASUREMENT>` — Specific measurement names to audit (can be specified multiple times). At least one of --measurement or --filter must be provided. Multiple measurements use OR logic. Example: -m timer -m memory
* `-n`, `--max-count <MAX_COUNT>` — Limit the number of previous commits considered. HEAD is included in this count

  Default value: `40`
* `--since <SINCE>` — Only include commits more recent than a specific date. Accepts all formats that `git log --since` accepts, e.g. "2025-01-01", "2 weeks ago", "yesterday"
* `--until <UNTIL>` — Only include commits older than a specific date. Accepts all formats that `git log --until` accepts, e.g. "2025-12-31", "yesterday"
* `-s`, `--selectors <SELECTORS>` — Key-value pair separated by "=" with no whitespaces to subselect measurements
* `-f`, `--filter <FILTER>` — Filter measurements by regex pattern (can be specified multiple times). At least one of --measurement or --filter must be provided. If any filter matches, the measurement is included (OR logic). Patterns are unanchored by default. Use ^pattern$ for exact matches. Examples: -f "bench_.*" (prefix), -f ".*_x64$" (suffix), -f "^perf_" (anchored prefix)
* `-S`, `--separate-by <SEPARATE_BY>` — Create separate audit groups by grouping with the value of this selector. Can be specified multiple times to split on multiple dimensions (e.g., -S os -S arch). Multiple splits create combined group labels like "os=ubuntu/arch=x64". Each group is audited independently with its own statistical validation
* `--min-measurements <MIN_MEASUREMENTS>` — Minimum number of historic measurements needed. If less, pass test and assume more measurements are needed. A minimum of two historic measurements are needed for proper evaluation of standard deviation. If specified on CLI, applies to ALL measurements (overrides config). If not specified, uses per-measurement config or defaults to 2
* `-a`, `--aggregate-by <AGGREGATE_BY>` — What to aggregate the measurements in each group with. If not specified, uses the value from .gitperfconfig file, or defaults to min

  Possible values: `min`, `max`, `median`, `mean`

* `-d`, `--sigma <SIGMA>` — Multiple of the dispersion after which an outlier is detected. If the HEAD measurement is within the acceptable range based on this threshold, it is considered acceptable. If not specified, uses the value from .gitperfconfig file, or defaults to 4.0
* `-D`, `--dispersion-method <DISPERSION_METHOD>` — Method for calculating statistical dispersion. Choose between:

   **stddev**: Standard deviation - sensitive to outliers, use for normally distributed data where you want to detect all changes.

   **mad**: Median Absolute Deviation - robust to outliers, use when data has occasional spikes or you want to focus on typical changes.

   If not specified, uses the value from .gitperfconfig file, or defaults to stddev.

  Possible values: `stddev`, `mad`

* `--max-cov <MAX_COV>` — Flag measurements with high Coefficient of Variation (CoV = σ/μ × 100%). Tail CoV is computed from per-commit aggregated values and reflects cross-run baseline stability. Head CoV is computed from the raw measurements at HEAD and reflects within-run repeatability. A warning is emitted when either exceeds this percentage threshold. If not specified, uses the value from .gitperfconfig file, or no flagging
* `--no-change-point-warning` — Suppress warning when change points are detected in the current epoch. By default, audit will warn if a change point (regime shift) is detected within the current measurement epoch, as this may affect z-score reliability



## `git-perf bump-epoch`

Accept HEAD commit's measurement for audit, even if outside of range. This is allows to accept expected performance changes. This is accomplished by starting a new epoch for the given measurement. The epoch is configured in the git perf config file. A change to the epoch therefore has to be committed and will result in a new HEAD for which new measurements have to be taken

**Usage:** `git-perf bump-epoch --measurement <MEASUREMENTS>`

###### **Options:**

* `-m`, `--measurement <MEASUREMENTS>`



## `git-perf remove`

Remove all performance measurements for commits that have been committed at or before the specified time (inclusive boundary, uses <=).

By default, this command automatically prunes orphaned measurements after removal (measurements for commits that no longer exist or are unreachable). Use --no-prune to skip this automatic cleanup.

Note: Only published measurements (i.e., those that have been pushed to the remote repository) can be removed. Local unpublished measurements are not affected by this operation.

**Usage:** `git-perf remove [OPTIONS] --older-than <OLDER_THAN>`

###### **Options:**

* `--older-than <OLDER_THAN>`
* `--no-prune` — Skip automatic pruning of orphaned measurements after removal

  Default value: `false`
* `--dry-run` — Preview what would be removed without actually removing



## `git-perf prune`

Remove all performance measurements for non-existent/unreachable objects. Will refuse to work if run on a shallow clone

**Usage:** `git-perf prune`



## `git-perf status`

Show pending measurements that haven't been pushed

Lists local measurements that exist in write branches but haven't been pushed to the remote repository. Similar to `git status` for tracking which changes are pending.

Pending measurements are those created with `add`, `measure`, or `import` that haven't been published via `push`. These can be safely discarded with the `reset` command.

Examples: git perf status                    # Show summary of pending measurements git perf status --detailed         # Show per-commit breakdown

**Usage:** `git-perf status [OPTIONS]`

###### **Options:**

* `-d`, `--detailed` — Show detailed per-commit breakdown



## `git-perf reset`

Drop locally pending measurements that haven't been pushed

Removes measurements from local write branches that haven't been pushed to the remote repository. This is useful for discarding test or debug measurements before publishing.

IMPORTANT: This only affects local pending measurements. Measurements that have been pushed to the remote are not affected. Use `remove` or `prune` for managing published measurements.

Removes ALL pending measurements.

Examples: git perf reset                        # Remove all pending measurements git perf reset --dry-run              # Preview what would be reset git perf reset --force                # Skip confirmation prompt

**Usage:** `git-perf reset [OPTIONS]`

###### **Options:**

* `--dry-run` — Preview what would be reset without actually resetting
* `-f`, `--force` — Skip confirmation prompt (dangerous)



## `git-perf list-commits`

List all commits that have performance measurements.

Outputs one commit SHA-1 hash per line. This can be used to identify which commits have measurements stored in the performance notes branch.

Example: git perf list-commits | wc -l  # Count commits with measurements git perf list-commits | head   # Show first few commits

**Usage:** `git-perf list-commits`



## `git-perf size`

Estimate storage size of live performance measurements

This command calculates the total size of performance measurement data stored in git notes (refs/notes/perf-v3). Use --detailed to see a breakdown by measurement name.

By default, shows logical object sizes (uncompressed). Use --disk-size to see actual on-disk sizes accounting for compression.

Examples: git perf size                    # Show total size in human-readable format git perf size --detailed         # Show breakdown by measurement name git perf size --format bytes     # Show size in raw bytes git perf size --disk-size        # Show actual on-disk sizes git perf size --include-objects  # Include git repository statistics

**Usage:** `git-perf size [OPTIONS]`

###### **Options:**

* `-d`, `--detailed` — Show detailed breakdown by measurement name
* `-f`, `--format <FORMAT>` — Output format (human-readable or bytes)

  Default value: `human`

  Possible values:
  - `human`:
    Human-readable format (e.g., "1.2 MB")
  - `bytes`:
    Raw bytes as integer

* `--disk-size` — Use on-disk size (compressed) instead of logical size
* `--include-objects` — Include git repository statistics for context



## `git-perf study`

Analyze cross-runner variance to recommend .gitperfconfig settings

Reads measurements at HEAD that were collected from multiple independent runner instances (each tagged with a group key via --key-value group=N), computes between-group CoV (Coefficient of Variation), and outputs recommended .gitperfconfig parameters.

## Workflow

1. Run the benchmark on N independent CI runners (use a matrix strategy). Each runner stores its measurements and tags them with a group key: `git-perf measure -n 10 -m MEASUREMENT --key-value group=$INSTANCE -- COMMAND` `git-perf push`

2. After all runners finish, pull and run study: `git-perf pull` `git-perf study -m MEASUREMENT`

## Interpretation

- CoV < 10%: stable benchmark — use the recommended config as-is - CoV 10–20%: moderate noise — monitor with max_cov and consider increasing workload - CoV > 20%: too noisy — improve benchmark isolation before merging

## Examples

```bash # Analyze cross-runner variance for a specific measurement git-perf study -m bench::add_measurements/add_measurement/1::median

# Fail if CoV exceeds 20% (useful as a CI gate on PRs) git-perf study -m bench::my_benchmark --max-cov 20 ```

**Usage:** `git-perf study [OPTIONS] --measurement <NAME>`

###### **Options:**

* `-m`, `--measurement <NAME>` — Name of the measurement to analyze
* `-n`, `--max-count <MAX_COUNT>` — Limit the number of previous commits considered (HEAD is included)

  Default value: `50`
* `--max-cov <MAX_COV>` — Fail if between-group CoV exceeds this percentage
* `--commit <COMMIT>` — Target commit to study (default: HEAD)
* `--group-by <GROUP_BY>` — Metadata key used to identify independent runner groups. Each matrix instance should tag its measurements with `--key-value <KEY>=<INSTANCE_NUMBER>`

  Default value: `group`



## `git-perf config`

Manage git-perf configuration

Display and query git-perf configuration settings, including git context (branch name, repository location), configuration sources, and measurement-specific settings. This follows the git config pattern.

Examples: git perf config --list                  # Show configuration summary git perf config --list --detailed       # Show all measurement settings git perf config --list --json           # Output as JSON git perf config --list --validate       # Check for config issues git perf config --list --measurement M  # Show specific measurement

**Usage:** `git-perf config [OPTIONS]`

###### **Options:**

* `-l`, `--list` — List all configuration settings (similar to git config --list)
* `-d`, `--detailed` — Show detailed configuration including all measurements
* `-f`, `--format <FORMAT>` — Output format (human-readable or JSON)

  Default value: `human`

  Possible values:
  - `human`:
    Human-readable format
  - `json`:
    JSON format for machine parsing

* `-v`, `--validate` — Validate configuration and report issues
* `-m`, `--measurement <MEASUREMENT>` — Show specific measurement configuration only



<hr/>

<small><i>
    This document was generated automatically by
    <a href="https://crates.io/crates/clap-markdown"><code>clap-markdown</code></a>.
</i></small>