A project should have some guiding metrics. Lavatube has these:
- Experiments - It is meant to be a sandbox for experimenting with trace tool technologies. The number of useful experiments done with it is its most important metric of success.
- Performance - It is meant to be fast, measured in time to completion, completion cost in watt and memory usage (for capture, postprocessing and replay - but we care more about replay).
- Autogeneration - It is mean to to be easy to maintain and change (due to metric 1), measured in how many commands and structs are hardcoded vs autogenerated.
- Tests - We have no testers and no test farm, so we need to rely on a large number of integrated tests that can rapidly be run after every commit.
How do we stack up?
- Lavatube was the first and is still the only Vulkan trace tool that does multi-threaded replay, proving that it is not only possible, but also can be done performant and sometimes increase performance by a lot.
- Lavatube was not the first to use virtual swapchains (that was Google's gapid), but the first one to use it integrated rather than as a layer and some of our lessons were provided as feedback to other tools.
- Feature and extension usage detection and unused removal code was developed here and later
ported to
gfxreconstruct-arm. - Lavatube is being used a bring-up testbed for the development of the spirv-simulator to enable Vulkan tracing for the bindless age. This will be used in other projects later.
- Lavatube is used to try out AI skills and AI coding, as code slop and mistakes can more easily be fixed later here and no users are harmed. Several useful AI skills have been develop in it that have later been copied into other projects.
- As an experiment in autogeneration, it has found quite a few bugs in the Khronos specs that have been reported and fixed.
- We are experimenting with making agentic usage and control easier through both a CLI interface meant for agent interaction and output optimized for token parsing and our own TUI which could use a local model for lower latency and cheaper operation.
We use Kishonti's recently open-sourced Kishonti GFXBench as our benchmark of choice here, running the Aztec Ruins scene with 100fps fixed-rate and comparing against gfxreconstruct. It seems completely CPU-bound on high-end desktop. We pick average of three runs.
How to measure:
- Time to completion and time to completion:
/usr/bin/time -f "%e,%M" <command> - Power:
perf stat -e power/energy-pkg/ <command>Ifpower/energy-gpu/is available, we should gather this one as well.
Latest results (less is better):
| Metric | Lavatube | Gfxr | Native |
|---|---|---|---|
| Capture power (J) | 3243 | 4541 | 1141 |
| Capture time (s) | 58 | 86 | 22 |
| Capture memory (mb) | 502 | 378 | 343 |
| Replay power (J) | 744 | 1285 | 1141 |
| Replay time (s) | 10 | 18 | 22 |
| Replay memory (mb) | 636 | 339 | 343 |
Total autogenerated commands: 1292 out of 1349 (95.77%) Total autogenerated structs: 2565 out of 2579 (99.46%)
We are over 2000 tests that are being run with make test (or ctest) now. While the sheer number
of tests is not a reliable indicator of the quality or coverage of testing, it is still a useful
indicator.