feat: pop bench overhead benchmarking

[chungquantin]

This PR is a part of a large PR #424. It implements the pop bench overhead command.

Deliverables

• Prompt to select runtime
• Prompt to select genesis builder
• Prompt to input the directory path of weight files
• Skip interactive UI feature

How to test the PR?

Run the below command to test the feature quickly

pop bench overhead --warmup=1 --repeat=1

There is a new command flag provided to skip the interactive UI

pop bench overhead --warmup=1 --repeat=1 -i

[Daanvdplas]

Some comments and open questions:

• The final command would be great to output here as well.
• I see a lot of flags that can be used but none of them appear in the interactive UI?
• What is your view on tests regarding all those flags, the tests right now are very minimal which leaves a lot of room for bugs.
• I tried pop bench overhead --warmup=1 --repeat=1 -i --runtime runtime/testnet/:

I have no idea what I'm doing wrong.

In general I think the tests need some attention, most probably for the entire benchmark feature. See for example the tests for building a spec:

pop-cli/crates/pop-cli/src/commands/build/spec.rs

Line 588 in 16693b4

mod tests {

[chungquantin]

@Daanvdplas

The final command would be great to output here as well.

Resolved in: #3b0e48f

I see a lot of flags that can be used but none of them appear in the interactive UI?

As defined in the spec, we only have interactive UI for pallet benchmarking because there are a lot of works needed to cover all params of other commands like storage, overhead and machine. The second point would be these commands are not used frequently compared to the main command pop bench pallet. The goal is to keep non-frequent commands' implementation minimal as possible.

What is your view on tests regarding all those flags, the tests right now are very minimal which leaves a lot of room for bugs.

I don't cover tests for those flags because the logic of the implementation is simply passing arguments to the method cmd.run_with_default_builder_and_spec provided by OverheadCmd in frame-benchmarking-cli. They are supposed to be tested by the frame-benchmarking-cli not in pop-cli.

I tried pop bench overhead --warmup=1 --repeat=1 -i --runtime runtime/testnet/:

--runtime only accepts path to the runtime WASM binary, not directory. It is checked inside the OverheadCmd from frame-benchmarking-cli. I am not sure if this is something we need to handle in our code because that behaviour is expected. User is expected to provide the direct path to the WASM binary (same applied for frame-omni-bencher) or keep it empty so we find it for them.

[Daanvdplas]

I left a reaction and I am also curious why we are not outputting the command by default. Other than that good to go!