Parallel tank RPC command execution

[pinheadmz]

Currently, scenarios execute RPC commands in order, blocking, from a central pod. Ideally, some if not all tanks can act somewhat autonomously, in addition to the coordinated central rpc-0 scenarios. Double-ideally, a central scenario can "launch" a series of behaviors in tanks.

Here's some possible strategies:

1. Write a bash script with bitcoin-cli commands, exec into the tank, write the file, and execute it

• pros: requires no new images or cluster configuration
• cons: bash is great for very simple things but awful when you want loops, if/else logic etc

2. Rebuild the tank image WITH python and the bitcoind/warnet test_framework package, send warnet scenarios written in python to tanks

• pros: no new language involved, can program articulate behaviors and loops (like "when mempool is half full, send 1000 txs", etc)
• cons: requires larger tank images and rebuilding all tank images (are we still maintaining all these old versions of bitcoind?)

3. Add a sidecar container to every tank pod with python, and the test_framework package

• pros: still get the flexiblity of writing scenarios in python, will work with any existing tank image, only need to maintain one sidecar image
• cons: doubles the number of running containers on the cluster.

(2) and (3) will also require some fiddling with the warnet test framework / test_framework_bridge.py or perhaps just another subclass like our current bridge but specifically only interacts with one single node. In fact, we might be able to ditch the framework altogether and just use TestNode. Nice thing about any of these strategies is we can use hard-coded rpcuser, rpcpassword and rpcport values because all tanks have the same settings.

Related:
#272
#390

[m3dwards]

When I wrote a scenario that required multiple things to happen at the same time I just span up a bunch of threads in the scenario. Seemed to work. Distributed running of tasks does sound great but it would add some complexity such as monitoring and communication back to a controller (are tasks truly going to be fire and forget?).

Does threading in the RPC pod's scenario get us there?

[pinheadmz]

Hm good point Ill expiriment with threads. Maybe if need be we can specify extra resource allocation to the RPC pod, Im imagining 250 nodes all creating random transactions

[mplsgrant]

As Max mentioned, using threads from rpc-0 seems like it would work pretty well for this. If we end up with many threads, I like the idea of experimenting with asyncio.

To push some functionality out into the nodes, my gut tells me that allowing some kind of simple bash script functionality like Will's #272 combined with optional sidecars for more complex functionality.

[m3dwards]

If threads are not sufficient, of the two options presented, I like the idea of a warnet container in remote / distributed mode with the full python stack and testing helpers.

[pinheadmz]

Threads seems to be working well: #395

[willcl-ark]

If you find time, would like to see comparison with #396 on the big network...

May write some more "persona" scripts (tx generatooors) and push them for easy use in testing.