Skip to content

Commit 58b8841

Browse files
skylar-simoncelliskylar-simoncelli
committed
feat: Add dynamic local environment
1 parent 04256ab commit 58b8841

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

57 files changed

+4899
-0
lines changed

‎dev/local-environment-dynamic/README.md

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
# Dynamic Local Test Environment
2+
3+
This document outlines the setup and operation of the dynamic local test environment for Partner Chains. This environment is designed for flexibility, allowing developers to easily configure and launch a large-scale network of Partner Chains nodes with automated setup and registration.
4+
5+
Unlike the previous static `local-environment`, which used a fixed number of validators with pre-generated keys, this dynamic environment automates the entire lifecycle, including key generation, funding, and on-chain registration.
6+
7+
## Key Features
8+
9+
- **Dynamic Configuration**: Easily set the number of "permissioned" and "registered" validator nodes to launch.
10+
- **Automated Lifecycle Management**: Handles the entire process of key generation, Cardano address funding, and SPO registration for all validators.
11+
- **Dynamic Node Discovery**: Nodes use dynamic node-keys, PeerIDs, and public addresses for automatic peer discovery within the Substrate network.
12+
- **Comprehensive Stack**: Includes all necessary components for a fully functional test environment: Cardano node, DB-Sync, Ogmios, and the Partner Chains nodes.
13+
14+
## System Requirements
15+
16+
Running the local environment requires a machine with adequate resources.
17+
18+
## Configuration
19+
20+
### Node Count
21+
22+
At the top of the `setup.sh` script, you can configure the size of the network:
23+
24+
```sh
25+
NUM_PERMISSIONED_NODES_TO_PROCESS=10
26+
NUM_REGISTERED_NODES_TO_PROCESS=10
27+
```
28+
29+
- `NUM_PERMISSIONED_NODES_TO_PROCESS`: Defines the number of initial, permissioned nodes.
30+
- `NUM_REGISTERED_NODES_TO_PROCESS`: Defines the number of additional "registered" nodes that will be spun up and registered as SPOs.
31+
32+
### Custom Node Image
33+
34+
To use a custom Partner Chains node image, simply update the `PARTNER_CHAINS_NODE_IMAGE` variable at the top of the `setup.sh` script.
35+
36+
## Automated Setup Process Explained
37+
38+
The automation is handled by a series of scripts that execute in a specific order.
39+
40+
### 1. `setup.sh`
41+
42+
This is the main orchestration script. When you run `bash setup.sh`, it performs the following steps:
43+
44+
1. **System Checks**: Detects the OS and prompts for configuration choices (ports, resource limits, etc.) in interactive mode.
45+
2. **Configuration Generation**:
46+
- Creates a `.env` file with all the environment variables for Docker Compose.
47+
- Generates individual entrypoint scripts for each Partner Chains node inside `configurations/partner-chains-nodes/`. These entrypoints include a staggered start delay (`sleep`) to prevent all nodes from starting at once.
48+
3. **Docker Compose Manifest Generation**:
49+
- Constructs the main `docker-compose.yml` file by adding service definitions for each Partner Chains node and appending the core Cardano stack (`cardano-node-1`, `postgres`, etc.) from the `modules/` directory.
50+
51+
A key feature of the generated configurations is the use of dynamic discovery flags for the Substrate node:
52+
53+
```bash
54+
/usr/local/bin/partner-chains-node \\
55+
# ... other flags
56+
--node-key="$(openssl rand -hex 32)" \\
57+
--listen-addr=/ip4/0.0.0.0/tcp/30333 \\
58+
--public-addr="/dns4/partner-chains-node-[...]/tcp/30333/p2p/\$PEER_ID" &
59+
```
60+
61+
- A unique `--node-key` is generated for each node on startup, from which a `PEER_ID` is derived. The node then advertises its public, DNS-based address for discovery.
62+
63+
### 2. `cardano/entrypoint.sh` (inside `cardano-node-1`)
64+
65+
This script bootstraps the Cardano side of the environment.
66+
67+
1. **Starts Cardano Node**: It waits for configuration files from the setup container, then starts the `cardano-node`.
68+
2. **SPO Registration**: It runs a loop that watches the `/shared/spo-certs/` directory. When a new validator's `.cert` file appears, it submits it to the Cardano network, officially registering the validator as an SPO.
69+
70+
### 3. `partner-chains-setup/entrypoint.sh` (inside `partner-chains-setup`)
71+
72+
This is the core script for automating the validator lifecycle.
73+
74+
1. **Setup**: It waits for the Cardano node and Ogmios, then generates the dynamic Cardano genesis files.
75+
2. **Key Generation**: It programmatically generates all necessary keys (payment, stake, cold, VRF, keystore) for every configured validator.
76+
3. **Funding**: It builds and submits Cardano transactions to fund the payment addresses of all validators from a genesis UTXO.
77+
4. **Certificate Generation**: For each validator, it generates the necessary stake address and stake pool registration certificates, placing the final certificate in `/shared/spo-certs/` to be picked up by the Cardano node.
78+
5. **Ready Signal**: Once complete, it creates a `partner-chains-setup.ready` file, signaling the Partner Chains nodes that they can start.
79+
80+
## Usage
81+
82+
### 1. Initialise the Environment
83+
84+
Run `setup.sh` to enter the setup wizard for initialising the environment (`.env` file) and `docker-compose.yml`.
85+
86+
```bash
87+
chmod +x setup.sh
88+
bash setup.sh
89+
```
90+
91+
Alternatively, run in non-interactive mode to accept default settings:
92+
```bash
93+
bash setup.sh --non-interactive
94+
```
95+
96+
### 2. Start the Environment
97+
98+
Once initialized, deploy the local environment with:
99+
100+
```bash
101+
docker compose up -d
102+
```
103+
104+
### 3. Monitor the Environment
105+
106+
We recommend using a visual Docker UI tool such as [lazydocker](https://github.com/jesseduffield/lazydocker) or Docker Desktop for following the live logs and container performance.
107+
108+
You can also monitor logs directly from the command line:
109+
- **Cardano SPO registrations**: `docker logs cardano-node-1 -f | grep -E "DEBUG|LOG|WARN"`
110+
- **Full stack setup progress**: `docker logs partner-chains-setup -f`
111+
- **Partner Chains node startup**: `docker logs partner-chains-node-permissioned-1 -f` (or any other node).
112+
113+
### 4. Stop the Environment
114+
115+
To tear down the environment and remove all data, it is mandatory to also wipe all volumes.
116+
117+
```bash
118+
docker compose down --volumes
119+
```
120+
121+
## Advanced Usage & Customization
122+
123+
The `setup.sh` script provides several flags for more advanced configurations. The help dialogue will always show the latest available features.
124+
125+
```bash
126+
$ bash setup.sh --help
127+
Usage: ./setup.sh [OPTION]...
128+
Initialize and configure the Docker environment.
129+
-n, --non-interactive Run with no interactive prompts and accept sensible default configuration settings.
130+
-d, --deployment-option Specify one of the custom deployment options (1, 2, 3, or 4).
131+
-p, --postgres-password Set a specific password for PostgreSQL (overrides automatic generation).
132+
-i, --node-image Specify a custom Partner Chains Node image.
133+
-t, --tests Include tests container.
134+
-h, --help Display this help dialogue and exit.
135+
```

‎dev/local-environment-dynamic/configurations/busybox

1.08 MB
Binary file not shown.

‎dev/local-environment-dynamic/configurations/cardano/config-pool1.json

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
{
2+
"AlonzoGenesisFile": "/shared/shelley/genesis.alonzo.json",
3+
"AlonzoGenesisHash": "7e94a15f55d1e82d10f09203fa1d40f8eede58fd8066542cf6566008068ed874",
4+
"ByronGenesisFile": "/shared/byron/genesis.json",
5+
"ByronGenesisHash": "83de1d7302569ad56cf9139a41e2e11346d4cb4a31c00142557b6ab3fa550761",
6+
"ConwayGenesisFile": "/shared/conway/genesis.conway.json",
7+
"ConwayGenesisHash": "9cc5084f02e27210eacba47af0872e3dba8946ad9460b6072d793e1d2f3987ef",
8+
"EnableP2P": true,
9+
"ExperimentalHardForksEnabled": true,
10+
"ExperimentalProtocolsEnabled": true,
11+
"LastKnownBlockVersion-Alt": 0,
12+
"LastKnownBlockVersion-Major": 3,
13+
"LastKnownBlockVersion-Minor": 1,
14+
"MinNodeVersion": "8.12.0",
15+
"PeerSharing": true,
16+
"Protocol": "Cardano",
17+
"RequiresNetworkMagic": "RequiresMagic",
18+
"NetworkMagic": 42,
19+
"ShelleyGenesisFile": "/shared/shelley/genesis.json",
20+
"ShelleyGenesisHash": "363498d1024f84bb39d3fa9593ce391483cb40d479b87233f868d6e57c3a400d",
21+
"TargetNumberOfActivePeers": 20,
22+
"TargetNumberOfEstablishedPeers": 50,
23+
"TargetNumberOfKnownPeers": 150,
24+
"TargetNumberOfRootPeers": 60,
25+
"TestAllegraHardForkAtEpoch": 0,
26+
"TestAlonzoHardForkAtEpoch": 0,
27+
"TestMaryHardForkAtEpoch": 0,
28+
"TestShelleyHardForkAtEpoch": 0,
29+
"TestBabbageHardForkAtEpoch": 0,
30+
"TestConwayHardForkAtEpoch": 0,
31+
"TraceAcceptPolicy": true,
32+
"TraceBlockFetchClient": false,
33+
"TraceBlockFetchDecisions": false,
34+
"TraceBlockFetchProtocol": false,
35+
"TraceBlockFetchProtocolSerialised": false,
36+
"TraceBlockFetchServer": false,
37+
"TraceChainDb": true,
38+
"TraceChainSyncBlockServer": false,
39+
"TraceChainSyncClient": false,
40+
"TraceChainSyncHeaderServer": false,
41+
"TraceChainSyncProtocol": false,
42+
"TraceConnectionManager": true,
43+
"TraceDNSResolver": true,
44+
"TraceDNSSubscription": true,
45+
"TraceDiffusionInitialization": true,
46+
"TraceErrorPolicy": true,
47+
"TraceForge": true,
48+
"TraceHandshake": true,
49+
"TraceInboundGovernor": true,
50+
"TraceIpSubscription": true,
51+
"TraceLedgerPeers": true,
52+
"TraceLocalChainSyncProtocol": false,
53+
"TraceLocalConnectionManager": true,
54+
"TraceLocalErrorPolicy": true,
55+
"TraceLocalHandshake": true,
56+
"TraceLocalRootPeers": true,
57+
"TraceLocalTxSubmissionProtocol": false,
58+
"TraceLocalTxSubmissionServer": false,
59+
"TraceMempool": true,
60+
"TraceMux": false,
61+
"TracePeerSelection": true,
62+
"TracePeerSelectionActions": true,
63+
"TracePublicRootPeers": true,
64+
"TraceServer": true,
65+
"TraceTxInbound": false,
66+
"TraceTxOutbound": false,
67+
"TraceTxSubmissionProtocol": false,
68+
"TracingVerbosity": "NormalVerbosity",
69+
"TurnOnLogMetrics": true,
70+
"TurnOnLogging": true,
71+
"defaultBackends": [
72+
"KatipBK"
73+
],
74+
"defaultScribes": [
75+
[
76+
"StdoutSK",
77+
"stdout"
78+
]
79+
],
80+
"hasEKG": 12788,
81+
"hasPrometheus": [
82+
"127.0.0.1",
83+
12798
84+
],
85+
"minSeverity": "Info",
86+
"options": {
87+
"mapBackends": {
88+
"cardano.node.metrics": [
89+
"EKGViewBK"
90+
],
91+
"cardano.node.resources": [
92+
"EKGViewBK"
93+
]
94+
},
95+
"mapSubtrace": {
96+
"cardano.node.metrics": {
97+
"subtrace": "Neutral"
98+
}
99+
}
100+
},
101+
"rotation": {
102+
"rpKeepFilesNum": 10,
103+
"rpLogLimitBytes": 5000000,
104+
"rpMaxAgeHours": 24
105+
},
106+
"setupBackends": [
107+
"KatipBK"
108+
],
109+
"setupScribes": [
110+
{
111+
"scFormat": "ScText",
112+
"scKind": "StdoutSK",
113+
"scName": "stdout",
114+
"scRotation": null
115+
}
116+
]
117+
}

0 commit comments

Comments
 (0)