fix: JVM heap headroom, in-container HTTP port conflicts, and ssh probe tunneling - #218
Merged
Merged
Conversation
The tiered heap table returned -Xmx2g for any resources.memory < 8GB, which equals the container limit verbatim (compose.go) — with the image's forced ZGC (Xms=Xmx) there is zero headroom and the JVM crash-loops with 'Failed to commit memory'. Below 8GB now cap -Xmx at floor(total/2) in MB (2GB->1024m, 3GB->1536m, 4GB->2g). jvm.heap_max still bypasses verbatim. Also fix the 1GB edge where -Xmn equaled -Xmx leaving zero old gen (512m heap now gets 128m new). Found in TX-167 functional testing.
…BFT) ports.http: 8091 rendered fullNodePort = solidityPort = 8091 because solidityPort defaulted to 8091 independently of ports.http — the shipped examples/private-network.yaml crash-looped its fullnode on bind conflict (HttpApiOnSolidityService, then HttpApiOnPBFTService once that was moved). RenderHOCON now bumps solidityPort/PBFTPort by +2 when they equal fullNodePort, with a >65535 overflow guard. Explicitness is judged by config_overrides only: ApplyDefaults pre-fills ports.solidity_http before render, so checking the intent field made the avoidance dead code in the real pipeline (caught in review, with an intent.Parse end-to-end regression test to pin it). Bumps only de-conflict the three node.http keys; cross-family clashes with grpc/p2p/jsonrpc/metrics are out of scope (documented in the function comment). The example also sets ports.solidity_http: 8093 explicitly. Found in TX-167 functional testing.
…endpoints
verify/health/diagnose/wait and MCP health probed ssh nodes either via
t.Exec('curl') — rejected by the ssh command allowlist — or by dialing
127.0.0.1 from the trond host, bypassing ssh entirely. On a firewalled
target (22 only) every liveness/sync probe failed while the node was
healthy; status/apply/inspect also reported endpoints as 127.0.0.1.
internal/target now offers a target-aware transport: an optional Dialer
interface (SSHTarget.DialContext = direct-tcpip over the existing ssh
client, restricted to loopback addrs; LocalTarget = net.Dialer), plus
HTTPClient/Get/Post/DialContext helpers with an enforceable timeout
(preemptable even when the underlying dial ignores ctx). Probes migrated:
verify, health, diagnose sync/peers/version/ports, apply LiveStatus jar
branch, wait --port/--http, MCP health. Endpoint reporting (status, apply
created/updated/no_change, inspect, network add/create, MCP status/
endpoints/monitoring, healthTool display) now uses EndpointHost so ssh
rigs report the target host. preflight checkPorts now dials through the
target too — it previously checked the local machine for ssh targets.
Found in TX-167 functional testing.
Sunny6889
approved these changes
Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Found during end-to-end functional testing of trond (TX-167). Three high-severity fixes plus one CI chore:
JVM heap headroom below 8GB —
resources.memory < 8GBproduced-Xmxequal to the container limit (the tier table floors at 2g), and the image's ZGC withXms=Xmxthen crash-loops withFailed to commit memory. Heap is now capped at 50% of container memory below 8GB (2GB→1024m, 3GB→1536m; ≥8GB tiers unchanged),jvm.heap_maxoverride still wins, andpreflightwarns on <8GB nodes referencing java-tron'sALLOW_MIN_MEMORY=8192guidance.In-container HTTP port conflicts rejected early — a fullnode with
ports.http: 8091(as shipped inexamples/private-network.yaml) collided in-container with the template's fixedsolidityPort: 8091(andPBFTPort), crash-looping the node. Render andconfig validatenow refuse such intents withVALIDATION_ERROR/RENDER_ERRORand suggestions; the example moves the fullnode to host port 8095 with an explanatory comment.Probe ssh nodes through the ssh tunnel — HTTP/TCP probes for ssh targets either exec'd remote
curl(rejected by the ssh allowlist) or dialed127.0.0.1from the operator machine, soverify/health/diagnose port_listening/MCPhealthfailed against healthy remote nodes. Adds aDialercapability to targets (ssh → direct-tcpip over the existing connection, local →net.Dialer) and routes every probe through a target-aware client;api_endpointsnow report the real target host instead of hard-coded loopback.CI chore — pause tag-triggered releases (manual
workflow_dispatchonly) until the release-CI work tracked in TX-167 lands; PR snapshot workflow and.goreleaser.yamluntouched.Test plan
go build ./... && go test ./...— all packages green (new table-driven tests: heap tiers, port-conflict validation, ssh/local dialers, HTTP client fallback)