CI flake fixes #24 #8699
Merged
CI flake fixes #24 #8699
+55
−38
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
If we see a close tx at the same time we see the channel reach announce depth (here, 104x1x0 confirms
and we process blocks 109 and 110), we see it go from:
CGOSSIP_WAITING_FOR_ANNOUNCE_DEPTH->CGOSSIP_CHANNEL_UNANNOUNCED_DYING (Channel closing before 6 confirms)
Then:
CGOSSIP_CHANNEL_UNANNOUNCED_DYING->CGOSSIP_CHANNEL_ANNOUNCED_DEAD
```
INFO 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#2: State changed from CHANNELD_NORMAL to CHANNELD_SHUTTING_DOWN
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#2: gossip state: CGOSSIP_WAITING_FOR_ANNOUNCE_DEPTH->CGOSSIP_CHANNEL_UNANNOUNCED_DYING (Channel closing before 6 confirms)
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#2: billboard: Channel ready for use. They've sent shutdown, waiting for ours
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#2: Trying commit
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#2: Can't send commit: nothing to send, feechange not wanted ({ RCVD_ADD_ACK_REVOCATION:3755 }) blockheight not wanted ({ RCVD_ADD_ACK_REVOCATION:109 })
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#2: peer_out WIRE_SHUTDOWN
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#2: billboard: Channel ready for use. Shutdown messages exchanged.
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: pid 35692, msgfd 86
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#2: Status closed, but not exited. Killing
INFO 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#2: State changed from CHANNELD_SHUTTING_DOWN to CLOSINGD_SIGEXCHANGE
DEBUG hsmd: new_client: 2
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: Expected closing weight = 772, fee 2895sat (min 1447sat, max 1006268sat)
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: out = 506268sat/500000sat
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: dustlimit = 546sat
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: fee = 2895sat
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: fee negotiation step = 50%
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: billboard perm: Negotiating closing fee between 1447sat and 1006268sat satoshi (ideal 2895sat) using step 50%
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: billboard: Waiting for their initial closing fee offer
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: peer_in WIRE_CLOSING_SIGNED
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: Making close tx at = 506268sat/500000sat fee 2895sat
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: Received fee offer 2895sat
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: ...offer is reasonable
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#2: Their actual closing tx fee is 2895sat vs previous 4220sat: weight is 772
INFO 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: performing quickclose in range 1447sat-8492sat
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: Making close tx at = 506268sat/500000sat fee 2895sat
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-hsmd: Got WIRE_HSMD_SIGN_MUTUAL_CLOSE_TX
DEBUG hsmd: Client: Received message 21 from client
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: sending fee offer 2895sat
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: peer_out WIRE_CLOSING_SIGNED
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-closingd-chan#2: billboard perm: We agreed on a closing fee of 2895 satoshi for tx:7685256e6e4632bd601f42bb85b6f22b7a0db8d55c468dfbb1703c3ab2e4d9ce
INFO 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#2: State changed from CLOSINGD_SIGEXCHANGE to CLOSINGD_COMPLETE
DEBUG wallet: Owning output 1 506268sat (p2tr) txid 7685256e6e4632bd601f42bb85b6f22b7a0db8d55c468dfbb1703c3ab2e4d9ce
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#2: We have 1 anchor points to use
DEBUG lightningd: Broadcasting txid 7685256e6e4632bd601f42bb85b6f22b7a0db8d55c468dfbb1703c3ab2e4d9ce
DEBUG lightningd: sendrawtransaction: 020000000001016f50e5481abf2482ed719dde6014cf9da9b3076fe12af7c8bbccfb705150ffbe0000000000ffffffff02d195070000000000225120eed745804da9784cc203f563efa99ffa54fdf01b137bc964e63c3124070ffbe69cb90700000000002251208a73bb281433f2b5db5461c6778aa2afd28e011de6bd04799a5991662c61d7a80400473044022015af4ffc277e44ab701507e017194b1e3ab8598397efab02c59dbe72340790b50220762065e91ce0a2c73f36fdf3a4d54732ef91d075c0887cfb28580220e081e81a01473044022050730b9d051a3fdecb368750ecbd558d625a1acaa42e59586c7187407319dda402202f2f6e099b2e092698fe528260e2b56be8de325bc985a17945c724514fb65b9c014752210259b3cb48220dc2016f4d320bb8105cc2c92bd8c79fc25c2ad96f37b496e491c62103bbee60c395056b8a1201e06ed79e2914c11a61d7b1aa781846468d02489dba6952ae00000000
DEBUG lightningd: Adding block 108: 4e6ba3059c6890797a29acd306e82d6f4e85ad62a6aa6d0ee5bc6ead3e6329cd
DEBUG hsmd: Client: Received message 5 from client
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: peer_downgrade
DEBUG plugin-bcli: sendrawtx exit 0 (bitcoin-cli -regtest -datadir=/tmp/ltests-laf51vlg/test_buy_liquidity_ad_check_bookkeeping_1/lightning-2/ -rpcclienttimeout=60 -rpcport=41847 -rpcuser=... -stdinrpcpass sendrawtransaction 020000000001016f50e5481abf2482ed719dde6014cf9da9b3076fe12af7c8bbccfb705150ffbe0000000000ffffffff02d195070000000000225120eed745804da9784cc203f563efa99ffa54fdf01b137bc964e63c3124070ffbe69cb90700000000002251208a73bb281433f2b5db5461c6778aa2afd28e011de6bd04799a5991662c61d7a80400473044022015af4ffc277e44ab701507e017194b1e3ab8598397efab02c59dbe72340790b50220762065e91ce0a2c73f36fdf3a4d54732ef91d075c0887cfb28580220e081e81a01473044022050730b9d051a3fdecb368750ecbd558d625a1acaa42e59586c7187407319dda402202f2f6e099b2e092698fe528260e2b56be8de325bc985a17945c724514fb65b9c014752210259b3cb48220dc2016f4d320bb8105cc2c92bd8c79fc25c2ad96f37b496e491c62103bbee60c395056b8a1201e06ed79e2914c11a61d7b1aa781846468d02489dba6952ae00000000)
DEBUG lightningd: Adding block 109: 540a47d6b316aec8d1b56efca22ffb384403659e3b29cddd3d68f87df72065a6
DEBUG lightningd: Adding block 110: 1931e9ea62324d0fb50188df525e84ee7021e1910cf3534b343db05cc4639851
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#2: Got UTXO spend for beff505170fbccbbc8f72ae16f07b3a99dcf1460de9d71ed8224bf1a48e5506f:0: 7685256e6e4632bd601f42bb85b6f22b7a0db8d55c468dfbb1703c3ab2e4d9ce
UNUSUAL 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#2: Peer permanent failure in CLOSINGD_COMPLETE: Funding transaction spent: onchain txid 7685256e6e4632bd601f42bb85b6f22b7a0db8d55c468dfbb1703c3ab2e4d9ce (reason=unknown)
UNUSUAL 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#2: Not dropping our unilateral close onchain since we already saw 7685256e6e4632bd601f42bb85b6f22b7a0db8d55c468dfbb1703c3ab2e4d9ce confirm.
INFO 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#2: State changed from CLOSINGD_COMPLETE to FUNDING_SPEND_SEEN
**BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#2: Illegal gossip state transition: CGOSSIP_CHANNEL_UNANNOUNCED_DYING->CGOSSIP_CHANNEL_ANNOUNCED_DEAD
DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-onchaind-chan#2: pid 35721, msgfd 86
```
Signed-off-by: Rusty Russell <[email protected]>
```
check_channel_moves(l1, expected_channel1)
> check_chain_moves(l1, expected_chain1)
tests/test_coinmoves.py:942:
...
'account_id': 'wallet',
'blockheight': 115,
'created_index': 15,
- 'credit_msat': Decimal('6358000.00000000'),
- 'debit_msat': 0,
? --
+ 'credit_msat': 6366000,
? +++ ++++++
+ 'debit_msat': 0,
'extra_tags': [],
- 'output_msat': Decimal('6358000.00000000'),
+ 'output_msat': 6366000,
'primary_tag': 'deposit',
'utxo': '4a16b7aceac233ed68bed0f18adfc1ab338b5fc55b696d1376eea173adeac8e3:1',
},
```
Signed-off-by: Rusty Russell <[email protected]>
Delay can cause bogus complaints: ``` 2025-11-13T23:50:03.6643632Z lightningd-3 2025-11-13T23:37:29.947Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: wake delay for WIRE_CHANNEL_REESTABLISH: 5708msec ``` Signed-off-by: Rusty Russell <[email protected]>
…v-memleak. We had a flake of form: ``` 2025-11-18T04:42:23.489Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-connectd: wake delay for WIRE_CHANNEL_REESTABLISH: 6789msec ``` Which happened as we're shutting down. Some investigation revealed the cause: `dev-memleak` can be extremely slow. Fair enough. So we change `dev-memleak` to call connectd first, and connectd uses that as a trigger to stop complaining about delays. Signed-off-by: Rusty Russell <[email protected]>
rustyrussell
force-pushed
the
guilt/flakes24
branch
from
15342a3 to
2626906
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changelog-None: all test flake fixes.