"Unable to blind the transaction properly." RPC Error sending a transaction consuming many Unpsent Items

[lolhill]

A "JSON RPC Exception occured: -6: Unable to blind the transaction properly. This should not happen." error is hit where the actions described below are taken (RPC via a Python Script - I'll attach/provide that plus the script I'm using to start the setup).

The error hits where a "large" send is performed with multiple small unsent items available. See also comments further below

Expected behavior
"Large" sendtoaddress RPC Call Succeeds.

Actual behavior
"Large" sendtoaddress RPC Call Fails with "A JSON RPC Exception occured: -6: Unable to blind the transaction properly. This should not happen."

To reproduce

• Spin up a single Bitcoin (22.0.0) and Elements (0.21.0.1) setup (As per https://elementsproject.org/elements-code-tutorial/overview without employing the second node and with only 610 initial free coins configured).
• Create default elements wallet, rescan blockchain to claim the 610 free coins.
• Using simple RPC Commands in a python script, for the single instance:
  : Loop around transferring a reducing balance of coin, starting at 600 (reduction factor per iteration = 0.999), new address on each transfer.
  : Every 20 iterations generate blocks (rpcuser.generatetoaddress(101,rpcuser.getnewaddress()))
  : With the command line I've run with, 660 iterations execute successfully and the script exits.
  : Mempool is clear
  : State is: 696 Unspent Items (max: 310.00791458, min 0.0004928), Wallet Bitcoin Balance showing as 610.
• Subsequently attempting to transfer 600 bitcoin again at this point (simply by re-running the script) results in the "A JSON RPC Exception occurred: -6: Unable to blind the transaction properly. This should not happen." error.
• Smaller value transfers are successful, so it looks like the issue may relate to the range of unspent items and the transfer value (See below).

Script used to start the environment (basically ripped from the Blockstream tutorial) and hacky RPC python below.

reset_start_610Free_OneElementsInstance.txt

SplitBitcoinSingleUser.py.txt

Python Execution Command Lines:
Execute the loop:
python SplitBitcoinSingleUser.py 1000 100 600 0.999 310
Re-execute the same command line on completion - The first iteration should trigger the error
python SplitBitcoinSingleUser.py 1000 100 600 0.999 310
Error is observed on the first send.

See also "Any extra information" below

System information
Bitcoin (22.0.0) - From bitcoin Core
Elements (0.21.0.1)
Single Node regtest Setup (as per tutorial mentioned above
Ubuntu 20.04
16 logical CPUs (11th Gen Intel(R) Core(TM) i7-11800H - Single socket, 8 Cores, hyper-threading). Tiger Lake
1 TB SSD
32GB RAM

Extra information
After experimenting with a few Test setups, it seems possible that a significant factor here could be the amount of unspent items built up in the test. After looking at a few configurations and test executions, it seems that the problem hits if the attempted error triggering sendtoaddress is for a value that exceeds the cumulative value of ~240 - 250 of the highest value unspent items, summed.

This might be incorrect - will leave it to the Devs to investigate further/determine, but looks like it to me.
Elements Debug Log Output across a triggering transaction.

2022-01-27T12:22:52Z Received a POST request for / from 127.0.0.1:52490
2022-01-27T12:22:52Z ThreadRPCServer method=getrawmempool user=liquidrpc
2022-01-27T12:22:52Z Received a POST request for / from 127.0.0.1:52490
2022-01-27T12:22:52Z ThreadRPCServer method=getblockcount user=liquidrpc
2022-01-27T12:22:52Z Received a POST request for / from 127.0.0.1:52490
2022-01-27T12:22:52Z ThreadRPCServer method=listunspent user=liquidrpc
2022-01-27T12:22:52Z Received a POST request for / from 127.0.0.1:52490
2022-01-27T12:22:52Z ThreadRPCServer method=getwalletinfo user=liquidrpc
2022-01-27T12:22:56Z Received a POST request for / from 127.0.0.1:52490
2022-01-27T12:22:56Z ThreadRPCServer method=getnewaddress user=liquidrpc
2022-01-27T12:22:56Z [default wallet] keypool reserve 853
2022-01-27T12:22:56Z [default wallet] keypool keep 853
2022-01-27T12:22:56Z Received a POST request for / from 127.0.0.1:52490
2022-01-27T12:22:56Z ThreadRPCServer method=sendtoaddress user=liquidrpc
2022-01-27T12:22:56Z [default wallet] keypool added 1 keys (0 internal), size=2000 (1000 internal)
2022-01-27T12:22:56Z [default wallet] keypool reserve 2984
2022-01-27T12:22:56Z FeeEst: 89 > 60% decay 0.99931: feerate: -1 from (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (0 - 1e+99) 100.00% 56.2/(56.2 0 mem 0.0 out)
2022-01-27T12:22:56Z FeeEst: 48 > 60% decay 0.99520: feerate: 99.0375 from (0 - 1e+99) 100.00% 25.3/(25.3 0 mem 0.0 out) Fail: (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out)
2022-01-27T12:22:56Z FeeEst: 12 > 60% decay 0.96200: feerate: -1 from (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (0 - 1e+99) 25.13% 0.3/(0.3 0 mem 1.0 out)
2022-01-27T12:22:56Z FeeEst: 178 > 85% decay 0.99931: feerate: -1 from (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (0 - 1e+99) 100.00% 56.2/(56.2 0 mem 0.0 out)
2022-01-27T12:22:56Z FeeEst: 48 > 85% decay 0.99520: feerate: 99.0375 from (0 - 1e+99) 100.00% 25.3/(25.3 0 mem 0.0 out) Fail: (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out)
2022-01-27T12:22:56Z FeeEst: 12 > 85% decay 0.96200: feerate: -1 from (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (0 - 1e+99) 25.13% 0.3/(0.3 0 mem 1.0 out)
2022-01-27T12:22:56Z FeeEst: 356 > 95% decay 0.99931: feerate: -1 from (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (0 - 1e+99) 100.00% 56.2/(56.2 0 mem 0.0 out)
2022-01-27T12:22:56Z FeeEst: 48 > 95% decay 0.99520: feerate: 99.0375 from (0 - 1e+99) 100.00% 25.3/(25.3 0 mem 0.0 out) Fail: (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out)
2022-01-27T12:22:56Z FeeEst: 12 > 95% decay 0.96200: feerate: -1 from (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (0 - 1e+99) 25.13% 0.3/(0.3 0 mem 1.0 out)
2022-01-27T12:22:56Z FeeEst: 3 > 60% decay 0.96200: feerate: -1 from (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (0 - 1e+99) 25.13% 0.3/(0.3 0 mem 1.0 out)
2022-01-27T12:22:56Z FeeEst: 6 > 85% decay 0.96200: feerate: -1 from (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (0 - 1e+99) 25.13% 0.3/(0.3 0 mem 1.0 out)
2022-01-27T12:22:56Z FeeEst: 12 > 95% decay 0.96200: feerate: -1 from (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (0 - 1e+99) 25.13% 0.3/(0.3 0 mem 1.0 out)
2022-01-27T12:22:56Z FeeEst: 12 > 95% decay 0.99520: feerate: 99.0375 from (0 - 1e+99) 96.22% 25.3/(25.3 0 mem 1.0 out) Fail: (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out)
2022-01-27T12:22:56Z FeeEst: 12 > 95% decay 0.99931: feerate: -1 from (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (0 - 1e+99) 98.25% 56.2/(56.2 0 mem 1.0 out)
2022-01-27T12:22:56Z FeeEst: 89 > 60% decay 0.99931: feerate: -1 from (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (0 - 1e+99) 100.00% 56.2/(56.2 0 mem 0.0 out)
2022-01-27T12:22:56Z FeeEst: 48 > 60% decay 0.99520: feerate: 99.0375 from (0 - 1e+99) 100.00% 25.3/(25.3 0 mem 0.0 out) Fail: (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out)
2022-01-27T12:22:56Z FeeEst: 12 > 60% decay 0.96200: feerate: -1 from (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (0 - 1e+99) 25.13% 0.3/(0.3 0 mem 1.0 out)
2022-01-27T12:22:56Z FeeEst: 178 > 85% decay 0.99931: feerate: -1 from (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (0 - 1e+99) 100.00% 56.2/(56.2 0 mem 0.0 out)
2022-01-27T12:22:56Z FeeEst: 48 > 85% decay 0.99520: feerate: 99.0375 from (0 - 1e+99) 100.00% 25.3/(25.3 0 mem 0.0 out) Fail: (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out)
2022-01-27T12:22:56Z FeeEst: 12 > 85% decay 0.96200: feerate: -1 from (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (0 - 1e+99) 25.13% 0.3/(0.3 0 mem 1.0 out)
2022-01-27T12:22:56Z FeeEst: 356 > 95% decay 0.99931: feerate: -1 from (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (0 - 1e+99) 100.00% 56.2/(56.2 0 mem 0.0 out)
2022-01-27T12:22:56Z FeeEst: 48 > 95% decay 0.99520: feerate: 99.0375 from (0 - 1e+99) 100.00% 25.3/(25.3 0 mem 0.0 out) Fail: (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out)
2022-01-27T12:22:56Z FeeEst: 12 > 95% decay 0.96200: feerate: -1 from (-1 - -1) 0.00% 0.0/(0.0 0 mem 0.0 out) Fail: (0 - 1e+99) 25.13% 0.3/(0.3 0 mem 1.0 out)
2022-01-27T12:22:56Z SelectCoins() best subset: 194.99999852 6.88657489 3.64882701 1.45557485 0.99999938 0.99899495 0.99799851 0.99700051 0.99600351 0.9950075 0.99401249 0.99301848 0.99202545 0.99103343 0.99004239 0.98905235 0.98806329 0.98707522 0.98608815 0.98510206 0.98411695 0.98313283 0.9821497 0.98116754 0.98018637 0.97920619 0.97822698 0.97724875 0.9762715 0.97529522 0.97431992 0.9733456 0.97237225 0.97139988 0.97042848 0.96945805 0.96848859 0.96752009 0.96655257 0.96558602 0.96462043 0.96365581 0.96269215 0.96172945 0.96076772 0.95980695 0.95884714 0.95788829 0.9569304 0.95597347 0.95501749 0.95406247 0.95310841 0.9521553 0.95120314 0.95025193 0.94930168 0.94835237 0.94740402 0.94645661 0.94551015 0.94456464 0.94362007 0.94267645 0.94173377 0.94079204 0.93985124 0.93891139 0.93797247 0.9370345 0.93609746 0.93516136 0.9342262 0.93329197 0.93235867 0.93142631 0.93049488 0.92956439 0.92863482 0.92770618 0.92677847 0.92585169 0.92492584 0.92400091 0.92307691 0.92215383 0.92123167 0.92031044 0.91939013 0.91847073 0.91755226 0.9166347 0.91571807 0.91480235 0.91388754 0.91297365 0.91206068 0.91114861 0.91023746 0.90932722 0.90841789 0.90750947 0.90660196 0.90569535 0.90478966 0.90388486 0.90298098 0.90207799 0.90117591 0.90027474 0.89937446 0.89847508 0.8975766 0.89667902 0.89578234 0.89488656 0.89399167 0.89309767 0.89220457 0.89131237 0.89042105 0.88953063 0.8886411 0.88775245 0.8868647 0.88597783 0.88509185 0.88420676 0.88332255 0.88243922 0.88155678 0.88067522 0.87979454 0.87891475 0.87803583 0.87715779 0.87628063 0.87540435 0.87452894 0.87365441 0.87278075 0.87190797 0.87103606 0.87016502 0.86929485 0.86842555 0.86755713 0.86668957 0.86582287 0.86495705 0.86409209 0.863228 0.86236476 0.8615024 0.86064089 0.85978025 0.85892047 0.85806154 0.85720348 0.85634627 0.85548992 0.85463443 0.8537798 0.85292601 0.85207308 0.85122101 0.85036979 0.84951941 0.84866989 0.84782122 0.8469734 0.84612642 0.84528029 0.84443501 0.84359057 0.84274698 0.84190423 0.84106232 0.84022126 0.83938103 0.83854165 0.8377031 0.8368654 0.83602853 0.8351925 0.83435731 0.83352295 0.83268942 0.83185673 0.83102487 0.83019384 0.82936365 0.82853428 0.82770574 0.82687803 0.82605115 0.8252251 0.82439987 0.82357547 0.82275189 0.82192914 0.82110721 0.8202861 0.81946581 0.81499751 0.81418251 0.81336832 0.81255495 0.8117424 0.81093065 0.8093096 0.80850028 0.80769178 0.79999751 0.79999751 0.79999743 0.79919751 0.79919751 0.79839831 0.79839831 0.79759991 0.79759991 0.79680231 0.79680231 0.7960055 0.7960055 0.79520949 0.79520949 0.79441428 0.79441428 0.79361987 0.79361987 0.79282624 0.79282624 0.79203341 0.79203341 0.79124138 0.79124138 0.79045013 0.79045013 0.78965968 0.78965968 0.78887002 0.78887002 0.78808115 0.78808115 0.78729306 0.78729306 0.78650577 0.78650577 0.78571926 0.78571926 0.78493354 0.78493354 0.7841486 0.7841486 0.78336445 0.78336445 0.78258108 0.78258108 0.7817985 0.7817985 0.7810167 0.78023568 0.77945544 0.77867598 0.77789731 0.77711941 0.77634228 0.77556594 0.77479037 0.77401558 0.77324156 0.77246832 0.77169584 0.77092415 0.77015322 0.76938306 0.76861368 0.76784506 0.76707721 0.76631014 0.76554382 0.76477828 0.7640135 0.76324948 0.76248623 0.76172374 0.76096201 0.76020105 0.75944084 0.7586814 0.75792272 0.75716479 0.75640762 0.75565121 0.75489556 0.75414066 0.75338652 0.75263313 0.7518805 0.75112861 0.75037748 0.7496271 0.74887747 0.74812859 0.74738046 0.74663308 0.74588644 0.74514055 0.74439541 0.74365101 0.74290736 0.74216445 0.74142228 0.74068086 0.73994017 0.73920023 0.73846103 0.73772256 0.73698484 0.73624785 0.7355116 0.73477609 0.73404131 0.73330727 0.73257396 0.73184138 0.73110954 0.73037842 0.72964804 0.72891839 0.72818947 0.72746128 0.72673382 0.72600708 0.72528107 0.72455579 0.72383123 0.72310739 0.72238428 0.7216619 0.72094023 0.72021929 0.71949907 0.71877957 0.71806078 0.71734272 0.71662538 0.71590875 0.71519284 0.71447764 0.71376316 0.7130494 0.71233634 0.71162401 0.71091238 0.71020146 0.70949126 0.70878177 0.70807298 0.70736491 0.70665754 0.70595088 0.70524493 0.70453968 0.70383514 0.7031313 0.70242816 0.70172573 0.70102401 0.70032298 0.69962265 0.69892303 0.6982241 0.69752588 0.69682835 0.69613152 0.69543538 0.69473995 0.6940452 0.69335116 0.6926578 0.69196514 0.69127317 0.6905819 0.68989131 0.68920142 0.68851222 0.6878237 0.68713588 0.68644874 0.68576229 0.68507652 0.68439144 0.68370705 0.68302334 0.68234031 0.68165797 0.68097631 0.68029533 0.67961503 0.67893542 0.67825648 0.67757822 0.67690064 0.67622373 0.67554751 0.67487196 0.67419708 0.67352288 0.67284936 0.67217651 0.67150433 0.67083282 0.67016199 0.66949182 0.66882233 0.6681535 0.66748535 0.66681786 0.66615104 0.66548488 0.6648194 0.66415458 0.66349042 0.66282693 0.6621641 0.66150193 0.66084042 0.66017958 0.6595194 0.65885988 0.65820102 0.65754281 0.65688527 0.65622838 0.65557215 0.65491657 0.65426165 0.65360739 0.65295378 0.65230082 0.65164852 0.65099687 0.65034587 0.64969552 0.64904582 0.64839678 0.64774838 0.64710063 0.64645352 0.64580707 0.64516126 0.64451609 0.64387157 0.6432277 0.64258447 0.64194188 0.64129994 0.64065864 0.64001798 0.63937796 0.63873857 0.63809983 0.63746173 0.63682427 0.63618744 0.63555125 0.6349157 0.63428078 0.6336465 0.63301285 0.63237983 0.63174745 0.6311157 0.63048458 0.62985409 0.62922424 0.62859501 0.62796641 0.62733844 0.6267111 0.62608439 0.62483284 0.62420801 0.6235838 0.62296021 0.62171491 0.62109319 0.62047209 0.61923177 0.61861253 0.61614178 0.61552564 0.61306721 0.61245414 0.61184169 0.61061861 0.61000799 0.14929082 0.01011723 total 600.01004322
2022-01-27T12:22:56Z [default wallet] keypool return 2984
2022-01-27T12:22:58Z Flushing wallet.dat
2022-01-27T12:22:58Z Flushed wallet.dat 46ms

[pavel-kokolemin]

I also have the same problem in the latest version (23.2.1).
I tried to debug it and it fails for me here:

// FIXME [hardfork] Elements currently cannot handle surjection proofs on transactions
//  with more than 256 inputs. The Elements verification code will always try to give
//  secp-zkp the complete list of inputs, and if this exceeds 256 then surjectionproof_verify
//  will always return false, so there is no way to work around this situation at signing time
if (surjection_targets.size() > SECP256K1_SURJECTIONPROOF_MAX_N_INPUTS) {
    // We must return false here to avoid triggering an assertion within
    // secp256k1_surjectionproof_initialize on the next line.
    return false;
}

SECP256K1_SURJECTIONPROOF_MAX_N_INPUTS is 256.
So it might fail if you need to transfer a large amount and there are many inputs in the wallet. The selection of UTXOs is random, so sometimes it works even if the wallet has many UTXOs.
I guess a workaround would be to limit the number of UTXOs in the wallet.

[apoelstra]

I'm sorry to hear that we've finally hit this in production and we still haven't done the Bulletproofs hardfork to fix it.

This RPC error is also really unhelpful -- though it's better than the old "assertion failure, node shuts down" behavior that we had when we discovered this.

Basically yeah, for now the wallet needs to work around this by first consolidating UTXOs, in batches of < 256, until coin selection works.

[opacey]

I hit this issue last week in production on a custom Elements chain and was pleased to find this thread explaining the problem. Indeed the the error message isn't explicit so it took quite a bit of searching to find this thread.

How are plans for the bulletproof work currently?

[opacey]

Hi @apoelstra, would you elaborate on what you meant by "until coin selection works."? I haven't yet studied why my wallet ended up with >256 utxos, I assumed it was a quirk of the way I was issuing roc commands to it, but is there something innate to the wallet that would trend to large utxo sets anyway?

[apoelstra]

It's pretty common and sensible to have large UTXO sets. This isn't the wallet's fault. This would happen no matter what the wallet did if, for example, hundreds of people sent you coins separately and before your wallet had a chance to consolidate. It's also not always sensible to consolidate even if your wallet is online. It's bad for privacy and may be expensive depending on fee market dynamics.

What I meant by "until coin selection works" is "until we update to BPs and the 256 limit doesn't apply, or until somebody writes wallet logic in Elements that understands the 256 limit and uses multiple transactions to effect one spend".

The timeline on the former is "after Simplicity and once we have pressure to upgrade" -- we expected pressure to come from feerates on Liquid but actually this bug might be a more urgent reason -- and the timeline on the latter (rearchitecting the wallet to do some multi-tx thing) is "likely never". I'm not on the Liquid team and can't really say anything more specific.

[delta1]

Regarding improved error messaging for this specific case, I need to finish up #1288