1cb416397b psbt: avoid duplicate taproot leaf script keys when merging (Shuvam Pandey)
Pull request description:
Follow-up to #35665, which fixed the same combiner defect for `PSBT_GLOBAL_XPUB`. thomasbuilds
and winterrdog asked for this one as its own PR when I reported it there.
`m_tap_scripts` maps a leaf script to a set of control blocks, but is serialized as one record
per control block, keyed by the control block (`SerializeToVector(s, PSBT_IN_TAP_LEAF_SCRIPT,
std::span{control_block})`). `PSBTInput::Merge` unions it by the map key, so two PSBTs that map
the same control block to different leaf scripts merge into an input that serializes the `0x15`
key twice. Duplicate keys make a PSBT invalid, so it is the same `combinepsbt` then
`decodepsbt` failure as the xpub case, at the input level. Present since #22558 (v24.0).
Both decode on their own, and differ only in the leaf script the control block maps to, `OP_1`
against `OP_1 OP_1`:
```
$ A=cHNidP8BADwCAAAAAaqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAIhXAUJKbdMGgSVS3i0tgNel6XgeKWg8o7JbVR7/ums6AOsACUcAAAA==
$ B=cHNidP8BADwCAAAAAaqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAIhXAUJKbdMGgSVS3i0tgNel6XgeKWg8o7JbVR7/ums6AOsADUVHAAAA=
$ bitcoin-cli -regtest decodepsbt "$(bitcoin-cli -regtest combinepsbt "[\"$A\",\"$B\"]")"
error code: -22
error message:
TX decode failed Duplicate Key, input key "15c050929b74c1a04954b78b4b6035e97a5e078a5a0f28ec96d547bfee9ace803ac0" already provided: unspecified iostream_category error
```
winterrdog reproduced it on the #35665 thread with another pair.
Merge the records rather than the map entries, keeping the leaf script already there. BIP 174
lets the combiner "pick arbitrarily when conflicts occur", and unknown and proprietary records
already resolve that way. Refusing to combine is the BIP's other option, but that would fail
`combinepsbt` on input it accepts today.
Merging by map key drops records as well. `std::map::insert` leaves existing keys alone, so
when both PSBTs carry the same leaf script with different control blocks, the incoming set was
dropped. Those keys do not conflict, so merging per record keeps them.
The control blocks already present are collected once per merge rather than searched for per
incoming record, which would be quadratic in the size of the two PSBTs `combinepsbt` takes from
the caller.
Since this is the second field with this shape I checked the rest. `m_xpubs` (#35665) and
`m_tap_scripts` are the only two whose record key comes from the value, so two map entries can
serialize the same key. `partial_sigs` is keyed by `CKeyID` and serialized under the pubkey,
but the pubkey determines the `CKeyID`, so those records stay distinct. The others key the
record by the map key, `m_proprietary` included, and `PSBTOutput` has no such field.
The test fails on master on both counts, and covers the merges that do not conflict as well.
I found this with a local assertion in the psbt fuzz target that a combined PSBT must
roundtrip. That assertion can go in a follow-up.
Tested:
```
./build/bin/test_bitcoin --run_test=psbt_tests
./build/bin/test_bitcoin --run_test=psbt_wallet_tests
./build/test/functional/test_runner.py rpc_psbt.py rpc_rawtransaction.py wallet_taproot.py wallet_signer.py feature_taproot.py wallet_basic.py
```
ACKs for top commit:
achow101:
ACK 1cb416397b
winterrdog:
re-ACK 1cb416397b
Tree-SHA512: 2599beffe701ba9b672e8dcc3d43844f3853ceeb3d86fc53798a428d3288aa8edd2e42f338b32a1046fa558d6cc5cfa5d55b3a0aaf960278b0f3002158381623
436921eb46 test: check joinpsbts preserves global xpubs and proprietary fields (Thomas)
011094b282 rpc: preserve global xpubs and proprietary fields in joinpsbts (Thomas)
Pull request description:
`joinpsbts` collects the global xpubs of all the joined PSBTs into `merged_psbt`, but returns a separately constructed `shuffled_psbt` into which only the inputs, outputs, and unknown fields are copied. The collected `PSBT_GLOBAL_XPUB` records are silently dropped, and `PSBT_GLOBAL_PROPRIETARY` records are not collected at all.
The xpub collection was added in #17034, which was written against a `joinpsbts` that still returned `merged_psbt`, but was merged after #16512 had introduced the `shuffled_psbt` rebuild, so the collected xpubs have never reached the result.
Shuffle the inputs and outputs of `merged_psbt` in place instead of rebuilding a new PSBT, so that all global data is preserved, and union the global proprietary records in the merge loop, matching the `combinepsbt` behavior from #34893.
ACKs for top commit:
jpk68:
ACK 436921eb46
achow101:
ACK 436921eb46
winterrdog:
tACK 436921eb46
Tree-SHA512: d9de34c25aecc29b6b4fb80d6584fa919cc5ff9b7ef2f4d8ce35c4043fe7638fefb8af10448f2cd14021f5d25e149f0efc8798c5b8c9bc8b5582c6152010e891
b42f7fade0 descriptor: don't prepend key origins twice (Shuvam Pandey)
7b15e2cb44 descriptor: fix duplicate check for hardened keys (Shuvam Pandey)
Pull request description:
Fixes#34273.
Importing a descriptor that uses the same `musig()` participants twice in one
tapleaf, with different musig subderivations, fails with
`is not sane: contains duplicate public keys`. It only fails when one of the
participants is a private key on a hardened path. The all-xpub version of the
same descriptor imports fine. That's what gave it away.
The duplicate check (`KeyCompare`) resolves each key expression to a pubkey and
compares the results. It does this at index 0, and the old code used an empty
signing provider. With that empty provider, a `musig()` expression can't resolve
when one of its participants is on a hardened path, because deriving that
participant needs its private key, so the whole aggregate key comes back empty.
Two different musig expressions both came back empty, so the check treated them
as duplicates. The fix derives against the signing provider populated during
parsing, which holds the private keys, and only compares the expression strings
when neither side resolves. 151henry151 had suggested looking at the empty
signing provider on the issue.
scgbckbone found a second, separate bug in the same descriptors. When another
expression that reuses those participants is handled in the same expansion, its
participant origin in the PSBT is added twice, so `m/86h/1h/0h` becomes
`m/86h/1h/0h/86h/1h/0h` in both the input and output Taproot BIP32 derivation
maps. `OriginPubkeyProvider::GetPubKey()` now derives into a temporary provider,
merges it, and writes the corrected origin once, so a later expression can't
prepend the same origin again.
Tested:
```
./build/bin/test_bitcoin --run_test=descriptor_tests
./build/bin/test_bitcoin --run_test=miniscript_tests
./build/bin/test_bitcoin --run_test=bip328_tests
./build/bin/test_bitcoin --run_test=psbt_wallet_tests
./build/test/functional/test_runner.py wallet_musig.py --jobs=1
```
ACKs for top commit:
achow101:
ACK b42f7fade0
scgbckbone:
ACK b42f7fade0
Tree-SHA512: ab36caa6bc484fa1fc3289c79e9a3d713278d82f80de478e53e1bdbe645037e07776ac798eba085733abc139c11a9dbf0d3f49d3c0eee9632e4ddf33d2242f92
m_tap_scripts maps a leaf script to a set of control blocks, but is serialized
one record per control block, keyed by the control block. PSBTInput::Merge
unions it by the map key, so two PSBTs that map the same control block to
different leaf scripts merge into an input serializing the 0x15 key twice.
Duplicate keys are invalid, so combinepsbt hands back a PSBT that can no longer
be decoded. Present since #22558 (v24.0).
Merge the records instead of the map entries, keeping the leaf script already
there, as BIP 174 lets the Combiner pick arbitrarily when conflicts occur. The
control blocks already present are collected once rather than searched for per
incoming record, which would be quadratic in the size of the PSBTs.
Control blocks under a leaf script that both PSBTs carry are now kept as well,
where the map level union dropped them.
The test covers conflicting and non-conflicting merges, including an incoming
leaf script whose control blocks only partly conflict, so records that do not
conflict are not dropped alongside those that do.
bd4b1524ea init: do not count file descriptors for HTTPServer if -server=0 (Matthew Zipkin)
b08662060d init: account for maximum file descriptors needed by HTTP (Matthew Zipkin)
cc2acebefb http: configure simultaneous connection limit with -rpcmaxconnections (Matthew Zipkin)
b3d6d2d1a7 http: limit connected clients to 16 (Matthew Zipkin)
86651d8197 scripted-diff: Rename nUserBind, nBind, nMaxConnections to snake_case (Matthew Zipkin)
Pull request description:
Introduces a new configuration option `-rpcmaxconnections` with default value `16`. This is used to limit the number of simultaneous `HTTPClient` connected to the `HTTPServer`. When the limit is reached, new pending connections remain queued in the kernel's socket buffer. Those connections have complete TCP handshakes with the kernel but do not occupy any application memory.
The previous libevent-based HTTP server had no limit on connections but it did have a limit on the kernel socket queue:
e7ff4ef2b4/http.c (L3510)
```c
if (listen(fd, 128) == -1) {
```
The current HTTP server, like the p2p server, uses a platform constant here:
b6becf3534/src/httpserver.cpp (L743)
(on my macOS `SOMAXCONN` is `128` but on my Debian machine it's `4096`)
The default of 16 was chosen as a reasonable upper bound for single-user RPC use cases. Systems designed to handle more simultaneous HTTP connections than this (previously relying on the absence of a limit) can adjust the setting.
## File descriptors
Because of the connection limit, we can now account for the maximum number of file descriptors needed by the HTTP server. This addresses several issues (#11368#11322 maybe #27732) that could have been fixed by a PR waiting in vain for a libevent release (#27731).
## Bonus performance improvement
The new limit is managed in a loop that drains the kernel's socket queue with `accept()`. All pending connections from the queue (up to the limit) are processed in one single call to `SocketHandlerListening()`. The previous code would only accept one connection from the queue on each I/O loop tick, with a `SELECT_TIMEOUT` (50ms) sleep between each.
ACKs for top commit:
fjahr:
tACK bd4b1524ea
janb84:
ACK bd4b1524ea
winterrdog:
tested ACK bd4b1524ea
hodlinator:
Concept ACK bd4b1524ea
willcl-ark:
ACK bd4b1524ea
Tree-SHA512: 2ef7a96da4d7037c7343ec0ea03fda5bb55d10c2a071fce4929141297515923b203d3d338dbcb6599849768f52aa3c9da509fb5d1d6f7c574a1d2034ea2a9e74
73a94b4545 psbt: avoid aborting on invalid MuSig2 derivations (Lőrinc)
e3d1e75a51 test: characterize MuSig2 derivation aborts (Lőrinc)
Pull request description:
**Problem:** A PSBT may contain MuSig2 derivation metadata with a hardened child index or a path that derives to a different key.
The hardened index aborts during public derivation, while the mismatched key aborts at the result assertion.
`analyzepsbt`, `finalizepsbt`, and `descriptorprocesspsbt` all reach this code without a wallet.
Even the read-only `analyzepsbt` can force a co-signer service to restart its node after unexpected input.
**Fix:** Return failure when a MuSig2 derivation path contains a hardened child index, and skip only the current aggregate when the path derives to a different key so another matching aggregate can still be tried.
This follows [#35154](https://github.com/bitcoin/bitcoin/pull/35154), with the related contributions credited in the commits.
ACKs for top commit:
jeanpablojp:
ACK 73a94b4545
achow101:
ACK 73a94b4545
andrewtoth:
ACK 73a94b4545
Tree-SHA512: d8e28c5a4184154a4427c644ce62423cbcccdc3d82a6293f36fe99055fa04714598bc92c43b526fbcc7c99b231140669c2d0f1b853999d7dc33f949564c90504
7f9c4e2928 doc: add release notes (ismaelsadeeq)
e18d392689 test: add mempool estimator i/o fuzz test (ismaelsadeeq)
970f02096d fees: persist mempool policy estimator data (ismaelsadeeq)
7dcb37989d fees: move fee_estimates.dat into fees directory (ismaelsadeeq)
0db2b69e6d rpc: add verbosity option to estimatesmartfee options (ismaelsadeeq)
06bb65730e fees: gate mempool estimates on recent block coverage (ismaelsadeeq)
cfe585df25 validation: emit block mempool removal signal from ConnectTip (ismaelsadeeq)
0d88558f95 fees: return mempool estimates when it's lower than block policy (ismaelsadeeq)
693b1351af fees: add caching to MemPoolFeeRateEstimator (ismaelsadeeq)
c9bb3df29f fees: add MemPoolFeeRateEstimator class (ismaelsadeeq)
9cacf677a9 rpc: add fee_rate_estimator option to estimatesmartfee (ismaelsadeeq)
ba6c61bbdd fees: add FeeRateEstimatorManager class (ismaelsadeeq)
2cb6b831e0 fees: add EstimateFeeRate and MaximumTarget to CBlockPolicyEstimator (ismaelsadeeq)
5adb2ab084 refactor: test block policy estimator directly (ismaelsadeeq)
9c8309a890 test: rename policy estimator tests to block policy estimator tests (ismaelsadeeq)
e3d5ef1b5f fees: move StringForBlockPolicyEstimateReason to block policy estimator (ismaelsadeeq)
74245c20e0 fees: split wallet and estimator fee reasons (ismaelsadeeq)
Pull request description:
This PR is another attempt to fix#27995 using a better approach.
For background and motivation, see #27995 and the discussion in the Delving Bitcoin post [Mempool Based Fee Estimation on Bitcoin Core](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703).
This PR is currently limited to using the mempool only to lower what is recommended by the Block Policy Estimator.
Accurate and safe fee estimation using the mempool is challenging. There are open questions about how to prevent mempool games that are theoretically possible for miners [(a variant of the Finney attack)](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/6).
This is one reason this PR uses the mempool only to lower the Block Policy Estimator result. The Block Policy Estimator itself is not gameable in this way, so the combined estimate is not susceptible to this attack increasing the returned feerate.
The underlying assumption is that, with the current tools and work done to make RBF and CPFP feasible and reliable (TRUC transaction relay, ephemeral anchors, cluster size 2 package RBF), underestimation is safer than overestimation. We now assume it is relatively easy to fee-bump later if a transaction does not confirm, whereas once a fee is overestimated there is no way to recover from that.
Another open question when using the mempool for fee estimation is how to account for incoming transaction inflow.
[Bitcoin Augur](https://github.com/block/bitcoin-augur) does this by using past inflow plus a constant expected inflow to predict future inflow. I find this unconvincing for fee estimation and potentially prone to more overestimation, as past conditions are not always representative of the future. See my [review of the Augur fee rate estimator and open questions](https://github.com/block/bitcoin-augur/issues/3).
This PR uses a much simpler approach based on current user behavior, similar to the widely used method employed by mempool.space: looking at the top block of the mempool and selecting a percentile feerate depending on whether the user is economical or conservative.
Empirical data from both myself and Clara Shikhelman shows that the 75th percentile feerate for economical users and the 50th percentile feerate for conservative users provide positive confirmation guarantees, hence this is what is used in this PR.
Parallel research by Rene Pickhardt and his student suggests that using the [average fee per byte of the block template performs well](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/12).
All of these are constants that can be adjusted. There is parallel work exploring these constants and running benchmarks across fee estimators to find a sweet spot.
See also work in LND, the [LND Budget Sweeper](https://delvingbitcoin.org/t/lnds-deadline-aware-budget-sweeper/1512), which applies this idea successfully. Their approach is to estimate fees initially with bitcoind, then increment gradually as the confirmation deadline approaches, using a fixed fee budget.
Historical data indicates that this PR's approach can [reduce overestimation quite significantly (~29%)](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/8).
This is particularly useful in scenarios where the Block Policy Estimator recommends a high feerate while the mempool is empty.
<img width="1800" height="1090" alt="56f3ba26c0184521c42bb82ec9d8c9f2224d4f8e" src="https://github.com/user-attachments/assets/c035c40c-8ece-42a7-b290-d29f1ac9bf4d" />
As seen in the image above, there is only one remaining unfixed case: when there is a sudden inflow of transactions and the feerate rises, the Block Policy Estimator takes time to reflect this. In that case, users will continue to see a low feerate estimate until it slowly updates. From the historical data linked above, [this occurs about ~26% of the time](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/8).
Overall, we observe a **73% success rate with 0% overestimation, and 26% underestimation** with this approach.
See https://bitcoincorefeerate.com/stats for recent running stats that have almost identical data.
This PR also includes refactors that enable this work. Rather than splitting the PR and implementing changes incrementally, I opted for an end-to-end implementation:
### 1. Refactors
* Split the mixed fee reason enum into separate wallet and block policy concepts. The wallet now has a `FeeReason` enum for why the wallet selected a fee rate (`FEE_RATE_ESTIMATOR`, `MEMPOOL_MIN`, `USER_SPECIFIED`, `FALLBACK`, `REQUIRED`), while the Block Policy Estimator uses `BlockPolicyEstimateReason` for its internal threshold details.
* Move `StringForBlockPolicyEstimateReason` to the Block Policy Estimator code, keeping the estimator-specific strings with the estimator.
* Move detailed Block Policy Estimator logging out of wallet transaction creation and into the estimator path. Wallet transaction creation now logs the selected fee and wallet fee reason instead of leaking estimator internals.
* Keep the wallet RPC `fee_reason` field name for compatibility, but update its meaning to report the wallet fee reason instead of the Block Policy Estimator's internal threshold reason.
* Rename policy estimator tests and files to block-policy-specific names where appropriate.
* Update Block Policy Estimator unit tests to be independent of the mempool and validation interface.
### 2. Introduce Mempool-Based Fee Estimator and Fee Estimator Manager
* Introduce `FeeRateEstimation` and `FeeRateEstimationError` as common estimator result types, avoiding new out-parameters for fee estimation results.
* Add `FeeRateEstimatorType` to identify the estimator that produced a result.
* Add `FeeRateEstimatorManager`, responsible for owning the Block Policy Estimator and Mempool Fee Rate Estimator.
* Update the node context to store a `std::unique_ptr` to `FeeRateEstimatorManager` instead of `CBlockPolicyEstimator`.
* Update `CBlockPolicyEstimator` to no longer subscribe directly to the validation interface; instead, `FeeRateEstimatorManager` subscribes and forwards relevant notifications.
* Add a mempool fee estimator that generates a block template when called, calculates a percentile feerate, and returns the 75th percentile for economical mode or the 50th percentile for conservative mode.
* When the selected estimate is below the node's fee floor, `estimatesmartfee` still returns at least the max of `mempoolminfee` and `minrelaytxfee`.
* Add caching to the mempool estimator so new estimates are generated at most every 7 seconds while the chain tip is unchanged, assuming enough [transactions have propagated](https://bitcoin.stackexchange.com/questions/125776/how-long-does-it-take-for-a-transaction-to-propagate-through-the-network/125777#125777) to make a meaningful difference.
This heuristic will likely be replaced by requesting block templates via the general-purpose block template cache proposed here: https://github.com/bitcoin/bitcoin/issues/33389
* Update `MempoolTransactionsRemovedForBlock` to receive the connected block as well as the transactions removed from the mempool.
* Track the weight of block transactions and mempool transactions removed due to block connection after each block connection.
This data is tracked for the last 6 mined blocks. A mempool feerate estimate is returned only when the ratio of mempool transaction weight removed due to block connection to block transaction weight is greater than 75% across the tracked window. This heuristic provides rough confidence that the node's mempool matches that of the majority of the hashrate. The 75% threshold is arbitrary and can be adjusted.
There is a caveat when transactions in the local mempool are consistently not mined by the network, as described in #27995 (e.g. due to filtering).
Accounting for these transactions during fee estimation is not necessary, as they should be evicted from the mempool itself (see #33510). Handling this again within fee estimation would be redundant.
* Persist statistics for the 6 most recent mined blocks to `fees/mempool_policy_estimator.dat` during periodic flushes and shutdown, so this data is available after restarts.
* Move Block Policy Estimator data from `fee_estimates.dat` to `fees/block_policy_estimates.dat`, migrating the legacy file during startup when needed.
* Add `fee_rate_estimator` to the `estimatesmartfee` options object. Supported values are `"none"` (default combined behavior), `"block_policy"` (use only the Block Policy Estimator), and `"mempool_policy"` (use only the Mempool Fee Rate Estimator). Unknown values are treated as `"none"`.
* Add `verbosity` to the `estimatesmartfee` options object. With `verbosity >= 2`, the RPC returns recent mempool health statistics.
* Expose the selected fee rate estimator in `estimatesmartfee` results when `fee_rate_estimator` is `"none"` and the estimate succeeds.
* Add unit, functional, and fuzz test coverage for the new estimator behavior, persistence, RPC options, and estimator I/O.
<details>
<summary>see example output</summary>
```bash
bitcoin-cli estimatesmartfee 1 economical '{"verbosity": 2, "fee_rate_estimator": "none"}'
```
```json
{
"feerate": 0.00002133,
"estimator": "mempool_policy",
"blocks": 2,
"mempool_health_statistics": [
{
"block_height": 927953,
"block_weight": 3991729,
"mempool_txs_weight": 3942409
}
]
}
```
</details>
ACKs for top commit:
willcl-ark:
reACK 7f9c4e2928
jsarenik:
Approach ACK 7f9c4e2
Tree-SHA512: c35b423eea0eb34524cf5ad07822c0ab8d53e2ab78965b58c8738044c61c77352184822360ed077a51bfbf83d0226d221e988f7156b1948023707c7e1fb31495
Persist MemPoolFeeRateEstimator's recent mined-block statistics
to fees/mempool_policy_estimator.dat and reload them at startup.
Without this, the mempool estimator starts cold after each restart
and treats the mempool as unhealthy until MEMPOOL_HEALTH_WINDOW_BLOCKS
blocks have been observed, causing the default combined estimatesmartfee
request to return a mempool fee rate estimator error.
Files with more stats than MEMPOOL_HEALTH_WINDOW_BLOCKS,
non-consecutive block heights, or a final block that does not match the
active chain tip are rejected on read, preserving the invariant that
loaded stats describe the current chain.
Add MempoolPolicyEstimatorPath(), pass the path through
FeeRateEstimatorManager, and flush both block-policy
and mempool-policy estimator files on interval and shutdown.
Move block policy fee estimates from fee_estimates.dat to
fees/block_policy_estimates.dat.
On startup, migrate the legacy file to the new path when only the legacy
file exists. If both files exist, keep the new file and remove the
legacy file.
Rename the block policy estimator args source files to the generic
estimator_args.{cpp,h} names and rename FeeestPath to
BlockPolicyFeeEstPath while the path helper is moved into the shared fee
estimator argument code.
Add a verbosity option to the existing estimatesmartfee options object.
The default verbosity remains 1.
When verbosity is at least 2 include mempool_health_statistics in the response.
The array reports the mined blocks tracked by the mempool fee rate estimator in
most-recent-first order, with each entry containing:
- block_height
- block_weight: total non-coinbase transaction weight in the block
- mempool_txs_weight: weight of transactions removed from our mempool
for that block
Expose these stats through the fee rate estimator manager so RPC users
can inspect the block coverage data used by the mempool health check.
Integrate MemPoolFeeRateEstimator into FeeRateEstimatorManager.
When both estimators succeed, select the lower of the block policy
and mempool estimates.
When either estimator fails, return its error instead of falling back
to the block policy estimate: if the mempool estimator cannot produce
an estimate, the combined estimate fails.
Callers that want a block-policy-only estimate can request it explicitly
via fee_rate_estimator option.
estimatesmartfee now emits the estimator field only for successful
manager-selected estimates.
Add a test that ensures estimatesmartfee returns the mempool fee rate
estimate when it is lower than the block policy estimate, and can request
the mempool policy estimator explicitly
Two wallet functional tests also need adjusting. When the mempool is
too sparse to fill its percentile buckets, MemPoolFeeRateEstimator
returns a relayable floor of max(min relay fee, mempool min fee), so in
regtest getFeeRateEstimate now returns the min relay fee where the
wallet previously had no estimate and fell back to a higher rate:
- wallet_taproot.py: the cleanup sendall used automatic fee estimation.
GetMinimumFeeRate previously fell back to the wallet fallback fee
(fallbackfee, 20 sat/vB in the test framework); it now uses the min
relay fee floor. At that lower feerate the wallet's underestimate of
the taproot script-path witness size drops the effective feerate
below min relay, so the transaction is rejected. Pin fee_rate=20 to
match the framework fallbackfee.
- wallet_bumpfee.py: GetDiscardRate() previously fell back to the
wallet discard rate (-discardfee); it now takes the minimum of that
and the estimate, so the min relay fee floor collapses the discard
rate down to the dust relay feerate. The lower discard rate reduces
the cost of change, so the ~614 sat leftover change in
test_dust_to_fee is now retained instead of being dropped to fee.
Rework the test to leave a sub-dust (20/270 sat) change that is
dropped regardless of the discard rate.
Co-authored-by: willcl-ark <will@256k1.dev>
Add MemPoolFeeRateEstimator, which calls Bitcoin Core's block
assembler with the mempool and chainstate to build a block template and
use its chunk fee rates for fee rate estimation.
Add CalculateMaxWeightPercentiles to return the 50th and 75th
percentile chunk feerates by cumulative block weight. If sparse,
EstimateFeeRate uses the higher of the minimum relay fee rate and the
current mempool minimum fee rate.
The 50th percentile is returned as the conservative estimate, and the
75th percentile as the economical estimate.
Wire MemPoolFeeRateEstimator into FeeRateEstimatorManager and add
FeeRateEstimatorType::MEMPOOL_POLICY for result attribution.
Add unit tests for the mempool fee rate estimator and fee estimator
string conversions, plus fuzz coverage for the string conversions.
Co-authored-by: willcl-ark <will@256k1.dev>
6d387af562 psbt: remove write-only global xpub tracking set (Thomas)
3b7051c7e3 test: check combinepsbt with conflicting global xpub origins (Thomas)
7c632c0e2a psbt: avoid duplicate global xpub keys when merging (Thomas)
Pull request description:
Global xpubs are stored in a map of key origin to set of xpubs, while the serialization writes one record per xpub, keyed by the xpub. `Merge` unions the map origin-by-origin, so when the combined PSBTs provide different key origins for the same xpub, the result serializes the same `PSBT_GLOBAL_XPUB` key twice. BIP 174 declares PSBTs with duplicate keys invalid and the deserializer rejects them, so `combinepsbt` returns a PSBT that no RPC can parse again. This affects all releases since the merge loop was added in #17034 (v23.0).
<details><summary>Reproduction on master</summary>
The PSBTs share the unsigned transaction and xpub, and differ only in the master fingerprint of the global xpub record (`00000000` vs `11111111`):
```
$ A=cHNidP8BADwCAAAAAaqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqAAAAAAD/////AQAAAAAAAAAAAAAAAABPAQQ1h88AAAAAAAAAAACHPf+BwC9SViP9H+UWfqw6VaBJ3j0xS7Qu4if/7TfVCAM5o2ATMBWX2u9B++WToCzFE9C1VSfsLfEFDi6P9JyFwgQAAAAAAAAA
$ B=cHNidP8BADwCAAAAAaqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqAAAAAAD/////AQAAAAAAAAAAAAAAAABPAQQ1h88AAAAAAAAAAACHPf+BwC9SViP9H+UWfqw6VaBJ3j0xS7Qu4if/7TfVCAM5o2ATMBWX2u9B++WToCzFE9C1VSfsLfEFDi6P9JyFwgQRERERAAAA
$ bitcoin-cli -regtest decodepsbt "$(bitcoin-cli -regtest combinepsbt "[\"$A\",\"$B\"]")"
error code: -22
error message:
TX decode failed Duplicate Key, global key "01043587cf00...9c85c2" already provided: iostream error
```
</details>
Deduplicate by xpub when merging, keeping the origin that is already present: BIP 174 lets the Combiner "pick arbitrarily when conflicts occur", and conflicting unknown and proprietary records are already resolved the same way. The logic is shared between `combinepsbt` and `joinpsbts` through a new `MergeGlobalXPubs` helper. The second commit adds a test that fails on master with the error above, and the last commit removes the `global_xpubs` tracking set in `Unserialize`, write-only since the generic duplicate key check introduced in #21283 (1e2d146b47) replaced the explicit one.
Note: the xpub loop in `joinpsbts` currently has no observable effect, since the collected xpubs never reach the returned PSBT. My #35516 fixes that, so this PR should land first: on its own, #35516 would make the same duplicate key issue reachable through `joinpsbts`, while with the shared helper in place it never becomes reachable. I will rebase #35516 on top afterwards.
ACKs for top commit:
Bicaru20:
tACK 6d387af562.
achow101:
ACK 6d387af562
winterrdog:
tACK 6d387af562
Tree-SHA512: e2a9e02617eeec22a9240d7cf9386ee880a5f3639b143df7de4d8ea3e7b808f8c123f0b0410ff4a22e9a564bd86b2335a5c4aa3b2281af47d111484a6f1fd108
8454fb2bd7 test: sync funding block before isolating nodes (shaurya2k06)
Pull request description:
Fixes#35967
test_alternate_witness_tx mines the taproot funding output on node0 with
sync_fun=self.no_op and immediately disconnects. node1 later includes the
script-path spend via generateblock. If the funding block has not reached
node1, that call fails with bad-txns-inputs-missingorspent.
Drop the no_op so generate() uses the default sync_all before the partition.
Later generate* calls keep no_op because the nodes are then disconnected.
Seen twice this week in hebasto bitcoin-core-nightly NetBSD jobs:
https://github.com/hebasto/bitcoin-core-nightly/actions/runs/31350308484/job/93339698854https://github.com/hebasto/bitcoin-core-nightly/actions/runs/31765546925/job/94660585799
The modified test is test/functional/wallet_listtransactions.py. I ran it
locally three times with build/test/functional/wallet_listtransactions.py.
ACKs for top commit:
achow101:
ACK 8454fb2bd7
furszy:
utACK 8454fb2bd7
Tree-SHA512: 6b8fdcdc9ce9c57c2939caff34850e88450864c909fe226ba9b6e02ffcefa3625623589b8ecd2f09b9ca16bcaec3a64bd07642ff2c47ae6e8d250de478d34b53
fada80192b test: Print os exit code on failure (MarcoFalke)
Pull request description:
Printing the exit code (like printing the stderr) seems independently useful, but should also help to debug the Windows CI failures, which have an empty stderr and truncated combined log:
* https://github.com/bitcoin/bitcoin/issues/34925
* https://github.com/bitcoin/bitcoin/issues/34367
* ...
ACKs for top commit:
sedited:
tACK fada80192b
Tree-SHA512: 085201532ccce9da27cf996136d48436b7800b00a8c8011977d37fcfe152ca029e093428b06ba058759ae10f6dd8e94500b34df712f13d6064f6a7499539bcdc
fe7d475d45 private broadcast: bound broadcast attempts per tx to 1k (Gregory Sanders)
Pull request description:
Since rebroacasts introduce additional state, bound the state growth by capping the number of rebroadcasts. With ~72 bytes per record, 10k transactions rebroadcasting for ~42 hours will result about 703 MiB allocated with overhead.
ACKs for top commit:
andrewtoth:
ACK fe7d475d45
frankomosh:
ReACK fe7d475d45
sedited:
ACK fe7d475d45
Tree-SHA512: e4ec5156b90ad24d68b561df03ad09bdf0ac7535886ff56891cb698cf64ff0e1e484075b76040bba6194baf874c9237028c82debf7405136447ba5b5faee589c
c079288967 psbt: update output metadata without inputs (Lőrinc)
4f5712476a test: characterize P2WSH miniscript output (Lőrinc)
e24e8fa2a6 test: characterize PSBT output metadata (Lőrinc)
Pull request description:
**Problem:** PSBTv2 permits outputs to be added before inputs.
An authenticated `descriptorprocesspsbt` request can abort the node while updating metadata for one of those outputs because `UpdatePSBTOutput()` traverses the output script with a signature creator for input index 0.
ECDSA signing or a miniscript timelock check can then access the missing input.
**Fix:** Make `UpdatePSBTOutput()` traverse output scripts with a temporary one-input transaction while continuing to take the output from the PSBT's unsigned transaction.
`MutableTransactionSignatureCreator` continues to require a valid input index.
Output metadata traversal still records scripts and key origins, allowing outputs to be updated before inputs are added.
ACKs for top commit:
jeanpablojp:
tACK c079288967
achow101:
ACK c079288967
w0xlt:
ACK c079288967
polespinasa:
ACK c079288967
Tree-SHA512: 0d8cda74b8a56c0f4713b2669e5a3e5b0551ecda4fdfceb38a80e5b98a9d208d447f2045a1cc9fee74fe33b2fc8f7a60997cd60b2961de5cea871f53831895fe
9cc7dc50bd p2p: reconsider orphans when missing inputs are mined (Greg Sanders)
Pull request description:
We reconsider for mempool entry of missing inputs, we should reconsider for mining of them too.
ACKs for top commit:
yuvicc:
ACK 9cc7dc50bd
l0rinc:
Lightly tested code review ACK 9cc7dc50bd
marcofleon:
ACK 9cc7dc50bd
Tree-SHA512: 9acfb6898e3b286ce23bc2ca3369ae951fadee5f175baad634a6bd23039108d97283814e47972fb857ae459505535f621866f2514b705e94cf841979c37a3933
de2adc308a qa: Disable Qt's glib event dispatcher for GUI tests on OpenBSD (Hennadii Stepanov)
Pull request description:
When `bitcoin-gui` is built against OpenBSD's system Qt packages (which have GLib support), shutdown emits "GLib-CRITICAL **: g_main_context_pop_thread_default: assertion 'stack != NULL' failed" messages on `stderr`, which the test framework treats as a failure.
Set `QT_NO_GLIB=1` so Qt falls back to its poll-based event dispatcher, which avoids the GLib thread-default context entirely.
Fixes https://github.com/bitcoin/bitcoin/issues/35851.
See the CI log here: https://github.com/hebasto/bitcoin-core-nightly/actions/runs/31510478034.
ACKs for top commit:
maflcko:
lgtm ACK de2adc308a
Tree-SHA512: edc991c7a174bc304a4da0ca29ec97bcaece463289de3da5350a046f1133ce6d830c37d5188536ca2ce238d462e56de8f2167fdeeb1d1e5ecca38d60c8495cce
e07d826e0e rpc: Fix type in ApplyTypeStrOverride (Shuvam Pandey)
c94074fa1b rpc: Surface OBJ_USER_KEYS description for openrpc (sedited)
c020c21d54 rpc: Handle skip type args for openrpc (sedited)
Pull request description:
This was initially motivated by testing the dump of the schema against open-rpc-generator, which crashed with:
```
open-rpc-generator generate -t client -l rust -n bitcoin_client -d ./openrpc.gen.json -o ./generated
There was error at generator runtime:
TypeError: Cannot convert undefined or null to object
```
The changes here fix this crash (albeit perfectly valid existing schema), but I think creating a more complete output is helpful on its own. The openrpc schema dumps can eventually be re-used for the rpc docs and to track rpc interface changes more accurately. Adding the CreateTxDoc outputs section seems useful for that.
Also includes a type tightening from number to integer in `ApplyTypeStrOverride` to reflect the actual behaviour in the rpc calls, where only integers are accepted.
ACKs for top commit:
achow101:
ACK e07d826e0e
willcl-ark:
ACK e07d826e0e
Tree-SHA512: d0454a71b4f1dab1daf8a0d5b1e5bf1c1b8f1a16d26638d4a64a2652402ad74230366cabf0cf4135a16d0bdab584d3d4b2a47f2968a4eff605348ce85e8dbadb
Add a string fee_rate_estimator option (default "none") to
estimatesmartfee options. "block_policy" consults only the block
policy fee rate estimator, "none" uses the fee rate estimator
manager selected behaviour, and unknown values are treated as
"none". Unknown option keys are rejected.
Still only the block policy fee rate estimator, so the result is
unchanged; a subsequent commit will change the default behaviour.
Also adds GetFeeRateEstimate(FeeRateEstimatorType, target, conservative)
to FeeRateEstimatorManager so callers can query a single estimator by
type; NONE returns the manager-selected combined estimate.
9954aa7728 http: don't parse any new requests from a client if m_req_busy = true (Matthew Zipkin)
c7db3ae1f9 test: cover HTTPRequest state machine (Matthew Zipkin)
90676e24ad Add state to HTTPRequest to avoid duplicate work over I/O cycles (Matthew Zipkin)
507e528e84 http: reuse HTTPHeaders to parse chunked trailer (Matthew Zipkin)
902d8908c9 http: only read one HTTPRequest at a time per client (Matthew Zipkin)
Pull request description:
This PR reduces the memory consumption of the HTTP Server when reading data from connected clients, and improves performance especially when requests are large (i.e. requiring multiple TCP packets).
In https://github.com/bitcoin/bitcoin/pull/35182 the server copies as much data as it can from the socket into application memory, and then tries to parse as many complete HTTP requests as possible from that data. If a request is discovered to be incomplete, the in-progress request is abandoned. The server tries again on the next I/O cycle to read the same data from the buffer, duplicating work as many times as it takes before the client finishes sending the request (or times out).
This PR implements two improvements to this:
1. Only parse one request at a time from the receive buffer. The server processes requests from each client in series anyway.
2. Add state to `HTTPRequest` so it can be filled with data from the receive buffer over multiple I/O loop iterations without losing progress.
If a client sends large or multiple requests, that data will sit in the kernel's socket buffer instead of the application memory. Eventually the socket buffer will fill up and TCP backpressure will kick in, dropping the TCP window to 0 and blocking the client from sending any more.
A state machine for `HTTPRemoteClient` was [discussed previously](https://github.com/bitcoin/bitcoin/pull/35182#pullrequestreview-4322490068) to control resource consumption. Another nice benefit of this model (for a follow-up PR) will be to insert the RPC authentication check after reading 8kB-limited headers but before the 32MB-limited request body.
ACKs for top commit:
winterrdog:
re-ACK 9954aa7728
janb84:
re ACK 9954aa7728
frankomosh:
ACK 9954aa7728.
fjahr:
ACK 9954aa7728
Tree-SHA512: b7c913114283fbf1f360b40f6c65a01390a26731bf3b166f460ec260f9206f25d738b3a06887bfa839911c1c6aaf634448181da47a752a9a881aebd907e44868
fabe100c2b test: Use throwing config parser getters without fallback (MarcoFalke)
fa8acd57cd test: Write true/false values in config.ini (MarcoFalke)
Pull request description:
Currently, the called `getboolean` member function is *not* the throwing https://docs.python.org/3/library/configparser.html#configparser.ConfigParser.getboolean, but a non-throwing member function on a dict-like proxy object.
This is confusing and brittle, because tests shouldn't silently skip when a config key is missing. Instead, tests should loudly fail, e.g. when the config key is renamed in one place, but not the other.
ACKs for top commit:
jeanpablojp:
tACK fabe100c2b
willcl-ark:
ACK fabe100c2b
Tree-SHA512: a970d74ad285372b8adcce8e2a52b01f5a3b563899dfc5262e6ffbf3d8aba43e72f7b03111e8d5188924c7d3d789992407cddb5d182d9be5e42f07896d8ad4a3
b3d77ea027 test: Speedup fee estimation functional test with batching (sedited)
Pull request description:
The fee estimation functional test is currently the slowest one by a good margin. It is a bit annoying, because it also increases the total runtime of the functional tests.
It seems like most of the slowness comes from the transactions propagating between the nodes. This patch helps them do that by submitting them directly to all the nodes. Also take this opportunity to batch the transaction submissions.
On my machine this speeds up the fee estimation functional test from around 71 seconds to 25 seconds.
ACKs for top commit:
151henry151:
tACK b3d77ea027
maflcko:
review ACK b3d77ea027🐇
ismaelsadeeq:
ACK b3d77ea027
Tree-SHA512: f76415dca7997577ca39ac6b95dfdf32b930dd64b4311e3da34e34adb16107ff4ea2d9fa679f3ca50540e80c38af7f9390b44f21ad1b8107fbbc3586edb2ef19
25bed560be test: add forward-compat functional test for txindex (sedited)
703304ed8c doc: add release notes for txindex disk usage and downgrading (Andrew Toth)
8e5320a2d2 tests: cover txindex hash prefix collisions and legacy fallback (Andrew Toth)
b75efa19ba txindex: skip bloom filters and legacy lookups for new databases (Andrew Toth)
004d7c098c txindex: hash key prefixes and pack block positions (Andrew Toth)
5a255970fd refactor: move txindex db constants and legacy key to txindex_key.h (Andrew Toth)
327660134c txindex: pass the full block to DB::WriteTxs (Andrew Toth)
42771e7998 txindex: use a new block locator for downgrade safety (Andrew Toth)
4b08baed72 txindex: return optional tx and block hash from FindTx (Andrew Toth)
Pull request description:
The current txindex uses the full 32-byte txid as keys, which takes up about 66 GB of disk space today on mainnet. Using a 5-byte key prefix instead drops the disk usage to 26 GB - cutting the size to less than half.
Using the full 32-bytes is unnecessary since a 5-byte salted siphash will produce collisions in about 1 in 1.1 trillion. Some collisions will occur, but the penalty is just an extra disk read, deserialization and hash.
The tx position can be appended to the key instead of used as a value, and a LevelDB iterator can seek to the prefix and then scan for the correct tx. This is an almost identical approach to `txospenderindex`.
Also instead of storing the file position of the block, we can store only the sequence of the connected block and offset of the transaction in the block. This can be packed into a 6-byte key suffix using 3-byte representations of the sequence and offset in the block. The block file can be recovered by the CBlockIndex that is already in memory. The sequence is mapped to the block hash in the db, so we can lookup the block hash to find the CBlockIndex during reads.
If a tx is not found with this method, we fallback to looking up the legacy entry. With this method a user with an existing db can opt to erase the `indexes/txindex` folder and reindex, or keep the current index and new entries will be appended with the smaller footprint.
The time to index was faster on my machine with this method, 1h19m vs current 1h50m.
Lookups are roughly the same, around 0.2ms per lookup with `getrawtransaction`.
When testing on mainnet, I got 894,549 2-way collisions, 395 3-way collision, and 1 4-way collision that worst case could cause an extra 3 false positives when reading.
ACKs for top commit:
l0rinc:
diff reACK 25bed560be
sedited:
ACK 25bed560be
ajtowns:
ACK 25bed560be
Tree-SHA512: a25c79ca7e722e2f372b65f5fc11c8b194ad49f2240b4881c7e606306aabbd3604aede3f1c33606b467486affac3a3f503638f513c896935cebbc02709cb60d8
c3945bfd2b doc: use derivehdkey in multisig tutorial (Sjors Provoost)
3662e33669 test: use derivehdkey in M-of-N multisig demo (Sjors Provoost)
d9570f0838 rpc: add derivehdkey (Sjors Provoost)
62da9f9614 wallet: add GetExtKey helper (Sjors Provoost)
aaf1548475 wallet: generalize GetActiveHDPubKeys helper (Sjors Provoost)
3821452c4a refactor: add hardened derivation helper (Sjors Provoost)
0ab61caafd rpc: ParsePathBIP32 helper (Sjors Provoost)
e36c4b76e1 util: reject out-of-range BIP32 keypath indices (Sjors Provoost)
ba78c31a00 fuzz: check ParseHDKeypath/WriteHDKeypath round-trip (Sjors Provoost)
8cce969085 Have ParseHDKeypath handle h derivation marker (Sjors Provoost)
fc53077762 test: move parse_hd_keypath test to bip32_tests (Sjors Provoost)
dab525eb77 key: add DeriveExtKey() helper (Sjors Provoost)
Pull request description:
Adds a `derivehdkey` RPC that returns an xpub, or optionally the xprv, at an arbitrary BIP32 path (with at least one hardened step), derived from a wallet HD key.
The main use case is coordinating a multisig setup, where each participant shares an xpub derived at a hardened path (e.g. `m/87h/0h/0h`) distinct from their default single-signature descriptors. See the (updated) `doc/multisig-tutorial.md` and (updated) functional test to see how that workflow improves.
The first commits are some helpful helpers:
- _key: add DeriveExtKey() helper_ - performs the actual derivation
- _test: move parse_hd_keypath test to bip32_tests_ - from `psbt_wallet_tests`
- _Have ParseHDKeypath handle h derivation marker_
- _util: reject out-of-range BIP32 keypath indices_ - `ParseHDKeypath` would previously map overflowing values without `h` to hardened.
- _fuzz: check ParseHDKeypath/WriteHDKeypath round-trip_
- _rpc: ParsePathBIP32 helper_
- _refactor: add hardened derivation helper_ - `HasHardenedDerivation()`, to enforce the "at least one hardened step" rule
- _wallet: generalize GetActiveHDPubKeys helper_ - extracts code from `gethdkeys` which `derivehdkey` needs
- _wallet: add GetExtKey helper_ - reconstruct an xprv from a wallet xpub (analog of `GetKey()`); behavior-preserving prep, also simplifies `gethdkeys`.
Meat and potatoes:
- _rpc: add derivehdkey_ - the RPC itself, plus the `UnusedKey` filter on `GetHDPubKeys` that drives key selection.
- _test: use derivehdkey in M-of-N multisig demo_ - rewrites the functional multisig test to use the RPC and `<0;1>` syntax.
- _doc: use derivehdkey in multisig tutorial_ - same for the prose tutorial.
ACKs for top commit:
pseudoramdom:
code review ACK c3945bfd2b
achow101:
ACK c3945bfd2b
w0xlt:
That being the case, ACK c3945bfd2b
Tree-SHA512: 661f17c9bfe26017eb14c27ba7af37093387100d3baa25f5d29bba9c1aedc40d19afe1bdfc126a18d018857bb02f1fc84386f10b8f4f4b8e9d6f4b0691d9e302
Rather than rebroadcasting forever, bound attempts at
private broadcast, report remaining attempts over RPC
results, and allow exhausted transactions to be
retried when submitted.
test_alternate_witness_tx mines the taproot funding output on node0
with sync_fun=self.no_op and immediately disconnects. node1 later
includes the script-path spend via generateblock. If the funding
block has not reached node1, that call fails with
bad-txns-inputs-missingorspent.
Drop the no_op so generate() uses the default sync_all before the
partition.
Signed-off-by: shaurya2k06 <shaurya2k06@gmail.com>
PSBTv2 permits outputs to be added before inputs.
`UpdatePSBTOutput()` traverses output scripts with a signature creator for input index 0, so ECDSA signing or a miniscript relative-timelock check can access a missing input and abort.
Construct a standalone transaction with one input for metadata traversal.
Continue taking the output from the unsigned PSBT transaction so scripts and key origins remain associated with the correct output while traversal no longer depends on the PSBT input list.
`MutableTransactionSignatureCreator` continues to require callers to provide a valid input index.
Co-authored-by: Ava Chow <github@achow101.com>
Document the scripts and key origins collected for common output descriptors before changing zero-input handling.
Use a valid input for ECDSA-backed outputs because their zero-input paths currently abort.
f280f5eb47 wallet: rpc: deprecate removeprunedfunds (David Gumberg)
e5b7785447 test: wallet: resend: avoid internal behavior via removeprunedfunds (David Gumberg)
Pull request description:
Originally added in https://github.com/bitcoin/bitcoin/pull/7558 as a companion to `importprunedfunds`, this RPC has no known helpful use while being both dangerous and a maintenance burden.
Despite what the name says, it allows the deletion of arbitrary transactions, and `importprunedfunds` does not allow the importing of transactions not belonging to the user, and `listtransactions` does not list transactions not belonging to the wallet, so this RPC can only be used to delete transactions actually belonging to the wallet, and in the unlikely event that transactions not belonging to the wallet are present, they cause no harm except for occupying a few bytes on the users disk.
ACKs for top commit:
achow101:
ACK f280f5eb47
polespinasa:
ACK f280f5eb47
pablomartin4btc:
reACK f280f5eb47
Tree-SHA512: ed9c30c50be514d637999b4c8f3fa9b9b1446a5553e3974703638b45d8f55f1291f5cfb82dd2ead6d0743e424e9e3b1edbd56fc04dae3dcdee4d175e2a1ce061
4e8c4bc794 test: classify SOCKS5 peers via getpeerinfo addrbind (Henry Romp)
Pull request description:
p2p_private_broadcast.py classifies each SOCKS5 connection by scanning the node's debug log for `trying v. connection (...) to <addr>:<port>`, then attaches a fake peer for that type. The helper returned the first match in the whole log, so when a feeler selected a clearnet address that private broadcast had used earlier in the run (in the CI failure, `[50::1]:8333`, about 10 seconds apart), the feeler was labelled private-broadcast, was given the `NoRelayP2PInterface`, and disconnected as a feeler rather than with the expected "connected in vain" message.
Instead of relying on the debug log, identify the connection via the SOCKS5 proxy client socket's source address, which equals the node's `addrbind` for that peer, and read `connection_type` from getpeerinfo. The proxy replies to the SOCKS5 request before invoking `destinations_factory`, so the node has already registered the peer by the time classification runs. This also stops treating debug.log contents as a stable test interface. Dropping the log scrape removes a full re-read of debug.log per SOCKS5 connection; `p2p_private_broadcast.py` goes from ~23s to ~14s locally.
Fixes#35843
Tested with:
`build/test/functional/test_runner.py p2p_private_broadcast.py p2p_private_broadcast_retry_v1.py --timeout-factor=2`, and against the forced-feeler repro from the issue, which no longer mislabels the feeler.
ACKs for top commit:
jeanpablojp:
tACK 4e8c4bc794
andrewtoth:
ACK 4e8c4bc794
mzumsande:
Code Review ACK 4e8c4bc794
Tree-SHA512: ce2db418787d7ecf518bd49b37d7d664748fee5991a2924522dfaf42b27d90ca001caa0611011310636b453d3aada1061d086f20bb866eb66605645935f55c74
While this is usually used where the types are not enforced strictly,
adding the description is both useful to the developer implementing a
client and for potentially using the openrpc output as a basis for
documentation.
```diff
diff interim_dump.json new_dump.json
465c465,466
< }
---
> },
> "description": "A key-value pair. The key (string) is the bitcoin address,\nthe value (float or string) is the amount in BTC"
777c778,779
< }
---
> },
> "description": "A key-value pair. The key (string) is the bitcoin address, the value (float or string) is the amount in BTC"
903c905,906
< }
---
> },
> "description": "A key-value pair. The key (string) is the bitcoin address, the value (float or string) is the amount in BTC"
12915c12918,12919
< }
---
> },
> "description": "A key-value pair. The key (string) is the bitcoin address,\nthe value (float or string) is the amount in BTC"
13692c13696,13697
< }
---
> },
> "description": "A key-value pair. The key (string) is the bitcoin address,\nthe value (float or string) is the amount in BTC"
14002c14007,14008
< }
---
> },
> "description": "A key-value pair. The key (string) is the bitcoin address, the value (float or string) is the amount in BTC"
14254c14260,14261
< }
---
> },
> "description": "The bitcoin address is the key, the numeric amount (can be string) in BTC is the value"
16046c16053,16054
< }
---
> },
> "description": "A key-value pair. The key (string) is the bitcoin address,\nthe value (float or string) is the amount in BTC"
```
Instead of filling them in with empty object and array args be a bit
more friendly to the consumer by giving type hints, while retaining type
flexibility:
Keep the empty object, but fill the array with the hinted at types by
recursing through them. Add an additional argument to the openrpc
functions (`in_skip_type_check`) to keep track of when a loosely typed
argument is under evaluation. Note that the array is evaluated stricter
than before: If it contains items, they need to match the nested
objects.
The change from oneOf->anyOf and removing additionalProperties: false
should still convey to the reader that this is a loosely typed object
and prevent collisions between these loosely-typed schemas.
This was initially motivated by testing the dump of the schema against
open-rpc-generator, which crashed with:
```
open-rpc-generator generate -t client -l rust -n bitcoin_client -d ./openrpc.gen.json -o ./generated
There was error at generator runtime:
TypeError: Cannot convert undefined or null to object
```
The changes here fix this crash, but I think creating a friendlier
output is helpful on its own.
This patch changes the schema exported from CreateTxDoc as follows:
```diff
diff -U6 <(jq '.methods[] | select(.name == "createrawtransaction")' dump.json) \
<(jq '.methods[] | select(.name == "createrawtransaction")' new_dump.json)
@@ -37,13 +37,43 @@
{
"name": "outputs",
"required": true,
"schema": {
"oneOf": [
{
- "type": "array"
+ "type": "array",
+ "items": {
+ "anyOf": [
+ {
+ "type": "object",
+ "additionalProperties": {
+ "oneOf": [
+ {
+ "type": "number"
+ },
+ {
+ "type": "string"
+ }
+ ]
+ }
+ },
+ {
+ "type": "object",
+ "properties": {
+ "data": {
+ "type": "string",
+ "pattern": "^[0-9a-fA-F]+$",
+ "description": "A key-value pair. The key must be \"data\", the value is hex-encoded data that becomes a part of an OP_RETURN output"
+ }
+ },
+ "required": [
+ "data"
+ ]
+ }
+ ]
+ }
},
{
"type": "object"
}
]
},
```
OriginPubkeyProvider::GetPubKey() derives its sub-provider straight into
the output provider and then prepends its origin to the entry it finds
there. The sub-providers insert with emplace(), so if the same key was
already expanded by another key expression the insert is a no-op and the
entry that gets the origin prepended is the one that already has it.
musig() is where this shows up, because it expands its participants into
the provider it is given, so two musig() expressions in the same key
expression list end up expanding a shared participant twice. A
participant origin of m/86h/1h/0h then comes out as
m/86h/1h/0h/86h/1h/0h in the input and output taproot BIP32 derivation
maps of a PSBT, so a signer that follows them derives the wrong key.
When the two expressions declare different origins for the participant,
the fingerprint of one is combined with the path of both.
Derive into a temporary provider and apply the origin there. Merge the
rest into the output, then insert the clean origin separately. This
avoids stacking paths and makes an explicit origin replace an implicit
one already present for the same key.
The miniscript duplicate key check compares two key expressions by
deriving each of them at index 0, and it did that with an empty signing
provider. Any expression with a hardened step could therefore not be
derived and came back empty, so two of them compared equal and the
descriptor was rejected with "contains duplicate public keys" even
though the keys were different. musig() makes this easy to run into,
since one participant on a hardened path keeps the whole aggregate key
from resolving, but plain key expressions are affected just the same.
Derive with the signing provider that is filled while parsing, or with
the one we are inferring from, since that is where the private keys for
the hardened steps are. If both keys still cannot be derived, compare
the key expressions instead, so that two different expressions are not
mistaken for one another.
A duplicate written two different ways can still be missed if either
spelling cannot be derived. The private key that is missing for the
comparison is also needed to expand the descriptor, so deriveaddresses
and importdescriptors both refuse it. Once the needed private keys are
available, both expressions resolve and the duplicate is caught again.
4ca182ca40 doc: clarify alternate_wtxids is empty when only one witness variant (pablomartin4btc)
fa48b5d28e test: assert listsinceblock "removed" reports current canonical wtxid (pablomartin4btc)
9b96ee1288 wallet, test: add unit test for variant txid validation in CWalletTx deserializer (pablomartin4btc)
9de6543cb5 wallet: post-#35501 cleanup in CWalletTx (pablomartin4btc)
Pull request description:
Follow-up cleanups and clarifications after #35501 was merged.
Commit breakdown:
1. _post-[#35501](https://github.com/bitcoin/bitcoin/pull/35501) cleanup in_ `CWalletTx`
- Rename `arg_state` → `new_state` in `Update()` for consistency
- Simplify `RecomputeCanonical()` using `std::ranges::min_element` with a projection lambda (14 lines → 3 lines)
- Add variant txid validation in the `CWalletTx` deserialise constructor: throws `std::runtime_error` if any variant's txid doesn't match the canonical txid deserialized from the stream
- Move `Init()` to `private` and extend it to clear `m_txs` and reset `m_canonical_wtxid`, so a full re-deserialise via `Unserialize()` starts from a clean state
All [suggested](https://github.com/bitcoin/bitcoin/pull/35501#pullrequestreview-4854519083) by ajtowns.
2. _add unit test for variant txid validation in_ `CWalletTx` _deserializer_
3. _assert_ `listsinceblock` "removed" _reports current canonical wtxid_
Documents that removed entries reflect the wallet's current `CWalletTx` state, not a snapshot of the detached block. A future followup could improve this (requires per-block tracking of which witness variant was included).
[Suggested](https://github.com/bitcoin/bitcoin/pull/35501#discussion_r3632044472) by w0xlt.
4. _clarify_ `alternate_wtxids` _is empty when only one witness variant_
[Suggested](https://github.com/bitcoin/bitcoin/pull/35501#discussion_r3632113003) by polespinasa.
ACKs for top commit:
jeanpablojp:
re-ACK 4ca182ca40
achow101:
ACK 4ca182ca40
polespinasa:
ACK 4ca182ca40
Tree-SHA512: 64eadeb11372d904c79edbfd264c4d8dc1b4fe4ce5e3acc301bfeba9e556efb5dce2c684f0e58c7b74e2c687cc7dd77389970662b3fd629ed034a97bcfdfb71c
