Since 31.0 has a compatibility issue with wallets containing miniscript
descriptors, this should be in the test, with a test for the failure
condition.
Instead of treating the descriptor ID as something which has a meaning
which can be verified, treat the ID read from disk as some opaque blob
used solely to identify and tie together specific records from disk.
This removes the usage of the ID for duplication checks or comparison,
and removes the check that the read ID matches a computed ID.
When writing new descriptors to disk, the ID is still calculated from
the old Descriptor ID method for backwards compatibility. But this fact
is opaque to all further usages of the ID.
If a descriptor is being reimported, we should only update the metadata
and cache from the other one, rather than overwriting the entire thing.
This avoids a potential issue where the on-disk record is overwritten
with a backwards incompatible string.
Instead of creating a DescriptorSPKM that doesn't have a descriptor,
only to then generate the descriptor, combine SetupDescriptorGeneration
into the GenerateNewSingleSig factory function, and within that
function, generate the descriptor first before constructing the new
DescriptorSPKM.
Previous versions did not pass down StringType::COMPAT when that was
given as the serialization string type. As COMPAT is used for descriptor
id calculation, we need to maintain the previous (incorrect) behavior of
not passing StringType::COMPAT.
1cb416397b psbt: avoid duplicate taproot leaf script keys when merging (Shuvam Pandey)
Pull request description:
Follow-up to #35665, which fixed the same combiner defect for `PSBT_GLOBAL_XPUB`. thomasbuilds
and winterrdog asked for this one as its own PR when I reported it there.
`m_tap_scripts` maps a leaf script to a set of control blocks, but is serialized as one record
per control block, keyed by the control block (`SerializeToVector(s, PSBT_IN_TAP_LEAF_SCRIPT,
std::span{control_block})`). `PSBTInput::Merge` unions it by the map key, so two PSBTs that map
the same control block to different leaf scripts merge into an input that serializes the `0x15`
key twice. Duplicate keys make a PSBT invalid, so it is the same `combinepsbt` then
`decodepsbt` failure as the xpub case, at the input level. Present since #22558 (v24.0).
Both decode on their own, and differ only in the leaf script the control block maps to, `OP_1`
against `OP_1 OP_1`:
```
$ A=cHNidP8BADwCAAAAAaqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAIhXAUJKbdMGgSVS3i0tgNel6XgeKWg8o7JbVR7/ums6AOsACUcAAAA==
$ B=cHNidP8BADwCAAAAAaqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAIhXAUJKbdMGgSVS3i0tgNel6XgeKWg8o7JbVR7/ums6AOsADUVHAAAA=
$ bitcoin-cli -regtest decodepsbt "$(bitcoin-cli -regtest combinepsbt "[\"$A\",\"$B\"]")"
error code: -22
error message:
TX decode failed Duplicate Key, input key "15c050929b74c1a04954b78b4b6035e97a5e078a5a0f28ec96d547bfee9ace803ac0" already provided: unspecified iostream_category error
```
winterrdog reproduced it on the #35665 thread with another pair.
Merge the records rather than the map entries, keeping the leaf script already there. BIP 174
lets the combiner "pick arbitrarily when conflicts occur", and unknown and proprietary records
already resolve that way. Refusing to combine is the BIP's other option, but that would fail
`combinepsbt` on input it accepts today.
Merging by map key drops records as well. `std::map::insert` leaves existing keys alone, so
when both PSBTs carry the same leaf script with different control blocks, the incoming set was
dropped. Those keys do not conflict, so merging per record keeps them.
The control blocks already present are collected once per merge rather than searched for per
incoming record, which would be quadratic in the size of the two PSBTs `combinepsbt` takes from
the caller.
Since this is the second field with this shape I checked the rest. `m_xpubs` (#35665) and
`m_tap_scripts` are the only two whose record key comes from the value, so two map entries can
serialize the same key. `partial_sigs` is keyed by `CKeyID` and serialized under the pubkey,
but the pubkey determines the `CKeyID`, so those records stay distinct. The others key the
record by the map key, `m_proprietary` included, and `PSBTOutput` has no such field.
The test fails on master on both counts, and covers the merges that do not conflict as well.
I found this with a local assertion in the psbt fuzz target that a combined PSBT must
roundtrip. That assertion can go in a follow-up.
Tested:
```
./build/bin/test_bitcoin --run_test=psbt_tests
./build/bin/test_bitcoin --run_test=psbt_wallet_tests
./build/test/functional/test_runner.py rpc_psbt.py rpc_rawtransaction.py wallet_taproot.py wallet_signer.py feature_taproot.py wallet_basic.py
```
ACKs for top commit:
achow101:
ACK 1cb416397b
winterrdog:
re-ACK 1cb416397b
Tree-SHA512: 2599beffe701ba9b672e8dcc3d43844f3853ceeb3d86fc53798a428d3288aa8edd2e42f338b32a1046fa558d6cc5cfa5d55b3a0aaf960278b0f3002158381623
436921eb46 test: check joinpsbts preserves global xpubs and proprietary fields (Thomas)
011094b282 rpc: preserve global xpubs and proprietary fields in joinpsbts (Thomas)
Pull request description:
`joinpsbts` collects the global xpubs of all the joined PSBTs into `merged_psbt`, but returns a separately constructed `shuffled_psbt` into which only the inputs, outputs, and unknown fields are copied. The collected `PSBT_GLOBAL_XPUB` records are silently dropped, and `PSBT_GLOBAL_PROPRIETARY` records are not collected at all.
The xpub collection was added in #17034, which was written against a `joinpsbts` that still returned `merged_psbt`, but was merged after #16512 had introduced the `shuffled_psbt` rebuild, so the collected xpubs have never reached the result.
Shuffle the inputs and outputs of `merged_psbt` in place instead of rebuilding a new PSBT, so that all global data is preserved, and union the global proprietary records in the merge loop, matching the `combinepsbt` behavior from #34893.
ACKs for top commit:
jpk68:
ACK 436921eb46
achow101:
ACK 436921eb46
winterrdog:
tACK 436921eb46
Tree-SHA512: d9de34c25aecc29b6b4fb80d6584fa919cc5ff9b7ef2f4d8ce35c4043fe7638fefb8af10448f2cd14021f5d25e149f0efc8798c5b8c9bc8b5582c6152010e891
e2ab8ae551 wallet: spkm: Only notify CanGetAddressesChanged on change (David Gumberg)
0892f16f91 refactor: moveonly: Pair CanGetAddressesChanged notifications with desc range. (David Gumberg)
e6adae3db2 wallet: `NotifyCanGetAddressesChanged` when advancing `next_index` (David Gumberg)
Pull request description:
Even though `TopUp()` notifies, advancing `next_index` after can deplete available addresses, so make sure to notify any time it's changed.
This would manifest as users seeing a clickable `Receive` button in the GUI when in fact no address can be generated in some edge cases, e.g. when a user has a watch only wallet with a hardened derivation path and runs out of keys.
This feels like it's begging for:
1) a refactor to make it impossible to modify `next_index` or `range_end` without firing `CanGetAddressesChanged`
2) a test
I banged my head against the keyboard for a bit but I couldn't get either of these to fall out, I also tried massaging a few clankers into doing it but I couldn't get any results that seemed reasonable to me, still seems like a worthwhile fix so opening PR anyway.
I also included a moveonly commit to pair code that can change the result of `CanGetAddresses()` with the notification firing
ACKs for top commit:
achow101:
ACK e2ab8ae551
polespinasa:
ACK e2ab8ae551
furszy:
utACK e2ab8ae551
Tree-SHA512: 5bb00d1ef4909a3e55535d283e5995df75e4288a151647f4a368b2086e2f2f4140693f43cae4f72727eafb49c9050aea8604cd6ddddc8646f7ac47b1357ed287
d837bb38a4 contrib/init: fix unused variables in openrc script (jpk68)
Pull request description:
- Makes it so that `${BITCOIND_BIN}` is actually used as `command=`, rather than the hardcoded `/usr/bin/bitcoind`.
- Passes `BITCOIND_GROUP` to `start-stop-daemon`, so that the daemon process itself runs under it.
ACKs for top commit:
jeanpablojp:
utACK d837bb38a4
thomasbuilds:
ACK d837bb38
winterrdog:
utACK d837bb38a4
Tree-SHA512: 78c237224d65cc47ade606df4808fbf4ca70109d95301c35d1b336eead1b25138a83ce6f2314775b8a11a6745af2220af9027640530b9edb7e0167ccea081b6b
9e115edd39 test: Remove `BOOST_CHECK_CLOSE` in favor of exact comparison (rustaceanrob)
Pull request description:
`max_cache` is known ahead of time in this test as a `size_t` of `10000`, and each of these calculations should be known ahead of time (500.0, 9500.0). This test can truncate the double and assert exact equality rather than use a tolerance. Found in #35713 whereby this is the only use of this macro in the unit tests. IMO it is appropriate to tighten this test and remove the macro.
ACKs for top commit:
maflcko:
lgtm ACK 9e115edd39
josibake:
ACK 9e115edd39
Tree-SHA512: 738f05650bd1426e8e29e94955685e8ad3cd62e57f1c65b7af045161e4ca5af64bbd1f0cfed8da427a1f778620e3c8f7dfb835881f8de169806096213056e863
7ea36e985a test: preserve index crash test state (Lőrinc)
5aa15df60c test: expose missing index crash checkpoint (Lőrinc)
Pull request description:
**Problem:** #35847 moved the unclean-shutdown test into the shared base index tests, but it checked only that each index could reopen and start background sync.
Both checks also pass when the index reopens at height 0, so they do not verify that a height-100 checkpoint was saved before the simulated crash and reloaded afterward.
**Fix:** The first commit records the existing false positive by asserting that each index reopens at height 0 before background sync.
The second commit establishes a durable checkpoint at height 100, drains its setup notification, and changes the same assertion to the pre-crash height.
ACKs for top commit:
jeanpablojp:
tACK 7ea36e985a
mzumsande:
ACK 7ea36e985a
Tree-SHA512: 0dca2bdd978c5df4acbb01692bb2058e74efa70da3d7191687628075a680a57848ddda8087629b5d743d9a1648d7dc849fda9ff487252136a0c91cdfda33ba32
fad1e6bf23 util: refactor: Remove deprecated SetMockTime(i64) alias (MarcoFalke)
faf87c3535 test: refactor: Use FakeNodeClock over manual/global SetMockTime (MarcoFalke)
Pull request description:
The deprecated test-only alias is only used in a few places and required in none.
In fact, it is incorrectly used in two unit tests, so first fixup those, and then remove it.
ACKs for top commit:
rustaceanrob:
ACK fad1e6bf23
Tree-SHA512: 1fa49e363bf8ccad07d61a77d3bc55c84724cd4cf034756b534cdb56614f334c0009d62dbdb08a940ddf46e0143ec6ecd6b4608baa2f8f581f56a6eef0f0abb8
`max_cache` is known ahead of time in this test as a `size_t` of
`10000`, and each of these calculations should be known ahead of time
(500.0, 9500.0). This test can truncate the double and assert exact
equality rather than use a tolerance.
Using SetMockTime in tests is problematic, because it often requires
verbose calls to
`SetMockTime(GetTime<std::chrono::seconds>() + offset)`.
Also, it requires manual `SetMockTime(0);` at the end.
Fix both issues by using FakeNodeClock.
b42f7fade0 descriptor: don't prepend key origins twice (Shuvam Pandey)
7b15e2cb44 descriptor: fix duplicate check for hardened keys (Shuvam Pandey)
Pull request description:
Fixes#34273.
Importing a descriptor that uses the same `musig()` participants twice in one
tapleaf, with different musig subderivations, fails with
`is not sane: contains duplicate public keys`. It only fails when one of the
participants is a private key on a hardened path. The all-xpub version of the
same descriptor imports fine. That's what gave it away.
The duplicate check (`KeyCompare`) resolves each key expression to a pubkey and
compares the results. It does this at index 0, and the old code used an empty
signing provider. With that empty provider, a `musig()` expression can't resolve
when one of its participants is on a hardened path, because deriving that
participant needs its private key, so the whole aggregate key comes back empty.
Two different musig expressions both came back empty, so the check treated them
as duplicates. The fix derives against the signing provider populated during
parsing, which holds the private keys, and only compares the expression strings
when neither side resolves. 151henry151 had suggested looking at the empty
signing provider on the issue.
scgbckbone found a second, separate bug in the same descriptors. When another
expression that reuses those participants is handled in the same expansion, its
participant origin in the PSBT is added twice, so `m/86h/1h/0h` becomes
`m/86h/1h/0h/86h/1h/0h` in both the input and output Taproot BIP32 derivation
maps. `OriginPubkeyProvider::GetPubKey()` now derives into a temporary provider,
merges it, and writes the corrected origin once, so a later expression can't
prepend the same origin again.
Tested:
```
./build/bin/test_bitcoin --run_test=descriptor_tests
./build/bin/test_bitcoin --run_test=miniscript_tests
./build/bin/test_bitcoin --run_test=bip328_tests
./build/bin/test_bitcoin --run_test=psbt_wallet_tests
./build/test/functional/test_runner.py wallet_musig.py --jobs=1
```
ACKs for top commit:
achow101:
ACK b42f7fade0
scgbckbone:
ACK b42f7fade0
Tree-SHA512: ab36caa6bc484fa1fc3289c79e9a3d713278d82f80de478e53e1bdbe645037e07776ac798eba085733abc139c11a9dbf0d3f49d3c0eee9632e4ddf33d2242f92
m_tap_scripts maps a leaf script to a set of control blocks, but is serialized
one record per control block, keyed by the control block. PSBTInput::Merge
unions it by the map key, so two PSBTs that map the same control block to
different leaf scripts merge into an input serializing the 0x15 key twice.
Duplicate keys are invalid, so combinepsbt hands back a PSBT that can no longer
be decoded. Present since #22558 (v24.0).
Merge the records instead of the map entries, keeping the leaf script already
there, as BIP 174 lets the Combiner pick arbitrarily when conflicts occur. The
control blocks already present are collected once rather than searched for per
incoming record, which would be quadratic in the size of the PSBTs.
Control blocks under a leaf script that both PSBTs carry are now kept as well,
where the map level union dropped them.
The test covers conflicting and non-conflicting merges, including an incoming
leaf script whose control blocks only partly conflict, so records that do not
conflict are not dropped alongside those that do.
bd4b1524ea init: do not count file descriptors for HTTPServer if -server=0 (Matthew Zipkin)
b08662060d init: account for maximum file descriptors needed by HTTP (Matthew Zipkin)
cc2acebefb http: configure simultaneous connection limit with -rpcmaxconnections (Matthew Zipkin)
b3d6d2d1a7 http: limit connected clients to 16 (Matthew Zipkin)
86651d8197 scripted-diff: Rename nUserBind, nBind, nMaxConnections to snake_case (Matthew Zipkin)
Pull request description:
Introduces a new configuration option `-rpcmaxconnections` with default value `16`. This is used to limit the number of simultaneous `HTTPClient` connected to the `HTTPServer`. When the limit is reached, new pending connections remain queued in the kernel's socket buffer. Those connections have complete TCP handshakes with the kernel but do not occupy any application memory.
The previous libevent-based HTTP server had no limit on connections but it did have a limit on the kernel socket queue:
e7ff4ef2b4/http.c (L3510)
```c
if (listen(fd, 128) == -1) {
```
The current HTTP server, like the p2p server, uses a platform constant here:
b6becf3534/src/httpserver.cpp (L743)
(on my macOS `SOMAXCONN` is `128` but on my Debian machine it's `4096`)
The default of 16 was chosen as a reasonable upper bound for single-user RPC use cases. Systems designed to handle more simultaneous HTTP connections than this (previously relying on the absence of a limit) can adjust the setting.
## File descriptors
Because of the connection limit, we can now account for the maximum number of file descriptors needed by the HTTP server. This addresses several issues (#11368#11322 maybe #27732) that could have been fixed by a PR waiting in vain for a libevent release (#27731).
## Bonus performance improvement
The new limit is managed in a loop that drains the kernel's socket queue with `accept()`. All pending connections from the queue (up to the limit) are processed in one single call to `SocketHandlerListening()`. The previous code would only accept one connection from the queue on each I/O loop tick, with a `SELECT_TIMEOUT` (50ms) sleep between each.
ACKs for top commit:
fjahr:
tACK bd4b1524ea
janb84:
ACK bd4b1524ea
winterrdog:
tested ACK bd4b1524ea
hodlinator:
Concept ACK bd4b1524ea
willcl-ark:
ACK bd4b1524ea
Tree-SHA512: 2ef7a96da4d7037c7343ec0ea03fda5bb55d10c2a071fce4929141297515923b203d3d338dbcb6599849768f52aa3c9da509fb5d1d6f7c574a1d2034ea2a9e74
5be248341a bugfix: compare real chunk weight against block weight limit (ismaelsadeeq)
fc98790869 test: `TestChunkBlockLimits` uses incorrect weight for comparison (ismaelsadeeq)
Pull request description:
Partially fixes#35596
When assembling a block template, `BlockAssembler::addChunks()` adds chunks of transactions until the block is close to being full. For each chunk, `TestChunkBlockLimits()` checks both the weight and the sigop-cost limits before the chunk is included.
The weight check compared the chunk's **sigops-adjusted** weight against `block_max_weight`:
```cpp
if (nBlockWeight + chunk_feerate.size >= m_options.block_max_weight) {
return false;
}
```
Whereas `nBlockWeight` accumulates the actual chunk weight.
A chunk whose sigop-adjusted weight exceeds the actual weight can be wrongly skipped even though the block sigop limit is enforced independently on the next line, and that could pass. Those chunks pay higher fees, so this could potentially cause miners to needlessly forfeit some fees revenue.
This PR fixes this by passing the chunk's real weight (sum of `GetTxWeight()`, accumulated in the same loop that already sums sigop cost) to `TestChunkBlockLimits()`. The separate sigop-cost check is unchanged.
- The first commit adds `TestSigOpsAdjustedWeightChunkLimit`: it builds one sigop-dense transaction sized to fit by real weight but not by adjusted weight, and asserts that the tx is skipped and only the coinbase is mined.
- The second commit applies the fix and flips the assertion to show the transaction is now included.
ACKs for top commit:
pablomartin4btc:
Code Review ACK 5be248341a
sedited:
ACK 5be248341a
Tree-SHA512: b3fe9bfaa6d83d713d0243c0fc0d0fb8e68e1060bf6d606e43d9a52bd1ec07e42c561a1ba3426f180903726826c4a664bb131ddc5160354a96e3454d538fbf6c
73a94b4545 psbt: avoid aborting on invalid MuSig2 derivations (Lőrinc)
e3d1e75a51 test: characterize MuSig2 derivation aborts (Lőrinc)
Pull request description:
**Problem:** A PSBT may contain MuSig2 derivation metadata with a hardened child index or a path that derives to a different key.
The hardened index aborts during public derivation, while the mismatched key aborts at the result assertion.
`analyzepsbt`, `finalizepsbt`, and `descriptorprocesspsbt` all reach this code without a wallet.
Even the read-only `analyzepsbt` can force a co-signer service to restart its node after unexpected input.
**Fix:** Return failure when a MuSig2 derivation path contains a hardened child index, and skip only the current aggregate when the path derives to a different key so another matching aggregate can still be tried.
This follows [#35154](https://github.com/bitcoin/bitcoin/pull/35154), with the related contributions credited in the commits.
ACKs for top commit:
jeanpablojp:
ACK 73a94b4545
achow101:
ACK 73a94b4545
andrewtoth:
ACK 73a94b4545
Tree-SHA512: d8e28c5a4184154a4427c644ce62423cbcccdc3d82a6293f36fe99055fa04714598bc92c43b526fbcc7c99b231140669c2d0f1b853999d7dc33f949564c90504
Flush the chainstate at the current tip and drain its notification before registering any index.
This lets each end-of-sync `Commit()` persist the pre-crash height and keeps the setup callback out of the simulated crash window.
Replace the TODO-marked `0` expectation with the captured tip height.
The check runs before background sync, so rebuilding cannot hide a missing checkpoint.
`index_unclean_shutdown` previously checked only that each index could reopen and start background sync after the simulated crash.
An empty index at height 0 satisfies both checks, so the test could pass without preserving any pre-crash checkpoint.
Assert the current reopened height before background sync to make the false positive explicit.
dd669f40b9 util: set os-level thread names on Windows (ViniciusCestarii)
Pull request description:
Update SetThreadName to set os-level thread names on Windows too.
This is useful for debugging-ergonomics on Windows. Threads currently show up unnamed in debuggers, crash dumps on Windows and mismatch what's documented under https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#threads.
Tested with the mingw cross build running on Windows 11, print from WinDbg:
<img width="713" height="631" alt="image" src="https://github.com/user-attachments/assets/05e03383-c9b1-4e6b-91f3-9088b2fc7e90" />
ACKs for top commit:
l0rinc:
code review ACK dd669f40b9
hebasto:
ACK dd669f40b9, tested Guix-built `bitcoind.exe` on Windows 11 Pro using WinDbg:
winterrdog:
utACK dd669f40b9
Tree-SHA512: 3632584f5f0612414a53ad6d868b9f832e8e7f1fad19f212f292172a4a6516f6e0055ec6ac8fbb71b22acdd003d09c9a5f97c0c15a137e1ad242580f997aca6f
9eba3aafa6 test: avoid undersized Boost.Test signal stacks (Lőrinc)
Pull request description:
**Problem:** Boost.Test can fail while an Alpine CI test binary is starting, before any tests run.
The required signal stack size depends on the runner's CPU features, while musl provides a fixed size.
**Fix:** Use the [regular process stack](https://www.boost.org/doc/libs/latest/libs/test/doc/html/boost_test/utf_reference/link_references/config_disable_alt_stack.html) for Boost.Test signal handling in both unit-test binaries.
Fixes#36026
ACKs for top commit:
maflcko:
lgtm ACK 9eba3aafa6
Tree-SHA512: 177a31ab325f4ebce0ad7e4679b171cd5a28787595bf0e117fc66731b8bf9157c4f5bef74cc6bb7651021bafb616fe5e9a25709abf30b7689d10ddd0fbd484cb
7f9c4e2928 doc: add release notes (ismaelsadeeq)
e18d392689 test: add mempool estimator i/o fuzz test (ismaelsadeeq)
970f02096d fees: persist mempool policy estimator data (ismaelsadeeq)
7dcb37989d fees: move fee_estimates.dat into fees directory (ismaelsadeeq)
0db2b69e6d rpc: add verbosity option to estimatesmartfee options (ismaelsadeeq)
06bb65730e fees: gate mempool estimates on recent block coverage (ismaelsadeeq)
cfe585df25 validation: emit block mempool removal signal from ConnectTip (ismaelsadeeq)
0d88558f95 fees: return mempool estimates when it's lower than block policy (ismaelsadeeq)
693b1351af fees: add caching to MemPoolFeeRateEstimator (ismaelsadeeq)
c9bb3df29f fees: add MemPoolFeeRateEstimator class (ismaelsadeeq)
9cacf677a9 rpc: add fee_rate_estimator option to estimatesmartfee (ismaelsadeeq)
ba6c61bbdd fees: add FeeRateEstimatorManager class (ismaelsadeeq)
2cb6b831e0 fees: add EstimateFeeRate and MaximumTarget to CBlockPolicyEstimator (ismaelsadeeq)
5adb2ab084 refactor: test block policy estimator directly (ismaelsadeeq)
9c8309a890 test: rename policy estimator tests to block policy estimator tests (ismaelsadeeq)
e3d5ef1b5f fees: move StringForBlockPolicyEstimateReason to block policy estimator (ismaelsadeeq)
74245c20e0 fees: split wallet and estimator fee reasons (ismaelsadeeq)
Pull request description:
This PR is another attempt to fix#27995 using a better approach.
For background and motivation, see #27995 and the discussion in the Delving Bitcoin post [Mempool Based Fee Estimation on Bitcoin Core](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703).
This PR is currently limited to using the mempool only to lower what is recommended by the Block Policy Estimator.
Accurate and safe fee estimation using the mempool is challenging. There are open questions about how to prevent mempool games that are theoretically possible for miners [(a variant of the Finney attack)](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/6).
This is one reason this PR uses the mempool only to lower the Block Policy Estimator result. The Block Policy Estimator itself is not gameable in this way, so the combined estimate is not susceptible to this attack increasing the returned feerate.
The underlying assumption is that, with the current tools and work done to make RBF and CPFP feasible and reliable (TRUC transaction relay, ephemeral anchors, cluster size 2 package RBF), underestimation is safer than overestimation. We now assume it is relatively easy to fee-bump later if a transaction does not confirm, whereas once a fee is overestimated there is no way to recover from that.
Another open question when using the mempool for fee estimation is how to account for incoming transaction inflow.
[Bitcoin Augur](https://github.com/block/bitcoin-augur) does this by using past inflow plus a constant expected inflow to predict future inflow. I find this unconvincing for fee estimation and potentially prone to more overestimation, as past conditions are not always representative of the future. See my [review of the Augur fee rate estimator and open questions](https://github.com/block/bitcoin-augur/issues/3).
This PR uses a much simpler approach based on current user behavior, similar to the widely used method employed by mempool.space: looking at the top block of the mempool and selecting a percentile feerate depending on whether the user is economical or conservative.
Empirical data from both myself and Clara Shikhelman shows that the 75th percentile feerate for economical users and the 50th percentile feerate for conservative users provide positive confirmation guarantees, hence this is what is used in this PR.
Parallel research by Rene Pickhardt and his student suggests that using the [average fee per byte of the block template performs well](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/12).
All of these are constants that can be adjusted. There is parallel work exploring these constants and running benchmarks across fee estimators to find a sweet spot.
See also work in LND, the [LND Budget Sweeper](https://delvingbitcoin.org/t/lnds-deadline-aware-budget-sweeper/1512), which applies this idea successfully. Their approach is to estimate fees initially with bitcoind, then increment gradually as the confirmation deadline approaches, using a fixed fee budget.
Historical data indicates that this PR's approach can [reduce overestimation quite significantly (~29%)](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/8).
This is particularly useful in scenarios where the Block Policy Estimator recommends a high feerate while the mempool is empty.
<img width="1800" height="1090" alt="56f3ba26c0184521c42bb82ec9d8c9f2224d4f8e" src="https://github.com/user-attachments/assets/c035c40c-8ece-42a7-b290-d29f1ac9bf4d" />
As seen in the image above, there is only one remaining unfixed case: when there is a sudden inflow of transactions and the feerate rises, the Block Policy Estimator takes time to reflect this. In that case, users will continue to see a low feerate estimate until it slowly updates. From the historical data linked above, [this occurs about ~26% of the time](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/8).
Overall, we observe a **73% success rate with 0% overestimation, and 26% underestimation** with this approach.
See https://bitcoincorefeerate.com/stats for recent running stats that have almost identical data.
This PR also includes refactors that enable this work. Rather than splitting the PR and implementing changes incrementally, I opted for an end-to-end implementation:
### 1. Refactors
* Split the mixed fee reason enum into separate wallet and block policy concepts. The wallet now has a `FeeReason` enum for why the wallet selected a fee rate (`FEE_RATE_ESTIMATOR`, `MEMPOOL_MIN`, `USER_SPECIFIED`, `FALLBACK`, `REQUIRED`), while the Block Policy Estimator uses `BlockPolicyEstimateReason` for its internal threshold details.
* Move `StringForBlockPolicyEstimateReason` to the Block Policy Estimator code, keeping the estimator-specific strings with the estimator.
* Move detailed Block Policy Estimator logging out of wallet transaction creation and into the estimator path. Wallet transaction creation now logs the selected fee and wallet fee reason instead of leaking estimator internals.
* Keep the wallet RPC `fee_reason` field name for compatibility, but update its meaning to report the wallet fee reason instead of the Block Policy Estimator's internal threshold reason.
* Rename policy estimator tests and files to block-policy-specific names where appropriate.
* Update Block Policy Estimator unit tests to be independent of the mempool and validation interface.
### 2. Introduce Mempool-Based Fee Estimator and Fee Estimator Manager
* Introduce `FeeRateEstimation` and `FeeRateEstimationError` as common estimator result types, avoiding new out-parameters for fee estimation results.
* Add `FeeRateEstimatorType` to identify the estimator that produced a result.
* Add `FeeRateEstimatorManager`, responsible for owning the Block Policy Estimator and Mempool Fee Rate Estimator.
* Update the node context to store a `std::unique_ptr` to `FeeRateEstimatorManager` instead of `CBlockPolicyEstimator`.
* Update `CBlockPolicyEstimator` to no longer subscribe directly to the validation interface; instead, `FeeRateEstimatorManager` subscribes and forwards relevant notifications.
* Add a mempool fee estimator that generates a block template when called, calculates a percentile feerate, and returns the 75th percentile for economical mode or the 50th percentile for conservative mode.
* When the selected estimate is below the node's fee floor, `estimatesmartfee` still returns at least the max of `mempoolminfee` and `minrelaytxfee`.
* Add caching to the mempool estimator so new estimates are generated at most every 7 seconds while the chain tip is unchanged, assuming enough [transactions have propagated](https://bitcoin.stackexchange.com/questions/125776/how-long-does-it-take-for-a-transaction-to-propagate-through-the-network/125777#125777) to make a meaningful difference.
This heuristic will likely be replaced by requesting block templates via the general-purpose block template cache proposed here: https://github.com/bitcoin/bitcoin/issues/33389
* Update `MempoolTransactionsRemovedForBlock` to receive the connected block as well as the transactions removed from the mempool.
* Track the weight of block transactions and mempool transactions removed due to block connection after each block connection.
This data is tracked for the last 6 mined blocks. A mempool feerate estimate is returned only when the ratio of mempool transaction weight removed due to block connection to block transaction weight is greater than 75% across the tracked window. This heuristic provides rough confidence that the node's mempool matches that of the majority of the hashrate. The 75% threshold is arbitrary and can be adjusted.
There is a caveat when transactions in the local mempool are consistently not mined by the network, as described in #27995 (e.g. due to filtering).
Accounting for these transactions during fee estimation is not necessary, as they should be evicted from the mempool itself (see #33510). Handling this again within fee estimation would be redundant.
* Persist statistics for the 6 most recent mined blocks to `fees/mempool_policy_estimator.dat` during periodic flushes and shutdown, so this data is available after restarts.
* Move Block Policy Estimator data from `fee_estimates.dat` to `fees/block_policy_estimates.dat`, migrating the legacy file during startup when needed.
* Add `fee_rate_estimator` to the `estimatesmartfee` options object. Supported values are `"none"` (default combined behavior), `"block_policy"` (use only the Block Policy Estimator), and `"mempool_policy"` (use only the Mempool Fee Rate Estimator). Unknown values are treated as `"none"`.
* Add `verbosity` to the `estimatesmartfee` options object. With `verbosity >= 2`, the RPC returns recent mempool health statistics.
* Expose the selected fee rate estimator in `estimatesmartfee` results when `fee_rate_estimator` is `"none"` and the estimate succeeds.
* Add unit, functional, and fuzz test coverage for the new estimator behavior, persistence, RPC options, and estimator I/O.
<details>
<summary>see example output</summary>
```bash
bitcoin-cli estimatesmartfee 1 economical '{"verbosity": 2, "fee_rate_estimator": "none"}'
```
```json
{
"feerate": 0.00002133,
"estimator": "mempool_policy",
"blocks": 2,
"mempool_health_statistics": [
{
"block_height": 927953,
"block_weight": 3991729,
"mempool_txs_weight": 3942409
}
]
}
```
</details>
ACKs for top commit:
willcl-ark:
reACK 7f9c4e2928
jsarenik:
Approach ACK 7f9c4e2
Tree-SHA512: c35b423eea0eb34524cf5ad07822c0ab8d53e2ab78965b58c8738044c61c77352184822360ed077a51bfbf83d0226d221e988f7156b1948023707c7e1fb31495
950bdb763e bench: Construct CTxOut and COutPoint in a single expression (Alexander Wiederin)
Pull request description:
Replaces field-by-field mutation of `CTxOut` and `COutPoint` in two bench files with brace initialisation, which requires the `size_t` conversions to be made explicit.
Noticed while looking at #35994, where switching the proposed fix-it to `{}` surfaces implicit narrowing conversions like these.
The constructed values are unchanged.
*Note: Only the sites where a conversion is involved are included in this PR; the remaining field-by-field construction in `bench/` would be covered by #35994's follow-ups.*
ACKs for top commit:
l0rinc:
code review ACK 950bdb763e
maflcko:
review ACK 950bdb763e🐯
Tree-SHA512: c664d41eeeb9241d381e5fa27689f18d9445a942055734463978498f60d0f2c495a0d7a58131493bb63267371bb6fe2d45ac124d910c919c8905322948d17a23
Replace separate member assignments with construction, using brace
initialization so the size_t to CAmount conversions have to be explicit.
Use uint32_t for the loop index feeding COutPoint::n, which avoids
conversion entirely.
ef501a63d9 consensus: document merkle mutation root invariant (Lőrinc)
Pull request description:
**Problem:** `ComputeMerkleRoot`'s optional mutation flag and the reasoning behind its per-level check are undocumented, and the behavior is only exercised indirectly by merkle_test through random duplications and old-vs-new comparisons, so a refactor could silently change it, as the discussions in #22046 and #28430 illustrate.
**Fix:** Document the flag on the function declaration, explain inside the inner loop why the mutation check runs at every tree level even after a duplicate is found, and add direct API coverage for the CVE-2012-2459 construction.
**Coverage check:** Both `merkle_test` and the new `merkle_test_mutated_return_value` would fail under a refactor that stops the outer reduction once mutation is detected, e.g.:
<details><summary>Hypothetical regression</summary>
```patch
diff --git a/src/consensus/merkle.cpp b/src/consensus/merkle.cpp
index dfa23cf897..40bc3f8efa 100644
--- a/src/consensus/merkle.cpp
+++ b/src/consensus/merkle.cpp
@@ -59,6 +59,7 @@ uint256 ComputeMerkleRoot(std::vector<uint256> hashes, bool* mutated) {
if (hashes[pos] == hashes[pos + 1]) mutation = true;
}
}
+ if (mutation) break;
if (hashes.size() & 1) {
hashes.push_back(hashes.back());
}
```
</details>
Fixes#28457
ACKs for top commit:
optout21:
reACK ef501a63d9
achow101:
ACK ef501a63d9
w0xlt:
reACK ef501a63d9
hodlinator:
ACK ef501a63d9
Tree-SHA512: 5a54eed071079a0a37333d5ba7c2d8eb81ae318ee4c84e15e3c050198daea6282453d4f7727b75f7dab90696b3d9bc946b8b33e6456a7f190b2297c15aca390c
Boost.Test registers a `SIGSTKSZ` alternate signal stack as each test binary starts, before any tests run.
On musl, this can be smaller than Linux's hardware-dependent minimum, causing `sigaltstack()` to fail with `ENOMEM` during setup.
Disable Boost.Test's alternate stack in both test entry points.
Signal handlers continue to use the regular process stack. Boost.Test can no longer report stack overflows.
Fixes#36026
Co-authored-by: MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz>
Persist MemPoolFeeRateEstimator's recent mined-block statistics
to fees/mempool_policy_estimator.dat and reload them at startup.
Without this, the mempool estimator starts cold after each restart
and treats the mempool as unhealthy until MEMPOOL_HEALTH_WINDOW_BLOCKS
blocks have been observed, causing the default combined estimatesmartfee
request to return a mempool fee rate estimator error.
Files with more stats than MEMPOOL_HEALTH_WINDOW_BLOCKS,
non-consecutive block heights, or a final block that does not match the
active chain tip are rejected on read, preserving the invariant that
loaded stats describe the current chain.
Add MempoolPolicyEstimatorPath(), pass the path through
FeeRateEstimatorManager, and flush both block-policy
and mempool-policy estimator files on interval and shutdown.
Move block policy fee estimates from fee_estimates.dat to
fees/block_policy_estimates.dat.
On startup, migrate the legacy file to the new path when only the legacy
file exists. If both files exist, keep the new file and remove the
legacy file.
Rename the block policy estimator args source files to the generic
estimator_args.{cpp,h} names and rename FeeestPath to
BlockPolicyFeeEstPath while the path helper is moved into the shared fee
estimator argument code.
Add a verbosity option to the existing estimatesmartfee options object.
The default verbosity remains 1.
When verbosity is at least 2 include mempool_health_statistics in the response.
The array reports the mined blocks tracked by the mempool fee rate estimator in
most-recent-first order, with each entry containing:
- block_height
- block_weight: total non-coinbase transaction weight in the block
- mempool_txs_weight: weight of transactions removed from our mempool
for that block
Expose these stats through the fee rate estimator manager so RPC users
can inspect the block coverage data used by the mempool health check.
Gate the mempool fee rate estimator on a coverage check: recent
connected blocks must be well represented by transactions removed from
our mempool.
Track per-block weight for the last MEMPOOL_HEALTH_WINDOW_BLOCKS blocks.
AddMinedBlockStats drops stats at or above a connected block's height
before appending it, and resets the window on a forward height gap so
tracked heights stay consecutive.
Only apply the coverage ratio once the window holds at least one block
of transactions; below that activity is too low for the ratio to be
meaningful, so treat the mempool as healthy.
Replace a boolean health check with a MempoolHealth enum so
EstimateFeeRate() can report whether estimation is unavailable because
too few recent blocks have been tracked (INSUFFICIENT_DATA) or because
recent blocks poorly represent the mempool (LOW_COVERAGE).
Return the removed mempool transaction info from
CTxMemPool::removeForBlock instead of dispatching the
MempoolTransactionsRemovedForBlock notification from the mempool.
Emit it from ConnectTip after mempool removal and before BlockConnected,
passing the connected block, the removed mempool transactions, and the
block height to the callback.
Because the signal now originates from ConnectTip, where the IBD state is
known, gate it on !IsInitialBlockDownload(): the notification is no longer
fired for blocks connected during initial block download or reindex, while
the mempool removal in removeForBlock still runs unconditionally. This keeps
fee rate estimators from recording blocks connected before the node is
synced.
