bd4b1524ea init: do not count file descriptors for HTTPServer if -server=0 (Matthew Zipkin)
b08662060d init: account for maximum file descriptors needed by HTTP (Matthew Zipkin)
cc2acebefb http: configure simultaneous connection limit with -rpcmaxconnections (Matthew Zipkin)
b3d6d2d1a7 http: limit connected clients to 16 (Matthew Zipkin)
86651d8197 scripted-diff: Rename nUserBind, nBind, nMaxConnections to snake_case (Matthew Zipkin)
Pull request description:
Introduces a new configuration option `-rpcmaxconnections` with default value `16`. This is used to limit the number of simultaneous `HTTPClient` connected to the `HTTPServer`. When the limit is reached, new pending connections remain queued in the kernel's socket buffer. Those connections have complete TCP handshakes with the kernel but do not occupy any application memory.
The previous libevent-based HTTP server had no limit on connections but it did have a limit on the kernel socket queue:
e7ff4ef2b4/http.c (L3510)
```c
if (listen(fd, 128) == -1) {
```
The current HTTP server, like the p2p server, uses a platform constant here:
b6becf3534/src/httpserver.cpp (L743)
(on my macOS `SOMAXCONN` is `128` but on my Debian machine it's `4096`)
The default of 16 was chosen as a reasonable upper bound for single-user RPC use cases. Systems designed to handle more simultaneous HTTP connections than this (previously relying on the absence of a limit) can adjust the setting.
## File descriptors
Because of the connection limit, we can now account for the maximum number of file descriptors needed by the HTTP server. This addresses several issues (#11368#11322 maybe #27732) that could have been fixed by a PR waiting in vain for a libevent release (#27731).
## Bonus performance improvement
The new limit is managed in a loop that drains the kernel's socket queue with `accept()`. All pending connections from the queue (up to the limit) are processed in one single call to `SocketHandlerListening()`. The previous code would only accept one connection from the queue on each I/O loop tick, with a `SELECT_TIMEOUT` (50ms) sleep between each.
ACKs for top commit:
fjahr:
tACK bd4b1524ea
janb84:
ACK bd4b1524ea
winterrdog:
tested ACK bd4b1524ea
hodlinator:
Concept ACK bd4b1524ea
willcl-ark:
ACK bd4b1524ea
Tree-SHA512: 2ef7a96da4d7037c7343ec0ea03fda5bb55d10c2a071fce4929141297515923b203d3d338dbcb6599849768f52aa3c9da509fb5d1d6f7c574a1d2034ea2a9e74
5be248341a bugfix: compare real chunk weight against block weight limit (ismaelsadeeq)
fc98790869 test: `TestChunkBlockLimits` uses incorrect weight for comparison (ismaelsadeeq)
Pull request description:
Partially fixes#35596
When assembling a block template, `BlockAssembler::addChunks()` adds chunks of transactions until the block is close to being full. For each chunk, `TestChunkBlockLimits()` checks both the weight and the sigop-cost limits before the chunk is included.
The weight check compared the chunk's **sigops-adjusted** weight against `block_max_weight`:
```cpp
if (nBlockWeight + chunk_feerate.size >= m_options.block_max_weight) {
return false;
}
```
Whereas `nBlockWeight` accumulates the actual chunk weight.
A chunk whose sigop-adjusted weight exceeds the actual weight can be wrongly skipped even though the block sigop limit is enforced independently on the next line, and that could pass. Those chunks pay higher fees, so this could potentially cause miners to needlessly forfeit some fees revenue.
This PR fixes this by passing the chunk's real weight (sum of `GetTxWeight()`, accumulated in the same loop that already sums sigop cost) to `TestChunkBlockLimits()`. The separate sigop-cost check is unchanged.
- The first commit adds `TestSigOpsAdjustedWeightChunkLimit`: it builds one sigop-dense transaction sized to fit by real weight but not by adjusted weight, and asserts that the tx is skipped and only the coinbase is mined.
- The second commit applies the fix and flips the assertion to show the transaction is now included.
ACKs for top commit:
pablomartin4btc:
Code Review ACK 5be248341a
sedited:
ACK 5be248341a
Tree-SHA512: b3fe9bfaa6d83d713d0243c0fc0d0fb8e68e1060bf6d606e43d9a52bd1ec07e42c561a1ba3426f180903726826c4a664bb131ddc5160354a96e3454d538fbf6c
73a94b4545 psbt: avoid aborting on invalid MuSig2 derivations (Lőrinc)
e3d1e75a51 test: characterize MuSig2 derivation aborts (Lőrinc)
Pull request description:
**Problem:** A PSBT may contain MuSig2 derivation metadata with a hardened child index or a path that derives to a different key.
The hardened index aborts during public derivation, while the mismatched key aborts at the result assertion.
`analyzepsbt`, `finalizepsbt`, and `descriptorprocesspsbt` all reach this code without a wallet.
Even the read-only `analyzepsbt` can force a co-signer service to restart its node after unexpected input.
**Fix:** Return failure when a MuSig2 derivation path contains a hardened child index, and skip only the current aggregate when the path derives to a different key so another matching aggregate can still be tried.
This follows [#35154](https://github.com/bitcoin/bitcoin/pull/35154), with the related contributions credited in the commits.
ACKs for top commit:
jeanpablojp:
ACK 73a94b4545
achow101:
ACK 73a94b4545
andrewtoth:
ACK 73a94b4545
Tree-SHA512: d8e28c5a4184154a4427c644ce62423cbcccdc3d82a6293f36fe99055fa04714598bc92c43b526fbcc7c99b231140669c2d0f1b853999d7dc33f949564c90504
Flush the chainstate at the current tip and drain its notification before registering any index.
This lets each end-of-sync `Commit()` persist the pre-crash height and keeps the setup callback out of the simulated crash window.
Replace the TODO-marked `0` expectation with the captured tip height.
The check runs before background sync, so rebuilding cannot hide a missing checkpoint.
`index_unclean_shutdown` previously checked only that each index could reopen and start background sync after the simulated crash.
An empty index at height 0 satisfies both checks, so the test could pass without preserving any pre-crash checkpoint.
Assert the current reopened height before background sync to make the false positive explicit.
dd669f40b9 util: set os-level thread names on Windows (ViniciusCestarii)
Pull request description:
Update SetThreadName to set os-level thread names on Windows too.
This is useful for debugging-ergonomics on Windows. Threads currently show up unnamed in debuggers, crash dumps on Windows and mismatch what's documented under https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#threads.
Tested with the mingw cross build running on Windows 11, print from WinDbg:
<img width="713" height="631" alt="image" src="https://github.com/user-attachments/assets/05e03383-c9b1-4e6b-91f3-9088b2fc7e90" />
ACKs for top commit:
l0rinc:
code review ACK dd669f40b9
hebasto:
ACK dd669f40b9, tested Guix-built `bitcoind.exe` on Windows 11 Pro using WinDbg:
winterrdog:
utACK dd669f40b9
Tree-SHA512: 3632584f5f0612414a53ad6d868b9f832e8e7f1fad19f212f292172a4a6516f6e0055ec6ac8fbb71b22acdd003d09c9a5f97c0c15a137e1ad242580f997aca6f
9eba3aafa6 test: avoid undersized Boost.Test signal stacks (Lőrinc)
Pull request description:
**Problem:** Boost.Test can fail while an Alpine CI test binary is starting, before any tests run.
The required signal stack size depends on the runner's CPU features, while musl provides a fixed size.
**Fix:** Use the [regular process stack](https://www.boost.org/doc/libs/latest/libs/test/doc/html/boost_test/utf_reference/link_references/config_disable_alt_stack.html) for Boost.Test signal handling in both unit-test binaries.
Fixes#36026
ACKs for top commit:
maflcko:
lgtm ACK 9eba3aafa6
Tree-SHA512: 177a31ab325f4ebce0ad7e4679b171cd5a28787595bf0e117fc66731b8bf9157c4f5bef74cc6bb7651021bafb616fe5e9a25709abf30b7689d10ddd0fbd484cb
7f9c4e2928 doc: add release notes (ismaelsadeeq)
e18d392689 test: add mempool estimator i/o fuzz test (ismaelsadeeq)
970f02096d fees: persist mempool policy estimator data (ismaelsadeeq)
7dcb37989d fees: move fee_estimates.dat into fees directory (ismaelsadeeq)
0db2b69e6d rpc: add verbosity option to estimatesmartfee options (ismaelsadeeq)
06bb65730e fees: gate mempool estimates on recent block coverage (ismaelsadeeq)
cfe585df25 validation: emit block mempool removal signal from ConnectTip (ismaelsadeeq)
0d88558f95 fees: return mempool estimates when it's lower than block policy (ismaelsadeeq)
693b1351af fees: add caching to MemPoolFeeRateEstimator (ismaelsadeeq)
c9bb3df29f fees: add MemPoolFeeRateEstimator class (ismaelsadeeq)
9cacf677a9 rpc: add fee_rate_estimator option to estimatesmartfee (ismaelsadeeq)
ba6c61bbdd fees: add FeeRateEstimatorManager class (ismaelsadeeq)
2cb6b831e0 fees: add EstimateFeeRate and MaximumTarget to CBlockPolicyEstimator (ismaelsadeeq)
5adb2ab084 refactor: test block policy estimator directly (ismaelsadeeq)
9c8309a890 test: rename policy estimator tests to block policy estimator tests (ismaelsadeeq)
e3d5ef1b5f fees: move StringForBlockPolicyEstimateReason to block policy estimator (ismaelsadeeq)
74245c20e0 fees: split wallet and estimator fee reasons (ismaelsadeeq)
Pull request description:
This PR is another attempt to fix#27995 using a better approach.
For background and motivation, see #27995 and the discussion in the Delving Bitcoin post [Mempool Based Fee Estimation on Bitcoin Core](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703).
This PR is currently limited to using the mempool only to lower what is recommended by the Block Policy Estimator.
Accurate and safe fee estimation using the mempool is challenging. There are open questions about how to prevent mempool games that are theoretically possible for miners [(a variant of the Finney attack)](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/6).
This is one reason this PR uses the mempool only to lower the Block Policy Estimator result. The Block Policy Estimator itself is not gameable in this way, so the combined estimate is not susceptible to this attack increasing the returned feerate.
The underlying assumption is that, with the current tools and work done to make RBF and CPFP feasible and reliable (TRUC transaction relay, ephemeral anchors, cluster size 2 package RBF), underestimation is safer than overestimation. We now assume it is relatively easy to fee-bump later if a transaction does not confirm, whereas once a fee is overestimated there is no way to recover from that.
Another open question when using the mempool for fee estimation is how to account for incoming transaction inflow.
[Bitcoin Augur](https://github.com/block/bitcoin-augur) does this by using past inflow plus a constant expected inflow to predict future inflow. I find this unconvincing for fee estimation and potentially prone to more overestimation, as past conditions are not always representative of the future. See my [review of the Augur fee rate estimator and open questions](https://github.com/block/bitcoin-augur/issues/3).
This PR uses a much simpler approach based on current user behavior, similar to the widely used method employed by mempool.space: looking at the top block of the mempool and selecting a percentile feerate depending on whether the user is economical or conservative.
Empirical data from both myself and Clara Shikhelman shows that the 75th percentile feerate for economical users and the 50th percentile feerate for conservative users provide positive confirmation guarantees, hence this is what is used in this PR.
Parallel research by Rene Pickhardt and his student suggests that using the [average fee per byte of the block template performs well](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/12).
All of these are constants that can be adjusted. There is parallel work exploring these constants and running benchmarks across fee estimators to find a sweet spot.
See also work in LND, the [LND Budget Sweeper](https://delvingbitcoin.org/t/lnds-deadline-aware-budget-sweeper/1512), which applies this idea successfully. Their approach is to estimate fees initially with bitcoind, then increment gradually as the confirmation deadline approaches, using a fixed fee budget.
Historical data indicates that this PR's approach can [reduce overestimation quite significantly (~29%)](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/8).
This is particularly useful in scenarios where the Block Policy Estimator recommends a high feerate while the mempool is empty.
<img width="1800" height="1090" alt="56f3ba26c0184521c42bb82ec9d8c9f2224d4f8e" src="https://github.com/user-attachments/assets/c035c40c-8ece-42a7-b290-d29f1ac9bf4d" />
As seen in the image above, there is only one remaining unfixed case: when there is a sudden inflow of transactions and the feerate rises, the Block Policy Estimator takes time to reflect this. In that case, users will continue to see a low feerate estimate until it slowly updates. From the historical data linked above, [this occurs about ~26% of the time](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/8).
Overall, we observe a **73% success rate with 0% overestimation, and 26% underestimation** with this approach.
See https://bitcoincorefeerate.com/stats for recent running stats that have almost identical data.
This PR also includes refactors that enable this work. Rather than splitting the PR and implementing changes incrementally, I opted for an end-to-end implementation:
### 1. Refactors
* Split the mixed fee reason enum into separate wallet and block policy concepts. The wallet now has a `FeeReason` enum for why the wallet selected a fee rate (`FEE_RATE_ESTIMATOR`, `MEMPOOL_MIN`, `USER_SPECIFIED`, `FALLBACK`, `REQUIRED`), while the Block Policy Estimator uses `BlockPolicyEstimateReason` for its internal threshold details.
* Move `StringForBlockPolicyEstimateReason` to the Block Policy Estimator code, keeping the estimator-specific strings with the estimator.
* Move detailed Block Policy Estimator logging out of wallet transaction creation and into the estimator path. Wallet transaction creation now logs the selected fee and wallet fee reason instead of leaking estimator internals.
* Keep the wallet RPC `fee_reason` field name for compatibility, but update its meaning to report the wallet fee reason instead of the Block Policy Estimator's internal threshold reason.
* Rename policy estimator tests and files to block-policy-specific names where appropriate.
* Update Block Policy Estimator unit tests to be independent of the mempool and validation interface.
### 2. Introduce Mempool-Based Fee Estimator and Fee Estimator Manager
* Introduce `FeeRateEstimation` and `FeeRateEstimationError` as common estimator result types, avoiding new out-parameters for fee estimation results.
* Add `FeeRateEstimatorType` to identify the estimator that produced a result.
* Add `FeeRateEstimatorManager`, responsible for owning the Block Policy Estimator and Mempool Fee Rate Estimator.
* Update the node context to store a `std::unique_ptr` to `FeeRateEstimatorManager` instead of `CBlockPolicyEstimator`.
* Update `CBlockPolicyEstimator` to no longer subscribe directly to the validation interface; instead, `FeeRateEstimatorManager` subscribes and forwards relevant notifications.
* Add a mempool fee estimator that generates a block template when called, calculates a percentile feerate, and returns the 75th percentile for economical mode or the 50th percentile for conservative mode.
* When the selected estimate is below the node's fee floor, `estimatesmartfee` still returns at least the max of `mempoolminfee` and `minrelaytxfee`.
* Add caching to the mempool estimator so new estimates are generated at most every 7 seconds while the chain tip is unchanged, assuming enough [transactions have propagated](https://bitcoin.stackexchange.com/questions/125776/how-long-does-it-take-for-a-transaction-to-propagate-through-the-network/125777#125777) to make a meaningful difference.
This heuristic will likely be replaced by requesting block templates via the general-purpose block template cache proposed here: https://github.com/bitcoin/bitcoin/issues/33389
* Update `MempoolTransactionsRemovedForBlock` to receive the connected block as well as the transactions removed from the mempool.
* Track the weight of block transactions and mempool transactions removed due to block connection after each block connection.
This data is tracked for the last 6 mined blocks. A mempool feerate estimate is returned only when the ratio of mempool transaction weight removed due to block connection to block transaction weight is greater than 75% across the tracked window. This heuristic provides rough confidence that the node's mempool matches that of the majority of the hashrate. The 75% threshold is arbitrary and can be adjusted.
There is a caveat when transactions in the local mempool are consistently not mined by the network, as described in #27995 (e.g. due to filtering).
Accounting for these transactions during fee estimation is not necessary, as they should be evicted from the mempool itself (see #33510). Handling this again within fee estimation would be redundant.
* Persist statistics for the 6 most recent mined blocks to `fees/mempool_policy_estimator.dat` during periodic flushes and shutdown, so this data is available after restarts.
* Move Block Policy Estimator data from `fee_estimates.dat` to `fees/block_policy_estimates.dat`, migrating the legacy file during startup when needed.
* Add `fee_rate_estimator` to the `estimatesmartfee` options object. Supported values are `"none"` (default combined behavior), `"block_policy"` (use only the Block Policy Estimator), and `"mempool_policy"` (use only the Mempool Fee Rate Estimator). Unknown values are treated as `"none"`.
* Add `verbosity` to the `estimatesmartfee` options object. With `verbosity >= 2`, the RPC returns recent mempool health statistics.
* Expose the selected fee rate estimator in `estimatesmartfee` results when `fee_rate_estimator` is `"none"` and the estimate succeeds.
* Add unit, functional, and fuzz test coverage for the new estimator behavior, persistence, RPC options, and estimator I/O.
<details>
<summary>see example output</summary>
```bash
bitcoin-cli estimatesmartfee 1 economical '{"verbosity": 2, "fee_rate_estimator": "none"}'
```
```json
{
"feerate": 0.00002133,
"estimator": "mempool_policy",
"blocks": 2,
"mempool_health_statistics": [
{
"block_height": 927953,
"block_weight": 3991729,
"mempool_txs_weight": 3942409
}
]
}
```
</details>
ACKs for top commit:
willcl-ark:
reACK 7f9c4e2928
jsarenik:
Approach ACK 7f9c4e2
Tree-SHA512: c35b423eea0eb34524cf5ad07822c0ab8d53e2ab78965b58c8738044c61c77352184822360ed077a51bfbf83d0226d221e988f7156b1948023707c7e1fb31495
950bdb763e bench: Construct CTxOut and COutPoint in a single expression (Alexander Wiederin)
Pull request description:
Replaces field-by-field mutation of `CTxOut` and `COutPoint` in two bench files with brace initialisation, which requires the `size_t` conversions to be made explicit.
Noticed while looking at #35994, where switching the proposed fix-it to `{}` surfaces implicit narrowing conversions like these.
The constructed values are unchanged.
*Note: Only the sites where a conversion is involved are included in this PR; the remaining field-by-field construction in `bench/` would be covered by #35994's follow-ups.*
ACKs for top commit:
l0rinc:
code review ACK 950bdb763e
maflcko:
review ACK 950bdb763e🐯
Tree-SHA512: c664d41eeeb9241d381e5fa27689f18d9445a942055734463978498f60d0f2c495a0d7a58131493bb63267371bb6fe2d45ac124d910c919c8905322948d17a23
Replace separate member assignments with construction, using brace
initialization so the size_t to CAmount conversions have to be explicit.
Use uint32_t for the loop index feeding COutPoint::n, which avoids
conversion entirely.
ef501a63d9 consensus: document merkle mutation root invariant (Lőrinc)
Pull request description:
**Problem:** `ComputeMerkleRoot`'s optional mutation flag and the reasoning behind its per-level check are undocumented, and the behavior is only exercised indirectly by merkle_test through random duplications and old-vs-new comparisons, so a refactor could silently change it, as the discussions in #22046 and #28430 illustrate.
**Fix:** Document the flag on the function declaration, explain inside the inner loop why the mutation check runs at every tree level even after a duplicate is found, and add direct API coverage for the CVE-2012-2459 construction.
**Coverage check:** Both `merkle_test` and the new `merkle_test_mutated_return_value` would fail under a refactor that stops the outer reduction once mutation is detected, e.g.:
<details><summary>Hypothetical regression</summary>
```patch
diff --git a/src/consensus/merkle.cpp b/src/consensus/merkle.cpp
index dfa23cf897..40bc3f8efa 100644
--- a/src/consensus/merkle.cpp
+++ b/src/consensus/merkle.cpp
@@ -59,6 +59,7 @@ uint256 ComputeMerkleRoot(std::vector<uint256> hashes, bool* mutated) {
if (hashes[pos] == hashes[pos + 1]) mutation = true;
}
}
+ if (mutation) break;
if (hashes.size() & 1) {
hashes.push_back(hashes.back());
}
```
</details>
Fixes#28457
ACKs for top commit:
optout21:
reACK ef501a63d9
achow101:
ACK ef501a63d9
w0xlt:
reACK ef501a63d9
hodlinator:
ACK ef501a63d9
Tree-SHA512: 5a54eed071079a0a37333d5ba7c2d8eb81ae318ee4c84e15e3c050198daea6282453d4f7727b75f7dab90696b3d9bc946b8b33e6456a7f190b2297c15aca390c
Boost.Test registers a `SIGSTKSZ` alternate signal stack as each test binary starts, before any tests run.
On musl, this can be smaller than Linux's hardware-dependent minimum, causing `sigaltstack()` to fail with `ENOMEM` during setup.
Disable Boost.Test's alternate stack in both test entry points.
Signal handlers continue to use the regular process stack. Boost.Test can no longer report stack overflows.
Fixes#36026
Co-authored-by: MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz>
`ParseOutputs` iterates a `UniValue` object's keys and looks up each value by key.
Each lookup scans the key vector from the beginning, making the lookup work quadratic.
Walk the parallel key and value vectors together to avoid repeated scans.
This preserves output order and validation behavior.
Persist MemPoolFeeRateEstimator's recent mined-block statistics
to fees/mempool_policy_estimator.dat and reload them at startup.
Without this, the mempool estimator starts cold after each restart
and treats the mempool as unhealthy until MEMPOOL_HEALTH_WINDOW_BLOCKS
blocks have been observed, causing the default combined estimatesmartfee
request to return a mempool fee rate estimator error.
Files with more stats than MEMPOOL_HEALTH_WINDOW_BLOCKS,
non-consecutive block heights, or a final block that does not match the
active chain tip are rejected on read, preserving the invariant that
loaded stats describe the current chain.
Add MempoolPolicyEstimatorPath(), pass the path through
FeeRateEstimatorManager, and flush both block-policy
and mempool-policy estimator files on interval and shutdown.
Move block policy fee estimates from fee_estimates.dat to
fees/block_policy_estimates.dat.
On startup, migrate the legacy file to the new path when only the legacy
file exists. If both files exist, keep the new file and remove the
legacy file.
Rename the block policy estimator args source files to the generic
estimator_args.{cpp,h} names and rename FeeestPath to
BlockPolicyFeeEstPath while the path helper is moved into the shared fee
estimator argument code.
Add a verbosity option to the existing estimatesmartfee options object.
The default verbosity remains 1.
When verbosity is at least 2 include mempool_health_statistics in the response.
The array reports the mined blocks tracked by the mempool fee rate estimator in
most-recent-first order, with each entry containing:
- block_height
- block_weight: total non-coinbase transaction weight in the block
- mempool_txs_weight: weight of transactions removed from our mempool
for that block
Expose these stats through the fee rate estimator manager so RPC users
can inspect the block coverage data used by the mempool health check.
Gate the mempool fee rate estimator on a coverage check: recent
connected blocks must be well represented by transactions removed from
our mempool.
Track per-block weight for the last MEMPOOL_HEALTH_WINDOW_BLOCKS blocks.
AddMinedBlockStats drops stats at or above a connected block's height
before appending it, and resets the window on a forward height gap so
tracked heights stay consecutive.
Only apply the coverage ratio once the window holds at least one block
of transactions; below that activity is too low for the ratio to be
meaningful, so treat the mempool as healthy.
Replace a boolean health check with a MempoolHealth enum so
EstimateFeeRate() can report whether estimation is unavailable because
too few recent blocks have been tracked (INSUFFICIENT_DATA) or because
recent blocks poorly represent the mempool (LOW_COVERAGE).
Return the removed mempool transaction info from
CTxMemPool::removeForBlock instead of dispatching the
MempoolTransactionsRemovedForBlock notification from the mempool.
Emit it from ConnectTip after mempool removal and before BlockConnected,
passing the connected block, the removed mempool transactions, and the
block height to the callback.
Because the signal now originates from ConnectTip, where the IBD state is
known, gate it on !IsInitialBlockDownload(): the notification is no longer
fired for blocks connected during initial block download or reindex, while
the mempool removal in removeForBlock still runs unconditionally. This keeps
fee rate estimators from recording blocks connected before the node is
synced.
Integrate MemPoolFeeRateEstimator into FeeRateEstimatorManager.
When both estimators succeed, select the lower of the block policy
and mempool estimates.
When either estimator fails, return its error instead of falling back
to the block policy estimate: if the mempool estimator cannot produce
an estimate, the combined estimate fails.
Callers that want a block-policy-only estimate can request it explicitly
via fee_rate_estimator option.
estimatesmartfee now emits the estimator field only for successful
manager-selected estimates.
Add a test that ensures estimatesmartfee returns the mempool fee rate
estimate when it is lower than the block policy estimate, and can request
the mempool policy estimator explicitly
Two wallet functional tests also need adjusting. When the mempool is
too sparse to fill its percentile buckets, MemPoolFeeRateEstimator
returns a relayable floor of max(min relay fee, mempool min fee), so in
regtest getFeeRateEstimate now returns the min relay fee where the
wallet previously had no estimate and fell back to a higher rate:
- wallet_taproot.py: the cleanup sendall used automatic fee estimation.
GetMinimumFeeRate previously fell back to the wallet fallback fee
(fallbackfee, 20 sat/vB in the test framework); it now uses the min
relay fee floor. At that lower feerate the wallet's underestimate of
the taproot script-path witness size drops the effective feerate
below min relay, so the transaction is rejected. Pin fee_rate=20 to
match the framework fallbackfee.
- wallet_bumpfee.py: GetDiscardRate() previously fell back to the
wallet discard rate (-discardfee); it now takes the minimum of that
and the estimate, so the min relay fee floor collapses the discard
rate down to the dust relay feerate. The lower discard rate reduces
the cost of change, so the ~614 sat leftover change in
test_dust_to_fee is now retained instead of being dropped to fee.
Rework the test to leave a sub-dust (20/270 sat) change that is
dropped regardless of the discard rate.
Co-authored-by: willcl-ark <will@256k1.dev>
fa8762da62 build: ci/doc win64-cross build via nix (MarcoFalke)
fafe7205cc doc: Clarify that cygwin/msys2 are not tested/supported (MarcoFalke)
Pull request description:
Release cross-builds to win64 are done in guix. There are also docs to use Debian/Ubuntu for those cross-builds and this approach is used in CI. However, there are many problems:
* The CI is intended to mirror the guix build, but often it is not possible to find the major versions used for mingw and GCC in the guix build in the `apt` packages for an LTS distro.
* Users on older distro releases may lack released bugfixes, such as 8e06daa36d in mingw 13 (e.g. Debian Trixie with mingw 12, https://packages.debian.org/trixie/mingw-w64-x86-64-dev).
* When using the UCRT variant of the build, this uncovers bugs such as https://bugs.launchpad.net/ubuntu/+source/mingw-w64/+bug/2106420 or https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1121403.
So add a way to use nix to do the cross build. This allows to closer mimic the guix build.
Also, clarify that cygwin/msys2 are not tested/supported.
ACKs for top commit:
willcl-ark:
reACK fa8762da62
hebasto:
re-ACK fa8762da62.
Tree-SHA512: de94f8bc4bb6ed9a75352cd909aa227c5954e336bf9b14969d7412b5cedfbe6cd6b2d8b476d5b1b0bcfc93bdb32fc229015855082350c8c507189e34b9b3ef3f
Cache previous mempool fee rate estimates.
Cached estimates are tagged with the chain tip they were computed on
(the template's hashPrevBlock). They are only served while they are
not stale and the chain tip has not changed. This avoids generating
block templates too often.
The estimator lock is not held while building a block template, so
concurrent callers may duplicate estimation work; the tip tag keeps
stale results out of the cache.
Co-authored-by: willcl-ark <will@256k1.dev>
Add MemPoolFeeRateEstimator, which calls Bitcoin Core's block
assembler with the mempool and chainstate to build a block template and
use its chunk fee rates for fee rate estimation.
Add CalculateMaxWeightPercentiles to return the 50th and 75th
percentile chunk feerates by cumulative block weight. If sparse,
EstimateFeeRate uses the higher of the minimum relay fee rate and the
current mempool minimum fee rate.
The 50th percentile is returned as the conservative estimate, and the
75th percentile as the economical estimate.
Wire MemPoolFeeRateEstimator into FeeRateEstimatorManager and add
FeeRateEstimatorType::MEMPOOL_POLICY for result attribution.
Add unit tests for the mempool fee rate estimator and fee estimator
string conversions, plus fuzz coverage for the string conversions.
Co-authored-by: willcl-ark <will@256k1.dev>
5a431c957d qt: Update `src/qt/locale/bitcoin_en.ts` translation source file (Hennadii Stepanov)
Pull request description:
This PR follows our [Release Process](4df077d7cd/doc/release-process.md).
It is required for the translation string freeze, as https://github.com/bitcoin-core/gui/pull/957 introduced a new translatable string after the soft translation string freeze.
Steps to reproduce the diff:
```console
cmake --preset dev-mode
cmake --build build_dev_mode --target translate
```
ACKs for top commit:
polespinasa:
ACK 5a431c957d
Tree-SHA512: 89ee1721c5091fa0deb11f9c014515846b8af781c03d3aaed2ff757e475927d5e9bd5a16a496de0f5269e6371f2ad27d5065ef656c9376793332a05aa707272b
fef99e6563 qt: fix out-of-bounds read in RPCParseCommandLine on empty command (sayed nabhan)
Pull request description:
When a console line has no command name (it starts with `)`, or is `()`, `(`, or `,`), RPCParseCommandLine reaches the command-execution branch while the current argument frame is still empty, so `stack.back()[0]` reads out of bounds and the argument list built from `stack.back().begin() + 1` to `end()` is an invalid iterator range (throws std::length_error in practice, UBSan flags the null-pointer reference otherwise).
The `(` branch already guards the frame with `stack.back().size() > 0`, so I add the same check to the `)`/newline branch and the empty frame is skipped. To be clear, `(` alone isn't safe on master either: it fails via the `\n` branch, not via the `(` branch itself (the state there isn't `STATE_ARGUMENT`), and `,` alone fails the same way.
Since there's no command to run in any of these cases, the parser now returns `false` so the console reports an invalid command line, consistent with other fully-invalid input like a bare `'` or `"`, rather than silently ignoring it.
Regression cases added to rpcNestedTests for `)`, `()`, `(` and `,` (all abort on master without the guard), plus `getblockchaininfo)` which stays tolerated.
ACKs for top commit:
hebasto:
ACK fef99e6563, tested on Ubuntu 26.04.
Tree-SHA512: 15822a0525402878483d5b2d0fe7b9e27916e514c1a8ad4a697f11b1e5cfe33f5fcd1f089efefb976bcf0d84cdf5166a58a070593924c2d4c2e1f8224d06590f
15e5c35c45 doc: Correct comments after HTTPRequest::m_client was changed from shared to weak pointer (Hodlinator)
Pull request description:
There were lingering comments from when `HTTPRequest::m_client` was a `shared_ptr`.
Prompted by https://github.com/bitcoin/bitcoin/pull/36007#issuecomment-5329937262.
Follow-up to #36007.
ACKs for top commit:
winterrdog:
ACK 15e5c35c45
Tree-SHA512: 021427258e46d2a6a19f5304167c957084b174a4f1fd0e223605f9b06ee771225d023e5f546db9045e1bfdfd5558f933e54d9ee76e5393949612628be9815bc4
fa0fe212f5 test: [refactor] Properly use BOOST_CHECK_EXCEPTION (MarcoFalke)
Pull request description:
The exception checking in unit tests is partly verbose, fragile, inconsistent and thus confusing.
Fix all those issues by using `BOOST_CHECK_EXCEPTION` consistently:
* The test code is less bloated and follows a standard pattern; Extra state and dead code like `exceptionThrown = false;` or `BOOST_CHECK(0)` can be removed.
* The checks are more strict, because they use `HasReason{...}` or a similar predicate.
ACKs for top commit:
l0rinc:
ACK fa0fe212f5
janb84:
ACK fa0fe212f5
jonatack:
Light ACK fa0fe212f5
Tree-SHA512: f3a9abfe02988299f6d2604643feb630e078c2177287899d4fcc191802536d41d09e6365d81d0184bf3533583987969cb1b85ab3e112e7364fdb5b85a3405cb7
joinpsbts collects the global xpubs of all the joined PSBTs into
merged_psbt, but returns a separately constructed shuffled_psbt into
which only the inputs, outputs, and unknown fields are copied. The
collected PSBT_GLOBAL_XPUB records are silently dropped, and
PSBT_GLOBAL_PROPRIETARY records are not collected at all.
The xpub collection was added in #17034, which was written against a
joinpsbts that still returned merged_psbt, but was merged after #16512
had introduced the shuffled_psbt rebuild, so the collected xpubs have
never reached the result.
Shuffle the inputs and outputs of merged_psbt in place instead of
rebuilding a new PSBT, so that all global data is preserved, and union
the global proprietary records in the merge loop, matching the
combinepsbt behavior from #34893.
6d387af562 psbt: remove write-only global xpub tracking set (Thomas)
3b7051c7e3 test: check combinepsbt with conflicting global xpub origins (Thomas)
7c632c0e2a psbt: avoid duplicate global xpub keys when merging (Thomas)
Pull request description:
Global xpubs are stored in a map of key origin to set of xpubs, while the serialization writes one record per xpub, keyed by the xpub. `Merge` unions the map origin-by-origin, so when the combined PSBTs provide different key origins for the same xpub, the result serializes the same `PSBT_GLOBAL_XPUB` key twice. BIP 174 declares PSBTs with duplicate keys invalid and the deserializer rejects them, so `combinepsbt` returns a PSBT that no RPC can parse again. This affects all releases since the merge loop was added in #17034 (v23.0).
<details><summary>Reproduction on master</summary>
The PSBTs share the unsigned transaction and xpub, and differ only in the master fingerprint of the global xpub record (`00000000` vs `11111111`):
```
$ A=cHNidP8BADwCAAAAAaqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqAAAAAAD/////AQAAAAAAAAAAAAAAAABPAQQ1h88AAAAAAAAAAACHPf+BwC9SViP9H+UWfqw6VaBJ3j0xS7Qu4if/7TfVCAM5o2ATMBWX2u9B++WToCzFE9C1VSfsLfEFDi6P9JyFwgQAAAAAAAAA
$ B=cHNidP8BADwCAAAAAaqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqAAAAAAD/////AQAAAAAAAAAAAAAAAABPAQQ1h88AAAAAAAAAAACHPf+BwC9SViP9H+UWfqw6VaBJ3j0xS7Qu4if/7TfVCAM5o2ATMBWX2u9B++WToCzFE9C1VSfsLfEFDi6P9JyFwgQRERERAAAA
$ bitcoin-cli -regtest decodepsbt "$(bitcoin-cli -regtest combinepsbt "[\"$A\",\"$B\"]")"
error code: -22
error message:
TX decode failed Duplicate Key, global key "01043587cf00...9c85c2" already provided: iostream error
```
</details>
Deduplicate by xpub when merging, keeping the origin that is already present: BIP 174 lets the Combiner "pick arbitrarily when conflicts occur", and conflicting unknown and proprietary records are already resolved the same way. The logic is shared between `combinepsbt` and `joinpsbts` through a new `MergeGlobalXPubs` helper. The second commit adds a test that fails on master with the error above, and the last commit removes the `global_xpubs` tracking set in `Unserialize`, write-only since the generic duplicate key check introduced in #21283 (1e2d146b47) replaced the explicit one.
Note: the xpub loop in `joinpsbts` currently has no observable effect, since the collected xpubs never reach the returned PSBT. My #35516 fixes that, so this PR should land first: on its own, #35516 would make the same duplicate key issue reachable through `joinpsbts`, while with the shared helper in place it never becomes reachable. I will rebase #35516 on top afterwards.
ACKs for top commit:
Bicaru20:
tACK 6d387af562.
achow101:
ACK 6d387af562
winterrdog:
tACK 6d387af562
Tree-SHA512: e2a9e02617eeec22a9240d7cf9386ee880a5f3639b143df7de4d8ea3e7b808f8c123f0b0410ff4a22e9a564bd86b2335a5c4aa3b2281af47d111484a6f1fd108
cf36df070b Wallet: Check crypter return values (benthecarman)
b76afff274 Wallet: Use unsigned KDF iteration count (benthecarman)
Pull request description:
CMasterKey::nDeriveIterations values are deserialized from wallet files
as unsigned 32-bit integers, but key derivation narrowed the count to a
signed int. A count above INT_MAX became negative in the conversion, and
the derivation loop counter then overflowed, which is undefined
behavior.
Keep the count unsigned through the derivation path to match the
serialized type, and add tests for zero and normal counts.
Also check key-derivation calibration failures and validate calculated
iteration counts before conversion. Keep the output master key unchanged
until derivation and encryption succeed, and mark fallible crypter
methods as [[nodiscard]].
ACKs for top commit:
l0rinc:
code review ACK cf36df070b
achow101:
ACK cf36df070b
Tree-SHA512: 95d5db2655fef8ca499af7da0f0258b4bee90975468286cb88e424257c4c5bf36407d2b2816b538e2ac3227e2a3a75c8d775211c65fe3ec6d12189da0b05fba1
777aee77d1 refactor: deduplicate keypath element parsing (pythcoiner)
7d8fddfba2 refactor: define BIP32_HARDENED and BIP32_UNHARDENED constants (pythcoiner)
Pull request description:
The codebase used raw `0x80000000` (and implicit `0`) as the bip32 hardened / unhardened flag.
`ParseHDKeypath` and `ParseKeyPathNum` were two separate parsers for BIP32 keypath elements, #32784 aligned their rules (both accept ' and h as hardened marker and reject indexes > 0x7FFFFFFF), but the parsing logic itself was still duplicated.
This PR:
- Define `BIP32_HARDENED_FLAG` / `BIP32_UNHARDENED_FLAG` constants to replace magic `0x80000000` and `0` literals.
- Add `ParseKeyPathElement` as bip32 parsing util and use it consistantly in `ParseHDKeyPath` and the descriptor keypath parser.
ACKs for top commit:
Sjors:
ACK 777aee77d1
achow101:
ACK 777aee77d1
Tree-SHA512: fb096eef82bb5a90baa7de41f5562b935665ee4b64c41586cf89e06c5633be44063a35a89d01a1432c8e63222d94bd06840bafc2a837f252013e317de0f5837a
465bca734e contrib: reject divergent verify-commits history (Lőrinc)
b3d1dca338 contrib: fail on verify-commits ancestry errors (Lőrinc)
Pull request description:
**Problem:** `verify-commits.py` checks a Git commit's history for trusted signatures and tree hashes back to configured roots.
The documented workflow runs this check after fetching a commit and before checkout, proceeding only when the script succeeds.
A commit that is an ancestor of a configured root is intentionally accepted without checking earlier history.
The script also takes this success path after Git errors or for divergent commits, even though neither establishes that relationship.
**Fix:** Require Git to prove the ancestor relationship before taking this success path.
**Reproducers:** Each commit can be validated manually.
<details><summary>Manual reproducer: Git error</summary>
Run this on `master` and at this PR's head:
```bash
contrib/verify-commits/verify-commits.py 0000000000000000000000000000000000000000 && echo ❌ || echo ✅
```
`master` exits successfully without verifying the missing commit, while the PR head rejects the Git error.
</details>
<details><summary>Manual reproducer: divergent history</summary>
On `master` and at this PR's head, create an unreferenced sibling of the trusted root and run the verifier:
```bash
root=$(head -n1 contrib/verify-commits/trusted-git-root)
divergent_commit=$(git commit-tree "$root^{tree}" -p "$root^" -m 'divergent commit')
contrib/verify-commits/verify-commits.py "$divergent_commit" && echo ❌ || echo ✅
```
`master` exits successfully without verifying the sibling commit, while the PR head rejects divergent history.
</details>
This issue was also found and disclosed responsibly by the Red Team 🟥.
ACKs for top commit:
151henry151:
tACK 465bca734e
jeanpablojp:
tACK 465bca734e
achow101:
ACK 465bca734e
sedited:
ACK 465bca734e
maflcko:
review ACK 465bca734e🥜
Tree-SHA512: 72b8cd9902d881e59a1d99fda8e5d511806826fa27c05a2c21a7d2eb62b2a5a0b1b6bdc67e8d19d57f9171278f4858fd019eb7890b960df0475ba4713683f0ac
