mirror of
https://github.com/bitcoin/bitcoin.git
synced 2026-09-11 21:20:39 +02:00
7dcb7f09ed
7f9c4e2928doc: add release notes (ismaelsadeeq)e18d392689test: add mempool estimator i/o fuzz test (ismaelsadeeq)970f02096dfees: persist mempool policy estimator data (ismaelsadeeq)7dcb37989dfees: move fee_estimates.dat into fees directory (ismaelsadeeq)0db2b69e6drpc: add verbosity option to estimatesmartfee options (ismaelsadeeq)06bb65730efees: gate mempool estimates on recent block coverage (ismaelsadeeq)cfe585df25validation: emit block mempool removal signal from ConnectTip (ismaelsadeeq)0d88558f95fees: return mempool estimates when it's lower than block policy (ismaelsadeeq)693b1351affees: add caching to MemPoolFeeRateEstimator (ismaelsadeeq)c9bb3df29ffees: add MemPoolFeeRateEstimator class (ismaelsadeeq)9cacf677a9rpc: add fee_rate_estimator option to estimatesmartfee (ismaelsadeeq)ba6c61bbddfees: add FeeRateEstimatorManager class (ismaelsadeeq)2cb6b831e0fees: add EstimateFeeRate and MaximumTarget to CBlockPolicyEstimator (ismaelsadeeq)5adb2ab084refactor: test block policy estimator directly (ismaelsadeeq)9c8309a890test: rename policy estimator tests to block policy estimator tests (ismaelsadeeq)e3d5ef1b5ffees: move StringForBlockPolicyEstimateReason to block policy estimator (ismaelsadeeq)74245c20e0fees: split wallet and estimator fee reasons (ismaelsadeeq) Pull request description: This PR is another attempt to fix #27995 using a better approach. For background and motivation, see #27995 and the discussion in the Delving Bitcoin post [Mempool Based Fee Estimation on Bitcoin Core](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703). This PR is currently limited to using the mempool only to lower what is recommended by the Block Policy Estimator. Accurate and safe fee estimation using the mempool is challenging. There are open questions about how to prevent mempool games that are theoretically possible for miners [(a variant of the Finney attack)](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/6). This is one reason this PR uses the mempool only to lower the Block Policy Estimator result. The Block Policy Estimator itself is not gameable in this way, so the combined estimate is not susceptible to this attack increasing the returned feerate. The underlying assumption is that, with the current tools and work done to make RBF and CPFP feasible and reliable (TRUC transaction relay, ephemeral anchors, cluster size 2 package RBF), underestimation is safer than overestimation. We now assume it is relatively easy to fee-bump later if a transaction does not confirm, whereas once a fee is overestimated there is no way to recover from that. Another open question when using the mempool for fee estimation is how to account for incoming transaction inflow. [Bitcoin Augur](https://github.com/block/bitcoin-augur) does this by using past inflow plus a constant expected inflow to predict future inflow. I find this unconvincing for fee estimation and potentially prone to more overestimation, as past conditions are not always representative of the future. See my [review of the Augur fee rate estimator and open questions](https://github.com/block/bitcoin-augur/issues/3). This PR uses a much simpler approach based on current user behavior, similar to the widely used method employed by mempool.space: looking at the top block of the mempool and selecting a percentile feerate depending on whether the user is economical or conservative. Empirical data from both myself and Clara Shikhelman shows that the 75th percentile feerate for economical users and the 50th percentile feerate for conservative users provide positive confirmation guarantees, hence this is what is used in this PR. Parallel research by Rene Pickhardt and his student suggests that using the [average fee per byte of the block template performs well](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/12). All of these are constants that can be adjusted. There is parallel work exploring these constants and running benchmarks across fee estimators to find a sweet spot. See also work in LND, the [LND Budget Sweeper](https://delvingbitcoin.org/t/lnds-deadline-aware-budget-sweeper/1512), which applies this idea successfully. Their approach is to estimate fees initially with bitcoind, then increment gradually as the confirmation deadline approaches, using a fixed fee budget. Historical data indicates that this PR's approach can [reduce overestimation quite significantly (~29%)](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/8). This is particularly useful in scenarios where the Block Policy Estimator recommends a high feerate while the mempool is empty. <img width="1800" height="1090" alt="56f3ba26c0184521c42bb82ec9d8c9f2224d4f8e" src="https://github.com/user-attachments/assets/c035c40c-8ece-42a7-b290-d29f1ac9bf4d" /> As seen in the image above, there is only one remaining unfixed case: when there is a sudden inflow of transactions and the feerate rises, the Block Policy Estimator takes time to reflect this. In that case, users will continue to see a low feerate estimate until it slowly updates. From the historical data linked above, [this occurs about ~26% of the time](https://delvingbitcoin.org/t/mempool-based-fee-estimation-on-bitcoin-core/703/8). Overall, we observe a **73% success rate with 0% overestimation, and 26% underestimation** with this approach. See https://bitcoincorefeerate.com/stats for recent running stats that have almost identical data. This PR also includes refactors that enable this work. Rather than splitting the PR and implementing changes incrementally, I opted for an end-to-end implementation: ### 1. Refactors * Split the mixed fee reason enum into separate wallet and block policy concepts. The wallet now has a `FeeReason` enum for why the wallet selected a fee rate (`FEE_RATE_ESTIMATOR`, `MEMPOOL_MIN`, `USER_SPECIFIED`, `FALLBACK`, `REQUIRED`), while the Block Policy Estimator uses `BlockPolicyEstimateReason` for its internal threshold details. * Move `StringForBlockPolicyEstimateReason` to the Block Policy Estimator code, keeping the estimator-specific strings with the estimator. * Move detailed Block Policy Estimator logging out of wallet transaction creation and into the estimator path. Wallet transaction creation now logs the selected fee and wallet fee reason instead of leaking estimator internals. * Keep the wallet RPC `fee_reason` field name for compatibility, but update its meaning to report the wallet fee reason instead of the Block Policy Estimator's internal threshold reason. * Rename policy estimator tests and files to block-policy-specific names where appropriate. * Update Block Policy Estimator unit tests to be independent of the mempool and validation interface. ### 2. Introduce Mempool-Based Fee Estimator and Fee Estimator Manager * Introduce `FeeRateEstimation` and `FeeRateEstimationError` as common estimator result types, avoiding new out-parameters for fee estimation results. * Add `FeeRateEstimatorType` to identify the estimator that produced a result. * Add `FeeRateEstimatorManager`, responsible for owning the Block Policy Estimator and Mempool Fee Rate Estimator. * Update the node context to store a `std::unique_ptr` to `FeeRateEstimatorManager` instead of `CBlockPolicyEstimator`. * Update `CBlockPolicyEstimator` to no longer subscribe directly to the validation interface; instead, `FeeRateEstimatorManager` subscribes and forwards relevant notifications. * Add a mempool fee estimator that generates a block template when called, calculates a percentile feerate, and returns the 75th percentile for economical mode or the 50th percentile for conservative mode. * When the selected estimate is below the node's fee floor, `estimatesmartfee` still returns at least the max of `mempoolminfee` and `minrelaytxfee`. * Add caching to the mempool estimator so new estimates are generated at most every 7 seconds while the chain tip is unchanged, assuming enough [transactions have propagated](https://bitcoin.stackexchange.com/questions/125776/how-long-does-it-take-for-a-transaction-to-propagate-through-the-network/125777#125777) to make a meaningful difference. This heuristic will likely be replaced by requesting block templates via the general-purpose block template cache proposed here: https://github.com/bitcoin/bitcoin/issues/33389 * Update `MempoolTransactionsRemovedForBlock` to receive the connected block as well as the transactions removed from the mempool. * Track the weight of block transactions and mempool transactions removed due to block connection after each block connection. This data is tracked for the last 6 mined blocks. A mempool feerate estimate is returned only when the ratio of mempool transaction weight removed due to block connection to block transaction weight is greater than 75% across the tracked window. This heuristic provides rough confidence that the node's mempool matches that of the majority of the hashrate. The 75% threshold is arbitrary and can be adjusted. There is a caveat when transactions in the local mempool are consistently not mined by the network, as described in #27995 (e.g. due to filtering). Accounting for these transactions during fee estimation is not necessary, as they should be evicted from the mempool itself (see #33510). Handling this again within fee estimation would be redundant. * Persist statistics for the 6 most recent mined blocks to `fees/mempool_policy_estimator.dat` during periodic flushes and shutdown, so this data is available after restarts. * Move Block Policy Estimator data from `fee_estimates.dat` to `fees/block_policy_estimates.dat`, migrating the legacy file during startup when needed. * Add `fee_rate_estimator` to the `estimatesmartfee` options object. Supported values are `"none"` (default combined behavior), `"block_policy"` (use only the Block Policy Estimator), and `"mempool_policy"` (use only the Mempool Fee Rate Estimator). Unknown values are treated as `"none"`. * Add `verbosity` to the `estimatesmartfee` options object. With `verbosity >= 2`, the RPC returns recent mempool health statistics. * Expose the selected fee rate estimator in `estimatesmartfee` results when `fee_rate_estimator` is `"none"` and the estimate succeeds. * Add unit, functional, and fuzz test coverage for the new estimator behavior, persistence, RPC options, and estimator I/O. <details> <summary>see example output</summary> ```bash bitcoin-cli estimatesmartfee 1 economical '{"verbosity": 2, "fee_rate_estimator": "none"}' ``` ```json { "feerate": 0.00002133, "estimator": "mempool_policy", "blocks": 2, "mempool_health_statistics": [ { "block_height": 927953, "block_weight": 3991729, "mempool_txs_weight": 3942409 } ] } ``` </details> ACKs for top commit: willcl-ark: reACK7f9c4e2928jsarenik: Approach ACK7f9c4e2Tree-SHA512: c35b423eea0eb34524cf5ad07822c0ab8d53e2ab78965b58c8738044c61c77352184822360ed077a51bfbf83d0226d221e988f7156b1948023707c7e1fb31495
Bitcoin Core
Setup
Bitcoin Core is the original Bitcoin client and it builds the backbone of the network. It downloads and, by default, stores the entire history of Bitcoin transactions, which requires several hundred gigabytes or more of disk space. Depending on the speed of your computer and network connection, the synchronization process can take anywhere from a few hours to several days or more.
To download Bitcoin Core, visit bitcoincore.org.
Running
The following are some helpful notes on how to run Bitcoin Core on your native platform.
Unix
Unpack the files into a directory and run:
bin/bitcoin-qt(GUI) orbin/bitcoind(headless)bin/bitcoin(wrapper command)
The bitcoin command supports subcommands like bitcoin gui, bitcoin node, and bitcoin rpc exposing different functionality. Subcommands can be listed with bitcoin help.
Windows
Unpack the files into a directory, and then run bitcoin-qt.exe.
macOS
Drag Bitcoin Core to your applications folder, and then run Bitcoin Core.
Need Help?
- See the documentation at the Bitcoin Wiki for help and more information.
- Ask for help on Bitcoin StackExchange.
- Ask for help on #bitcoin on Libera Chat. If you don't have an IRC client, you can use web.libera.chat.
- Ask for help on the BitcoinTalk forums, in the Technical Support board.
Building
The following are developer notes on how to build Bitcoin Core on your native platform. They are not complete guides, but include notes on the necessary libraries, compile flags, etc.
- Dependencies
- macOS Build Notes
- Unix Build Notes
- Windows Build Notes
- FreeBSD Build Notes
- OpenBSD Build Notes
- NetBSD Build Notes
Development
The Bitcoin repo's root README contains relevant information on the development process and automated testing.
- Developer Notes
- Productivity Notes
- Release Process
- Source Code Documentation (External Link)
- Translation Process
- Translation Strings Policy
- JSON-RPC Interface
- Unauthenticated REST Interface
- BIPS
- Dnsseed Policy
- Benchmarking
- Internal Design Docs
Resources
- Discuss on the BitcoinTalk forums, in the Development & Technical Discussion board.
- Discuss project-specific development on #bitcoin-core-dev on Libera Chat. If you don't have an IRC client, you can use web.libera.chat.
Miscellaneous
- Assets Attribution
- bitcoin.conf Configuration File
- CJDNS Support
- Files
- Fuzz-testing
- I2P Support
- Init Scripts (systemd/upstart/openrc)
- Managing Wallets
- Multisig Tutorial
- Offline Signing Tutorial
- P2P bad ports definition and list
- PSBT support
- Reduce Memory
- Reduce Traffic
- Tor Support
- Transaction Relay Policy
- ZMQ
License
Distributed under the MIT software license.