Persist MemPoolFeeRateEstimator's recent mined-block statistics
to fees/mempool_policy_estimator.dat and reload them at startup.
Without this, the mempool estimator starts cold after each restart
and treats the mempool as unhealthy until MEMPOOL_HEALTH_WINDOW_BLOCKS
blocks have been observed, causing the default combined estimatesmartfee
request to return a mempool fee rate estimator error.
Files with more stats than MEMPOOL_HEALTH_WINDOW_BLOCKS,
non-consecutive block heights, or a final block that does not match the
active chain tip are rejected on read, preserving the invariant that
loaded stats describe the current chain.
Add MempoolPolicyEstimatorPath(), pass the path through
FeeRateEstimatorManager, and flush both block-policy
and mempool-policy estimator files on interval and shutdown.
Move block policy fee estimates from fee_estimates.dat to
fees/block_policy_estimates.dat.
On startup, migrate the legacy file to the new path when only the legacy
file exists. If both files exist, keep the new file and remove the
legacy file.
Rename the block policy estimator args source files to the generic
estimator_args.{cpp,h} names and rename FeeestPath to
BlockPolicyFeeEstPath while the path helper is moved into the shared fee
estimator argument code.
Add a verbosity option to the existing estimatesmartfee options object.
The default verbosity remains 1.
When verbosity is at least 2 include mempool_health_statistics in the response.
The array reports the mined blocks tracked by the mempool fee rate estimator in
most-recent-first order, with each entry containing:
- block_height
- block_weight: total non-coinbase transaction weight in the block
- mempool_txs_weight: weight of transactions removed from our mempool
for that block
Expose these stats through the fee rate estimator manager so RPC users
can inspect the block coverage data used by the mempool health check.
Integrate MemPoolFeeRateEstimator into FeeRateEstimatorManager.
When both estimators succeed, select the lower of the block policy
and mempool estimates.
When either estimator fails, return its error instead of falling back
to the block policy estimate: if the mempool estimator cannot produce
an estimate, the combined estimate fails.
Callers that want a block-policy-only estimate can request it explicitly
via fee_rate_estimator option.
estimatesmartfee now emits the estimator field only for successful
manager-selected estimates.
Add a test that ensures estimatesmartfee returns the mempool fee rate
estimate when it is lower than the block policy estimate, and can request
the mempool policy estimator explicitly
Two wallet functional tests also need adjusting. When the mempool is
too sparse to fill its percentile buckets, MemPoolFeeRateEstimator
returns a relayable floor of max(min relay fee, mempool min fee), so in
regtest getFeeRateEstimate now returns the min relay fee where the
wallet previously had no estimate and fell back to a higher rate:
- wallet_taproot.py: the cleanup sendall used automatic fee estimation.
GetMinimumFeeRate previously fell back to the wallet fallback fee
(fallbackfee, 20 sat/vB in the test framework); it now uses the min
relay fee floor. At that lower feerate the wallet's underestimate of
the taproot script-path witness size drops the effective feerate
below min relay, so the transaction is rejected. Pin fee_rate=20 to
match the framework fallbackfee.
- wallet_bumpfee.py: GetDiscardRate() previously fell back to the
wallet discard rate (-discardfee); it now takes the minimum of that
and the estimate, so the min relay fee floor collapses the discard
rate down to the dust relay feerate. The lower discard rate reduces
the cost of change, so the ~614 sat leftover change in
test_dust_to_fee is now retained instead of being dropped to fee.
Rework the test to leave a sub-dust (20/270 sat) change that is
dropped regardless of the discard rate.
Co-authored-by: willcl-ark <will@256k1.dev>
Add a string fee_rate_estimator option (default "none") to
estimatesmartfee options. "block_policy" consults only the block
policy fee rate estimator, "none" uses the fee rate estimator
manager selected behaviour, and unknown values are treated as
"none". Unknown option keys are rejected.
Still only the block policy fee rate estimator, so the result is
unchanged; a subsequent commit will change the default behaviour.
Also adds GetFeeRateEstimate(FeeRateEstimatorType, target, conservative)
to FeeRateEstimatorManager so callers can query a single estimator by
type; NONE returns the manager-selected combined estimate.
8966352df3 doc: add release notes (ismaelsadeeq)
704a09fe71 test: ensure fee estimator provide fee rate estimate < 1 s/vb (ismaelsadeeq)
243e48cf49 fees: delete unused dummy field (ismaelsadeeq)
fc4fbda42a fees: bump fees file version (ismaelsadeeq)
b54dedcc85 fees: reduce `MIN_BUCKET_FEERATE` to 100 (ismaelsadeeq)
Pull request description:
This is a simple PR that updates the block policy estimator’s `MIN_BUCKET_FEERATE` constant to be 100, which is identical to the policy `DEFAULT_MIN_RELAY_TX_FEE`.
This change enables the block policy fee rate estimator to return sub-1 sat/vB fee rate estimates.
The change is correct because the estimator creates buckets of fee rates from
`MIN_BUCKET_FEERATE`,
`MIN_BUCKET_FEERATE` + `FEE_SPACING`,
`MIN_BUCKET_FEERATE` + `2 * FEE_SPACING`,
… up to `MAX_BUCKET_FEERATE`.
This means it will record sub-1 sat/vB fee rates in the buckets and may return them as a fee rate estimate when that bucket is the lowest one with sufficient transactions for a given target.
---
While touching this part of the fee estimator code, this PR got rid of the dummy value persisted in the file
ACKs for top commit:
achow101:
ACK 8966352df3
musaHaruna:
ACK [8966352](8966352df3)
polespinasa:
ACK 8966352df3
Tree-SHA512: 9424e0820fcbe124adf5e5d9101a8a2983ba802887101875b2d1856d700c8b563d7a6f6ca3e3db29cb939f786719603aadbf5480d59d55d0951ed1c0caa49868
9b75cfda4d test: retain the intended behavior of `feature_fee_estimation.py` nodes (ismaelsadeeq)
5c1236f04a test: fix incorrect subtest in `feature_fee_estimation.py` (ismaelsadeeq)
Pull request description:
Attempt to fix#32461
In the `estimatesmartfee` RPC, we return the maximum of the following: the feerate estimate for the target, `minrelaytxfee`, and `mempoolminfee`.
9a05b45da6/src/rpc/fees.cpp (L85)
The test `test_feerate_mempoolminfee`, originally introduced in ea31caf6b4, is incorrect.
It should calculate the fee rate ceiling by taking the maximum of the custom `minrelaytxfee`, `mempoolminfee`, and the highest fee rate observed during the test (`check_smart_estimates`). This is necessary because:
* There is no guarantee that the generated fee rates will exceed both `minrelaytxfee` and `mempoolminfee`.
* Users can start a node with custom fee settings.
Due to the non-deterministic nature of the `feature_fee_estimation.py` test, it often passes by chance. The randomly generated fees typically include a value higher than the custom `minrelaytxfee`, inadvertently hiding the issue.
Issue #32461 identified a random seeds that consistently fails the test because the generated fees never exceed the custom `minrelaytxfee`:
e.g
```
build/test/functional/feature_fee_estimation.py --random=3450808900320758527
```
This PR has two commits which :
* Correctly fixes the test by calculating the fee rate ceiling as the maximum of the node `minrelaytxfee`, `mempoolminfee`, and the highest seen fee rate, when verifying smart fee estimates.
* Improves the subtest name and comment for clarity.
* Restores the original test behavior by appending 4000 WU to the custom `blockmaxweight`.
ACKs for top commit:
achow101:
ACK 9b75cfda4d
glozow:
ACK 9b75cfda4d
theStack:
Light ACK 9b75cfda4d
Tree-SHA512: 0f7fb0496b50a399b58f6fb1afd95414fad454795fbc0046e22dfc54a2062ae0c519a12ebfeb6ad7ef547438868d99eca2351c0d19d0346adaadb500eff6f15f
- Increase block weight by 4000 for all nodes with custom -blockmaxweight.
Prior to this commit, we generated blocks with 4000 weight units less worth of transactions.
See https://github.com/bitcoin/bitcoin/issues/32461#issuecomment-2925282272 for details.
This commit fixes it by increasing the block weight by 4000.
- Update `check_smart_estimates` to calculate the fee rate ceiling
by taking the maximum of fees seen, minrelaytxfee, and mempoolminfee.
- Improve the subtest name and comments.
Since the previous commit, CTransaction object calls to the
methods `.rehash()` and `.calc_sha256()` are effectively no-ops
if the returned value is not used, so we can just remove them.
In the functional tests there are lots of cases where we assert != which
this new util will replace, we also are adding the imports and the new assertion
ec317bc44b test: update satoshi_round function (naiyoma)
Pull request description:
This PR refactors `satoshi_round` to accept different rounding modes and make rounding a required argument.
Continuation of https://github.com/bitcoin/bitcoin/pull/23225
ACKs for top commit:
maflcko:
review ACK ec317bc44b
achow101:
ACK ec317bc44b
AngusP:
ACK ec317bc44b
Tree-SHA512: 070f0aa6f66e58bff7412cae6b71f5f4ab8c718c7b5eeba4bb604fe22c6416a1ced0474294f504e1c28943ddc073104466b5944b43bae27e42bee5ca85afa468
Refactor satoshi_round function to accept different rounding modes.
Updated call site to use the revised `satoshi_round` function.
Co-authored-by: Kate Salazar <52637275+katesalazar@users.noreply.github.com>
25bf86a225 [test]: ensure `estimatesmartfee` default mode is `economical` (ismaelsadeeq)
41a2545046 [fees]: change `estimatesmartfee` default mode to `economical` (ismaelsadeeq)
Pull request description:
Fixes#30009
This PR changes the `estimatesmartfee` default mode to `economical`.
This was also suggested on IRC https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2024-04-26#1021609
- `conservative` mode: This is the `estimatesmartfee` RPC mode which considers a longer history of blocks. It potentially returns a higher fee rate and is more likely to be sufficient for the desired target, but it is not as responsive to short-term drops in the prevailing fee market.
- `economical` mode: This is the `estimatesmartfee` RPC mode where estimates are potentially lower and more responsive to short-term drops in the prevailing fee market.
Since users are likely to use the default mode, this change will reduce overestimation for many users. The conservative mode remains available for those who wish to opt-in.
For an in-depth analysis of how significantly the `conservative` mode overestimates, see
https://delvingbitcoin.org/t/bitcoind-policy-estimator-modes-analysis/964.
ACKs for top commit:
instagibbs:
reACK 25bf86a225
glozow:
ACK 25bf86a225
willcl-ark:
ACK 25bf86a225
Tree-SHA512: 78ebda667eb9c8f87dcc2f0e6c14968bd1de30358dc77a13611b186fb8427ad97d9f537bad6e32e0a1aa477ccd8c64fee4d41e19308ef3cb184ff1664e6ba8a6
Instead of passing the datadir and chain name to os.path.join, just use
the existing properties, which are the same.
-BEGIN VERIFY SCRIPT-
sed -i --regexp-extended 's|\.datadir, self\.chain, .wallets.|.wallets_path|g' $(git grep -l '\.datadir, self\.chain,')
sed -i --regexp-extended 's|\.datadir, self\.chain,|.chain_path,|g' $(git grep -l '\.datadir, self\.chain,')
-END VERIFY SCRIPT-
This commit adds tests to ensure that old fee_estimates.dat files
are not read and that fee_estimates are periodically flushed to the
fee_estimates.dat file.
Additionaly it tests the -regtestonly option -acceptstalefeeestimates.
feature_fee_estimation has a lot of loops that hit the RPC many times in
succession in order to setup scenarios. Using batched requests for these
can reduce the test's runtime without effecting the test's behavior.
This simplifies the code, and slightly speeds up the test.
Running `./test/functional/test_runner.py -j15 $(printf 'feature_fee_estimation %.0s' {1..15})`
on master 3 times gives:
- Before:
ALL | ✓ Passed | 788 s (accumulated)
ALL | ✓ Passed | 818 s (accumulated)
ALL | ✓ Passed | 873 s (accumulated)
- After:
ALL | ✓ Passed | 763 s (accumulated)
ALL | ✓ Passed | 798 s (accumulated)
ALL | ✓ Passed | 731 s (accumulated)
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
The previous diff touched most files in ./test/, so bump the headers to
avoid having to touch them again for a bump later.
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./test/
-END VERIFY SCRIPT-
This will provide better estimates which would be closer to fee paid in actual
transactions.
The test has also been changed such that when the node is restarted with a
high mempoolMinFee, the estimatesmartfee still returns a feeRate greater
than or equal to the mempoolMinFee, minRelayTxFee.(just like the feeRate of actual transactions)
