3281824ecf fuzz: prevent invalid `FRESH` entries and surface `BatchWrite` errors (Lőrinc)
780f460635 fuzz: avoid invalid `AddCoin` overwrites (Lőrinc)
d7e0d510f2 fuzz: make `AddCoins` query view for overwrites (Lőrinc)
b8fa6f0f70 util: introduce `TrySub` to prevent unsigned underflow (Lőrinc)
Pull request description:
### Problem
This is an alternative approach to #34647, fixes#34645.
### Fix
First, add `CheckedSub` and use it for decrements of `m_dirty_count` and `cachedCoinsUsage`, so unsigned underflows turn into immediate failures instead of silently wrapping and only failing later.
<details><summary>Assertion `j <= i' failed.</summary>
```bash
util/overflow.h:44 T CheckedSub(const T, const U) [T = unsigned long, U = bool]: Assertion `j <= i' failed.
==72817== ERROR: libFuzzer: deadly signal
#0 0x556e9225eab5 in __sanitizer_print_stack_trace (/mnt/my_storage/bitcoin/build_fuzz/bin/fuzz+0x191dab5) (BuildId: d77c4d5f9dfd38ea06fab463f49341735205e109)
#1 0x556e921acafc in fuzzer::PrintStackTrace() (/mnt/my_storage/bitcoin/build_fuzz/bin/fuzz+0x186bafc) (BuildId: d77c4d5f9dfd38ea06fab463f49341735205e109)
#2 0x556e92191bb7 in fuzzer::Fuzzer::CrashCallback() (/mnt/my_storage/bitcoin/build_fuzz/bin/fuzz+0x1850bb7) (BuildId: d77c4d5f9dfd38ea06fab463f49341735205e109)
#3 0x7164cfc458cf (/lib/x86_64-linux-gnu/libc.so.6+0x458cf) (BuildId: ae7440bbdce614e0e79280c3b2e45b1df44e639c)
#4 0x7164cfca49bb in __pthread_kill_implementation nptl/pthread_kill.c:43:17
#5 0x7164cfca49bb in __pthread_kill_internal nptl/pthread_kill.c:89:10
#6 0x7164cfca49bb in pthread_kill nptl/pthread_kill.c:100:10
#7 0x7164cfc4579d in raise signal/../sysdeps/posix/raise.c:26:13
#8 0x7164cfc288cc in abort stdlib/abort.c:73:3
#9 0x556e92f9d591 in assertion_fail(std::source_location const&, std::basic_string_view<char, std::char_traits<char>>) /mnt/my_storage/bitcoin/src/util/check.cpp:41:5
#10 0x556e9250daf0 in bool&& inline_assertion_check<false, bool>(bool&&, std::source_location const&, std::basic_string_view<char, std::char_traits<char>>) /mnt/my_storage/bitcoin/src/util/check.h:90:13
#11 0x556e9250daf0 in unsigned long CheckedSub<unsigned long, bool>(unsigned long, bool) /mnt/my_storage/bitcoin/src/util/overflow.h:44:5
#12 0x556e9250daf0 in CoinsViewCacheCursor::NextAndMaybeErase(std::pair<COutPoint const, CCoinsCacheEntry>&) /mnt/my_storage/bitcoin/src/coins.h:282:25
#13 0x556e92507eb2 in (anonymous namespace)::MutationGuardCoinsViewCache::BatchWrite(CoinsViewCacheCursor&, uint256 const&) /mnt/my_storage/bitcoin/src/test/fuzz/coins_view.cpp:90:75
#14 0x556e92c17a2b in CCoinsViewCache::Flush(bool) /mnt/my_storage/bitcoin/src/coins.cpp:282:11
#15 0x556e924fb732 in TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_1::operator()() const /mnt/my_storage/bitcoin/src/test/fuzz/coins_view.cpp:135:34
#16 0x556e924fb732 in unsigned long CallOneOf<TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_0, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_1, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_2, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_3, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_4, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_5, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_6, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_7, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_8, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_9, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_10, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_11>(FuzzedDataProvider&, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_0, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_1, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_2, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_3, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_4, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_5, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_6, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_7, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_8, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_9, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_10, TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool)::$_11) /mnt/my_storage/bitcoin/src/test/fuzz/util.h:42:27
#17 0x556e924fb732 in TestCoinsView(FuzzedDataProvider&, CCoinsViewCache&, CCoinsView&, bool) /mnt/my_storage/bitcoin/src/test/fuzz/coins_view.cpp:114:9
#18 0x556e92503b0c in coins_view_overlay_fuzz_target(std::span<unsigned char const, 18446744073709551615ul>) /mnt/my_storage/bitcoin/src/test/fuzz/coins_view.cpp:404:5
#19 0x556e92bcb7a5 in std::function<void (std::span<unsigned char const, 18446744073709551615ul>)>::operator()(std::span<unsigned char const, 18446744073709551615ul>) const /usr/lib/gcc/x86_64-linux-gnu/15/../../../../include/c++/15/bits/std_function.h:593:9
#20 0x556e92bcb7a5 in test_one_input(std::span<unsigned char const, 18446744073709551615ul>) /mnt/my_storage/bitcoin/src/test/fuzz/fuzz.cpp:88:5
#21 0x556e92bcb7a5 in LLVMFuzzerTestOneInput /mnt/my_storage/bitcoin/src/test/fuzz/fuzz.cpp:216:5
#22 0x556e9219318f in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/mnt/my_storage/bitcoin/build_fuzz/bin/fuzz+0x185218f) (BuildId: d77c4d5f9dfd38ea06fab463f49341735205e109)
#23 0x556e92192799 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) (/mnt/my_storage/bitcoin/build_fuzz/bin/fuzz+0x1851799) (BuildId: d77c4d5f9dfd38ea06fab463f49341735205e109)
#24 0x556e92194139 in fuzzer::Fuzzer::MutateAndTestOne() (/mnt/my_storage/bitcoin/build_fuzz/bin/fuzz+0x1853139) (BuildId: d77c4d5f9dfd38ea06fab463f49341735205e109)
#25 0x556e92194c95 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile>>&) (/mnt/my_storage/bitcoin/build_fuzz/bin/fuzz+0x1853c95) (BuildId: d77c4d5f9dfd38ea06fab463f49341735205e109)
#26 0x556e92181255 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/mnt/my_storage/bitcoin/build_fuzz/bin/fuzz+0x1840255) (BuildId: d77c4d5f9dfd38ea06fab463f49341735205e109)
#27 0x556e921ad696 in main (/mnt/my_storage/bitcoin/build_fuzz/bin/fuzz+0x186c696) (BuildId: d77c4d5f9dfd38ea06fab463f49341735205e109)
#28 0x7164cfc2a577 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#29 0x7164cfc2a63a in __libc_start_main csu/../csu/libc-start.c:360:3
#30 0x556e921757e4 in _start (/mnt/my_storage/bitcoin/build_fuzz/bin/fuzz+0x18347e4) (BuildId: d77c4d5f9dfd38ea06fab463f49341735205e109)
NOTE: libFuzzer has rudimentary signal handlers.
Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
MS: 2 PersAutoDict-CopyPart- DE: "\005\000"-; base unit: ecb626aff8724f0fdde38a0a6965718f2096d474
artifact_prefix='/tmp/fuzz_artifacts/'; Test unit written to /tmp/fuzz_artifacts/crash-1d19026c1a23f08bfe693fd684a56ce51187c6e5
./build_fuzz/bin/fuzz /tmp/fuzz_corpus/coins_view_overlay -max_total_time=3600 -rss_limit_mb=2560 -artifact_prefix=/tmp/fuzz_artifacts/ >fuzz-16.log 2>&1
```
</details>
The coins view fuzz targets can call `AddCoin`/`AddCoins` and construct `BatchWrite` cursors in ways that violate `CCoinsViewCache` caller contracts. These invalid states can trigger `BatchWrite` `std::logic_error` and can desync dirty-entry accounting (caught by `Assume(m_dirty_count == 0)` currently).
Make the fuzzer avoid generating invalid states instead of catching and resetting:
* Derive `AddCoin`’s `possible_overwrite` from `PeekCoin`, so `possible_overwrite=false` is only used when the outpoint is absent - similarly to 67c0d1798e/src/test/fuzz/coinscache_sim.cpp (L312-L317)
- Only use `AddCoins(check=false)` when we have confirmed the txid has no unspent outputs; otherwise fall back to `check=true` so `AddCoins` determines overwrites via the view.
- When constructing a `CoinsViewCacheCursor`, avoid setting `FRESH` when the parent already has an unspent coin, and ensure `FRESH` implies `DIRTY`.
### Fuzzing
The original error could be reproduced in ~10 minutes using `coins_view_overlay`. I ran the `coins_view`, `coins_view_db`, `coins_view_overlay`, and `coinscache_sim` fuzzers for this PR overnight and they didn't fail anymore.
ACKs for top commit:
achow101:
ACK 3281824ecf
sipa:
ACK 3281824ecf. Ran the 4 relevant fuzz tests for ~1 CPU day each. Will run more overnight.
andrewtoth:
ACK 3281824ecf
Tree-SHA512: b8155e8d21740eb7800e373c27a8a1457eb84468c24af879bac5a1ed251ade2aec99c34a350a31f2ebb74e41bb7380bf20214d38d14fe23310a43282d2434fb7
5cd57943b8 test: verify node state after restart in assumeutxo (Yash Bhutwala)
Pull request description:
## Summary
This PR replaces the TODO comment in `wallet_assumeutxo.py` (line 242) with actual test assertions that verify node and wallet behavior after a restart during assumeutxo background sync.
## Changes
The new tests verify:
- Two chainstates exist (background validation not complete)
- Background chainstate is still at `START_HEIGHT`
- Snapshot chainstate has synced to at least `PAUSE_HEIGHT`
- Wallets cannot be loaded after restart (expected behavior)
- Wallet backup from before snapshot height cannot be restored
## Motivation
During implementation, I discovered that **wallets cannot be loaded after a node restart during assumeutxo background sync**. This is expected behavior because:
- The wallet loading code checks if required blocks are available for rescanning
- During assumeutxo background sync, blocks before the snapshot are not available
- This applies to all wallets, including watch-only wallets created at the snapshot height
This is a valuable test addition because it documents this expected behavior and ensures it doesn't regress. Users should be aware that if they restart their node during assumeutxo background sync, they won't be able to load their wallets until the background sync completes.
## Related
refs #28648
Addresses the TODO comment that was originally added as part of the assumeutxo wallet test implementation.
ACKs for top commit:
Bicaru20:
Code review ACK 5cd57943b8
achow101:
ACK 5cd57943b8
fjahr:
ACK 5cd57943b8
sedited:
ACK 5cd57943b8
polespinasa:
code lgtm ACK 5cd57943b8
Tree-SHA512: 4a125c5247168da2bbf4d855b4150ca453bb5e4cce1a62e633ce5e43acdc2c58883a6a94dcc46b38f8b4c44206fe42cec4db151a76aded53d8ea433ea5eb2562
50cf6838e6 wallet: rpc: manpage: fix example missing `fee_rate` argument (SomberNight)
Pull request description:
The function signature for the `send` RPC is:
```
send [{"address":amount,...},{"data":"hex"},...] ( conf_target "estimate_mode" fee_rate options version )
```
The last example in the manpage is missing the `fee_rate` arg, but is trying to specify the `options` arg, by index. The parser confuses the intended `options` arg as the missing `fee_rate` arg.
See:
```
$ bitcoin-cli -rpcuser=doggman -rpcpassword=donkey -rpcport=18554 -regtest send '{"bcrt1qusm48zmlzwr32csxdw4ar7atw260h22c9ten9l": 0.1}' 1 economical '{"add_to_wallet": false, "inputs": [{"txid":"0b7e1a471dc948b7a6187936b16e6d7d9833629b2f9dd8a392eb89928f63aaad", "vout":0}]}'
error code: -8
error message:
Cannot specify both conf_target and fee_rate. Please provide either a confirmation target in blocks for automatic fee estimation, or an explicit fee rate.
```
vs
```
$ bitcoin-cli -rpcuser=doggman -rpcpassword=donkey -rpcport=18554 -regtest send '{"bcrt1qusm48zmlzwr32csxdw4ar7atw260h22c9ten9l": 0.1}' 1 economical null '{"add_to_wallet": false, "inputs": [{"txid":"0b7e1a471dc948b7a6187936b16e6d7d9833629b2f9dd8a392eb89928f63aaad", "vout":0}]}'
{
"psbt": "cHNidP8BAHECAAAAAa2qY4+SieuSo9idL5tiM5h9bW6xNnkYprdIyR1HGn4LAAAAAAD9////AkR2DwQAAAAAFgAUpLDwJu+wFRHLQAgKAb0psk7UVd2AlpgAAAAAABYAFOQ3U4t/E4cVYgZrq9H7q3K0+6lYAAAAAAABAIUCAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/////wQC4wMA/////wLIF6gEAAAAABYAFLMY1zihXrefAA0DA5nld4MCPjkrAAAAAAAAAAAmaiSqIant4vYcP3HR3v0/qZnfo2lTdVxpBol5mWK0i+vYNpdOjPkAAAAAAQEfyBeoBAAAAAAWABSzGNc4oV63nwANAwOZ5XeDAj45KwEIawJHMEQCIElTV4pbUrsPR9qHWcioowVv3QVWHizxwevfD0u/I8YyAiBCY3OzF81PSLM00h4ueQkehYuxDFZu7Jk51iejphKnnwEhA0VKdYVSyBpWoxBwTDOupB58Fi3mEBs+u+OOqEYVd2sZACICA98YLWyH7dBCfXVxe7woiLSTgV1mJN8Zc8KgZ77pVSg+GNBMeT5UAACAAQAAgAAAAIABAAAAbAAAAAAA",
"txid": "625b71b314a6ac4f738634e29dc007cd5edc0427c1ae96ab706d06a62910cea2",
"hex": "02000000000101adaa638f9289eb92a3d89d2f9b6233987d6d6eb1367918a6b748c91d471a7e0b0000000000fdffffff0244760f0400000000160014a4b0f026efb01511cb40080a01bd29b24ed455dd8096980000000000160014e437538b7f13871562066babd1fbab72b4fba9580247304402204953578a5b52bb0f47da8759c8a8a3056fdd05561e2cf1c1ebdf0f4bbf23c6320220426373b317cd4f48b334d21e2e79091e858bb10c566eec9939d627a3a612a79f012103454a758552c81a56a310704c33aea41e7c162de6101b3ebbe38ea84615776b1900000000",
"complete": true
}
```
ACKs for top commit:
svanstaa:
tACK 50cf6838e6
kannapoix:
Tested ACK 50cf6838e6
achow101:
ACK 50cf6838e6
rkrux:
tACK 50cf6838e6
theStack:
Tested ACK 50cf6838e6
Tree-SHA512: 499701729038cd863b612698098a73ce995589fc5ab08a2962f8edf1ff3cb3de6f7090e04722ca13ba7707a566fa3750ae549b6ad55750a3d01127eb6b94a79f
e8f8b74a46 test: index, improve txospenderindex_initial_sync() test code (furszy)
ac3bea07cd test: improve rpc_gettxspendingprevout.py code (furszy)
Pull request description:
Fixes#34637.
Was reviewing #34637 and, while reading the new txospender index
test code for the first time, found it could use some cleanups. Finding
stuff in there is harder than it should be due to the amount of dup code.
The first commit cleans up `rpc_gettxspendingprevout.py` by introducing
helper functions to avoid repeating the same dicts everywhere, using
for-loops instead of duplicating the same checks for each node, and
renaming variables to better reflect what they actually represent.
The second commit reorganizes `txospenderindex_initial_sync()`
moving index initialization after the test setup phase, since the index
doesn't participate in it anyway. It adds a post-sync check to catch
cases where `Sync()` aborted prematurely.
Note:
This is just a pre-work for deeper index changes I'm cooking.
ACKs for top commit:
achow101:
ACK e8f8b74a46
sedited:
Re-ACK e8f8b74a46
w0xlt:
reACK e8f8b74a46
Tree-SHA512: 3f7026712ab20a43f376afa28c683dcd5daec8ed1bbf1c36d7ec6bbf231f468d4de74efae4aa8295ff3afb83986286ccaf31c03b34e45fc9971652f064791ed0
The index is now initialized after the setup phase (chain generation
and txs creation), since it doesn't participate on it at all.
This improves readability and splits setup from what we actually
want to check.
This also adds a check after Sync() to verify the index best block hash
matches the tip, so we know it fully synced before checking the
processed data. This will help catching errors as Sync() could have
aborted prematurely.
As a happy side effect, the SyncWithValidationInterfaceQueue() call at
the end of the test is no longer needed and has been removed.
a28eedb8c2 ci: use LLVM 22 in sanitizer tasks (fanquake)
Pull request description:
Clang/LLVM 22 is out, and upstream apt issues seem to be resolved. Use it in the sanitizer related tasks in CI. Not changing tidy here (needs more changes).
ACKs for top commit:
maflcko:
lgtm ACK a28eedb8c2
hebasto:
ACK a28eedb8c2.
Tree-SHA512: 311cca7d1ebc5769812e63c1946e8bad1bd2da728bfe3a4b91182eca8c9a9036f7506aff4ea99665982f57c14e4523b674a8689a2390e30936c16ae2548bc759
The error was added in commit 1ea7e45a1f,
because there was an additional confusing `AssertionError: [node 0]
Error: no RPC connection` instead of just a single `FileNotFoundError:
[Errno 2] No such file or directory`.
This is no longer needed on current master.
Also, the test is incomplete, because it was just checking bitcoind and
bitcoin-cli, not any other missing binaries.
Also, after the previous commit, it would not work in combination with
--valgrind.
Instead of trying to make it complete, and work in all combinations,
just remove it, because the already existing error will be clear in any
case.
This can be tested via:
```sh
./test/get_previous_releases.py
mv releases releases_backup
# Confirm the test is skipped due to missing releases
./bld-cmake/test/functional/wallet_migration.py
# Confirm the test fails due to missing releases
./bld-cmake/test/functional/wallet_migration.py --previous-releases
mv releases_backup releases
mv ./releases/v28.2 ./releases/v28.2_backup
# Confirm the test fails with a single FileNotFoundError
./bld-cmake/test/functional/wallet_migration.py
mv ./releases/v28.2_backup ./releases/v28.2
# Confirm the test runs and passes
./bld-cmake/test/functional/wallet_migration.py
rm ./bld-cmake/bin/bitcoind
# Confirm the test fails with a single "No such file or directory",
# testing with and without --valgrind
./bld-cmake/test/functional/wallet_migration.py
./bld-cmake/test/functional/wallet_migration.py --valgrind
```
Prior to this commit, tool_bitcoin.py was failing:
```sh
$ ./bld-cmake/test/functional/tool_bitcoin.py --valgrind
TestFramework (ERROR): Unexpected exception
Traceback (most recent call last):
File "./test/functional/test_framework/test_framework.py", line 138, in main
self.setup()
~~~~~~~~~~^^
File "./test/functional/test_framework/test_framework.py", line 269, in setup
self.setup_network()
~~~~~~~~~~~~~~~~~~^^
File "./test/functional/tool_bitcoin.py", line 38, in setup_network
assert all(node.args[:len(node_argv)] == node_argv for node in self.nodes)
~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
```
This commit fixes this issue by running `bitcoin` under valgrind. Also,
it comes with other improvements:
* Drop the outdated valgrind 3.14 requirement, because there is no
distro that ships a version that old anymore.
* Drop the VALGRIND_SUPPRESSIONS_FILE env var handling, because it was
presumably never used since it was introduced. Also, the use-case
seems limited.
Review note:
The set_cmd_args was ignoring the --valgrind test option.
In theory, this could be fixed by refactoring Binaries::node_argv() to
be used here. However, for now, just re-implement the node_argv logic in
set_cmd_args to prepend the valgrind cmd.
fcaec2544b doc: release note for IPC cooldown and interrupt (Sjors Provoost)
1e82fa498c mining: add interrupt() (Sjors Provoost)
a11297a904 mining: add cooldown argument to createNewBlock() (Sjors Provoost)
Pull request description:
As reported in #33994, connected mining clients will receive a flood of new templates if the node is still going through IBD or catching up on the last 24 hours. This PR fixes that using an _optional_ cooldown mechanism, only applied to `createNewBlock()`.
First, cooldown waits for IBD. Then, as the tip keeps moving forward, it waits a few seconds to see if the tip updated. If so, it restarts the timer and waits again. The trade-offs for this mechanism are explained below.
Because this PR changes `createNewBlock()` from a method that returns quickly to one that can block for minutes, we rely on #34568 to fix a bug in our `.capnp` definition, adding the missing `context` to `createNewBlock` (and `checkBlock`).
The second commit then adds an `interrupt()` method so that clients can cleanly disconnect.
---
## Rationale
The cooldown argument is optional, and not used by internal non-IPC code, for two reasons:
1. The mechanism wreaks havoc on the functional test suite, which would require very careful mock time handling to work around. But that's pointless, because only IPC clients need it.
2. It needs to be optional for IPC clients too, because in some situations, like a signet with only one miner, waiting for IBD can mean being stuck forever.
The reason it's only applied to `createNewBlock()` is that this is the first method called by clients; `waitNext()` is a method on the interface returned by `createNewBlock()`, at which point the cooldown is done.
After IBD, we wait N seconds if the header is N blocks ahead of the tip, with a minimum of 3 and a maximum of 20 seconds. The minimum waiting time is short enough that it shouldn't be annoying or confusing for someone manually starting up a client. While the maximum should be harmless if it happens spuriously (which it shouldn't).
If the minimum wait is too short, clients get a burst of templates, as observed in the original issue. We can't entirely rule this out without a lot of additional complexity (like scanning our own log file for heuristics). This PR should make it a lot less likely, and thanks to the IBD wait also limit it to one day worth of blocks (`-maxtipage`).
Some test runs on an M4 MacBook Pro, where I had a node catch up on the last few days worth of blocks:
<img width="872" height="972" alt="Schermafbeelding 2026-02-04 om 18 21 17" src="https://github.com/user-attachments/assets/7902a0f2-0e0b-4604-9688-cec2da073261" />
As the chart shows, sometimes it takes longer than 3 seconds. But it turns out that in all those cases there were quite a few headers ahead of the tip. It also demonstrates that it's important to first wait for IBD, because it's less likely a random tip update takes longer than 20 seconds.
- modified sv2-apps: https://github.com/Sjors/sv2-apps/tree/2026/02/cooldown
- test script: https://gist.github.com/Sjors/feb6122c97acc2b9e6d66b168614609c#file-run_mainnet_pool_loop-zsh
- chart script: https://gist.github.com/Sjors/feb6122c97acc2b9e6d66b168614609c#file-tip_interval_charts-py
ACKs for top commit:
ryanofsky:
Code review ACK fcaec2544b. Only changes since last review were removing two cooldown arguments from the mining IPC test to simplify it
enirox001:
ACK fcaec2544b
Tree-SHA512: 08b75470f7c5c80a583a2fdb918fad145e7d5377309e5c599f67fc0d0e3139d09881067ba50c74114f117e69da17ee50666838259491691c031b1feaf050853f
c462e54f9d test: don't always assert NUM_PRIVATE_BROADCAST_PER_TX broadcasts (Vasil Dimov)
3710566305 test: move abortprivatebroadcast test at the end (Vasil Dimov)
Pull request description:
_test: move abortprivatebroadcast test at the end_
The piece of `p2p_private_broadcast.py` which tests the correctness of
`abortprivatebroadcast` issues a new `sendrawtransaction` call. That
call schedules up to 3 new connections: peer=13, peer=14 and possibly
peer=15 before it gets aborted.
These up to 3 in-the-process-of-opening private broadcast connections
have `CNode::m_connected` set early - when the `CNode` object is
created. Later in the test the mock time is advanced by 20 minutes and
those "old" connections pick a transaction for rebroadcast but that
triggers `PRIVATE_BROADCAST_MAX_CONNECTION_LIFETIME` immediately:
```
2026-02-21T13:28:14.209766Z [privbcast] [net.cpp:4006] [CNode] [net] Added connection peer=20
2026-02-21T13:28:14.309792Z (mocktime: 2026-02-21T13:48:14Z) [msghand] [net.cpp:4074] [PushMessage] [net] sending inv (37 bytes) peer=20
2026-02-21T13:28:14.309801Z (mocktime: 2026-02-21T13:48:14Z) [msghand] [net_processing.cpp:5745] [SendMessages] [privatebroadcast] Disconnecting: did not complete the transaction send within 180 seconds, peer=20
```
This prematurely stops the private broadcast connection and results in
a failure like:
```
AssertionError: ... not({} == {'ping': 1, 'tx': 1})
```
---
_test: don't always assert NUM_PRIVATE_BROADCAST_PER_TX broadcasts_
In `p2p_private_broadcast.py` in the function `check_broadcasts()` we
should assert that the broadcast was done to `broadcasts_to_expect`
peers, not to `NUM_PRIVATE_BROADCAST_PER_TX`. This is because in the
"Basic" test we check the first broadcast manually because it is done to
`nodes[1]` and then check the other two by
`check_broadcasts(..., NUM_PRIVATE_BROADCAST_PER_TX - 1, ...)`.
The first broadcast might not have fully concluded by the time we call
`check_broadcasts()` to check the remaining 2.
Demanding always `NUM_PRIVATE_BROADCAST_PER_TX` can lead to:
```
Traceback (most recent call last):
File "/home/vd/gh/bitcoin/bitcoin/test/functional/test_framework/test_framework.py", line 142, in main
self.run_test()
~~~~~~~~~~~~~^^
File "/tmp/build/clang22/test/functional/p2p_private_broadcast.py", line 347, in run_test
self.check_broadcasts("Basic", txs[0], NUM_PRIVATE_BROADCAST_PER_TX - 1, NUM_INITIAL_CONNECTIONS + 1)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/build/clang22/test/functional/p2p_private_broadcast.py", line 313, in check_broadcasts
assert_greater_than_or_equal(sum(1 for p in peers if "received" in p), NUM_PRIVATE_BROADCAST_PER_TX)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vd/gh/bitcoin/bitcoin/test/functional/test_framework/util.py", line 94, in assert_greater_than_or_equal
raise AssertionError("%s < %s" % (str(thing1), str(thing2)))
AssertionError: 2 < 3
```
ACKs for top commit:
l0rinc:
ACK c462e54f9d
achow101:
ACK c462e54f9d
andrewtoth:
ACK c462e54f9d
Tree-SHA512: 0de8d0eae079eeedc3bfad39df8129a8fa0d7734bdc03b4fb3e520a2f13a187d68118ffc210556af125d634f0ff51a1b081b34a023ac68a1c6a0caf541cecb91
a9e59f7d95 rpc: add optimal result to getmempoolinfo (Greg Sanders)
a3fb3dd55c mempool: log if we detect a non-optimal mempool (Greg Sanders)
Pull request description:
Post-SFL #34023 I don't think we expect the mempool to be unordered for long periods of time. If we consider it likely to be a serious regression in production, it would be useful to expose the fact that the mempool is not known to be optimal.
1. do a MEMPOOL log after any `DoWork()` returns false, meaning non-optimal
2. expose it via getmempoolinfo, by calling `DoWork(0)`, which does nothing but return known-optimality
I'm not wedded to either approach, I just think something is better than nothing for the next release.
ACKs for top commit:
ajtowns:
ACK a9e59f7d95
ismaelsadeeq:
reACK a9e59f7d95 [c89b93b95..a9e59f7d95](c89b93b958..a9e59f7d95) fixed typo, added more logging for block/reorg additions to mempool, and fixed brittle test case.
sedited:
ACK a9e59f7d95
sipa:
ACK a9e59f7d95
Tree-SHA512: 1560ad21cc1606df7279c102f35f61d4555c0ac920f02208b2a6eb89b14d7e22befb6d7f510a00a9074c2f9931f32e9af86bcea3a8dd9a1d947b0398c84666dd
Modify fuzzer logic to avoid setting `FRESH` for an outpoint that already exists unspent in the parent view, and ensure `FRESH` implies `DIRTY`.
This keeps cursor invariants realistic and lets `BatchWrite` failures expose real bugs without resetting state.
The coins view fuzzer can call `AddCoin` with `possible_overwrite=false` for an outpoint that already exists unspent in the view, which violates the `AddCoin` caller contract.
Derive `possible_overwrite` from `PeekCoin` so `possible_overwrite=false` is only used when the outpoint is absent.
This matches the approach used by the `coinscache_sim` fuzzer, which derives the overwrite flag from simulated state.
In validation, `AddCoins(check_for_overwrite=false)` is only used after BIP30 has already ensured the transaction does not overwrite any unspent outputs in the UTXO view.
The coins view fuzz target can call `AddCoins` with arbitrary txids, so using the `check_for_overwrite=false` fast path on non-coinbase transactions may violate the `AddCoin` caller contract and trigger logic errors.
Only use `check_for_overwrite=false` when we have first confirmed that none of the outputs are currently unspent.
Otherwise, fall back to `check_for_overwrite=true` so `AddCoins` determines overwrites via the view.
Introduce `TrySub(T&, U)` which subtracts an unsigned integral `U` from an unsigned integral `T`, returning `false` on underflow.
Use with `Assume(TrySub(...))` at coins cache accounting decrement sites so invariant violations fail immediately rather than silently wrapping.
Co-authored-by: MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz>
Co-authored-by: Pieter Wuille <pieter@wuille.net>
Grouped changes to improve the overall readability and maintainability of the test.
A lot more can be done, but this is a good first step.
1) Use for-loops instead of duplicating lines to perform the same checks for each
node.
2) The {'txid': x, 'vout': y} dict is repeated everywhere in the test, both as
input to gettxspendingprevout and as part of its result when an output has no
known spender, making the test tedious to read and maintain.
This introduces a prevout(txid, vout) query helper and an unspent_out(txid, vout)
result helper to reduce the repetition. These two helpers are intentionally kept
separate to make it immediately clear whether a dict is an input to
gettxspendingprevout or an assertion on its result.
3) The same repetition problem mentioned above applies to other gettxspendingprevout
possible results:
Spent outputs returns {'txid': x, 'vout': y, 'spendingtxid': z} and
Spent outputs when requesting spending tx returns {'txid': x, 'vout': y,
'spendingtxid': z, 'blockhash': w, 'spendingtx': v}
To fix it, this introduces:
- spent_out(txid, vout, spending_tx_id): for outputs with a known spender
- spent_out_in_block(txid, vout, spending_tx_id, blockhash, spending_tx): for
outputs spent in a confirmed block, when full tx data is requested
4) Rename overloaded confirmed_utxo variable (used in three different tests) to more
descriptive names: root_utxo, reorg_replace_utxo, reorg_cancel_utxo to clarify
their roles in each of the tests.
In `p2p_private_broadcast.py` in the function `check_broadcasts()` we
should assert that the broadcast was done to `broadcasts_to_expect`
peers, not to `NUM_PRIVATE_BROADCAST_PER_TX`. This is because in the
"Basic" test we check the first broadcast manually because it is done to
`nodes[1]` and then check the other two by
`check_broadcasts(..., NUM_PRIVATE_BROADCAST_PER_TX - 1, ...)`.
The first broadcast might not have fully concluded by the time we call
`check_broadcasts()` to check the remaining 2.
Demanding always `NUM_PRIVATE_BROADCAST_PER_TX` can lead to:
```
Traceback (most recent call last):
File "/home/vd/gh/bitcoin/bitcoin/test/functional/test_framework/test_framework.py", line 142, in main
self.run_test()
~~~~~~~~~~~~~^^
File "/tmp/build/clang22/test/functional/p2p_private_broadcast.py", line 347, in run_test
self.check_broadcasts("Basic", txs[0], NUM_PRIVATE_BROADCAST_PER_TX - 1, NUM_INITIAL_CONNECTIONS + 1)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/build/clang22/test/functional/p2p_private_broadcast.py", line 313, in check_broadcasts
assert_greater_than_or_equal(sum(1 for p in peers if "received" in p), NUM_PRIVATE_BROADCAST_PER_TX)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vd/gh/bitcoin/bitcoin/test/functional/test_framework/util.py", line 94, in assert_greater_than_or_equal
raise AssertionError("%s < %s" % (str(thing1), str(thing2)))
AssertionError: 2 < 3
```
The piece of `p2p_private_broadcast.py` which tests the correctness of
`abortprivatebroadcast` issues a new `sendrawtransaction` call. That
call schedules up to 3 new connections: peer=13, peer=14 and possibly
peer=15 before it gets aborted.
These up to 3 in-the-process-of-opening private broadcast connections
have `CNode::m_connected` set early - when the `CNode` object is
created. Later in the test the mock time is advanced by 20 minutes and
those "old" connections pick a transaction for rebroadcast but that
triggers `PRIVATE_BROADCAST_MAX_CONNECTION_LIFETIME` immediately:
```
2026-02-21T13:28:14.209766Z [privbcast] [net.cpp:4006] [CNode] [net] Added connection peer=20
2026-02-21T13:28:14.309792Z (mocktime: 2026-02-21T13:48:14Z) [msghand] [net.cpp:4074] [PushMessage] [net] sending inv (37 bytes) peer=20
2026-02-21T13:28:14.309801Z (mocktime: 2026-02-21T13:48:14Z) [msghand] [net_processing.cpp:5745] [SendMessages] [privatebroadcast] Disconnecting: did not complete the transaction send within 180 seconds, peer=20
```
This prematurely stops the private broadcast connection and results in
a failure like:
```
AssertionError: ... not({} == {'ping': 1, 'tx': 1})
```
4b53cbd692 test: Test for musig() in various miniscript expressions (Ava Chow)
ec0f47b15c miniscript: Using Func and Expr when parsing keys, hashes, and locktimes (Ava Chow)
6fd780d4fb descriptors: Increment key_exp_index in ParsePubkey(Inner) (Ava Chow)
b12281bd86 miniscript: Use a reference to key_exp_index in KeyParser (Ava Chow)
ce4c66eb7c test: Test that key expression indexes match key count (Ava Chow)
Pull request description:
The miniscript parser currently only looks for the next `)` when parsing key, hash, and locktime expressions. This fails to parse when the expressions contain a nested expression. Currently, this is only possible with `musig()` inside of key expressions. However, this pattern can be generalized to handling hashes and locktimes, so I implemented those too.
Fixes#34076
ACKs for top commit:
rkrux:
ACK 4b53cbd692
sipa:
ACK 4b53cbd692
darosior:
Other than that, Approach ACK 4b53cbd692. That makes sense to me but i have not closely reviewed the code.
Tree-SHA512: 01040c7b07a59d8e3725ff11ab9543b256aea22535fb94059f490a5bb45319e859666af04c2f0a4edcb8cf1e6dfc7bd8a8271b21ad81143bafccd4d0a39cae9c
fd06157d14 test: Add coverage for restarted node without any block sync (Fabian Jahr)
3d7ab7ecb7 rpc, test: Address feedback from #29668 (Fabian Jahr)
312919c9dd test: Indices can not start based on block data without undo data (Fabian Jahr)
a9a3b29dd6 index: Check availability of undo data for indices (Fabian Jahr)
881ab4fc82 support multiple block status checks in CheckBlockDataAvailability (furszy)
Pull request description:
Currently, we check that `BLOCK_HAVE_DATA` is available for all blocks an index needs to sync during startup. However, for `coinstatsindex` and `blockfilterindex` we also need the undo data for these blocks. If that data is missing in the blocks, we are currently still starting to sync each of these indices and then crash later when we encounter the missing data.
This PR adds explicit knowledge of which block data is needed for each index and then checks its availability during startup before initializing the sync process on them.
This also addresses a few open comments from #29668 in the last commit.
ACKs for top commit:
achow101:
ACK fd06157d14
furszy:
utACK fd06157d14
sedited:
Re-ACK fd06157d14
Tree-SHA512: e2ed81c93372b02daa8ddf2819df4164f96d92de05b1d48855410ecac78d5fcd9612d7f0e63a9d57d7e75a0b46e1bea278e43ea87f2693af0220d1f9c600e416
231dd04b8d build: define CMAKE_COMPILE_WARNING_AS_ERROR as a cache option (will)
Pull request description:
`CMAKE_COMPILE_WARNING_AS_ERROR` is not a cache variable by default in CMake, so it has no value in the configure summary when not set, and even when set cannot be toggled in `ccmake`. Define it as an `option()` to make it a cache BOOL with a default of OFF.
From the original MR to cmake, this was deliberately not set as a cache variable: https://gitlab.kitware.com/cmake/cmake/-/merge_requests/7187 (see Brad King's reply to the collapsed comments from Marc Chevrier).
Most `CMAKE_*` variables which are expected to be toggled by users are (as far as I can research) cache variables by default. Those that are considered likely to be set by the project (e.g. `CMAKE_CXX_STANDARD` or `CMAKE_POSITION_INDEPENDENT_CODE`) are not, along with read-only variables, script/internal variables, platform sppecific variables, template variables. `CMAKE_COMPILE_WARNING_AS_ERROR` may be a slight outlier here.
I count ~ 600 documented CMAKE_* variables, of which ~ 60 are default cache variables.
I could only see a few of these like:
- CMAKE_COMPILE_WARNING_AS_ERROR
- CMAKE_CXX_STANDARD,
- CMAKE_POSITION_INDEPENDENT_CODE
- CMAKE_INTERPROCEDURAL_OPTIMIZATION
...that we (or any project) _might_ want to expose as user-togglable, and would have to add as an `option()` in CMakeLists.txt.
ACKs for top commit:
ajtowns:
Yeah, seems to work for me. ACK 231dd04b8d
fanquake:
ACK 231dd04b8d
hebasto:
ACK 231dd04b8d, tested on Fedora 43.
Tree-SHA512: e380d79188859c97a9f7e45215a789daff55b8bff945a3d61fe4de3daae54bb7196aca0f6a6f1165979e787238dc26e94d217cff35f9642031ff72f4b39655e1
Both waitTipChanged() and createNewBlock() can take a long time to
return. Add a way for clients to interrupt them.
The new m_interrupt_mining is safely accessed with a lock on
m_tip_block_mutex, but it has no guard annotation. A more thorough
solution is discussed here:
https://github.com/bitcoin/bitcoin/pull/34184#discussion_r2743566474
At startup, if the needs to catch up, connected mining clients will
receive a flood of new templates as new blocks are connected.
Fix this by adding a cooldown argument to createNewBlock(). When set
to true, block template creation is briefly paused while the best
header chain is ahead of the tip.
This wait only happens when the best header extends the current tip,
to ignore competing branches.
Additionally, cooldown waits for isInitialBlockDownload() to latch to
false, which happens when there is less than a day of blocks left to sync.
When cooldown is false createNewBlock() returns immediately. The argument
is optional, because many tests are negatively impacted by this
mechanism, and single miner signets could end up stuck if no block
was mined for a day.
The getblocktemplate RPC also opts out, because it would add a delay
to each call.
Fixes#33994
We expect this to be rare in practice, and to not be the
usual state of the mempool. If we we detect non-optimal
ordering after a DoWork() invocation, allow this to be
observed in MEMPOOL logs.
f700609e8a doc: Release notes for mining IPC interface bump (Ryan Ofsky)
9453c15361 ipc mining: break compatibility with existing clients (version bump) (Sjors Provoost)
70de5cc2d2 ipc mining: pass missing context to BlockTemplate methods (incompatible schema change) (Sjors Provoost)
2278f017af ipc mining: remove deprecated methods (incompatible schema change) (Ryan Ofsky)
c6638fa7c5 ipc mining: provide default option values (incompatible schema change) (Ryan Ofsky)
a4603ac774 ipc mining: declare constants for default field values (Ryan Ofsky)
ff995b50cf ipc test: add workaround to block_reserved_weight exception test (Ryan Ofsky)
b970cdf20f test framework: expand expected_stderr, expected_ret_code options (Ryan Ofsky)
df53a3e5ec rpc refactor: stop using deprecated getCoinbaseCommitment method (Ryan Ofsky)
Pull request description:
This PR increments the field number of the `Init.makeMining` method and makes the old `makeMining` method return an error, so IPC mining clients not using the latest schema file will get an error and not be able to access the Mining interface.
Normally, there shouldn't be a need to break compatibility this way, but the mining interface has evolved a lot since it was first introduced, with old clients using the original methods less stable and performant than newer clients. So now is a good time to introduce a cutoff, drop deprecated methods, and stop supporting old clients which can't function as well.
Bumping the field number is also an opportunity to make other improvements that would be awkward to implement compatibly:
- Making Cap'n Proto default parameter and field values match default values of corresponding C++ methods and structs.
- Adding missing Context parameters to Mining.createNewBlock and checkBlock methods so these methods will be executed on separate execution threads and not block the Cap'n Proto event loop thread.
More details about these changes are in the commit messages.
ACKs for top commit:
Sjors:
ACK f700609e8a
enirox001:
ACK f700609e8a
ismaelsadeeq:
ACK f700609e8a
sedited:
ACK f700609e8a
Tree-SHA512: 0901886af00214c138643b33cec21647de5671dfff2021afe06d78dfd970664a844cde9a1e28f685bb27edccaf6e0c3f2d1e6bb4164bde6b84f42955946e366d
0b96b9c600 Minimize mempool lock, sync txo spender index only when and if needed (sstone)
3d82ec5bdd Add a "tx output spender" index (sstone)
Pull request description:
This PR adds a new "tx output spender" index, which allows users to query which tx spent a given outpoint with the `gettxspendingprevout` RPC call that was added by https://github.com/bitcoin/bitcoin/pull/24408.
Such an index would be extremely useful for Lightning, and probably for most layer-2 protocols that rely on chains of unpublished transactions.
UPDATE: this PR is ready for review and issues have been addressed:
- using a watch-only wallet instead would not work if there is a significant number of outpoints to watch (see https://github.com/bitcoin/bitcoin/pull/24539#issuecomment-1276595646)
- this PR does not require `-txindex` anymore
We use a composite key with 2 parts (suggested by romanz): hash(spent outpoint) and tx position, with an empty value. Average composite key size is 15 bytes.
The spending tx can optionally be returned by `gettxspendingprevout` (even it `-txindex is not set`).
ACKs for top commit:
hodlinator:
re-ACK 0b96b9c600
sedited:
Re-ACK 0b96b9c600
fjahr:
ACK 0b96b9c600
w0xlt:
reACK 0b96b9c600
Tree-SHA512: 95c2c313ef4086e7d5bf1cf1a3c7b91cfe2bb1a0dcb4c9d3aa8a6e5bfde66aaca48d85a1f1251a780523c3e4356ec8a97fe6f5c7145bc6ccb6f820b26716ae01
* The bitcoin.conf related checks do not need any timeout, because the
logging happens in the main thread, before the node is fully started.
* The net thread related checks do need a timeout, because the threads
may be late to start after the node is fully started.
This is required to actually erase the orphan transaction when the
BlockConnected event is handled in the background validation interface
queue thread.
cae6d895f8 fuzz: add target for CoinsViewOverlay (Andrew Toth)
86eda88c8e fuzz: move backend mutating block to end of coins_view (Andrew Toth)
89824fb27b fuzz: pass coins_view_cache to TestCoinsView in coins_view (Andrew Toth)
73e99a5966 coins: don't mutate main cache when connecting block (Andrew Toth)
67c0d1798e coins: introduce CoinsViewOverlay (Andrew Toth)
69b01af0eb coins: add PeekCoin() (Andrew Toth)
Pull request description:
This is a slightly modified version of the first few commits of #31132, which can be merged as an independent change. It has a small benefit on its own, but will help in moving the parent PR forward.
When accessing coins via the `CCoinsViewCache`, methods like `GetCoin` can call `FetchCoin` which actually mutate `cacheCoins` internally to cache entries when they are pulled from the backing db. This is generally a performance improvement for single threaded access patterns, but it precludes us from accessing entries in a `CCoinsViewCache` from multiple threads without a lock.
Another aspect is that when we use the resettable `CCoinsViewCache` view backed by the main cache for use in `ConnectBlock()`, we will insert entries into the main cache even if the block is determined to be invalid. This is not the biggest concern, since an invalid block requires proof-of-work. But, an attacker could craft multiple invalid blocks to fill the main cache. This would make us `Flush` the cache more often than necessary. Obviously this would be very expensive to do on mainnet.
Introduce `CoinsViewOverlay`, a `CCoinsViewCache` subclass that reads coins without mutating the underlying cache via `FetchCoin()`.
Add `PeekCoin()` to look up a Coin through a stack of `CCoinsViewCache` layers without populating parent caches. This prevents the main cache from caching inputs pulled from disk for a block that has not yet been fully validated. Once `Flush()` is called on the view, these inputs will be added as spent to `coinsCache` in the main cache via `BatchWrite()`.
This is the foundation for async input fetching, where worker threads must not mutate shared state.
ACKs for top commit:
l0rinc:
ACK cae6d895f8
sipa:
reACK cae6d895f8
sedited:
Re-ACK cae6d895f8
willcl-ark:
ACK cae6d895f8
vasild:
Cursory ACK cae6d895f8
ryanofsky:
Code review ACK cae6d895f8. PR is basically back to the form I had acked the first time, implementing `PeekCoin()` by calling `GetCoin()`. This is not ideal because `PeekCoin()` is not supposed to modify caches and `GetCoin()` does that, but it at least avoids problems of the subsequent approach tried where `GetCoin()` calls `PeekCoin` and would result in bugs when subclasses implement `GetCoin` forgetting to override `PeekCoin`. Hopefully #34124 can clean all of this by making relevant methods pure virtual.
Tree-SHA512: a81a98e60ca9e47454933ad879840cc226cb3b841bc36a4b746c34b350e07c546cdb5ddc55ec1ff66cf65d1ec503d22201d3dc12d4e82a8f4d386ccc52ba6441
afea2af139 net: reduce log level for PCP/NAT-PMP NOT_AUTHORIZED failures (ANAVHEOBA)
Pull request description:
Cherry-picks (and tweaks) a commit from #34117 which the ANAVHEOBA
did not follow up with when changes were requested.
The tweak here is to log once at `LogWarning`, so that users have a chance
to spot misconfiguration.
----
Users running on home networks with routers that don't support PCP (Port
Control Protocol) or NAT-PMP port mapping receive frequent warning-level
log messages every few minutes:
"pcp: Mapping failed with result NOT_AUTHORIZED (code 2)"
This is expected behavior for many consumer routers that have PCP
disabled by default, not an actionable error.
Add explicit constants for the NOT_AUTHORIZED result code (value 2)
for both NAT-PMP and PCP protocols. Log the first NOT_AUTHORIZED
failure at warning level for visibility, then downgrade subsequent
occurrences to LogDebug to avoid log noise. Other failure types
continue to warn unconditionally.
Fixes#34114
ACKs for top commit:
achow101:
ACK afea2af139
sedited:
ACK afea2af139
Tree-SHA512: 43dad9f3cca0ef9b82446045a3ccd90418cd5595c9965e938d9d539bbba863dde6b4a358adbee56f8d97d6efbf947eb5ddbbaf426faefcf3b1e36e4c8edb0d94
fa90d44a22 test: Fix intermittent issues in feature_assumevalid.py (MarcoFalke)
Pull request description:
The test has many issues:
* The `send_blocks_until_disconnected` seems to imply to wait until a disconnect happens. However, in reality it will blindly send all blocks and then return early, when done or when a disconnect happens. This will cause test failures when python is running faster than Bitcoin Core, for example when using sanitizers.
* The `assert_debug_log` scopes are bloated, which makes finding the above issue harder. This is, because a test failure will basically print the 1000+ line debug log excerpt twice, with the second instance stripped of useful python test logs. Also, the checks are less precise, if they happen over a larger scope/snippet.
Fix all issues, by:
* Removing `send_blocks_until_disconnected` and just sending the blocks that are needed to get the disconnect and then explicitly waiting for the disconnect.
* Reduce the scopes of the debug log checks. This can be reviewed with the git options `--color-moved=dimmed-zebra --color-moved-ws=ignore-all-space`.
ACKs for top commit:
l0rinc:
ACK fa90d44a22
achow101:
ACK fa90d44a22
Tree-SHA512: 8a18fa0009fb40a47317976e20b230625182874391babe0da3c3dfd55d2cc8e2da7008ce8f38691d01cd37ae02cac7c9e88b8193c2ed66d3621ee3f792649f71
2a1d0db799 doc: Mention private broadcast RPCs in release notes (Andrew Toth)
c3378be10b test: Cover abortprivatebroadcast in p2p_private_broadcast (Andrew Toth)
557260ca14 rpc: Add abortprivatebroadcast (Andrew Toth)
15dff452eb test: Cover getprivatebroadcastinfo in p2p_private_broadcast (Andrew Toth)
996f20c18a rpc: Add getprivatebroadcastinfo (Andrew Toth)
5e64982541 net: Add PrivateBroadcast::GetBroadcastInfo (Andrew Toth)
573bb542be net: Store recipient node address in private broadcast (Andrew Toth)
Pull request description:
Follow up from #29415
Sending a transaction via private broadcast does not have any way for a user to track the status of the transaction before it gets returned by another peer. The default logs have been removed as well in #34267. Nor is there any way to abort a transaction once it has been added to the private broadcast queue.
This adds two new RPCs:
- `getprivatebroadastinfo` returns information about what transactions are in the private broadcast queue, including all the peers' addresses we have chosen and timestamps.
- `abortprivatebroadcast` stops broadcasting a transaction in the private broadcast queue.
ACKs for top commit:
nervana21:
tACK 2a1d0db799
achow101:
ACK 2a1d0db799
l0rinc:
ACK 2a1d0db799
danielabrozzoni:
tACK 2a1d0db799
sedited:
ACK 2a1d0db799
Tree-SHA512: cc8682d0be68a57b42bea6e3d091da2b80995d9e6d3b98644cb120a05c2b48a97c2e211173289b758c4f4e23f1d1a1f9be528a9b8c6644f71d1dd0ae5f673326
afb1bc120e validation: Use dirty entry count in flush warnings and disk space checks (Pieter Wuille)
b413491a1c coins: Keep track of number of dirty entries in `CCoinsViewCache` (Pieter Wuille)
7e52b1b945 fuzz: call `EmplaceCoinInternalDANGER` as well in `SimulationTest` (Lőrinc)
Pull request description:
### Problem
Now that non-wiping flushes are possible (#28280, #28233), the cache may be mostly clean at flush time.
But the flush warning, disk-space check, and benchmark logging still used total cache size, so a node with a 10 GiB cache that only needs to write a small fraction of dirty entries could still trigger a scary warning via the disk-space checks.
The previous `DynamicMemoryUsage` metric was also fundamentally wrong for estimating disk writes, even before non-wiping flushes. In-memory coin size differs from on-disk write size due to LevelDB overhead, log doubling, and compaction.
The warning also only fired in `FlushStateToDisk`, so `AssumeUTXO` snapshot loads never warned at all.
### Fix
This PR tracks the actual number of dirty entries via `m_dirty_count` in `CCoinsViewCache`, maintained alongside the existing dirty-flag linked list, `SanityCheck` cross-validating both counts.
The warning and benchmark log move from `FlushStateToDisk` down to `CCoinsViewDB::BatchWrite`, where the actual I/O happens. This is the single place all flush paths converge (regular flushes, syncs, and snapshot loads), so the warning now fires correctly for `AssumeUTXO` too.
The threshold changes from 1 GiB of memory to 10 million dirty entries, which is roughly equivalent but avoids the in-memory vs on-disk size confusion.
The disk-space safety check now uses `GetDirtyCount()` with the existing conservative 48-byte-per-entry estimate, preventing unnecessary shutdowns when the cache is large but mostly clean.
---
Note: the first commit adds fuzz coverage for `EmplaceCoinInternalDANGER` in `SimulationTest` to exercise the accounting paths before modifying them.
Note: this is a revival of #31703 with all outstanding review feedback addressed.
ACKs for top commit:
Eunovo:
Concept ACK afb1bc120e
andrewtoth:
re-ACK afb1bc120e
sipa:
Code review ACK afb1bc120e
sedited:
ACK afb1bc120e
Tree-SHA512: 4133c6669fd20836ae2fb62ed804cdf6ebaa61076927b54fc412e42455a2f0d4cadfab0844064f9c32431eacb1f5e47b78de8e5cde1b26ba7239a7becf92f369
726b3663cc http: properly respond to HTTP request during shutdown (furszy)
59d24bd5dd threadpool: make Submit return Expected instead of throwing (furszy)
Pull request description:
Fixes#34573.
As mentioned in https://github.com/bitcoin/bitcoin/issues/34573#issuecomment-3891596958, the ThreadPool PR (#33689) revealed an existing issue.
Before that PR, we were returning an incorrect error "Request rejected because http work queue depth exceeded" during shutdown for unhandled requests (we were not differentiating between "queue depth exceeded" and "server interrupted" errors). Now, with the ThreadPool inclusion, we return the proper error but we don't handle it properly.
This PR improves exactly that. Handling the missing error and properly returning it to the user.
The race can be reproduced as follows:
1) The server receives an http request.
2) Processing of the request is delayed, and shutdown is triggered in the meantime.
3) During shutdown, the libevent callback is unregistered and the threadpool interrupted.
4) The delayed request (step 2) resumes and tries to submit a task to the now-interrupted server.
Reproduction test can be found https://github.com/bitcoin/bitcoin/pull/34577#issuecomment-3902672521.
Also, to prevent this kind of issue from happening again, this PR changes task submission
to return the error as part of the function's return value using `util::Expected` instead of
throwing the exception. Unlike exceptions, which require extra try-catch blocks and can be
ignored, returning `Expected` forces callers to explicitly handle failures, and attributes
like `[[nodiscard]]` allow us catch unhandled ones at compile time.
ACKs for top commit:
achow101:
ACK 726b3663cc
sedited:
ACK 726b3663cc
pinheadmz:
re-ACK 726b3663cc
andrewtoth:
ACK 726b3663cc
hodlinator:
re-ACK 726b3663cc
Tree-SHA512: ef026e299adde1148c9fc575e7d937e957bf0ddedfc1cf081941b568736417c2eefcd8bc8c8aea795d7347040ed05da4371bddcdbda7d385e04bf4dc8d875780
We sync txospenderindex after we've checked the mempool for spending transaction, and only if search is not limited to the mempool and no
spending transactions have been found for some of the provided outpoints.
This should minimize the chance of having a block containing a spending transaction that is no longer in the mempool but has not been indexed yet.
8966352df3 doc: add release notes (ismaelsadeeq)
704a09fe71 test: ensure fee estimator provide fee rate estimate < 1 s/vb (ismaelsadeeq)
243e48cf49 fees: delete unused dummy field (ismaelsadeeq)
fc4fbda42a fees: bump fees file version (ismaelsadeeq)
b54dedcc85 fees: reduce `MIN_BUCKET_FEERATE` to 100 (ismaelsadeeq)
Pull request description:
This is a simple PR that updates the block policy estimator’s `MIN_BUCKET_FEERATE` constant to be 100, which is identical to the policy `DEFAULT_MIN_RELAY_TX_FEE`.
This change enables the block policy fee rate estimator to return sub-1 sat/vB fee rate estimates.
The change is correct because the estimator creates buckets of fee rates from
`MIN_BUCKET_FEERATE`,
`MIN_BUCKET_FEERATE` + `FEE_SPACING`,
`MIN_BUCKET_FEERATE` + `2 * FEE_SPACING`,
… up to `MAX_BUCKET_FEERATE`.
This means it will record sub-1 sat/vB fee rates in the buckets and may return them as a fee rate estimate when that bucket is the lowest one with sufficient transactions for a given target.
---
While touching this part of the fee estimator code, this PR got rid of the dummy value persisted in the file
ACKs for top commit:
achow101:
ACK 8966352df3
musaHaruna:
ACK [8966352](8966352df3)
polespinasa:
ACK 8966352df3
Tree-SHA512: 9424e0820fcbe124adf5e5d9101a8a2983ba802887101875b2d1856d700c8b563d7a6f6ca3e3db29cb939f786719603aadbf5480d59d55d0951ed1c0caa49868
e0463b4e8c rpc: add coinbase_tx field to getblock (Sjors Provoost)
Pull request description:
This adds a `coinbase_tx` field to the `getblock` RPC result, starting at verbosity level 1. It contains only fields guaranteed to be small, i.e. not the outputs.
Initial motivation for this was to more efficiently scan for BIP54 compliance. Without this change, it requires verbosity level 2 to get the coinbase, which makes such scan very slow. See https://github.com/bitcoin-inquisition/bitcoin/pull/99#issuecomment-3852370506.
Adding these fields should be useful in general though and hardly makes the verbosity 1 result longer.
```
bitcoin rpc help getblock
getblock "blockhash" ( verbosity )
If verbosity is 0, returns a string that is serialized, hex-encoded data for block 'hash'.
If verbosity is 1, returns an Object with information about block <hash>.
If verbosity is 2, returns an Object with information about block <hash> and information about each transaction.
...
Result (for verbosity = 1):
{ (json object)
"hash" : "hex", (string) the block hash (same as provided)
"confirmations" : n, (numeric) The number of confirmations, or -1 if the block is not on the main chain
"size" : n, (numeric) The block size
"strippedsize" : n, (numeric) The block size excluding witness data
"weight" : n, (numeric) The block weight as defined in BIP 141
"coinbase_tx" : { (json object) Coinbase transaction metadata
"version" : n, (numeric) The coinbase transaction version
"locktime" : n, (numeric) The coinbase transaction's locktime (nLockTime)
"sequence" : n, (numeric) The coinbase input's sequence number (nSequence)
"coinbase" : "hex", (string) The coinbase input's script
"witness" : "hex" (string, optional) The coinbase input's first (and only) witness stack element, if present
},
"height" : n, (numeric) The block height or index
"version" : n, (numeric) The block version
...
```
```
bitcoin rpc getblock 000000000000000000013c986f9aebe800a78454c835ccd07ecae2650bfad3f6 1
```
```json
{
"hash": "000000000000000000013c986f9aebe800a78454c835ccd07ecae2650bfad3f6",
"confirmations": 2,
"height": 935113,
"version": 561807360,
"...": "...",
"weight": 3993624,
"coinbase_tx": {
"version": 2,
"locktime": 0,
"sequence": 4294967295,
"coinbase": "03c9440e04307c84692f466f756e6472792055534120506f6f6c202364726f70676f6c642ffabe6d6d9a8624235259d3680c972b0dd42fa3fe1c45c5e5ae5a96fe10c182bda17080e70100000000000000184b17d3f138020000000000",
"witness": "0000000000000000000000000000000000000000000000000000000000000000"
},
"tx": [
"70eb053340c7978c5aa1b34d75e1ba9f9d1879c09896317f306f30c243536b62",
"5bcf8ed2900cb70721e808b8977898e47f2c9001fcee83c3ccd29e51c7775dcd",
"3f1991771aef846d7bb379d2931cccc04e8421a630ec9f52d22449d028d2e7f4",
"..."
]
}
```
ACKs for top commit:
sedited:
Re-ACK e0463b4e8c
darosior:
re-utACK e0463b4e8c
Tree-SHA512: 1b3e7111e6a0edffde8619c49b3e9bca833c8e90e416defc66811bd56dd00d45b69a84c8fd9715508f4e6515f77ac4fb5c59868ab997ae111017c78c05b74ba3
