0582932260e7de4e8aba01d63e7c8a9ddb9c3685 test: add test for fast rescan using block filters (top-up detection) (Sebastian Falbesoner)
ca48a4694f73e5be8f971ae482ebc2cce4caef44 rpc: doc: mention rescan speedup using `blockfilterindex=1` in affected wallet RPCs (Sebastian Falbesoner)
3449880b499d54bfbcf6caeed52851ce55259ed7 wallet: fast rescan: show log message for every non-skipped block (Sebastian Falbesoner)
935c6c4b234bbb0565cda6f58ee298048856acae wallet: take use of `FastWalletRescanFilter` (Sebastian Falbesoner)
70b35139040a2351c845a1cec1dafd2fbcd16e93 wallet: add `FastWalletRescanFilter` class for speeding up rescans (Sebastian Falbesoner)
c051026586fb269584bcba41de8a4a90280f5a7e wallet: add method for retrieving the end range for a ScriptPubKeyMan (Sebastian Falbesoner)
845279132b494f03b84d689c666fdcfad37f5a42 wallet: support fetching scriptPubKeys with minimum descriptor range index (Sebastian Falbesoner)
088e38d3bbea9694b319bc34e0d2e70d210c38b4 add chain interface methods for using BIP 157 block filters (Sebastian Falbesoner)
Pull request description:
## Description
This PR is another take of using BIP 157 block filters (enabled by `-blockfilterindex=1`) for faster wallet rescans and is a modern revival of #15845. For reviewers new to this topic I can highly recommend to read the corresponding PR review club (https://bitcoincore.reviews/15845).
The basic idea is to skip blocks for deeper inspection (i.e. looking at every single tx for matches) if our block filter doesn't match any of the block's spent or created UTXOs are relevant for our wallet. Note that there can be false-positives (see https://bitcoincore.reviews/15845#l-199 for a PR review club discussion about false-positive rates), but no false-negatives, i.e. it is safe to skip blocks if the filter doesn't match; if the filter *does* match even though there are no wallet-relevant txs in the block, no harm is done, only a little more time is spent extra.
In contrast to #15845, this solution only supports descriptor wallets, which are way more widespread now than back in the time >3 years ago. With that approach, we don't have to ever derive the relevant scriptPubKeys ourselves from keys before populating the filter, and can instead shift the full responsibility to that to the `DescriptorScriptPubKeyMan` which already takes care of that automatically. Compared to legacy wallets, the `IsMine` logic for descriptor wallets is as trivial as checking if a scriptPubKey is included in the ScriptPubKeyMan's set of scriptPubKeys (`m_map_script_pub_keys`): e191fac4f3/src/wallet/scriptpubkeyman.cpp (L1703-L1710)
One of the unaddressed issues of #15845 was that [the filter was only created once outside the loop](https://github.com/bitcoin/bitcoin/pull/15845#discussion_r343265997) and as such didn't take into account possible top-ups that have happened. This is solved here by keeping a state of ranged `DescriptorScriptPubKeyMan`'s descriptor end ranges and check at each iteration whether that range has increased since last time. If yes, we update the filter with all scriptPubKeys that have been added since the last filter update with a range index equal or higher than the last end range. Note that finding new scriptPubKeys could be made more efficient than linearly iterating through the whole `m_script_pub_keys` map (e.g. by introducing a bidirectional map), but this would mean introducing additional complexity and state and it's probably not worth it at this time, considering that the performance gain is already significant.
Output scripts from non-ranged `DescriptorScriptPubKeyMan`s (i.e. ones with a fixed set of output scripts that is never extended) are added only once when the filter is created first.
## Benchmark results
Obviously, the speed-up indirectly correlates with the wallet tx frequency in the scanned range: the more blocks contain wallet-related transactions, the less blocks can be skipped due to block filter detection.
In a [simple benchmark](https://github.com/theStack/bitcoin/blob/fast_rescan_functional_test_benchmark/test/functional/pr25957_benchmark.py), a regtest chain with 1008 blocks (corresponding to 1 week) is mined with 20000 scriptPubKeys contained (25 txs * 800 outputs) each. The blocks each have a weight of ~2500000 WUs and hence are about 62.5% full. A global constant `WALLET_TX_BLOCK_FREQUENCY` defines how often wallet-related txs are included in a block. The created descriptor wallet (default setting of `keypool=1000`, we have 8*1000 = 8000 scriptPubKeys at the start) is backuped via the `backupwallet` RPC before the mining starts and imported via `restorewallet` RPC after. The measured time for taking this import process (which involves a rescan) once with block filters (`-blockfilterindex=1`) and once without block filters (`-blockfilterindex=0`) yield the relevant result numbers for the benchmark.
The following table lists the results, sorted from worst-case (all blocks contain wallte-relevant txs, 0% can be skipped) to best-case (no blocks contain walltet-relevant txs, 100% can be skipped) where the frequencies have been picked arbitrarily:
wallet-related tx frequency; 1 tx per... | ratio of irrelevant blocks | w/o filters | with filters | speed gain
--------------------------------------------|-----------------------------|-------------|--------------|-------------
~ 10 minutes (every block) | 0% | 56.806s | 63.554s | ~0.9x
~ 20 minutes (every 2nd block) | 50% (1/2) | 58.896s | 36.076s | ~1.6x
~ 30 minutes (every 3rd block) | 66.67% (2/3) | 56.781s | 25.430s | ~2.2x
~ 1 hour (every 6th block) | 83.33% (5/6) | 58.193s | 15.786s | ~3.7x
~ 6 hours (every 36th block) | 97.22% (35/36) | 57.500s | 6.935s | ~8.3x
~ 1 day (every 144th block) | 99.31% (143/144) | 68.881s | 6.107s | ~11.3x
(no txs) | 100% | 58.529s | 5.630s | ~10.4x
Since even the (rather unrealistic) worst-case scenario of having wallet-related txs in _every_ block of the rescan range obviously doesn't take significantly longer, I'd argue it's reasonable to always take advantage of block filters if they are available and there's no need to provide an option for the user.
Feedback about the general approach (but also about details like naming, where I struggled a lot) would be greatly appreciated. Thanks fly out to furszy for discussing this subject and patiently answering basic question about descriptor wallets!
ACKs for top commit:
achow101:
ACK 0582932260e7de4e8aba01d63e7c8a9ddb9c3685
Sjors:
re-utACK 0582932260e7de4e8aba01d63e7c8a9ddb9c3685
aureleoules:
ACK 0582932260e7de4e8aba01d63e7c8a9ddb9c3685 - minor changes, documentation and updated test since last review
w0xlt:
re-ACK 0582932260
Tree-SHA512: 3289ba6e4572726e915d19f3e8b251d12a4cec8c96d041589956c484b5575e3708b14f6e1e121b05fe98aff1c8724de4564a5a9123f876967d33343cbef242e1
fa3da8307baa96ea7e51cc6ba4820aca7f93ec32 test: Check debug log as well in p2p_sendtxrcncl.py (MacroFake)
fae0439486e9af428829b4604b3be5bb97b67b24 test: Check correct disconnect reason in p2p_sendtxrcncl.py (MacroFake)
fa590cfaae887c927ffb4af92a15516332428d69 test: Fix intermittent issue in p2p_sendtxrcncl.py (MacroFake)
Pull request description:
Should fix https://github.com/bitcoin/bitcoin/issues/26364
I can't reproduce this, but my guess would be that `PeerNoVerack::on_version`, which sends the `wtxidrelay` message, is executed in the event loop and thus may run after the main thread sending `msg_verack`.
Also, fix another bug.
Finally, add some `assert_debug_log` to ensure the right code branch is executed (and not some random, unrelated disconnect).
ACKs for top commit:
naumenkogs:
ACK fa3da8307baa96ea7e51cc6ba4820aca7f93ec32
Tree-SHA512: ee2988372223ea22e94e9783ef6d37114a3945991b6d60f0157917f3850fb7077c566564c3771d915ee74ab28202a3b73c6d89ddec6e2c918462db9a3c02e6cf
aaaa7bd0ba60aa7114810d4794940882d987c0aa iwyu: Add missing includes (MacroFake)
fa9ebec096ae185576a54aa80bd2a9e57f867ed4 Remove g_parallel_script_checks (MacroFake)
fa7c834b9f988fa7f2ace2d67b1628211b7819df Move ::fCheckBlockIndex into ChainstateManager (MacroFake)
fa43188d86288fa6666307a77c106c8f069ebdbe Move ::fCheckpointsEnabled into ChainstateManager (MacroFake)
cccca83099453bf0882bce4f897f77eee5836e8b Move ::nMinimumChainWork into ChainstateManager (MacroFake)
fa29d0b57cdeb91c8798d5c90ba9cc18085e99fb Move ::hashAssumeValid into ChainstateManager (MacroFake)
faf44876db555f7488c8df96db9fa88b793f897c Move ::nMaxTipAge into ChainstateManager (MacroFake)
Pull request description:
It seems preferable to assign globals to a class (in this case `ChainstateManager`), than to leave them dangling. This should clarify scope for code-readers, as well as clarifying unit test behaviour.
ACKs for top commit:
dergoegge:
Code review ACK aaaa7bd0ba60aa7114810d4794940882d987c0aa
ryanofsky:
Code review ACK aaaa7bd0ba60aa7114810d4794940882d987c0aa. No changes since last review, other than rebase
aureleoules:
reACK aaaa7bd0ba60aa7114810d4794940882d987c0aa
Tree-SHA512: 83ec3ba0fb4f1dad95810d4bd4e578454e0718dc1bdd3a794cc4e48aa819b6f5dad4ac4edab3719bdfd5f89cbe23c2740a50fd56c1ff81c99e521c5f6d4e898d
9153ff3e274953ea0d92d53ddab4c72deeace1b1 rpc: add non-regression test about deriveaddresses crash when index is 2147483647 (muxator)
addf9d6502db12cebcc5976df3111cac1a369b82 rpc: fix crash in deriveaddresses when derivation index is 2147483647 (muxator)
Pull request description:
This PR is a proposal for fixing #26274 (better described there).
The problem is due to a signed int wrapping when the `index` parameter of the `deriveaddresses` RPC call has the value `2^31-1`.
```C++
for (int i = range_begin; i <= range_end; ++i) {
```
* the first commit adds a "temporary" test case (`test/functional/rpc_deriveaddresses_crash.py`) that shows the crash, and can be used to generate a core dump;
* the second commit fixes the problem giving an explicit size to the `i` variable in a for loop, from `int` to `int64_t`. The same commit also removes the ephemeral test case and adds a passing test to `test/functional/rpc_deriveaddresses.py`, in order to prevent future regressions.
This is my first submission to this project and I do not know its conventions. Please advise if something needs to be changed.
ACKs for top commit:
achow101:
ACK 9153ff3e274953ea0d92d53ddab4c72deeace1b1
Tree-SHA512: 0477b57b15dc2c682cf539d6002f100d44a8c7e668041aa3340c39dcdbd40e083c75dec6896b6c076b044a01c2e5254272ae6696d8a1467539391926f270940a
796b020c37c793674f9d614d5d70fd1ed65f0938 wallet: add taproot support to external signer (Sjors Provoost)
Pull request description:
Builds on #22558 (merged on 2022-06-28).
[HWI 2.1.0](https://github.com/bitcoin-core/HWI/releases/tag/2.1.0) or newer is required to import and use taproot descriptors. Older versions will work, but won't import a taproot descriptor.
Tested with HWI 2.1.1:
* Trezor T (firmware v2.5.1) on Signet: signs, change detection works
* Ledger Nano S (firmware 2.1.0, Bitcoin app 2.0.6): signs, change detection works
Only the most basic `tr(key)` descriptor is supported, script path spending is completely untested (if it works at all).
ACKs for top commit:
jb55:
utACK 796b020c37c793674f9d614d5d70fd1ed65f0938
achow101:
ACK 796b020c37c793674f9d614d5d70fd1ed65f0938
Tree-SHA512: 6dcb7eeb45421a3bbf2bdabeacd29979867db69077d7bf192bb77faa4bfefe446487b8df07bc40f9457009a88e598bdc09f769e6106fed2833ace7ef205a157a
When a block is initially read from a blk*.dat file during reindexing,
it can be added to the block index only if all of its ancestor blocks
have been added, which is rare. If the block's ancestors have not been
added, the block must be re-read from disk later when it can be added.
This commit: During the initial block read, deserialize only its header,
rather than the entire block, since this is sufficient to determine
if its parent (and thus all its ancestors) has been added. This is a
performance improvement.
e1eadaa72d6831d1d0a53ba97c215dc4cdb64436 Revert "test: check importing wallets when blocks are pruned throw an error" (Aurèle Oulès)
Pull request description:
The test doesn't pass (not detected by the normal CI, because it is an extended test):
```
Traceback (most recent call last):
File "/tmp/cirrus-ci-build/bitcoin-core/test/functional/test_framework/test_framework.py", line 133, in main
self.run_test()
File "/tmp/cirrus-ci-build/bitcoin-core/test/functional/feature_pruning.py", line 480, in run_test
self.wallet_test()
File "/tmp/cirrus-ci-build/bitcoin-core/test/functional/feature_pruning.py", line 361, in wallet_test
assert_raises_rpc_error(-4, "Importing wallets is disabled when blocks are pruned", self.nodes[2].importwallet, "abc")
File "/tmp/cirrus-ci-build/bitcoin-core/test/functional/test_framework/util.py", line 130, in assert_raises_rpc_error
assert try_rpc(code, message, fun, *args, **kwds), "No exception raised"
File "/tmp/cirrus-ci-build/bitcoin-core/test/functional/test_framework/util.py", line 145, in try_rpc
raise AssertionError(
AssertionError: Expected substring not found in error message:
substring: 'Importing wallets is disabled when blocks are pruned'
error message: 'Only legacy wallets are supported by this command'.
```
So revert it for now, which will be done anyway in https://github.com/bitcoin/bitcoin/pull/24865/commits. (This commit is taken from there)
ACKs for top commit:
andrewtoth:
ACK e1eadaa72d6831d1d0a53ba97c215dc4cdb64436
Tree-SHA512: 10f556ea1aa97dc9cb64f91055977eecb9534f658170aabb4909c3e85ca6c20588c7a021219356fab678e0e2bec999d347facd00054f07a9445ad393e6353b4c
dddd1acf58cb7bf328ce3e74d1dc0e8cbd503247 net: Set relay in version msg to peers with relay permission (MacroFake)
Pull request description:
Seems odd to set the `relay` permission in -blocksonly mode and also ask the peer not to relay transactions.
ACKs for top commit:
dergoegge:
ACK dddd1acf58cb7bf328ce3e74d1dc0e8cbd503247
naumenkogs:
ACK dddd1acf58
mzumsande:
ACK dddd1acf58cb7bf328ce3e74d1dc0e8cbd503247
Tree-SHA512: 7bb0e964993ea4982747ae2801fe963ff88586e2ded03015b60ab83172b5b61f2d50e9cde9d7711b7ab207f8639467ecafc4d011ea151ec6c82c722f510f4df7
deba6fe3158cd0b2283e0901a072e434ba5b594e test: update feature_config_args.py (josibake)
2e3826cbcd675dcd1d03970233ba5e143e09eb75 util: warn if reindex is used in conf (josibake)
5e744f423838fe7d45453541271bc1a07cd62eac util: disallow setting conf in bitcoin.conf (josibake)
Pull request description:
In help from `bitcoind -h` it specifes that `conf` can only be used from the commandline. However, if `conf` is set in a `bitcoin.conf` file, there is no error and from reading the logs it seems as if the `conf=<other file>` is being used, despite it being ignored. To recreate, you can setup a `bitcoin.conf` file in the default directory, add `conf=<some other file>.conf` and in the separate config file set whichever config value you want and verify that it is being ignored. alternatively, if you set `includeconf=<some other file>.conf` , your config in `<some other file>` will be picked up.
This PR fixes this by having the node error when reading the config file if `conf=` is set.
Additionally, it was mentioned in a recent [PR review club](https://bitcoincore.reviews/24858) that if `reindex=1` is set in the config file, the node will reindex on every startup, which is undesirable:
```irc
17:14 <larryruane> michaelfolkson: Reindex is requested by the user (node operator) as a configuration option (command line or in the config file, tho you probably would never put it in the file, or else it would reindex on every startup!)
```
This PR also has a commit to warn if `reindex=1` is set in the config file.
ACKs for top commit:
hebasto:
ACK deba6fe3158cd0b2283e0901a072e434ba5b594e, tested on Ubuntu 22.04.
aureleoules:
tACK deba6fe3158cd0b2283e0901a072e434ba5b594e
ryanofsky:
Code review ACK deba6fe3158cd0b2283e0901a072e434ba5b594e.
Tree-SHA512: 619fd0aa14e98af1166d6beb92651f5ba3f10d38b8ee132957f094f19c3a37313d9f4d7be2e4019f3fc9a2ca5fa42d03eb539ad820e27efec7ee58a26eb520b1
fafc96aaf4c92feec50074e34a3fc1edc13cab4c test: Test year 2106 block timestamps (MacroFake)
Pull request description:
Alternative to https://github.com/bitcoin/bitcoin/pull/21362 that closes https://github.com/bitcoin/bitcoin/issues/21356
ACKs for top commit:
Sjors:
utACK fafc96a
Tree-SHA512: 196d98f42d6f7f0222312b7bd1c68b3bd30cb6f0cbaccb900cfc5fcc689494adb2a7d7d6023c1ff1e8cf871047ec37eeca41386e31029d99cabf9343b4fd2a03
315fd4dbabb6b631b755811742a3bdf93e1241bf test: Test for out of bounds vout in sendall (Andrew Chow)
b132c85650afb2182f2e58e903f3d6f86fd3fb22 wallet: Check utxo prevout index out of bounds in sendall (Andrew Chow)
708b72b7151c855cb5dac2fb6a81e8f35153c46f test: Test that sendall works with watchonly spending specific utxos (Andrew Chow)
6bcd7e2a3b52f855db84cd23b5ee70d27be3434f wallet: Correctly check ismine for sendall (Andrew Chow)
Pull request description:
The `sendall` RPC would previously fail when used with a watchonly wallet and specified inputs. This failure was caused by checking isminetype equality with ISMINE_ALL rather than a bitwise AND as IsMine can never return ISMINE_ALL.
Also added a test.
ACKs for top commit:
w0xlt:
ACK 315fd4dbab
furszy:
ACK 315fd4db
Tree-SHA512: fb55cf6524e789964770b803f401027319f0351433ea084ffa7c5e6f1797567a608c956b7f7c5bd542aa172c4b7b38b07d0976f5ec587569efead27266e8664c
e133264c5b1f72e94dcb9cebd85cdb523fcf8070 Add test for PSBT input verification (Greg Sanders)
d25699280af1ea45bebc884f63a10da7ea275ef9 Verify PSBT inputs rather than check for fields being empty (Greg Sanders)
Pull request description:
In a few keys spots, PSBT finality is checked by looking for non-empty witness data.
This complicates a couple things:
1) Empty data can be valid in certain cases
2) User may be passed bogus final data by a counterparty during PSBT work happening, and end up with incorrect signatures that they may not be able to check in other contexts if the UTXO doesn't exist yet in chain/mempool, timelocks, etc.
On the whole I think these heavier checks are worth it in case someone is actually assuming the signatures are correct if our API is saying so.
ACKs for top commit:
achow101:
ACK e133264c5b1f72e94dcb9cebd85cdb523fcf8070
Tree-SHA512: 9de4fbb0be1257b081781f5df908fd55666e3acd5c4e36beb3b3f2f5a6aed69ff77068c44cde6127e159e773293fd9ced4c0bb47e693969f337e74dc8af030da
4aff7a48a4e0f1075306f181a276b8a74c857022 test: check importing wallets when blocks are pruned throw an error (brunoerg)
Pull request description:
This PR adds test coverage for the following error:
437b608df2/src/wallet/rpc/backup.cpp (L513-L518)
ACKs for top commit:
andrewtoth:
ACK 4aff7a48a4e0f1075306f181a276b8a74c857022
Tree-SHA512: fbbf6056cb3759f726b8a5ff25fca51bf47e973e5d655ec164e2bec88e2dbd3b243677869d2cf33af268ea635ca0f2e9f737c4734077fc5a936ac3a24ad4b88b
a8250e30f16f2919ea5aa122b2880b076bd398a3 doc: add release note about `/rest/deploymentinfo` (brunoerg)
5c960200242d237f2cf74309b8fd29e8162682ed doc: add `/deploymentinfo` in REST-interface (brunoerg)
3e44bee08eb93e086179b92007649d47652aa439 test: add coverage for `/rest/deploymentinfo` (brunoerg)
91497031cbd74a0665b7fc31eb6b73bfb7bd0d40 rest: add `/deploymentinfo` (brunoerg)
Pull request description:
#23508 added a new RPC named `getdeploymentinfo`, it moved the softfork section from `getblockchaininfo` into this new one. In the REST interface, we have an endpoint named`/rest/chaininfo.json` (which refers to `getblockchaininfo`), so, this PR adds a new REST endpoint named `/deploymentinfo` which refers to `getdeploymentinfo`.
You can use it by passing a block hash, e.g: '/rest/deploymentinfo/<BLOCKHASH>.json' or you can use it without passing a block hash to get the 'deploymentinfo' for the last block.
ACKs for top commit:
jonatack:
re-ACK a8250e30f16f2919ea5aa122b2880b076bd398a3 rebase-only since my last review at c65f82bb
achow101:
ACK a8250e30f16f2919ea5aa122b2880b076bd398a3
stickies-v:
re-ACK a8250e30f1
Tree-SHA512: 0735183b6828d51a72ed0e2be5a09b314ac4693f548982c6e9adaa0ef07a55aa428d3b2d1b1de70b83169811a663a8624b686166e5797f624dcc00178b9796e6
bff05bd7456d3634b0c83539293a753db6d76376 test: add functional test for -discover (brunoerg)
Pull request description:
This PR adds a functional test for `-discover`. It tests different scenarios where `localaddresses` should be empty or may contain the addresses. Obs: `localaddresses` is not always accurate, so it's not possible to ensure (100%) it will contain any addresses.
515200298b/src/init.cpp (L449)
Obs: See #24258 - It adds test coverage for this field but for nodes with proxy.
ACKs for top commit:
mzumsande:
Code review ACK bff05bd7456d3634b0c83539293a753db6d76376
achow101:
ACK bff05bd7456d3634b0c83539293a753db6d76376
rajarshimaitra:
tACK bff05bd7456d3634b0c83539293a753db6d76376
Tree-SHA512: 8782497c146bce1ba86fda6146f3847465d7069f2cb6b84f2afc8f3b43efa813442bffe7447e9ce02adee304100b60365409bf0e5d875dfb880038442feec2a6
626b7c8493ea1063f30ae4f62e1b36eb87adf685 fuzz: add scanblocks as safe for fuzzing (James O'Beirne)
94fe5453c7f8a86136dc9a6e0b370afd32564209 test: rpc: add scanblocks functional test (Jonas Schnelli)
6ef2566b68cb8570220c13df11c5cb5a5f4f7a4d rpc: add scanblocks - scan for relevant blocks with descriptors (Jonas Schnelli)
a4258f6e81a476ce1687e2d58f7d2bf16162a172 rpc: move-only: consolidate blockchain scan args (James O'Beirne)
Pull request description:
Revives #20664. All feedback from the previous PR has either been responded to inline or incorporated here.
---
Major changes from Jonas' PR:
- consolidated arguments for scantxoutset/scanblocks
- substantial cleanup of the functional test
Here's the range-diff (`git range-diff master jonasschnelli/2020/12/filterblocks_rpc jamesob/2021-11-scanblocks`): https://gist.github.com/jamesob/aa4a975344209f0316444b8de2ec1d18
### Original PR description
> The `scanblocks` RPC call allows one to get relevant blockhashes from a set of descriptors by scanning all blockfilters in a given range.
>
> **Example:**
>
> `scanblocks start '["addr(<bitcoin_address>)"]' 661000` (returns relevant blockhashes for `<bitcoin_address>` from blockrange 661000->tip)
>
> ## Why is this useful?
> **Fast wallet rescans**: get the relevant blocks and only rescan those via `rescanblockchain getblockheader(<hash>)[height] getblockheader(<hash>)[height])`. A future PR may add an option to allow to provide an array of blockhashes to `rescanblockchain`.
>
> **prune wallet rescans**: (_needs additional changes_): together with a call to fetch blocks from the p2p network if they have been pruned, it would allow to rescan wallets back to the genesis block in pruned mode (relevant #15946).
>
> **SPV mode** (_needs additional changes_): it would be possible to build the blockfilterindex from the p2p network (rather then deriving them from the blocks) and thus allow some sort of hybrid-SPV mode with moderate bandwidth consumption (related #9483)
ACKs for top commit:
furszy:
diff re-ACK 626b7c8
Tree-SHA512: f84e4dcb851b122b39e9700c58fbc31e899cdcf9b587df9505eaf1f45578cc4253e89ce2a45d1ff21bd213e31ddeedbbcad2c80810f46755b30acc17b07e2873
bf9597606166323158bbf631137b82d41f39334f doc: add note about snapshot chainstate init (James O'Beirne)
e4d799528696c5ede38c257afaffd367917e0de8 test: add testcases for snapshot initialization (James O'Beirne)
cced4e7336d93a2dc88e4a61c49941887766bd72 test: move-only-ish: factor out LoadVerifyActivateChainstate() (James O'Beirne)
51fc9241c08a00f1f407f1534853a5cddbbc0a23 test: allow on-disk coins and block tree dbs in tests (James O'Beirne)
3c361391b8f5971eb3c7b620aa7ad9b437cc515e test: add reset_chainstate parameter for snapshot unittests (James O'Beirne)
00b357c215ed900145bd770525a341ba0ed9c027 validation: add ResetChainstates() (James O'Beirne)
3a29dfbfb2c16a50d854f6f81428a68aa9180509 move-only: test: make snapshot chainstate setup reusable (James O'Beirne)
8153bd9247dad3982d54488bcdb3960470315290 blockmanager: avoid undefined behavior during FlushBlockFile (James O'Beirne)
ad67ff377c2b271cb4683da2fb25fd295557f731 validation: remove snapshot datadirs upon validation failure (James O'Beirne)
34d159033106cc595cfa852695610bfe419c989c add utilities for deleting on-disk leveldb data (James O'Beirne)
252abd1e8bc5cdf4368ad55e827a873240535b28 init: add utxo snapshot detection (James O'Beirne)
f9f1735f139b6a1f1c7fea50717ff90dc4ba2bce validation: rename snapshot chainstate dir (James O'Beirne)
d14bebf100aaaa25c7558eeed8b5c536da99885f db: add StoragePath to CDBWrapper/CCoinsViewDB (James O'Beirne)
Pull request description:
This is part of the [assumeutxo project](https://github.com/bitcoin/bitcoin/projects/11) (parent PR: https://github.com/bitcoin/bitcoin/pull/15606)
---
Half of the replacement for #24232. The original PR grew larger than expected throughout the review process.
This change adds the ability to initialize a snapshot-based chainstate during init if one is detected on disk. This is of course unused as of now (aside from in unittests) given that we haven't yet enabled actually loading snapshots.
Don't be scared! There are some big move-only commits in here.
Accompanying changes include:
- moving the snapshot coinsdb directory from being called `chainstate_[base blockhash]` to `chainstate_snapshot`, since we only support one snapshot in use at a time. This simplifies some logic, but it necessitates writing that base blockhash out to a file within the coinsdb dir. See [discussion here](https://github.com/bitcoin/bitcoin/pull/24232#discussion_r832762880).
- adding a simple fix in `FlushBlockFile()` that avoids a crash when attemping to flush to disk before `LoadBlockIndexDB()` is called, which happens when calling `MaybeRebalanceCaches()` during multiple chainstate init.
- improving the unittest to allow testing with on-disk chainstates - necessary to test a simulated restart and re-initialization.
ACKs for top commit:
naumenkogs:
utACK bf9597606166323158bbf631137b82d41f39334f
ariard:
Code Review ACK bf9597606
ryanofsky:
Code review ACK bf9597606166323158bbf631137b82d41f39334f. Changes since last review: rebasing, switching from CAutoFile to AutoFile, adding comments, switching from BOOST_CHECK to Assert in test util, using chainman.GetMutex() in tests, destroying one ChainstateManager before creating a new one in tests
fjahr:
utACK bf9597606166323158bbf631137b82d41f39334f
aureleoules:
ACK bf9597606166323158bbf631137b82d41f39334f
Tree-SHA512: 15ae75caf19f8d12a12d2647c52897904d27b265a7af6b4ae7b858592eeadb8f9da6c2394b6baebec90adc28742c053e3eb506119577dae7c1e722ebb3b7bcc0
9e386afb67bf8fa71b72f730da1695eeb11828cd tests: Test that PSBT_OUT_TAP_TREE is included correctly (Andrew Chow)
30ff25cf37eec4b09ab40424eb5d6a4a80410955 psbt: Only include m_tap_tree if it has scripts (Andrew Chow)
0577d423adda8e719d7611d03355680c8fbacab8 psbt: Change m_tap_tree to store just the tuples (Andrew Chow)
22c051ca70bae73e0430b05fb9d879591df27699 tests: Test that PSBT_OUT_TAP_TREE is combined correctly (Andrew Chow)
7df6e1bb77a96eac4fbcba424bbe780636b86650 psbt: Fix merging of m_tap_tree (Andrew Chow)
0652dc53b291bd295caff4093ec2854fd4b34645 [BugFix]: Do not allow deserializing PSBT with empty PSBT_OUT_TAP_TREE (Jeremy Rubin)
Pull request description:
PSBT_OUT_TAP_TREE should not be included for outputs that do not have such a tree. This should be disallowed during parsing, as well as prior to serialization when the field is populated during updating.
Also added some test cases.
Alternative to #25856
ACKs for top commit:
instagibbs:
ACK 9e386afb67
darosior:
ACK 9e386afb67bf8fa71b72f730da1695eeb11828cd
Tree-SHA512: ce5c02a69752d176dbd967c1e8d30129b1905c8f186aeeef034576c1de82059271a1ee846bd040f5be4e66bb77ba711dcf14ac1e597c5707d7e7e2293f6cfefb
fa08663344eb4cea335804d149b46ff4ff081b1c rpc: Return coinbase flag in scantxoutset (MacroFake)
Pull request description:
I guess it can't hurt to return this for someone that wants to know it
ACKs for top commit:
aureleoules:
ACK fa08663344eb4cea335804d149b46ff4ff081b1c
shaavan:
ACK fa08663344eb4cea335804d149b46ff4ff081b1c
Tree-SHA512: 04c554b3ed9877bab93ffcf0c1a4430cd41b30c5f4f3bf462a518fc8b3d68832dd85a29e81bd805eaa16e987856933d7a888a8c126f670bb2844bbd5ca1bf902
04526787b5f6613d1f1ad78434e1dd24ab88dd76 Validate `port` options (amadeuszpawlik)
f8387c42343867779170a0f96ef64e6acff5c481 Validate port value in `SplitHostPort` (amadeuszpawlik)
Pull request description:
Validate `port`-options, so that invalid values are rejected early in the startup.
Ports are `uint16_t`s, which effectively limits a port's value to <=65535. As discussed in https://github.com/bitcoin/bitcoin/pull/24116 and https://github.com/bitcoin/bitcoin/pull/24344, port "0" is considered invalid too.
Proposed in https://github.com/bitcoin/bitcoin/issues/21893#issuecomment-835784223
The `SplitHostPort(std::string in, uint16_t& portOut, std::string& hostOut)` now returns a bool that indicates whether the port value was set and within the allowed range. This is an improvement that can be used not only for port validation of options at startup, but also in rpc calls, etc,
ACKs for top commit:
luke-jr:
utACK 04526787b5f6613d1f1ad78434e1dd24ab88dd76
ryanofsky:
Code review ACK 04526787b5f6613d1f1ad78434e1dd24ab88dd76. Just suggested changes since last review: reverting some SplitHostPort changes, adding release notes, avoiding 'GetArgs[0]` problem.
Tree-SHA512: f1ac80bf98520b287a6413ceadb41bc3a93c491955de9b9319ee1298ac0ab982751905762a287e748997ead6198a8bb7a3bc8817ac9e3d2468e11ab4a0f8496d
Check `port` options for invalid values (ports are parsed as uint16, so
in practice values >65535 are invalid; port 0 is undefined and therefore
considered invalid too). This allows for an early rejection of faulty
values and an supplying an informative message to the user.
Splits tests in `feature_proxy.py` to cover both invalid `hostname`
and `port` values.
Adds a release-note as previously valid `-port` and `-rpcport` values
can now result in errors.
