Merge bitcoin/bitcoin#27596: assumeutxo (2)

edbed31066 chainparams: add signet assumeutxo param at height 160_000 (Sjors Provoost) b8cafe3871 chainparams: add testnet assumeutxo param at height 2_500_000 (Sjors Provoost) 99839bbfa7 doc: add note about confusing HaveTxsDownloaded name (James O'Beirne) 7ee46a755f contrib: add script to demo/test assumeutxo (James O'Beirne) 42cae39356 test: add feature_assumeutxo functional test (James O'Beirne) 0f64bac603 rpc: add getchainstates (James O'Beirne) bb05857794 refuse to activate a UTXO snapshot if mempool not empty (James O'Beirne) ce585a9a15 rpc: add loadtxoutset (James O'Beirne) 62ac519e71 validation: do not activate snapshot if behind active chain (James O'Beirne) 9511fb3616 validation: assumeutxo: swap m_mempool on snapshot activation (James O'Beirne) 7fcd21544a blockstorage: segment normal/assumedvalid blockfiles (James O'Beirne) 4c3b8ca35c validation: populate nChainTx value for assumedvalid chainstates (James O'Beirne) 49ef778158 test: adjust chainstate tests to use recognized snapshot base (James O'Beirne) 1019c39982 validation: pruning for multiple chainstates (James O'Beirne) 373cf91531 validation: indexing changes for assumeutxo (James O'Beirne) 1fffdd76a1 net_processing: validationinterface: ignore some events for bg chain (James O'Beirne) fbe0a7d7ca wallet: validationinterface: only handle active chain notifications (James O'Beirne) f073917a9e validationinterface: only send zmq notifications for active (James O'Beirne) 4d8f4dcb45 validation: pass ChainstateRole for validationinterface calls (James O'Beirne) 1e59acdf17 validation: only call UpdatedBlockTip for active chainstate (James O'Beirne) c6af23c517 validation: add ChainstateRole (James O'Beirne) 9f2318c76c validation: MaybeRebalanceCaches when chain leaves IBD (James O'Beirne) 434495a8c1 chainparams: add blockhash to AssumeutxoData (James O'Beirne) c711ca186f assumeutxo: remove snapshot during -reindex{-chainstate} (James O'Beirne) c93ef43e4f bugfix: correct is_snapshot_cs in VerifyDB (James O'Beirne) b73d3bbd23 net_processing: Request assumeutxo background chain blocks (Suhas Daftuar) Pull request description: - Background and FAQ: https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/proposal - Prior progress/project: https://github.com/bitcoin/bitcoin/projects/11 - Replaces https://github.com/bitcoin/bitcoin/pull/15606, which was closed due to Github slowness. Original description and commentary can be found there. --- This changeset finishes the first phase of the assumeutxo project. It makes UTXO snapshots loadable via RPC (`loadtxoutset`) and adds `assumeutxo` parameters to chainparams. It contains all the remaining changes necessary to both use an assumedvalid snapshot chainstate and do a full validation sync in the background. This may look like a lot to review, but note that - ~200 lines are a (non-essential) demo shell script - Many lines are functional test, documentation, and relatively dilute RPC code. So it shouldn't be as burdensome to review as the linecount might suggest. - **P2P**: minor changes are made to `init.cpp` and `net_processing.cpp` to make simultaneous IBD across multiple chainstates work. - **Pruning**: implement correct pruning behavior when using a background chainstate - **Blockfile separation**: to prevent "fragmentation" in blockfile storage, have background chainstates use separate blockfiles from active snapshot chainstates to avoid interleaving heights and impairing pruning. - **Indexing**: some `CValidationInterface` events are given with an additional parameter, ChainstateRole, and all indexers ignore events from ChainstateRole::ASSUMEDVALID so that indexation only happens sequentially. - Have `-reindex` properly wipe snapshot chainstates. - **RPC**: introduce RPC commands `loadtxoutset` and (hidden) `getchainstates`. - **Release docs & first assumeutxo commitment**: add notes and a particular assumeutxo hash value for first AU-enabled release. - This will complete the project and allow use of UTXO snapshots for faster node bootstrap. The next phase, if it were to be pursued, would be coming up with a way to distribute the UTXO snapshots over the P2P network. --- ### UTXO snapshots Create your own with `./contrib/devtools/utxo_snapshot.sh`, e.g. ```shell ./contrib/devtools/utxo_snapshot.sh 788000 utxo.dat ./src/bitcoin-cli -datadir=$(pwd)/testdata`) ``` or use the pre-generated ones listed below. - Testnet: **2'500'000** (Sjors): - torrent: `magnet:?xt=urn:btih:511e09f4bf853aefab00de5c070b1e031f0ecbe9&dn=utxo-testnet-2500000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969` - sha256: `79db4b025448cc0ac388d8589a28eab02de53055d181e34eb47391717aa16388` - Signet: **160'000** (Sjors): - torrent: `magnet:?xt=urn:btih:9da986cb27b3980ea7fd06b21e199b148d486880&dn=utxo-signet-160000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969` - sha256: `eeeca845385ba91e84ef58c09d38f98f246a24feadaad57fe1e5874f3f92ef8c` - Mainnet: **800'000** (Sjors): - Note: this needs the following commit cherry-picked in: 24deb2022b - torrent: `magnet:?xt=urn:btih:50ee955bef37f5ec3e5b0df4cf0288af3d715a2e&dn=utxo-800000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969` ### Testing #### For fun (~5min) If you want to do a quick test, you can run `./contrib/devtools/test_utxo_snapshots.sh` and follow the instructions. This is mostly obviated by the functional tests, though. #### For real (longer) If you'd like to experience a real usage of assumeutxo, you can do that too. I've cut a new snapshot at height 788'000 (http://img.jameso.be/utxo-788000.dat - but you can do it yourself with `./contrib/devtools/utxo_snapshot.sh` if you want). Download that, and then create a datadir for testing: ```sh $ cd ~/src/bitcoin # or whatever # get the snapshot $ curl http://img.jameso.be/utxo-788000.dat > utxo-788000.dat # you'll want to do this if you like copy/pasting $ export AU_DATADIR=/home/${USER}/au-test # or wherever $ mkdir ${AU_DATADIR} $ vim ${AU_DATADIR}/bitcoin.conf dbcache=8000 # or, you know, something high blockfilterindex=1 coinstatsindex=1 prune=3000 logthreadnames=1 ``` Obtain this branch, build it, and then start bitcoind: ```sh $ git remote add jamesob https://github.com/jamesob/bitcoin $ git fetch jamesob assumeutxo $ git checkout jamesob/assumeutxo $ ./configure $conf_args && make # (whatever you like to do here) # start 'er up and watch the logs $ ./src/bitcoind -datadir=${AU_DATADIR} ``` Then, in some other window, load the snapshot ```sh $ ./src/bitcoin-cli -datadir=${AU_DATADIR} loadtxoutset $(pwd)/utxo-788000.dat ``` You'll see some log messages about headers retrieval and waiting to see the snapshot in the headers chain. Once you get the full headers chain, you'll spend a decent amount of time (~10min) loading the snapshot, checking it, and flushing it to disk. After all that happens, you should be syncing to tip in pretty short order, and you'll see the occasional `[background validation]` log message go by. In yet another window, you can check out chainstate status with ```sh $ ./src/bitcoin-cli -datadir=${AU_DATADIR} getchainstates ``` as well as usual favorites like `getblockchaininfo`. ACKs for top commit: achow101: ACK edbed31066 Tree-SHA512: 6086fb9a38dc7df85fedc76b30084dd8154617a2a91e89a84fb41326d34ef8e7d7ea593107afba01369093bf8cc91770621d98f0ea42a5b3b99db868d2f14dc2
2025-11-10 14:08:40 +01:00 · 2023-10-02 17:00:28 -04:00
parent fd8ab08558 edbed31066
commit e7b0004b37
43 changed files with 1599 additions and 291 deletions
--- a/test/functional/feature_assumeutxo.py
+++ b/test/functional/feature_assumeutxo.py
@@ -0,0 +1,246 @@
+#!/usr/bin/env python3
+# Copyright (c) 2021 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+"""Test for assumeutxo, a means of quickly bootstrapping a node using
+a serialized version of the UTXO set at a certain height, which corresponds
+to a hash that has been compiled into bitcoind.
+
+The assumeutxo value generated and used here is committed to in
+`CRegTestParams::m_assumeutxo_data` in `src/chainparams.cpp`.
+
+## Possible test improvements
+
+- TODO: test submitting a transaction and verifying it appears in mempool
+- TODO: test what happens with -reindex and -reindex-chainstate before the
+      snapshot is validated, and make sure it's deleted successfully.
+
+Interesting test cases could be loading an assumeutxo snapshot file with:
+
+- TODO: An invalid hash
+- TODO: Valid hash but invalid snapshot file (bad coin height or truncated file or
+      bad other serialization)
+- TODO: Valid snapshot file, but referencing an unknown block
+- TODO: Valid snapshot file, but referencing a snapshot block that turns out to be
+      invalid, or has an invalid parent
+- TODO: Valid snapshot file and snapshot block, but the block is not on the
+      most-work chain
+
+Interesting starting states could be loading a snapshot when the current chain tip is:
+
+- TODO: An ancestor of snapshot block
+- TODO: Not an ancestor of the snapshot block but has less work
+- TODO: The snapshot block
+- TODO: A descendant of the snapshot block
+- TODO: Not an ancestor or a descendant of the snapshot block and has more work
+
+"""
+from test_framework.test_framework import BitcoinTestFramework
+from test_framework.util import assert_equal, wait_until_helper
+
+START_HEIGHT = 199
+SNAPSHOT_BASE_HEIGHT = 299
+FINAL_HEIGHT = 399
+COMPLETE_IDX = {'synced': True, 'best_block_height': FINAL_HEIGHT}
+
+
+class AssumeutxoTest(BitcoinTestFramework):
+
+    def set_test_params(self):
+        """Use the pregenerated, deterministic chain up to height 199."""
+        self.num_nodes = 3
+        self.rpc_timeout = 120
+        self.extra_args = [
+            [],
+            ["-fastprune", "-prune=1", "-blockfilterindex=1", "-coinstatsindex=1"],
+            ["-txindex=1", "-blockfilterindex=1", "-coinstatsindex=1"],
+        ]
+
+    def setup_network(self):
+        """Start with the nodes disconnected so that one can generate a snapshot
+        including blocks the other hasn't yet seen."""
+        self.add_nodes(3)
+        self.start_nodes(extra_args=self.extra_args)
+
+    def run_test(self):
+        """
+        Bring up two (disconnected) nodes, mine some new blocks on the first,
+        and generate a UTXO snapshot.
+
+        Load the snapshot into the second, ensure it syncs to tip and completes
+        background validation when connected to the first.
+        """
+        n0 = self.nodes[0]
+        n1 = self.nodes[1]
+        n2 = self.nodes[2]
+
+        # Mock time for a deterministic chain
+        for n in self.nodes:
+            n.setmocktime(n.getblockheader(n.getbestblockhash())['time'])
+
+        self.sync_blocks()
+
+        def no_sync():
+            pass
+
+        # Generate a series of blocks that `n0` will have in the snapshot,
+        # but that n1 doesn't yet see. In order for the snapshot to activate,
+        # though, we have to ferry over the new headers to n1 so that it
+        # isn't waiting forever to see the header of the snapshot's base block
+        # while disconnected from n0.
+        for i in range(100):
+            self.generate(n0, nblocks=1, sync_fun=no_sync)
+            newblock = n0.getblock(n0.getbestblockhash(), 0)
+
+            # make n1 aware of the new header, but don't give it the block.
+            n1.submitheader(newblock)
+            n2.submitheader(newblock)
+
+        # Ensure everyone is seeing the same headers.
+        for n in self.nodes:
+            assert_equal(n.getblockchaininfo()["headers"], SNAPSHOT_BASE_HEIGHT)
+
+        self.log.info("-- Testing assumeutxo + some indexes + pruning")
+
+        assert_equal(n0.getblockcount(), SNAPSHOT_BASE_HEIGHT)
+        assert_equal(n1.getblockcount(), START_HEIGHT)
+
+        self.log.info(f"Creating a UTXO snapshot at height {SNAPSHOT_BASE_HEIGHT}")
+        dump_output = n0.dumptxoutset('utxos.dat')
+
+        assert_equal(
+            dump_output['txoutset_hash'],
+            'ef45ccdca5898b6c2145e4581d2b88c56564dd389e4bd75a1aaf6961d3edd3c0')
+        assert_equal(dump_output['nchaintx'], 300)
+        assert_equal(n0.getblockchaininfo()["blocks"], SNAPSHOT_BASE_HEIGHT)
+
+        # Mine more blocks on top of the snapshot that n1 hasn't yet seen. This
+        # will allow us to test n1's sync-to-tip on top of a snapshot.
+        self.generate(n0, nblocks=100, sync_fun=no_sync)
+
+        assert_equal(n0.getblockcount(), FINAL_HEIGHT)
+        assert_equal(n1.getblockcount(), START_HEIGHT)
+
+        assert_equal(n0.getblockchaininfo()["blocks"], FINAL_HEIGHT)
+
+        self.log.info(f"Loading snapshot into second node from {dump_output['path']}")
+        loaded = n1.loadtxoutset(dump_output['path'])
+        assert_equal(loaded['coins_loaded'], SNAPSHOT_BASE_HEIGHT)
+        assert_equal(loaded['base_height'], SNAPSHOT_BASE_HEIGHT)
+
+        monitor = n1.getchainstates()
+        assert_equal(monitor['normal']['blocks'], START_HEIGHT)
+        assert_equal(monitor['snapshot']['blocks'], SNAPSHOT_BASE_HEIGHT)
+        assert_equal(monitor['snapshot']['snapshot_blockhash'], dump_output['base_hash'])
+
+        assert_equal(n1.getblockchaininfo()["blocks"], SNAPSHOT_BASE_HEIGHT)
+
+        PAUSE_HEIGHT = FINAL_HEIGHT - 40
+
+        self.log.info("Restarting node to stop at height %d", PAUSE_HEIGHT)
+        self.restart_node(1, extra_args=[
+            f"-stopatheight={PAUSE_HEIGHT}", *self.extra_args[1]])
+
+        # Finally connect the nodes and let them sync.
+        self.connect_nodes(0, 1)
+
+        n1.wait_until_stopped(timeout=5)
+
+        self.log.info("Checking that blocks are segmented on disk")
+        assert self.has_blockfile(n1, "00000"), "normal blockfile missing"
+        assert self.has_blockfile(n1, "00001"), "assumed blockfile missing"
+        assert not self.has_blockfile(n1, "00002"), "too many blockfiles"
+
+        self.log.info("Restarted node before snapshot validation completed, reloading...")
+        self.restart_node(1, extra_args=self.extra_args[1])
+        self.connect_nodes(0, 1)
+
+        self.log.info(f"Ensuring snapshot chain syncs to tip. ({FINAL_HEIGHT})")
+        wait_until_helper(lambda: n1.getchainstates()['snapshot']['blocks'] == FINAL_HEIGHT)
+        self.sync_blocks(nodes=(n0, n1))
+
+        self.log.info("Ensuring background validation completes")
+        # N.B.: the `snapshot` key disappears once the background validation is complete.
+        wait_until_helper(lambda: not n1.getchainstates().get('snapshot'))
+
+        # Ensure indexes have synced.
+        completed_idx_state = {
+            'basic block filter index': COMPLETE_IDX,
+            'coinstatsindex': COMPLETE_IDX,
+        }
+        self.wait_until(lambda: n1.getindexinfo() == completed_idx_state)
+
+
+        for i in (0, 1):
+            n = self.nodes[i]
+            self.log.info(f"Restarting node {i} to ensure (Check|Load)BlockIndex passes")
+            self.restart_node(i, extra_args=self.extra_args[i])
+
+            assert_equal(n.getblockchaininfo()["blocks"], FINAL_HEIGHT)
+
+            assert_equal(n.getchainstates()['normal']['blocks'], FINAL_HEIGHT)
+            assert_equal(n.getchainstates().get('snapshot'), None)
+
+            if i != 0:
+                # Ensure indexes have synced for the assumeutxo node
+                self.wait_until(lambda: n.getindexinfo() == completed_idx_state)
+
+
+        # Node 2: all indexes + reindex
+        # -----------------------------
+
+        self.log.info("-- Testing all indexes + reindex")
+        assert_equal(n2.getblockcount(), START_HEIGHT)
+
+        self.log.info(f"Loading snapshot into third node from {dump_output['path']}")
+        loaded = n2.loadtxoutset(dump_output['path'])
+        assert_equal(loaded['coins_loaded'], SNAPSHOT_BASE_HEIGHT)
+        assert_equal(loaded['base_height'], SNAPSHOT_BASE_HEIGHT)
+
+        monitor = n2.getchainstates()
+        assert_equal(monitor['normal']['blocks'], START_HEIGHT)
+        assert_equal(monitor['snapshot']['blocks'], SNAPSHOT_BASE_HEIGHT)
+        assert_equal(monitor['snapshot']['snapshot_blockhash'], dump_output['base_hash'])
+
+        self.connect_nodes(0, 2)
+        wait_until_helper(lambda: n2.getchainstates()['snapshot']['blocks'] == FINAL_HEIGHT)
+        self.sync_blocks()
+
+        self.log.info("Ensuring background validation completes")
+        wait_until_helper(lambda: not n2.getchainstates().get('snapshot'))
+
+        completed_idx_state = {
+            'basic block filter index': COMPLETE_IDX,
+            'coinstatsindex': COMPLETE_IDX,
+            'txindex': COMPLETE_IDX,
+        }
+        self.wait_until(lambda: n2.getindexinfo() == completed_idx_state)
+
+        for i in (0, 2):
+            n = self.nodes[i]
+            self.log.info(f"Restarting node {i} to ensure (Check|Load)BlockIndex passes")
+            self.restart_node(i, extra_args=self.extra_args[i])
+
+            assert_equal(n.getblockchaininfo()["blocks"], FINAL_HEIGHT)
+
+            assert_equal(n.getchainstates()['normal']['blocks'], FINAL_HEIGHT)
+            assert_equal(n.getchainstates().get('snapshot'), None)
+
+            if i != 0:
+                # Ensure indexes have synced for the assumeutxo node
+                self.wait_until(lambda: n.getindexinfo() == completed_idx_state)
+
+        self.log.info("Test -reindex-chainstate of an assumeutxo-synced node")
+        self.restart_node(2, extra_args=[
+            '-reindex-chainstate=1', *self.extra_args[2]])
+        assert_equal(n2.getblockchaininfo()["blocks"], FINAL_HEIGHT)
+        wait_until_helper(lambda: n2.getblockcount() == FINAL_HEIGHT)
+
+        self.log.info("Test -reindex of an assumeutxo-synced node")
+        self.restart_node(2, extra_args=['-reindex=1', *self.extra_args[2]])
+        self.connect_nodes(0, 2)
+        wait_until_helper(lambda: n2.getblockcount() == FINAL_HEIGHT)
+
+
+if __name__ == '__main__':
+    AssumeutxoTest().main()
--- a/test/functional/test_framework/test_framework.py
+++ b/test/functional/test_framework/test_framework.py
@@ -979,3 +979,7 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass):
    def is_bdb_compiled(self):
        """Checks whether the wallet module was compiled with BDB support."""
        return self.config["components"].getboolean("USE_BDB")
+
+    def has_blockfile(self, node, filenum: str):
+        blocksdir = os.path.join(node.datadir, self.chain, 'blocks', '')
+        return os.path.isfile(os.path.join(blocksdir, f"blk{filenum}.dat"))
--- a/test/functional/test_runner.py
+++ b/test/functional/test_runner.py
@@ -324,6 +324,7 @@ BASE_SCRIPTS = [
    'wallet_coinbase_category.py --descriptors',
    'feature_filelock.py',
    'feature_loadblock.py',
+    'feature_assumeutxo.py',
    'p2p_dos_header_tree.py',
    'p2p_add_connections.py',
    'feature_bind_port_discover.py',
--- a/test/lint/lint-shell.py
+++ b/test/lint/lint-shell.py
@@ -67,9 +67,13 @@ def main():
        '*.sh',
    ]
    files = get_files(files_cmd)
-    # remove everything that doesn't match this regex
    reg = re.compile(r'src/[leveldb,secp256k1,minisketch]')
-    files[:] = [file for file in files if not reg.match(file)]
+
+    def should_exclude(fname: str) -> bool:
+        return bool(reg.match(fname)) or 'test_utxo_snapshots.sh' in fname
+
+    # remove everything that doesn't match this regex
+    files[:] = [file for file in files if not should_exclude(file)]

    # build the `shellcheck` command
    shellcheck_cmd = [