Merge 14c85a9b3753b40e8cf9f2808530fa922c8a3473 into 5f4422d68dc3530c353af1f87499de1c864b60ad

Merge bitcoin/bitcoin#32010 : qa: Fix TxIndex race conditions
3301d2cbe8c3b76c97285d75fa59637cb6952d0b qa: Wait for txindex to avoid race condition (Hodlinator) 9bfb0d75ba10591cc6c9620f9fd1ecc0e55e7a48 qa: Remove unnecessary -txindex args (Hodlinator) 7ac281c19cd3d11f316dbbb3308eabf1ad4f26d6 qa: Add missing coverage of corrupt indexes (Hodlinator) Pull request description: - Add synchronization in 3 places where if the Transaction Index happens to be slow, we get rare test failures when querying it for transactions (one such case experienced on Windows, prompting investigation). - Remove unnecessary TxIndex initialization in some tests. - Add some test coverage where TxIndex aspect could be tested in feature_init.py. ACKs for top commit: fjahr: re-ACK 3301d2cbe8c3b76c97285d75fa59637cb6952d0b mzumsande: Code Review ACK 3301d2cbe8c3b76c97285d75fa59637cb6952d0b furszy: Code review ACK 3301d2cbe8c3b76c97285d75fa59637cb6952d0b Prabhat1308: Concept ACK [`3301d2c`](3301d2cbe8) Tree-SHA512: 7c2019e38455f344856aaf6b381faafbd88d53dc88d13309deb718c1dcfbee4ccca7c7f1b66917395503a6f94c3b216a007ad432cc8b93d0309db9805f38d602
2025-03-17 21:32:00 +01:00 · 2025-03-17 09:50:15 +07:00 · 2025-03-17 10:28:14 +08:00 · 2025-03-10 15:24:16 +01:00 · 2025-03-07 22:22:31 +01:00 · 2025-03-07 22:22:31 +01:00
7 changed files with 64 additions and 29 deletions
--- a/doc/dependencies.md
+++ b/doc/dependencies.md
@ -1,8 +1,8 @@
 # Dependencies

 These are the dependencies used by Bitcoin Core.
-You can find installation instructions in the `build-*.md` file for your platform.
-"Runtime" and "Version Used" are both in reference to the release binaries.
+You can find installation instructions in the `build-*.md` file for your platform, or self-compile
+them using [depends](/depends/README.md). "Version Used" refers to the release binaries.

 ## Compiler

@ -15,24 +15,37 @@ Bitcoin Core requires one of the following compilers.

 ## Required

-| Dependency | Releases | Version used | Minimum required | Runtime |
-| --- | --- | --- | --- | --- |
-| CMake | [link](https://cmake.org/) | N/A | [3.22](https://github.com/bitcoin/bitcoin/pull/30454) | No |
-| [Boost](../depends/packages/boost.mk) | [link](https://www.boost.org/users/download/) | [1.81.0](https://github.com/bitcoin/bitcoin/pull/26557) | [1.73.0](https://github.com/bitcoin/bitcoin/pull/29066) | No |
-| [libevent](../depends/packages/libevent.mk) | [link](https://github.com/libevent/libevent/releases) | [2.1.12-stable](https://github.com/bitcoin/bitcoin/pull/21991) | [2.1.8](https://github.com/bitcoin/bitcoin/pull/24681) | No |
-| glibc | [link](https://www.gnu.org/software/libc/) | N/A | [2.31](https://github.com/bitcoin/bitcoin/pull/29987) | Yes |
-| Linux Kernel (if building that platform) | [link](https://www.kernel.org/) | N/A | [3.17.0](https://github.com/bitcoin/bitcoin/pull/27699) | Yes |
+**Build:**
+
+| Dependency | Releases | Version used | Minimum required |
+| --- | --- | --- | --- |
+| [Boost](../depends/packages/boost.mk) | [link](https://www.boost.org/users/download/) | [1.81.0](https://github.com/bitcoin/bitcoin/pull/26557) | [1.73.0](https://github.com/bitcoin/bitcoin/pull/29066) |
+| CMake | [link](https://cmake.org/) | [3.24.2](https://github.com/bitcoin/bitcoin/pull/30730) | [3.22](https://github.com/bitcoin/bitcoin/pull/30454) |
+| [libevent](../depends/packages/libevent.mk) | [link](https://github.com/libevent/libevent/releases) | [2.1.12-stable](https://github.com/bitcoin/bitcoin/pull/21991) | [2.1.8](https://github.com/bitcoin/bitcoin/pull/24681) |
+
+**Runtime:**
+
+| Dependency | Releases | Minimum required |
+| --- | --- | --- |
+| glibc | [link](https://www.gnu.org/software/libc/) | [2.31](https://github.com/bitcoin/bitcoin/pull/29987)

 ## Optional

-| Dependency | Releases | Version used | Minimum required | Runtime |
-| --- | --- | --- | --- | --- |
-| [Fontconfig](../depends/packages/fontconfig.mk) (gui) | [link](https://www.freedesktop.org/wiki/Software/fontconfig/) | [2.12.6](https://github.com/bitcoin/bitcoin/pull/23495) | 2.6 | Yes |
-| [FreeType](../depends/packages/freetype.mk) (gui) | [link](https://freetype.org) | [2.11.0](https://github.com/bitcoin/bitcoin/commit/01544dd78ccc0b0474571da854e27adef97137fb) | 2.3.0 | Yes |
-| [qrencode](../depends/packages/qrencode.mk) (gui) | [link](https://fukuchi.org/works/qrencode/) | [4.1.1](https://github.com/bitcoin/bitcoin/pull/27312) | N/A | No |
-| [Qt](../depends/packages/qt.mk) (gui) | [link](https://download.qt.io/official_releases/qt/) | [5.15.16](https://github.com/bitcoin/bitcoin/pull/30774) | [5.11.3](https://github.com/bitcoin/bitcoin/pull/24132) | No |
-| [ZeroMQ](../depends/packages/zeromq.mk) (notifications) | [link](https://github.com/zeromq/libzmq/releases) | [4.3.4](https://github.com/bitcoin/bitcoin/pull/23956) | 4.0.0 | No |
-| [Berkeley DB](../depends/packages/bdb.mk) (legacy wallet) | [link](https://www.oracle.com/technetwork/database/database-technologies/berkeleydb/downloads/index.html) | 4.8.30 | 4.8.x | No |
-| [SQLite](../depends/packages/sqlite.mk) (wallet) | [link](https://sqlite.org) | [3.38.5](https://github.com/bitcoin/bitcoin/pull/25378) | [3.7.17](https://github.com/bitcoin/bitcoin/pull/19077) | No |
-| Python (scripts, tests) | [link](https://www.python.org) | N/A | [3.10](https://github.com/bitcoin/bitcoin/pull/30527) | No |
-| [systemtap](../depends/packages/systemtap.mk) ([tracing](tracing.md)) | [link](https://sourceware.org/systemtap/) | [4.8](https://github.com/bitcoin/bitcoin/pull/26945)| N/A | No |
+**Build:**
+
+| Dependency | Releases | Version used | Minimum required |
+| --- | --- | --- | --- |
+| [Berkeley DB](../depends/packages/bdb.mk) (legacy wallet) | [link](https://www.oracle.com/technetwork/database/database-technologies/berkeleydb/downloads/index.html) | 4.8.30 | 4.8.x |
+| Python (scripts, tests) | [link](https://www.python.org) | [3.10](https://github.com/bitcoin/bitcoin/pull/30527) | [3.10](https://github.com/bitcoin/bitcoin/pull/30527) |
+| [Qt](../depends/packages/qt.mk) (gui) | [link](https://download.qt.io/official_releases/qt/) | [5.15.16](https://github.com/bitcoin/bitcoin/pull/30774) | [5.11.3](https://github.com/bitcoin/bitcoin/pull/24132) |
+| [qrencode](../depends/packages/qrencode.mk) (gui) | [link](https://fukuchi.org/works/qrencode/) | [4.1.1](https://github.com/bitcoin/bitcoin/pull/27312) | N/A |
+| [SQLite](../depends/packages/sqlite.mk) (wallet) | [link](https://sqlite.org) | [3.38.5](https://github.com/bitcoin/bitcoin/pull/25378) | [3.7.17](https://github.com/bitcoin/bitcoin/pull/19077) |
+| [systemtap](../depends/packages/systemtap.mk) ([tracing](tracing.md)) | [link](https://sourceware.org/systemtap/) | [4.8](https://github.com/bitcoin/bitcoin/pull/26945)| N/A |
+| [ZeroMQ](../depends/packages/zeromq.mk) (notifications) | [link](https://github.com/zeromq/libzmq/releases) | [4.3.4](https://github.com/bitcoin/bitcoin/pull/23956) | 4.0.0 |
+
+**Runtime:**
+
+| Dependency | Releases | Minimum required |
+| --- | --- | --- |
+| [Fontconfig](../depends/packages/fontconfig.mk) (gui) | [link](https://www.freedesktop.org/wiki/Software/fontconfig/) | 2.6 |
+| [FreeType](../depends/packages/freetype.mk) (gui) | [link](https://freetype.org) | 2.3.0 |
--- a/test/functional/feature_init.py
+++ b/test/functional/feature_init.py
@ -88,7 +88,7 @@ class InitTest(BitcoinTestFramework):

        args = ['-txindex=1', '-blockfilterindex=1', '-coinstatsindex=1']
        for terminate_line in lines_to_terminate_after:
-            self.log.info(f"Starting node and will exit after line {terminate_line}")
+            self.log.info(f"Starting node and will terminate after line {terminate_line}")
            with node.busy_wait_for_debug_log([terminate_line]):
                if platform.system() == 'Windows':
                    # CREATE_NEW_PROCESS_GROUP is required in order to be able
@ -108,12 +108,22 @@ class InitTest(BitcoinTestFramework):
            'blocks/index/*.ldb': 'Error opening block database.',
            'chainstate/*.ldb': 'Error opening coins database.',
            'blocks/blk*.dat': 'Error loading block database.',
+            'indexes/txindex/MANIFEST*': 'LevelDB error: Corruption: CURRENT points to a non-existent file',
+            # Removing these files does not result in a startup error:
+            # 'indexes/blockfilter/basic/*.dat', 'indexes/blockfilter/basic/db/*.*', 'indexes/coinstats/db/*.*',
+            # 'indexes/txindex/*.log', 'indexes/txindex/CURRENT', 'indexes/txindex/LOCK'
        }

        files_to_perturb = {
            'blocks/index/*.ldb': 'Error loading block database.',
            'chainstate/*.ldb': 'Error opening coins database.',
            'blocks/blk*.dat': 'Corrupted block database detected.',
+            'indexes/blockfilter/basic/db/*.*': 'LevelDB error: Corruption',
+            'indexes/coinstats/db/*.*': 'LevelDB error: Corruption',
+            'indexes/txindex/*.log': 'LevelDB error: Corruption',
+            'indexes/txindex/CURRENT': 'LevelDB error: Corruption',
+            # Perturbing these files does not result in a startup error:
+            # 'indexes/blockfilter/basic/*.dat', 'indexes/txindex/MANIFEST*', 'indexes/txindex/LOCK'
        }

        for file_patt, err_fragment in files_to_delete.items():
@ -135,9 +145,10 @@ class InitTest(BitcoinTestFramework):
            self.stop_node(0)

        self.log.info("Test startup errors after perturbing certain essential files")
+        dirs = ["blocks", "chainstate", "indexes"]
        for file_patt, err_fragment in files_to_perturb.items():
-            shutil.copytree(node.chain_path / "blocks", node.chain_path / "blocks_bak")
-            shutil.copytree(node.chain_path / "chainstate", node.chain_path / "chainstate_bak")
+            for dir in dirs:
+                shutil.copytree(node.chain_path / dir, node.chain_path / f"{dir}_bak")
            target_files = list(node.chain_path.glob(file_patt))

            for target_file in target_files:
@ -151,10 +162,9 @@ class InitTest(BitcoinTestFramework):

            start_expecting_error(err_fragment)

-            shutil.rmtree(node.chain_path / "blocks")
-            shutil.rmtree(node.chain_path / "chainstate")
-            shutil.move(node.chain_path / "blocks_bak", node.chain_path / "blocks")
-            shutil.move(node.chain_path / "chainstate_bak", node.chain_path / "chainstate")
+            for dir in dirs:
+                shutil.rmtree(node.chain_path / dir)
+                shutil.move(node.chain_path / f"{dir}_bak", node.chain_path / dir)

    def init_pid_test(self):
        BITCOIN_PID_FILENAME_CUSTOM = "my_fancy_bitcoin_pid_file.foobar"
--- a/test/functional/mempool_accept.py
+++ b/test/functional/mempool_accept.py
@ -45,6 +45,7 @@ from test_framework.util import (
    assert_equal,
    assert_greater_than,
    assert_raises_rpc_error,
+    sync_txindex,
 )
 from test_framework.wallet import MiniWallet
 from test_framework.wallet_util import generate_keypair
@ -270,6 +271,7 @@ class MempoolAcceptanceTest(BitcoinTestFramework):

        self.log.info('A coinbase transaction')
        # Pick the input of the first tx we created, so it has to be a coinbase tx
+        sync_txindex(self, node)
        raw_tx_coinbase_spent = node.getrawtransaction(txid=node.decoderawtransaction(hexstring=raw_tx_in_block)['vin'][0]['txid'])
        tx = tx_from_hex(raw_tx_coinbase_spent)
        self.check_mempool_result(
--- a/test/functional/rpc_rawtransaction.py
+++ b/test/functional/rpc_rawtransaction.py
@ -34,6 +34,7 @@ from test_framework.util import (
    assert_equal,
    assert_greater_than,
    assert_raises_rpc_error,
+    sync_txindex,
 )
 from test_framework.wallet import (
    getnewdestination,
@ -70,7 +71,7 @@ class RawTransactionsTest(BitcoinTestFramework):
        self.num_nodes = 3
        self.extra_args = [
            ["-txindex"],
-            ["-txindex"],
+            [],
            ["-fastprune", "-prune=1"],
        ]
        # whitelist peers to speed up tx relay / mempool sync
@ -109,6 +110,7 @@ class RawTransactionsTest(BitcoinTestFramework):
            self.log.info(f"Test getrawtransaction {'with' if n == 0 else 'without'} -txindex")

            if n == 0:
+                sync_txindex(self, self.nodes[n])
                # With -txindex.
                # 1. valid parameters - only supply txid
                assert_equal(self.nodes[n].getrawtransaction(txId), tx['hex'])
--- a/test/functional/rpc_txoutproof.py
+++ b/test/functional/rpc_txoutproof.py
@ -12,6 +12,7 @@ from test_framework.test_framework import BitcoinTestFramework
 from test_framework.util import (
    assert_equal,
    assert_raises_rpc_error,
+    sync_txindex,
 )
 from test_framework.wallet import MiniWallet

@ -77,6 +78,7 @@ class MerkleBlockTest(BitcoinTestFramework):
        assert_equal(sorted(self.nodes[0].verifytxoutproof(self.nodes[0].gettxoutproof([txid1, txid2]))), sorted(txlist))
        assert_equal(sorted(self.nodes[0].verifytxoutproof(self.nodes[0].gettxoutproof([txid2, txid1]))), sorted(txlist))
        # We can always get a proof if we have a -txindex
+        sync_txindex(self, self.nodes[1])
        assert_equal(self.nodes[0].verifytxoutproof(self.nodes[1].gettxoutproof([txid_spent])), [txid_spent])
        # We can't get a proof if we specify transactions from different blocks
        assert_raises_rpc_error(-5, "Not all transactions found in specified or retrieved block", self.nodes[0].gettxoutproof, [txid1, txid3])
--- a/test/functional/test_framework/util.py
+++ b/test/functional/test_framework/util.py
@ -592,3 +592,10 @@ def find_vout_for_address(node, txid, addr):
        if addr == tx["vout"][i]["scriptPubKey"]["address"]:
            return i
    raise RuntimeError("Vout not found for address: txid=%s, addr=%s" % (txid, addr))
+
+
+def sync_txindex(test_framework, node):
+    test_framework.log.debug("Waiting for node txindex to sync")
+    sync_start = int(time.time())
+    test_framework.wait_until(lambda: node.getindexinfo("txindex")["txindex"]["synced"])
+    test_framework.log.debug(f"Synced in {time.time() - sync_start} seconds")
--- a/test/functional/wallet_avoid_mixing_output_types.py
+++ b/test/functional/wallet_avoid_mixing_output_types.py
@ -117,7 +117,6 @@ class AddressInputTypeGrouping(BitcoinTestFramework):
        self.extra_args = [
            [
                "-addresstype=bech32",
-                "-txindex",
            ],
            [
                "-addresstype=p2sh-segwit",
Author	SHA1	Message	Date
NicolaLS	d739bd7375	Merge 14c85a9b3753b40e8cf9f2808530fa922c8a3473 into 5f4422d68dc3530c353af1f87499de1c864b60ad	2025-03-17 09:50:15 +07:00
merge-script	5f4422d68d	Merge bitcoin/bitcoin#32010 : qa: Fix TxIndex race conditions 3301d2cbe8c3b76c97285d75fa59637cb6952d0b qa: Wait for txindex to avoid race condition (Hodlinator) 9bfb0d75ba10591cc6c9620f9fd1ecc0e55e7a48 qa: Remove unnecessary -txindex args (Hodlinator) 7ac281c19cd3d11f316dbbb3308eabf1ad4f26d6 qa: Add missing coverage of corrupt indexes (Hodlinator) Pull request description: - Add synchronization in 3 places where if the Transaction Index happens to be slow, we get rare test failures when querying it for transactions (one such case experienced on Windows, prompting investigation). - Remove unnecessary TxIndex initialization in some tests. - Add some test coverage where TxIndex aspect could be tested in feature_init.py. ACKs for top commit: fjahr: re-ACK 3301d2cbe8c3b76c97285d75fa59637cb6952d0b mzumsande: Code Review ACK 3301d2cbe8c3b76c97285d75fa59637cb6952d0b furszy: Code review ACK 3301d2cbe8c3b76c97285d75fa59637cb6952d0b Prabhat1308: Concept ACK [`3301d2c`](`3301d2cbe8`) Tree-SHA512: 7c2019e38455f344856aaf6b381faafbd88d53dc88d13309deb718c1dcfbee4ccca7c7f1b66917395503a6f94c3b216a007ad432cc8b93d0309db9805f38d602	2025-03-17 10:28:14 +08:00
Hodlinator	3301d2cbe8	qa: Wait for txindex to avoid race condition Can be verified to be necessary through adding std::this_thread::sleep_for(0.5s) at the beginning of TxIndex::CustomAppend.	2025-03-10 15:24:16 +01:00
Hodlinator	9bfb0d75ba	qa: Remove unnecessary -txindex args (Parent commit ensured indexes in feature_init.py are actually used, otherwise they would be removed here as well).	2025-03-07 22:22:31 +01:00
Hodlinator	7ac281c19c	qa: Add missing coverage of corrupt indexes	2025-03-07 22:22:31 +01:00
Nicola Leonardo Susca	14c85a9b37	doc: Order dependencies alphabetically	2025-02-18 10:42:20 -06:00
Nicola Leonardo Susca	ee4689f48d	doc: Add dependency self-compilation info The `dependencies.md` should mention that it is possible to self-compile the dependencies and reference `depends/README.md` for instructions.	2025-02-18 07:11:59 -06:00
Nicola Leonardo Susca	227cb8e6f9	doc: Add Python version used info	2025-02-18 07:10:48 -06:00
Nicola Leonardo Susca	d8035e7459	doc: Split dep. tables into build and runtime - Simplify distinguishing build/runtime dependencies (two tables instead of searching through the lines of one table looking at the `Runtime` column). - Consolidates build/runtime information better, rows in tables don't repeat the same information anymore (i.e. "No"/"Yes"). - Don't reference `Version used` for runtime dependencies as they are unknown unless the dependencies are built using `depends` in that case the versions used are apparent.	2025-02-18 07:10:43 -06:00
Nicola Leonardo Susca	69118dda49	doc: Add CMake version used info CMake was made a global requirement (#29725) and the version used is derived from guix time-machine recently bumped in #30730.	2025-02-18 07:06:29 -06:00
Nicola Leonardo Susca	b226a8fffd	doc: Remove Linux Kernel from dep. table Linux Kernel is neither an optional nor a required dependency for Bitcoin Core.	2025-02-18 07:06:08 -06:00