Merge #14121: Index for BIP 157 block filters

c7efb652f3 blockfilter: Update BIP 158 test vectors. (Jim Posen)
19308c9e21 rpc: Add getblockfilter RPC method. (Jim Posen)
ff35105096 init: Add CLI option to enable block filter index. (Jim Posen)
accc8b8b18 index: Access functions for global block filter indexes. (Jim Posen)
2bc90e4e7b test: Unit test for block filter index reorg handling. (Jim Posen)
6bcf0998c0 test: Unit tests for block index filter. (Jim Posen)
b5e8200db7 index: Implement lookup methods on block filter index. (Jim Posen)
75a76e3619 index: Implement block filter index with write operations. (Jim Posen)
2ad2338ef9 serialize: Serialization support for big-endian 32-bit ints. (Jim Posen)
ba6ff9a6f7 blockfilter: Functions to translate filter types to/from names. (Jim Posen)
62b7a4f094 index: Ensure block locator is not stale after chain reorg. (Jim Posen)
4368384f1d index: Allow atomic commits of index state to be extended. (Jim Posen)

Pull request description:

  This introduces a new BlockFilterIndex class, which is required for BIP 157 support.

  The index is uses the asynchronous BaseIndex infrastructure driven by the ValidationInterface callbacks. Filters are stored sequentially in flat files and the disk location of each filter is indexed in LevelDB along with the filter hash and header. The index is designed to ensure persistence of filters reorganized out of the main chain to simplify the BIP 157 net implementation.

  Stats (block height = 565500):
  - Syncing the index from scratch takes 45m
  - Total index size is 3.8 GiB

ACKs for commit c7efb6:
  MarcoFalke:
    utACK c7efb652f3
  ryanofsky:
    Slightly tested ACK c7efb652f3 (I just rebuilt the index with the updated PR and tested the RPC). Changes since last review: rebase, fixed compile errors in internal commits, new comments, updated error messages, tweaked cache size logic, renamed commit method, renamed constants and globals, fixed whitespace, extra BlockFilterIndex::Init error check.

Tree-SHA512: f8ed7a9b6f76df45933aa5eba92b27b3af83f6df2ccb3728a5c89eec80f654344dc14f055f6f63eb9b3a7649dd8af6553fe14969889e7e2fd2f8461574d18f28

test/functional/rpc_getblockfilter.py

@@ -0,0 +1,59 @@
+#!/usr/bin/env python3
+# Copyright (c) 2018 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+"""Test the getblockfilter RPC."""
+
+from test_framework.test_framework import BitcoinTestFramework
+from test_framework.util import (
+    assert_equal, assert_is_hex_string, assert_raises_rpc_error,
+    connect_nodes, disconnect_nodes, sync_blocks
+    )
+
+FILTER_TYPES = ["basic"]
+
+class GetBlockFilterTest(BitcoinTestFramework):
+    def set_test_params(self):
+        self.setup_clean_chain = True
+        self.num_nodes = 2
+        self.extra_args = [["-blockfilterindex"], []]
+
+    def run_test(self):
+        # Create two chains by disconnecting nodes 0 & 1, mining, then reconnecting
+        disconnect_nodes(self.nodes[0], 1)
+
+        self.nodes[0].generate(3)
+        self.nodes[1].generate(4)
+
+        assert_equal(self.nodes[0].getblockcount(), 3)
+        chain0_hashes = [self.nodes[0].getblockhash(block_height) for block_height in range(4)]
+
+        # Reorg node 0 to a new chain
+        connect_nodes(self.nodes[0], 1)
+        sync_blocks(self.nodes)
+
+        assert_equal(self.nodes[0].getblockcount(), 4)
+        chain1_hashes = [self.nodes[0].getblockhash(block_height) for block_height in range(4)]
+
+        # Test getblockfilter returns a filter for all blocks and filter types on active chain
+        for block_hash in chain1_hashes:
+            for filter_type in FILTER_TYPES:
+                result = self.nodes[0].getblockfilter(block_hash, filter_type)
+                assert_is_hex_string(result['filter'])
+
+        # Test getblockfilter returns a filter for all blocks and filter types on stale chain
+        for block_hash in chain0_hashes:
+            for filter_type in FILTER_TYPES:
+                result = self.nodes[0].getblockfilter(block_hash, filter_type)
+                assert_is_hex_string(result['filter'])
+
+        # Test getblockfilter with unknown block
+        bad_block_hash = "0123456789abcdef" * 4
+        assert_raises_rpc_error(-5, "Block not found", self.nodes[0].getblockfilter, bad_block_hash, "basic")
+
+        # Test getblockfilter with undefined filter type
+        genesis_hash = self.nodes[0].getblockhash(0)
+        assert_raises_rpc_error(-5, "Unknown filtertype", self.nodes[0].getblockfilter, genesis_hash, "unknown")
+
+if __name__ == '__main__':
+    GetBlockFilterTest().main()

test/functional/test_runner.py

@@ -144,6 +144,7 @@ BASE_SCRIPTS = [
     'wallet_txn_doublespend.py',
     'wallet_txn_clone.py --mineblock',
     'feature_notifications.py',
+    'rpc_getblockfilter.py',
     'rpc_invalidateblock.py',
     'feature_rbf.py',
     'mempool_packages.py',