Merge bitcoin/bitcoin#21800: mempool/validation: mempool ancestor/descendant limits for packages

accf3d5868 [test] mempool package ancestor/descendant limits (glozow)
2b6b26e57c [test] parameterizable fee for make_chain and create_child_with_parents (glozow)
313c09f7b7 [test] helper function to increase transaction weight (glozow)
f8253d69d6 extract/rename helper functions from rpc_packages.py (glozow)
3cd663a5d3 [policy] ancestor/descendant limits for packages (glozow)
c6e016aa13 [mempool] check ancestor/descendant limits for packages (glozow)
f551841d3e [refactor] pass size/count instead of entry to CalculateAncestorsAndCheckLimits (glozow)
97dd1c729d MOVEONLY: add helper function for calculating ancestors and checking limits (glozow)
f95bbf58aa misc package validation doc improvements (glozow)

Pull request description:

  This PR implements a function to calculate mempool ancestors for a package and enforces ancestor/descendant limits on them as a whole. It reuses a portion of `CalculateMemPoolAncestors()`; there's also a small refactor to move the reused code into a generic helper function. Instead of calculating ancestors and descendants on every single transaction in the package and their ancestors, we use a "worst case" heuristic, treating every transaction in the package as each other's ancestor and descendant. This may overestimate everyone's counts, but is still pretty accurate in the our main package use cases, in which at least one of the transactions in the package is directly related to all the others (e.g. 1 parent + 1 child, multiple parents with 1 child, or chains).

  Note on Terminology: While "package" is often used to describe groups of related transactions _within_ the mempool, here, I only use package to mean the group of not-in-mempool transactions we are currently validating.

  #### Motivation

  It would be a potential DoS vector to allow submission of packages to mempool without a proper guard for mempool ancestors/descendants. In general, the purpose of mempool ancestor/descendant limits is to limit the computational complexity of dealing with families during removals and additions. We want to be able to validate multiple transactions on top of the mempool, but also avoid these scenarios:

  - We underestimate the ancestors/descendants during package validation and end up with extremely complex families in our mempool (potentially a DoS vector).
  - We expend an unreasonable amount of resources calculating everyone's ancestors and descendants during package validation.

ACKs for top commit:
  JeremyRubin:
    utACK accf3d5
  ariard:
    ACK accf3d5.

Tree-SHA512: 0d18ce4b77398fe872e0b7c2cc66d3aac2135e561b64029584339e1f4de2a6a16ebab3dd5784f376e119cbafc4d50168b28d3bd95d0b3d01158714ade2e3624d

test/functional/test_framework/wallet.py

@@ -4,8 +4,10 @@
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 """A limited-functionality wallet, which may replace a real wallet in tests"""
 
+from copy import deepcopy
 from decimal import Decimal
 from enum import Enum
+from random import choice
 from typing import Optional
 from test_framework.address import ADDRESS_BCRT1_P2WSH_OP_TRUE
 from test_framework.key import ECKey
@@ -16,6 +18,7 @@ from test_framework.messages import (
     CTxIn,
     CTxInWitness,
     CTxOut,
+    tx_from_hex,
 )
 from test_framework.script import (
     CScript,
@@ -27,9 +30,11 @@ from test_framework.script import (
 )
 from test_framework.util import (
     assert_equal,
+    assert_greater_than_or_equal,
     satoshi_round,
 )
 
+DEFAULT_FEE = Decimal("0.0001")
 
 class MiniWalletMode(Enum):
     """Determines the transaction type the MiniWallet is creating and spending.
@@ -176,3 +181,75 @@ class MiniWallet:
     def sendrawtransaction(self, *, from_node, tx_hex):
         from_node.sendrawtransaction(tx_hex)
         self.scan_tx(from_node.decoderawtransaction(tx_hex))
+
+def make_chain(node, address, privkeys, parent_txid, parent_value, n=0, parent_locking_script=None, fee=DEFAULT_FEE):
+    """Build a transaction that spends parent_txid.vout[n] and produces one output with
+    amount = parent_value with a fee deducted.
+    Return tuple (CTransaction object, raw hex, nValue, scriptPubKey of the output created).
+    """
+    inputs = [{"txid": parent_txid, "vout": n}]
+    my_value = parent_value - fee
+    outputs = {address : my_value}
+    rawtx = node.createrawtransaction(inputs, outputs)
+    prevtxs = [{
+        "txid": parent_txid,
+        "vout": n,
+        "scriptPubKey": parent_locking_script,
+        "amount": parent_value,
+    }] if parent_locking_script else None
+    signedtx = node.signrawtransactionwithkey(hexstring=rawtx, privkeys=privkeys, prevtxs=prevtxs)
+    assert signedtx["complete"]
+    tx = tx_from_hex(signedtx["hex"])
+    return (tx, signedtx["hex"], my_value, tx.vout[0].scriptPubKey.hex())
+
+def create_child_with_parents(node, address, privkeys, parents_tx, values, locking_scripts, fee=DEFAULT_FEE):
+    """Creates a transaction that spends the first output of each parent in parents_tx."""
+    num_parents = len(parents_tx)
+    total_value = sum(values)
+    inputs = [{"txid": tx.rehash(), "vout": 0} for tx in parents_tx]
+    outputs = {address : total_value - fee}
+    rawtx_child = node.createrawtransaction(inputs, outputs)
+    prevtxs = []
+    for i in range(num_parents):
+        prevtxs.append({"txid": parents_tx[i].rehash(), "vout": 0, "scriptPubKey": locking_scripts[i], "amount": values[i]})
+    signedtx_child = node.signrawtransactionwithkey(hexstring=rawtx_child, privkeys=privkeys, prevtxs=prevtxs)
+    assert signedtx_child["complete"]
+    return signedtx_child["hex"]
+
+def create_raw_chain(node, first_coin, address, privkeys, chain_length=25):
+    """Helper function: create a "chain" of chain_length transactions. The nth transaction in the
+    chain is a child of the n-1th transaction and parent of the n+1th transaction.
+    """
+    parent_locking_script = None
+    txid = first_coin["txid"]
+    chain_hex = []
+    chain_txns = []
+    value = first_coin["amount"]
+
+    for _ in range(chain_length):
+        (tx, txhex, value, parent_locking_script) = make_chain(node, address, privkeys, txid, value, 0, parent_locking_script)
+        txid = tx.rehash()
+        chain_hex.append(txhex)
+        chain_txns.append(tx)
+
+    return (chain_hex, chain_txns)
+
+def bulk_transaction(tx, node, target_weight, privkeys, prevtxs=None):
+    """Pad a transaction with extra outputs until it reaches a target weight (or higher).
+    returns CTransaction object
+    """
+    tx_heavy = deepcopy(tx)
+    assert_greater_than_or_equal(target_weight, tx_heavy.get_weight())
+    while tx_heavy.get_weight() < target_weight:
+        random_spk = "6a4d0200"  # OP_RETURN OP_PUSH2 512 bytes
+        for _ in range(512*2):
+            random_spk += choice("0123456789ABCDEF")
+        tx_heavy.vout.append(CTxOut(0, bytes.fromhex(random_spk)))
+    # Re-sign the transaction
+    if privkeys:
+        signed = node.signrawtransactionwithkey(tx_heavy.serialize().hex(), privkeys, prevtxs)
+        return tx_from_hex(signed["hex"])
+    # OP_TRUE
+    tx_heavy.wit.vtxinwit = [CTxInWitness()]
+    tx_heavy.wit.vtxinwit[0].scriptWitness.stack = [CScript([OP_TRUE])]
+    return tx_heavy