highperfocused/bitcoin
1
0
Fork 0
mirror of https://github.com/bitcoin/bitcoin.git synced 2026-06-08 13:49:35 +02:00
Code Issues Packages Projects Releases Wiki Activity

Merge bitcoin/bitcoin#31553: cluster mempool: add TxGraph reorg functionality

Browse Source 
1632fc104b txgraph: Track multiple potential would-be clusters in Trim (improvement) (Pieter Wuille)
4608df37e0 txgraph: add Trim benchmark (benchmark) (Pieter Wuille)
9c436ff01c txgraph: add fuzz test scenario that avoids cycles inside Trim() (tests) (Pieter Wuille)
938e86f8fe txgraph: add unit test for TxGraph::Trim (tests) (glozow)
a04e205ab0 txgraph: Add ability to trim oversized clusters (feature) (Pieter Wuille)
eabcd0eb6f txgraph: remove unnecessary m_group_oversized (simplification) (Greg Sanders)
19b14e61ea txgraph: Permit transactions that exceed cluster size limit (feature) (Pieter Wuille)
c4287b9b71 txgraph: Add ability to configure maximum cluster size/weight (feature) (Pieter Wuille)

Pull request description:

  Part of cluster mempool (#30289).

  During reorganisations, it is possible that dependencies get added which would result in clusters that violate policy limits (cluster count, cluster weight), when linking the new from-block transactions to the old from-mempool transactions. Unlike RBF scenarios, we cannot simply reject the changes when they are due to received blocks. To accommodate this, add a `TxGraph::Trim()`, which removes some subset of transactions (including descendants) in order to make all resulting clusters satisfy the limits.

  Conceptually, the way this is done is by defining a rudimentary linearization for the entire would-be too-large cluster, iterating it from beginning to end, and reasoning about the counts and weights of the clusters that would be reached using transactions up to that point. If a transaction is encountered whose addition would violate the limit, it is removed, together with all its descendants.

  This rudimentary linearization is like a merge sort of the chunks of the clusters being combined, but respecting topology. More specifically, it is continuously picking the highest-chunk-feerate remaining transaction among those which have no unmet dependencies left. For efficiency, this rudimentary linearization is computed lazily, by putting all viable transactions in a heap, sorted by chunk feerate, and adding new transactions to it as they become viable.

  The `Trim()` function is rather unusual compared to the `TxGraph` functionality added in previous PRs, in that `Trim()` makes it own decisions about what the resulting graph contents will be, without good specification of how it makes that decision - it is just a best-effort attempt (which is improved in the last commit). All other `TxGraph` mutators are simply to inform the graph about changes the calling mempool code decided on; this one lets the decision be made by txgraph.

  As part of this, the "oversized" property is expanded to also encompass a configurable cluster weight limit (in addition to cluster count limit).

ACKs for top commit:
  instagibbs:
    reACK 1632fc104b
  glozow:
    reACK 1632fc104b via range-diff
  ismaelsadeeq:
    reACK 1632fc104b 🛰️

Tree-SHA512: ccacb54be8ad622bd2717905fc9b7e42aea4b07f824de1924da9237027a97a9a2f1b862bc6a791cbd2e1a01897ad2c7c73c398a2d5ccbce90bfbeac0bcebc9ce
This commit is contained in:
merge-script
2025-07-07 16:11:51 -04:00
parent d33c111448 1632fc104b
commit 87ab69155d
7 changed files with 1061 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
src/test/CMakeLists.txt
View File

@@ -106,6 +106,7 @@ add_executable(test_bitcoin
transaction_tests.cpp
translation_tests.cpp
txdownload_tests.cpp
txgraph_tests.cpp
txindex_tests.cpp
txpackage_tests.cpp
txreconciliation_tests.cpp

233
src/test/fuzz/txgraph.cpp
View File

@@ -56,9 +56,12 @@ struct SimTxGraph
/** Which transactions have been modified in the graph since creation, either directly or by
* being in a cluster which includes modifications. Only relevant for the staging graph. */
SetType modified;
/** The configured maximum total size of transactions per cluster. */
uint64_t max_cluster_size;
/** Construct a new SimTxGraph with the specified maximum cluster count. */
explicit SimTxGraph(DepGraphIndex max_cluster) : max_cluster_count(max_cluster) {}
/** Construct a new SimTxGraph with the specified maximum cluster count and size. */
explicit SimTxGraph(DepGraphIndex cluster_count, uint64_t cluster_size) :
max_cluster_count(cluster_count), max_cluster_size(cluster_size) {}
// Permit copying and moving.
SimTxGraph(const SimTxGraph&) noexcept = default;
@@ -66,6 +69,20 @@ struct SimTxGraph
SimTxGraph(SimTxGraph&&) noexcept = default;
SimTxGraph& operator=(SimTxGraph&&) noexcept = default;
/** Get the connected components within this simulated transaction graph. */
std::vector<SetType> GetComponents()
{
auto todo = graph.Positions();
std::vector<SetType> ret;
// Iterate over all connected components of the graph.
while (todo.Any()) {
auto component = graph.FindConnectedComponent(todo);
ret.push_back(component);
todo -= component;
}
return ret;
}
/** Check whether this graph is oversized (contains a connected component whose number of
* transactions exceeds max_cluster_count. */
bool IsOversized()
@@ -73,12 +90,11 @@ struct SimTxGraph
if (!oversized.has_value()) {
// Only recompute when oversized isn't already known.
oversized = false;
auto todo = graph.Positions();
// Iterate over all connected components of the graph.
while (todo.Any()) {
auto component = graph.FindConnectedComponent(todo);
for (auto component : GetComponents()) {
if (component.Count() > max_cluster_count) oversized = true;
todo -= component;
uint64_t component_size{0};
for (auto i : component) component_size += graph.FeeRate(i).size;
if (component_size > max_cluster_size) oversized = true;
}
}
return *oversized;
@@ -128,6 +144,8 @@ struct SimTxGraph
simmap[simpos] = std::make_shared<TxGraph::Ref>();
auto ptr = simmap[simpos].get();
simrevmap[ptr] = simpos;
// This may invalidate our cached oversized value.
if (oversized.has_value() && !*oversized) oversized = std::nullopt;
return ptr;
}
@@ -239,6 +257,31 @@ struct SimTxGraph
}
}
}
/** Verify that set contains transactions from every oversized cluster, and nothing from
* non-oversized ones. */
bool MatchesOversizedClusters(const SetType& set)
{
if (set.Any() && !IsOversized()) return false;
auto todo = graph.Positions();
if (!set.IsSubsetOf(todo)) return false;
// Walk all clusters, and make sure all of set doesn't come from non-oversized clusters
while (todo.Any()) {
auto component = graph.FindConnectedComponent(todo);
// Determine whether component is oversized, due to either the size or count limit.
bool is_oversized = component.Count() > max_cluster_count;
uint64_t component_size{0};
for (auto i : component) component_size += graph.FeeRate(i).size;
is_oversized |= component_size > max_cluster_size;
// Check whether overlap with set matches is_oversized.
if (is_oversized != set.Overlaps(component)) return false;
todo -= component;
}
return true;
}
};
} // namespace
@@ -254,20 +297,24 @@ FUZZ_TARGET(txgraph)
FuzzedDataProvider provider(buffer.data(), buffer.size());
/** Internal test RNG, used only for decisions which would require significant amount of data
* to be read from the provider, without realistically impacting test sensitivity. */
InsecureRandomContext rng(0xdecade2009added + buffer.size());
* to be read from the provider, without realistically impacting test sensitivity, and for
* specialized test cases that are hard to perform more generically. */
InsecureRandomContext rng(provider.ConsumeIntegral<uint64_t>());
/** Variable used whenever an empty TxGraph::Ref is needed. */
TxGraph::Ref empty_ref;
// Decide the maximum number of transactions per cluster we will use in this simulation.
auto max_count = provider.ConsumeIntegralInRange<DepGraphIndex>(1, MAX_CLUSTER_COUNT_LIMIT);
/** The maximum number of transactions per (non-oversized) cluster we will use in this
* simulation. */
auto max_cluster_count = provider.ConsumeIntegralInRange<DepGraphIndex>(1, MAX_CLUSTER_COUNT_LIMIT);
/** The maximum total size of transactions in a (non-oversized) cluster. */
auto max_cluster_size = provider.ConsumeIntegralInRange<uint64_t>(1, 0x3fffff * MAX_CLUSTER_COUNT_LIMIT);
// Construct a real graph, and a vector of simulated graphs (main, and possibly staging).
auto real = MakeTxGraph(max_count);
auto real = MakeTxGraph(max_cluster_count, max_cluster_size);
std::vector<SimTxGraph> sims;
sims.reserve(2);
sims.emplace_back(max_count);
sims.emplace_back(max_cluster_count, max_cluster_size);
/** Struct encapsulating information about a BlockBuilder that's currently live. */
struct BlockBuilderData
@@ -396,7 +443,7 @@ FUZZ_TARGET(txgraph)
// Otherwise, use smaller range which consume fewer fuzz input bytes, as just
// these are likely sufficient to trigger all interesting code paths already.
fee = provider.ConsumeIntegral<uint8_t>();
size = provider.ConsumeIntegral<uint8_t>() + 1;
size = provider.ConsumeIntegralInRange<uint32_t>(1, 0xff);
}
FeePerWeight feerate{fee, size};
// Create a real TxGraph::Ref.
@@ -534,7 +581,7 @@ FUZZ_TARGET(txgraph)
auto ref = pick_fn();
auto result = alt ? real->GetDescendants(*ref, use_main)
: real->GetAncestors(*ref, use_main);
assert(result.size() <= max_count);
assert(result.size() <= max_cluster_count);
auto result_set = sel_sim.MakeSet(result);
assert(result.size() == result_set.Count());
auto expect_set = sel_sim.GetAncDesc(ref, alt);
@@ -567,16 +614,20 @@ FUZZ_TARGET(txgraph)
auto ref = pick_fn();
auto result = real->GetCluster(*ref, use_main);
// Check cluster count limit.
assert(result.size() <= max_count);
assert(result.size() <= max_cluster_count);
// Require the result to be topologically valid and not contain duplicates.
auto left = sel_sim.graph.Positions();
uint64_t total_size{0};
for (auto refptr : result) {
auto simpos = sel_sim.Find(refptr);
total_size += sel_sim.graph.FeeRate(simpos).size;
assert(simpos != SimTxGraph::MISSING);
assert(left[simpos]);
left.Reset(simpos);
assert(!sel_sim.graph.Ancestors(simpos).Overlaps(left));
}
// Check cluster size limit.
assert(total_size <= max_cluster_size);
// Require the set to be connected.
auto result_set = sel_sim.MakeSet(result);
assert(sel_sim.graph.IsConnected(result_set));
@@ -774,6 +825,146 @@ FUZZ_TARGET(txgraph)
assert(sum == worst_chunk_feerate);
}
break;
} else if ((block_builders.empty() || sims.size() > 1) && command-- == 0) {
// Trim.
bool was_oversized = top_sim.IsOversized();
auto removed = real->Trim();
// Verify that something was removed if and only if there was an oversized cluster.
assert(was_oversized == !removed.empty());
if (!was_oversized) break;
auto removed_set = top_sim.MakeSet(removed);
// The removed set must contain all its own descendants.
for (auto simpos : removed_set) {
assert(top_sim.graph.Descendants(simpos).IsSubsetOf(removed_set));
}
// Something from every oversized cluster should have been removed, and nothing
// else.
assert(top_sim.MatchesOversizedClusters(removed_set));
// Apply all removals to the simulation, and verify the result is no longer
// oversized. Don't query the real graph for oversizedness; it is compared
// against the simulation anyway later.
for (auto simpos : removed_set) {
top_sim.RemoveTransaction(top_sim.GetRef(simpos));
}
assert(!top_sim.IsOversized());
break;
} else if ((block_builders.empty() || sims.size() > 1) &&
top_sim.GetTransactionCount() > max_cluster_count && !top_sim.IsOversized() && command-- == 0) {
// Trim (special case which avoids apparent cycles in the implicit approximate
// dependency graph constructed inside the Trim() implementation). This is worth
// testing separately, because such cycles cannot occur in realistic scenarios,
// but this is hard to replicate in general in this fuzz test.
// First, we need to have dependencies applied and linearizations fixed to avoid
// circular dependencies in implied graph; trigger it via whatever means.
real->CountDistinctClusters({}, false);
// Gather the current clusters.
auto clusters = top_sim.GetComponents();
// Merge clusters randomly until at least one oversized one appears.
bool made_oversized = false;
auto merges_left = clusters.size() - 1;
while (merges_left > 0) {
--merges_left;
// Find positions of clusters in the clusters vector to merge together.
auto par_cl = rng.randrange(clusters.size());
auto chl_cl = rng.randrange(clusters.size() - 1);
chl_cl += (chl_cl >= par_cl);
Assume(chl_cl != par_cl);
// Add between 1 and 3 dependencies between them. As all are in the same
// direction (from the child cluster to parent cluster), no cycles are possible,
// regardless of what internal topology Trim() uses as approximation within the
// clusters.
int num_deps = rng.randrange(3) + 1;
for (int i = 0; i < num_deps; ++i) {
// Find a parent transaction in the parent cluster.
auto par_idx = rng.randrange(clusters[par_cl].Count());
SimTxGraph::Pos par_pos = 0;
for (auto j : clusters[par_cl]) {
if (par_idx == 0) {
par_pos = j;
break;
}
--par_idx;
}
// Find a child transaction in the child cluster.
auto chl_idx = rng.randrange(clusters[chl_cl].Count());
SimTxGraph::Pos chl_pos = 0;
for (auto j : clusters[chl_cl]) {
if (chl_idx == 0) {
chl_pos = j;
break;
}
--chl_idx;
}
// Add dependency to both simulation and real TxGraph.
auto par_ref = top_sim.GetRef(par_pos);
auto chl_ref = top_sim.GetRef(chl_pos);
top_sim.AddDependency(par_ref, chl_ref);
real->AddDependency(*par_ref, *chl_ref);
}
// Compute the combined cluster.
auto par_cluster = clusters[par_cl];
auto chl_cluster = clusters[chl_cl];
auto new_cluster = par_cluster | chl_cluster;
// Remove the parent and child cluster from clusters.
std::erase_if(clusters, [&](const auto& cl) noexcept { return cl == par_cluster || cl == chl_cluster; });
// Add the combined cluster.
clusters.push_back(new_cluster);
// If this is the first merge that causes an oversized cluster to appear, pick
// a random number of further merges to appear.
if (!made_oversized) {
made_oversized = new_cluster.Count() > max_cluster_count;
if (!made_oversized) {
FeeFrac total;
for (auto i : new_cluster) total += top_sim.graph.FeeRate(i);
if (uint32_t(total.size) > max_cluster_size) made_oversized = true;
}
if (made_oversized) merges_left = rng.randrange(clusters.size());
}
}
// Determine an upper bound on how many transactions are removed.
uint32_t max_removed = 0;
for (auto& cluster : clusters) {
// Gather all transaction sizes in the to-be-combined cluster.
std::vector<uint32_t> sizes;
for (auto i : cluster) sizes.push_back(top_sim.graph.FeeRate(i).size);
auto sum_sizes = std::accumulate(sizes.begin(), sizes.end(), uint64_t{0});
// Sort from large to small.
std::sort(sizes.begin(), sizes.end(), std::greater{});
// In the worst case, only the smallest transactions are removed.
while (sizes.size() > max_cluster_count || sum_sizes > max_cluster_size) {
sum_sizes -= sizes.back();
sizes.pop_back();
++max_removed;
}
}
// Invoke Trim now on the definitely-oversized txgraph.
auto removed = real->Trim();
// Verify that the number of removals is within range.
assert(removed.size() >= 1);
assert(removed.size() <= max_removed);
// The removed set must contain all its own descendants.
auto removed_set = top_sim.MakeSet(removed);
for (auto simpos : removed_set) {
assert(top_sim.graph.Descendants(simpos).IsSubsetOf(removed_set));
}
// Something from every oversized cluster should have been removed, and nothing
// else.
assert(top_sim.MatchesOversizedClusters(removed_set));
// Apply all removals to the simulation, and verify the result is no longer
// oversized. Don't query the real graph for oversizedness; it is compared
// against the simulation anyway later.
for (auto simpos : removed_set) {
top_sim.RemoveTransaction(top_sim.GetRef(simpos));
}
assert(!top_sim.IsOversized());
break;
}
}
}
@@ -941,28 +1132,32 @@ FUZZ_TARGET(txgraph)
// Check its ancestors against simulation.
auto expect_anc = sim.graph.Ancestors(i);
auto anc = sim.MakeSet(real->GetAncestors(*sim.GetRef(i), main_only));
assert(anc.Count() <= max_count);
assert(anc.Count() <= max_cluster_count);
assert(anc == expect_anc);
// Check its descendants against simulation.
auto expect_desc = sim.graph.Descendants(i);
auto desc = sim.MakeSet(real->GetDescendants(*sim.GetRef(i), main_only));
assert(desc.Count() <= max_count);
assert(desc.Count() <= max_cluster_count);
assert(desc == expect_desc);
// Check the cluster the transaction is part of.
auto cluster = real->GetCluster(*sim.GetRef(i), main_only);
assert(cluster.size() <= max_count);
assert(cluster.size() <= max_cluster_count);
assert(sim.MakeSet(cluster) == component);
// Check that the cluster is reported in a valid topological order (its
// linearization).
std::vector<DepGraphIndex> simlin;
SimTxGraph::SetType done;
uint64_t total_size{0};
for (TxGraph::Ref* ptr : cluster) {
auto simpos = sim.Find(ptr);
assert(sim.graph.Descendants(simpos).IsSubsetOf(component - done));
done.Set(simpos);
assert(sim.graph.Ancestors(simpos).IsSubsetOf(done));
simlin.push_back(simpos);
total_size += sim.graph.FeeRate(simpos).size;
}
// Check cluster size.
assert(total_size <= max_cluster_size);
// Construct a chunking object for the simulated graph, using the reported cluster
// linearization as ordering, and compare it against the reported chunk feerates.
if (sims.size() == 1 || main_only) {

290
src/test/txgraph_tests.cpp Normal file
View File

@@ -0,0 +1,290 @@
// Copyright (c) 2023-present The Bitcoin Core developers
// Distributed under the MIT software license, see the accompanying
// file COPYING or https://opensource.org/license/mit/.
#include <txgraph.h>
#include <random.h>
#include <boost/test/unit_test.hpp>
#include <memory>
#include <vector>
BOOST_AUTO_TEST_SUITE(txgraph_tests)
BOOST_AUTO_TEST_CASE(txgraph_trim_zigzag)
{
// T T T T T T T T T T T T T T (50 T's)
// \ / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ /
// \ / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ /
// B B B B B B B B B B B B B (49 B's)
//
/** The maximum cluster count used in this test. */
static constexpr int MAX_CLUSTER_COUNT = 50;
/** The number of "bottom" transactions, which are in the mempool already. */
static constexpr int NUM_BOTTOM_TX = 49;
/** The number of "top" transactions, which come from disconnected blocks. These are re-added
* to the mempool and, while connecting them to the already-in-mempool transactions, we
* discover the resulting cluster is oversized. */
static constexpr int NUM_TOP_TX = 50;
/** The total number of transactions in the test. */
static constexpr int NUM_TOTAL_TX = NUM_BOTTOM_TX + NUM_TOP_TX;
static_assert(NUM_TOTAL_TX > MAX_CLUSTER_COUNT);
/** Set a very large cluster size limit so that only the count limit is triggered. */
static constexpr int32_t MAX_CLUSTER_SIZE = 100'000 * 100;
// Create a new graph for the test.
auto graph = MakeTxGraph(MAX_CLUSTER_COUNT, MAX_CLUSTER_SIZE);
// Add all transactions and store their Refs.
std::vector<TxGraph::Ref> refs;
refs.reserve(NUM_TOTAL_TX);
// First all bottom transactions: the i'th bottom transaction is at position i.
for (unsigned int i = 0; i < NUM_BOTTOM_TX; ++i) {
refs.push_back(graph->AddTransaction(FeePerWeight{200 - i, 100}));
}
// Then all top transactions: the i'th top transaction is at position NUM_BOTTOM_TX + i.
for (unsigned int i = 0; i < NUM_TOP_TX; ++i) {
refs.push_back(graph->AddTransaction(FeePerWeight{100 - i, 100}));
}
// Create the zigzag dependency structure.
// Each transaction in the bottom row depends on two adjacent transactions from the top row.
graph->SanityCheck();
for (unsigned int i = 0; i < NUM_BOTTOM_TX; ++i) {
graph->AddDependency(/*parent=*/refs[NUM_BOTTOM_TX + i], /*child=*/refs[i]);
graph->AddDependency(/*parent=*/refs[NUM_BOTTOM_TX + i + 1], /*child=*/refs[i]);
}
// Check that the graph is now oversized. This also forces the graph to
// group clusters and compute the oversized status.
graph->SanityCheck();
BOOST_CHECK_EQUAL(graph->GetTransactionCount(), NUM_TOTAL_TX);
BOOST_CHECK(graph->IsOversized(/*main_only=*/false));
// Call Trim() to remove transactions and bring the cluster back within limits.
auto removed_refs = graph->Trim();
graph->SanityCheck();
BOOST_CHECK(!graph->IsOversized(/*main_only=*/false));
// We only need to trim the middle bottom transaction to end up with 2 clusters each within cluster limits.
BOOST_CHECK_EQUAL(removed_refs.size(), 1);
BOOST_CHECK_EQUAL(graph->GetTransactionCount(), MAX_CLUSTER_COUNT * 2 - 2);
for (unsigned int i = 0; i < refs.size(); ++i) {
BOOST_CHECK_EQUAL(graph->Exists(refs[i]), i != (NUM_BOTTOM_TX / 2));
}
}
BOOST_AUTO_TEST_CASE(txgraph_trim_flower)
{
// We will build an oversized flower-shaped graph: all transactions are spent by 1 descendant.
//
// T T T T T T T T (100 T's)
// | | | | | | | |
// | | | | | | | |
// \---+---+---+-+-+---+---+---/
// |
// B (1 B)
//
/** The maximum cluster count used in this test. */
static constexpr int MAX_CLUSTER_COUNT = 50;
/** The number of "top" transactions, which come from disconnected blocks. These are re-added
* to the mempool and, connecting them to the already-in-mempool transactions, we discover the
* resulting cluster is oversized. */
static constexpr int NUM_TOP_TX = MAX_CLUSTER_COUNT * 2;
/** The total number of transactions in this test. */
static constexpr int NUM_TOTAL_TX = NUM_TOP_TX + 1;
/** Set a very large cluster size limit so that only the count limit is triggered. */
static constexpr int32_t MAX_CLUSTER_SIZE = 100'000 * 100;
auto graph = MakeTxGraph(MAX_CLUSTER_COUNT, MAX_CLUSTER_SIZE);
// Add all transactions and store their Refs.
std::vector<TxGraph::Ref> refs;
refs.reserve(NUM_TOTAL_TX);
// Add all transactions. They are in individual clusters.
refs.push_back(graph->AddTransaction({1, 100}));
for (unsigned int i = 0; i < NUM_TOP_TX; ++i) {
refs.push_back(graph->AddTransaction(FeePerWeight{500 + i, 100}));
}
graph->SanityCheck();
// The 0th transaction spends all the top transactions.
for (unsigned int i = 1; i < NUM_TOTAL_TX; ++i) {
graph->AddDependency(/*parent=*/refs[i], /*child=*/refs[0]);
}
graph->SanityCheck();
// Check that the graph is now oversized. This also forces the graph to
// group clusters and compute the oversized status.
BOOST_CHECK(graph->IsOversized(/*main_only=*/false));
// Call Trim() to remove transactions and bring the cluster back within limits.
auto removed_refs = graph->Trim();
graph->SanityCheck();
BOOST_CHECK(!graph->IsOversized(/*main_only=*/false));
// Since only the bottom transaction connects these clusters, we only need to remove it.
BOOST_CHECK_EQUAL(removed_refs.size(), 1);
BOOST_CHECK_EQUAL(graph->GetTransactionCount(false), MAX_CLUSTER_COUNT * 2);
BOOST_CHECK(!graph->Exists(refs[0]));
for (unsigned int i = 1; i < refs.size(); ++i) {
BOOST_CHECK(graph->Exists(refs[i]));
}
}
BOOST_AUTO_TEST_CASE(txgraph_trim_huge)
{
// The from-block transactions consist of 1000 fully linear clusters, each with 64
// transactions. The mempool contains 11 transactions that together merge all of these into
// a single cluster.
//
// (1000 chains of 64 transactions, 64000 T's total)
//
// T T T T T T T T
// | | | | | | | |
// T T T T T T T T
// | | | | | | | |
// T T T T T T T T
// | | | | | | | |
// T T T T T T T T
// (64 long) (64 long) (64 long) (64 long) (64 long) (64 long) (64 long) (64 long)
// | | | | | | | |
// | | / \ | / \ | | /
// \----------+--------/ \--------+--------/ \--------+-----+----+--------/
// | | |
// B B B
//
// (11 B's, each attaching to up to 100 chains of 64 T's)
//
/** The maximum cluster count used in this test. */
static constexpr int MAX_CLUSTER_COUNT = 64;
/** The number of "top" (from-block) chains of transactions. */
static constexpr int NUM_TOP_CHAINS = 1000;
/** The number of transactions per top chain. */
static constexpr int NUM_TX_PER_TOP_CHAIN = MAX_CLUSTER_COUNT;
/** The (maximum) number of dependencies per bottom transaction. */
static constexpr int NUM_DEPS_PER_BOTTOM_TX = 100;
/** The number of bottom transactions that are expected to be created. */
static constexpr int NUM_BOTTOM_TX = (NUM_TOP_CHAINS - 1 + (NUM_DEPS_PER_BOTTOM_TX - 2)) / (NUM_DEPS_PER_BOTTOM_TX - 1);
/** The total number of transactions created in this test. */
static constexpr int NUM_TOTAL_TX = NUM_TOP_CHAINS * NUM_TX_PER_TOP_CHAIN + NUM_BOTTOM_TX;
/** Set a very large cluster size limit so that only the count limit is triggered. */
static constexpr int32_t MAX_CLUSTER_SIZE = 100'000 * 100;
/** Refs to all top transactions. */
std::vector<TxGraph::Ref> top_refs;
/** Refs to all bottom transactions. */
std::vector<TxGraph::Ref> bottom_refs;
/** Indexes into top_refs for some transaction of each component, in arbitrary order.
* Initially these are the last transactions in each chains, but as bottom transactions are
* added, entries will be removed when they get merged, and randomized. */
std::vector<size_t> top_components;
FastRandomContext rng;
auto graph = MakeTxGraph(MAX_CLUSTER_COUNT, MAX_CLUSTER_SIZE);
// Construct the top chains.
for (int chain = 0; chain < NUM_TOP_CHAINS; ++chain) {
for (int chaintx = 0; chaintx < NUM_TX_PER_TOP_CHAIN; ++chaintx) {
// Use random fees, size 1.
int64_t fee = rng.randbits<27>() + 100;
FeePerWeight feerate{fee, 1};
top_refs.push_back(graph->AddTransaction(feerate));
// Add internal dependencies linked the chain transactions together.
if (chaintx > 0) {
graph->AddDependency(*(top_refs.rbegin()), *(top_refs.rbegin() + 1));
}
}
// Remember the last transaction in each chain, to attach the bottom transactions to.
top_components.push_back(top_refs.size() - 1);
}
graph->SanityCheck();
// Not oversized so far (just 1000 clusters of 64).
BOOST_CHECK(!graph->IsOversized());
// Construct the bottom transactions, and dependencies to the top chains.
while (top_components.size() > 1) {
// Construct the transaction.
int64_t fee = rng.randbits<27>() + 100;
FeePerWeight feerate{fee, 1};
auto bottom_tx = graph->AddTransaction(feerate);
// Determine the number of dependencies this transaction will have.
int deps = std::min<int>(NUM_DEPS_PER_BOTTOM_TX, top_components.size());
for (int dep = 0; dep < deps; ++dep) {
// Pick an transaction in top_components to attach to.
auto idx = rng.randrange(top_components.size());
// Add dependency.
graph->AddDependency(/*parent=*/top_refs[top_components[idx]], /*child=*/bottom_tx);
// Unless this is the last dependency being added, remove from top_components, as
// the component will be merged with that one.
if (dep < deps - 1) {
// Move entry top the back.
if (idx != top_components.size() - 1) std::swap(top_components.back(), top_components[idx]);
// And pop it.
top_components.pop_back();
}
}
bottom_refs.push_back(std::move(bottom_tx));
}
graph->SanityCheck();
// Now we are oversized (one cluster of 64011).
BOOST_CHECK(graph->IsOversized());
const auto total_tx_count = graph->GetTransactionCount();
BOOST_CHECK(total_tx_count == top_refs.size() + bottom_refs.size());
BOOST_CHECK(total_tx_count == NUM_TOTAL_TX);
// Call Trim() to remove transactions and bring the cluster back within limits.
auto removed_refs = graph->Trim();
BOOST_CHECK(!graph->IsOversized());
BOOST_CHECK(removed_refs.size() == total_tx_count - graph->GetTransactionCount());
graph->SanityCheck();
// At least 99% of chains must survive.
BOOST_CHECK(graph->GetTransactionCount() >= (NUM_TOP_CHAINS * NUM_TX_PER_TOP_CHAIN * 99) / 100);
}
BOOST_AUTO_TEST_CASE(txgraph_trim_big_singletons)
{
// Mempool consists of 100 singleton clusters; there are no dependencies. Some are oversized. Trim() should remove all of the oversized ones.
static constexpr int MAX_CLUSTER_COUNT = 64;
static constexpr int32_t MAX_CLUSTER_SIZE = 100'000;
static constexpr int NUM_TOTAL_TX = 100;
// Create a new graph for the test.
auto graph = MakeTxGraph(MAX_CLUSTER_COUNT, MAX_CLUSTER_SIZE);
// Add all transactions and store their Refs.
std::vector<TxGraph::Ref> refs;
refs.reserve(NUM_TOTAL_TX);
// Add all transactions. They are in individual clusters.
for (unsigned int i = 0; i < NUM_TOTAL_TX; ++i) {
// The 88th transaction is oversized.
// Every 20th transaction is oversized.
const FeePerWeight feerate{500 + i, (i == 88 || i % 20 == 0) ? MAX_CLUSTER_SIZE + 1 : 100};
refs.push_back(graph->AddTransaction(feerate));
}
graph->SanityCheck();
// Check that the graph is now oversized. This also forces the graph to
// group clusters and compute the oversized status.
BOOST_CHECK(graph->IsOversized(/*main_only=*/false));
// Call Trim() to remove transactions and bring the cluster back within limits.
auto removed_refs = graph->Trim();
graph->SanityCheck();
BOOST_CHECK_EQUAL(graph->GetTransactionCount(), NUM_TOTAL_TX - 6);
BOOST_CHECK(!graph->IsOversized(/*main_only=*/false));
// Check that all the oversized transactions were removed.
for (unsigned int i = 0; i < refs.size(); ++i) {
BOOST_CHECK_EQUAL(graph->Exists(refs[i]), i != 88 && i % 20 != 0);
}
}
BOOST_AUTO_TEST_SUITE_END()