f1d129d963
b2ea365648txgraph: Add Get{Ancestors,Descendants}Union functions (feature) (Pieter Wuille)54bceddd3atxgraph: Multiple inputs to Get{Ancestors,Descendant}Refs (preparation) (Pieter Wuille)aded047019txgraph: Add CountDistinctClusters function (feature) (Pieter Wuille)b685d322c9txgraph: Add DoWork function (feature) (Pieter Wuille)295a1ca8bbtxgraph: Expose ability to compare transactions (feature) (Pieter Wuille)22c68cd153txgraph: Allow Refs to outlive the TxGraph (feature) (Pieter Wuille)82fa3573e1txgraph: Destroying Ref means removing transaction (feature) (Pieter Wuille)6b037ceddftxgraph: Cache oversizedness of graphs (optimization) (Pieter Wuille)8c70688965txgraph: Add staging support (feature) (Pieter Wuille)c99c7300b4txgraph: Abstract out ClearLocator (refactor) (Pieter Wuille)34aa3da5adtxgraph: Group per-graph data in ClusterSet (refactor) (Pieter Wuille)36dd5edca5txgraph: Special-case removal of tail of cluster (Optimization) (Pieter Wuille)5801e0fb2btxgraph: Delay chunking while sub-acceptable (optimization) (Pieter Wuille)57f5499882txgraph: Avoid looking up the same child cluster repeatedly (optimization) (Pieter Wuille)1171953ac6txgraph: Avoid representative lookup for each dependency (optimization) (Pieter Wuille)64f69ec8c3txgraph: Make max cluster count configurable and "oversize" state (feature) (Pieter Wuille)1d27b74c8etxgraph: Add GetChunkFeerate function (feature) (Pieter Wuille)c80aecc24dtxgraph: Avoid per-group vectors for clusters & dependencies (optimization) (Pieter Wuille)ee57e93099txgraph: Add internal sanity check function (tests) (Pieter Wuille)05abf336f9txgraph: Add simulation fuzz test (tests) (Pieter Wuille)8ad3ed2681txgraph: Add initial version (feature) (Pieter Wuille)6eab3b2d73feefrac: Introduce tagged wrappers to distinguish vsize/WU rates (Pieter Wuille)d449773899scripted-diff: (refactor) ClusterIndex -> DepGraphIndex (Pieter Wuille)bfeb69f6e0clusterlin: Make IsAcyclic() a DepGraph member function (Pieter Wuille)0aa874a357clusterlin: Add FixLinearization function + fuzz test (Pieter Wuille) Pull request description: Part of cluster mempool: #30289. ### 1. Overview This introduces the `TxGraph` class, which encapsulates knowledge about the (effective) fees, sizes, and dependencies between all mempool transactions, but nothing else. In particular, it lacks knowledge about `CTransaction`, inputs, outputs, txids, wtxids, prioritization, validatity, policy rules, and a lot more. Being restricted to just those aspects of the mempool makes the behavior very easy to fully specify (ignoring the actual linearizations produced), and write simulation-based tests for (which are included in this PR). ### 2. Interface The interface can be largely categorized into: * Mutation functions: * `AddTransaction` (add a new transaction with specified feerate, and get a `Ref` object back to identify it). * `RemoveTransaction` (given a `Ref` object, remove the transaction). * `AddDependency` (given two `Ref` objects, add a dependency between them). * `SetTransactionFee` (modify the fee associated with a Ref object). * Inspector functions: * `GetAncestors` (get the ancestor set in the form of `Ref*` pointers) * `GetAncestorsUnion` (like above, but for the union of ancestors of multiple `Ref*` pointers) * `GetDescendants` (get the descendant set in the form of `Ref*` pointers) * `GetDescendantsUnion` (like above, but for the union of ancestors of multiple `Ref*` pointers) * `GetCluster` (get the connected component set in the form of `Ref*` pointers, in the order they would be mined). * `GetIndividualFeerate` (get the feerate of a transaction) * `GetChunkFeerate` (get the mining score of a transaction) * `CountDistinctClusters` (count the number of distinct clusters a list of `Ref`s belong to) * Staging functions: * `StartStaging` (make all future mutations operate on a proposed transaction graph) * `CommitStaging` (apply all the changes that are staged) * `AbortStaging` (discard all the changes that are staged) * Miscellaneous functions: * `DoWork` (do queued-up computations now, so that future operations are fast) This `TxGraph::Ref` type used as a "handle" on transactions in the graph can be inherited from, and the idea is that in the full cluster mempool implementation (#28676, after it is rebased on this), `CTxMempoolEntry` will inherit from it, and all actually used Ref objects will be `CTxMempoolEntry`s. With that, the mempool code can just cast any `Ref*` returned by txgraph to `CTxMempoolEntry*`. ### 3. Implementation Internally the graph data is kept in clustered form (partitioned into connected components), for which linearizations are maintained and updated as needed using the `cluster_linearize.h` algorithms under the hood, but this is hidden from the users of this class. Implementation-wise, mutations are generally applied lazily, appending to queues of to-be-removed transactions and to-be-added dependencies, so they can be batched for higher performance. Inspectors will generally only evaluate as much as is needed to answer queries, with roughly 5 levels of processing to go to fully instantiated and acceptable cluster linearizations, in order: 1. `ApplyRemovals` (take batches of to-be-removed transactions and translate them to "holes" in the corresponding Clusters/DepGraphs). 2. `SplitAll` (creating holes in Clusters may cause them to break apart into smaller connected components, so make turn them into separate Clusters/linearizations). 3. `GroupClusters` (figure out which Clusters will need to be combined in order to add requested to-be-added dependencies, as these may span clusters). 4. `ApplyDependencies` (actually merge Clusters as precomputed by `GroupClusters`, and add the dependencies between them). 5. `MakeAcceptable` (perform the LIMO linearization algorithm on Clusters to make sure their linearizations are acceptable). ### 4. Future work This is only an initial version of TxGraph, and some functionality is missing before #28676 can be rebased on top of it: * The ability to get comparative feerate diagrams before/after for the set of staged changes (to evaluate RBF incentive-compatibility). * Mining interface (ability to iterate transactions quickly in mining score order) (see #31444). * Eviction interface (reverse of mining order, plus memory usage accounting) (see #31444). * Ability to fix oversizedness of clusters (before or after committing) - this is needed for reorgs where aborting/rejecting the change just is not an option (see #31553). * Interface for controlling how much effort is spent on LIMO. In this PR it is hardcoded. Then there are further improvements possible which would not block other work: * Making Cluster a virtual class with different implementations based on transaction count (which could dramatically reduce memory usage, as most Clusters are just a single transaction, for which the current implementation is overkill). * The ability to have background thread(s) for improving cluster linearizations. ACKs for top commit: instagibbs: reACKb2ea365648ajtowns: reACKb2ea365648ismaelsadeeq: reACKb2ea365648🚀 glozow: ACKb2ea365648Tree-SHA512: 0f86f73d37651fe47d469db1384503bbd1237b4556e5d50b1d0a3dd27754792d6fc3481f77a201cf2ed36c6ca76e0e44c30e175d112aacb53dfdb9e11d8abc6b
Bitcoin Core integration/staging tree
For an immediately usable, binary version of the Bitcoin Core software, see https://bitcoincore.org/en/download/.
What is Bitcoin Core?
Bitcoin Core connects to the Bitcoin peer-to-peer network to download and fully validate blocks and transactions. It also includes a wallet and graphical user interface, which can be optionally built.
Further information about Bitcoin Core is available in the doc folder.
License
Bitcoin Core is released under the terms of the MIT license. See COPYING for more information or see https://opensource.org/licenses/MIT.
Development Process
The master branch is regularly built (see doc/build-*.md for instructions) and tested, but it is not guaranteed to be
completely stable. Tags are created
regularly from release branches to indicate new official, stable release versions of Bitcoin Core.
The https://github.com/bitcoin-core/gui repository is used exclusively for the development of the GUI. Its master branch is identical in all monotree repositories. Release branches and tags do not exist, so please do not fork that repository unless it is for development reasons.
The contribution workflow is described in CONTRIBUTING.md and useful hints for developers can be found in doc/developer-notes.md.
Testing
Testing and code review is the bottleneck for development; we get more pull requests than we can review and test on short notice. Please be patient and help out by testing other people's pull requests, and remember this is a security-critical project where any mistake might cost people lots of money.
Automated Testing
Developers are strongly encouraged to write unit tests for new code, and to
submit new unit tests for old code. Unit tests can be compiled and run
(assuming they weren't disabled during the generation of the build system) with: ctest. Further details on running
and extending unit tests can be found in /src/test/README.md.
There are also regression and integration tests, written
in Python.
These tests can be run (if the test dependencies are installed) with: build/test/functional/test_runner.py
(assuming build is your build directory).
The CI (Continuous Integration) systems make sure that every pull request is built for Windows, Linux, and macOS, and that unit/sanity tests are run automatically.
Manual Quality Assurance (QA) Testing
Changes should be tested by somebody other than the developer who wrote the code. This is especially important for large or high-risk changes. It is useful to add a test plan to the pull request description if testing the changes is not straightforward.
Translations
Changes to translations as well as new translations can be submitted to Bitcoin Core's Transifex page.
Translations are periodically pulled from Transifex and merged into the git repository. See the translation process for details on how this works.
Important: We do not accept translation changes as GitHub pull requests because the next pull from Transifex would automatically overwrite them again.