Commit Graph

7642 Commits

Author SHA1 Message Date
Suhas Daftuar
3e39ea8c30 Rewrite removeForReorg to avoid using sets
Also improve test coverage for removeForReorg by creating a scenario where
there are in-mempool descendants that are only invalidated due to an in-mempool
parent no longer spending a mature coin.
2025-11-30 11:01:57 -05:00
merge-script
7e129b644e Merge bitcoin/bitcoin#33893: test: add -alertnotify test for large work invalid chain warning
8343a9ffcc test: add `-alertnotify` test for large work invalid chain warning (Sebastian Falbesoner)

Pull request description:

  This PR adds missing test coverage for the `LARGE_WORK_INVALID_CHAIN` fork warning, checked with the `-alertnotify` option:
  ead849c9f1/src/validation.cpp (L2033-L2040)
  Found that this is missing during review of #32587. The test works by first creating a bunch of invalid blocks, that are first announced by headers and then submitted fully in reverse (invalid tip first), in order to set `m_best_invalid` to that value, finally leading to the best chain / invalid chain gap of >= 6 blocks. I'd be curious if there are other (more realistic?) ways to test this. One simple alternative is just to call `invalidateblock` twice (once at the tip, once at the base of the invalid chain).

  Note that the written warning doesn't include the exclamation mark, as it is removed via `SanitizeString` in the `AlertNotify` function.

ACKs for top commit:
  brunoerg:
    reACK 8343a9ffcc
  mzumsande:
    re-ACK 8343a9ffcc

Tree-SHA512: d81e9ce7622026498cad5cdcdb867a22068670983737502888c72c72209ca6ff183e77d7429f758765a42c25cda439e01f795884864ac6fe6ff258a98d0bbcbc
2025-11-25 11:45:07 +00:00
merge-script
fa283d28e2 Merge bitcoin/bitcoin#33629: Cluster mempool
17cf9ff7ef Use cluster size limit for -maxmempool bound, and allow -maxmempool=0 in general (Suhas Daftuar)
315e43e5d8 Sanity check `GetFeerateDiagram()` in CTxMemPool::check() (Suhas Daftuar)
de2e9a24c4 test: extend package rbf functional test to larger clusters (Suhas Daftuar)
4ef4ddb504 doc: update policy/packages.md for new package acceptance logic (Suhas Daftuar)
79f73ad713 Add check that GetSortedScoreWithTopology() agrees with CompareMiningScoreWithTopology() (Suhas Daftuar)
a86ac11768 Update comments for CTxMemPool class (Suhas Daftuar)
9567eaa66d Invoke TxGraph::DoWork() at appropriate times (Suhas Daftuar)
6c5c44f774 test: add functional test for new cluster mempool RPCs (Suhas Daftuar)
72f60c877e doc: Update mempool_replacements.md to reflect feerate diagram checks (Suhas Daftuar)
21693f031a Expose cluster information via rpc (Suhas Daftuar)
72e74e0d42 fuzz: try to add more code coverage for mempool fuzzing (Suhas Daftuar)
f107417490 bench: add more mempool benchmarks (Suhas Daftuar)
7976eb1ae7 Avoid violating mempool policy limits in tests (Suhas Daftuar)
84de685cf7 Stop tracking parents/children outside of txgraph (Suhas Daftuar)
88672e205b Rewrite GatherClusters to use the txgraph implementation (Suhas Daftuar)
1ca4f01090 Fix miniminer_tests to work with cluster limits (Suhas Daftuar)
1902111e0f Eliminate CheckPackageLimits, which no longer does anything (Suhas Daftuar)
3a646ec462 Rework RBF and TRUC validation (Suhas Daftuar)
19b8479868 Make getting parents/children a function of the mempool, not a mempool entry (Suhas Daftuar)
5560913e51 Rework truc_policy to use descendants, not children (Suhas Daftuar)
a4458d6c40 Use txgraph to calculate descendants (Suhas Daftuar)
c8b6f70d64 Use txgraph to calculate ancestors (Suhas Daftuar)
241a3e666b Simplify ancestor calculation functions (Suhas Daftuar)
b9cec7f0a1 Make removeConflicts private (Suhas Daftuar)
0402e6c780 Remove unused limits from CalculateMemPoolAncestors (Suhas Daftuar)
08be765ac2 Remove mempool logic designed to maintain ancestor/descendant state (Suhas Daftuar)
fc4e3e6bc1 Remove unused members from CTxMemPoolEntry (Suhas Daftuar)
ff3b398d12 mempool: eliminate accessors to mempool entry ancestor/descendant cached state (Suhas Daftuar)
b9a2039f51 Eliminate use of cached ancestor data in miniminer_tests and truc_policy (Suhas Daftuar)
ba09fc9774 mempool: Remove unused function CalculateDescendantMaximum (Suhas Daftuar)
8e49477e86 wallet: Replace max descendant count with cluster_count (Suhas Daftuar)
e031085fd4 Eliminate Single-Conflict RBF Carve Out (Suhas Daftuar)
cf3ab8e1d0 Stop enforcing descendant size/count limits (Suhas Daftuar)
89ae38f489 test: remove rbf carveout test from mempool_limit.py (Suhas Daftuar)
c0bd04d18f Calculate descendant information for mempool RPC output on-the-fly (Suhas Daftuar)
bdcefb8a8b Use mempool/txgraph to determine if a tx has descendants (Suhas Daftuar)
69e1eaa6ed Add test case for cluster size limits to TRUC logic (Suhas Daftuar)
9cda64b86c Stop enforcing ancestor size/count limits (Suhas Daftuar)
1f93227a84 Remove dependency on cached ancestor data in mini-miner (Suhas Daftuar)
9fbe0a4ac2 rpc: Calculate ancestor data from scratch for mempool rpc calls (Suhas Daftuar)
7961496dda Reimplement GetTransactionAncestry() to not rely on cached data (Suhas Daftuar)
feceaa42e8 Remove CTxMemPool::GetSortedDepthAndScore (Suhas Daftuar)
21b5cea588 Use cluster linearization for transaction relay sort order (Suhas Daftuar)
6445aa7d97 Remove the ancestor and descendant indices from the mempool (Suhas Daftuar)
216e693729 Implement new RBF logic for cluster mempool (Suhas Daftuar)
ff8f115dec policy: Remove CPFP carveout rule (Suhas Daftuar)
c3f1afc934 test: rewrite PopulateMempool to not violate mempool policy (cluster size) limits (Suhas Daftuar)
47ab32fdb1 Select transactions for blocks based on chunk feerate (Suhas Daftuar)
dec138d1dd fuzz: remove comparison between mini_miner block construction and miner (Suhas Daftuar)
6c2bceb200 bench: rewrite ComplexMemPool to not create oversized clusters (Suhas Daftuar)
1ad4590f63 Limit mempool size based on chunk feerate (Suhas Daftuar)
b11c89cab2 Rework miner_tests to not require large cluster limit (Suhas Daftuar)
95a8297d48 Check cluster limits when using -walletrejectlongchains (Suhas Daftuar)
95762e6759 Do not allow mempool clusters to exceed configured limits (Suhas Daftuar)
edb3e7cdf6 [test] rework/delete feature_rbf tests requiring large clusters (glozow)
435fd56711 test: update feature_rbf.py replacement test (Suhas Daftuar)
34e32985e8 Add new (unused) limits for cluster size/count (Suhas Daftuar)
838d7e3553 Add transactions to txgraph, but without cluster dependencies (Suhas Daftuar)
d5ed9cb3eb Add accessor for sigops-adjusted weight (Suhas Daftuar)
1bf3b51396 Add sigops adjusted weight calculator (Suhas Daftuar)
c18c68a950 Create a txgraph inside CTxMemPool (Suhas Daftuar)
29a94d5b2f Make CTxMemPoolEntry derive from TxGraph::Ref (Suhas Daftuar)
92b0079fe3 Allow moving CTxMemPoolEntry objects, disallow copying (Suhas Daftuar)
6c73e47448 mempool: Store iterators into mapTx in mapNextTx (Suhas Daftuar)
51430680ec Allow moving an Epoch::Marker (Suhas Daftuar)

Pull request description:

  [Reopening #28676 here as a new PR, because GitHub is slow to load the page making it hard to scroll through and see comments.  Also, that PR was originally opened with a prototype implementation which has changed significantly with the introduction of `TxGraph`.]

  This is an implementation of the [cluster mempool proposal](https://delvingbitcoin.org/t/an-overview-of-the-cluster-mempool-proposal/393).

  This branch implements the following observable behavior changes:

   - Maintains a partitioning of the mempool into connected clusters (via the `txgraph` class), which are limited in vsize to 101 kvB by default, and limited in count to 64 by default.
   - Each cluster is sorted ("linearized") to try to optimize for selecting highest-feerate-subsets of a cluster first
   - Transaction selection for mining is updated to use the cluster linearizations, selecting highest feerate "chunks" first for inclusion in a block template.
   - Mempool eviction is updated to use the cluster linearizations, selecting lowest feerate "chunks" first for removal.
   - The RBF rules are updated to: (a) drop the requirement that no new inputs are introduced; (b) change the feerate requirement to instead check that the feerate diagram of the mempool will strictly improve; (c) replace the direct conflicts limit with a directly-conflicting-clusters limit.
   - The CPFP carveout rule is eliminated (it doesn't make sense in a cluster-limited mempool)
   - The ancestor and descendant limits are no longer enforced.
   - New cluster count/cluster vsize limits are now enforced instead.
   - Transaction relay now uses chunk feerate comparisons to determine the order that newly received transactions are announced to peers.

  Additionally, the cached ancestor and descendant data are dropped from the mempool, along with the multi_index indices that were maintained to sort the mempool by ancestor and descendant feerates. For compatibility (eg with wallet behavior or RPCs exposing this), this information is now calculated dynamically instead.

ACKs for top commit:
  instagibbs:
    reACK 17cf9ff7ef
  glozow:
    reACK 17cf9ff7ef
  sipa:
    ACK 17cf9ff7ef

Tree-SHA512: bbde46d913d56f8d9c0426cb0a6c4fa80b01b0a4c2299500769921f886082fb4f51f1694e0ee1bc318c52e1976d7ebed8134a64eda0b8044f3a708c04938eee7
2025-11-25 10:35:11 +00:00
Sebastian Falbesoner
8343a9ffcc test: add -alertnotify test for large work invalid chain warning 2025-11-24 14:28:04 +01:00
Suhas Daftuar
17cf9ff7ef Use cluster size limit for -maxmempool bound, and allow -maxmempool=0 in general
Previously we would sanity check the -maxmempool configuration based on a
multiple of the descendant size limit, but with cluster mempool the maximum
evicted size is now the cluster size limit, so use that instead.

Also allow -maxmempool=0 in general (and not just if
-limitdescendantsize/-limitclustersize is set to 0).
2025-11-21 22:02:07 -05:00
Suhas Daftuar
de2e9a24c4 test: extend package rbf functional test to larger clusters
Co-Authored-By: Gregory Sanders <gsanders87@gmail.com>
2025-11-21 22:02:07 -05:00
Ava Chow
0690514d4f Merge bitcoin/bitcoin#33770: init: Require explicit -asmap filename
288b8c30be doc: Drop (default: none) from -i2psam description (Ryan Ofsky)
f6ec3519a3 init: Require explicit -asmap filename (Ryan Ofsky)

Pull request description:

  Currently, if `-asmap` is specified without a filename bitcoind tries to load `ip_asn.map` data file.

  This change now requires `-asmap=ip_asn.map` or another filename to be specified explicitly.

  The change is intended to make behavior of the option explicit and avoid confusion reported https://github.com/bitcoin/bitcoin/issues/33386 where documentation specifies a default file which is not actually loaded by default. It was originally implemented in
  https://github.com/bitcoin/bitcoin/pull/33631#issuecomment-3410302383 and various alternatives are discussed there.

ACKs for top commit:
  brunoerg:
    reACK 288b8c30be
  fjahr:
    re-ACK 288b8c30be
  vostrnad:
    utACK 288b8c30be
  achow101:
    ACK 288b8c30be

Tree-SHA512: 11a38a03892a58d6ccc1505cfbf915f58a86df9891761d89dc54b92d40593ee3cbb2d7c7bdbb922b871b3529072ef7f34cc98393aff6e8f0633b56352315b27c
2025-11-21 15:28:36 -08:00
Ava Chow
b2f88b53e0 Merge bitcoin/bitcoin#33286: doc: update multisig tutorial to use multipath descriptors
de7c3587cd doc: Update add checksum instructions in tutorial (Ben Westgate)
2a46e94a16 doc: Update multisig-tutorial.md to use multipath descriptors (Ben Westgate)

Pull request description:

  ### Summary

  Update `doc/multisig-tutorial.md` to use multipath descriptor format
  instead of separate external/internal descriptors. The tutorial now:

  - extracts a single `xpub_n` per participant
  - constructs a multipath `wsh(sortedmulti(...))` descriptor with `<0;1>`
    change index semantics
  - uses `getdescriptorinfo` to compute descriptor checksum
  - explains that `importdescriptors` expands the multipath descriptor
    into internal and external descriptors
  - update `/test/functional/wallet_multisig_descriptor_psbt.py` functional test / documentation to use multi-path descriptors

  ---

  ### Motivation

  A single multipath descriptor is the most convenient pattern for multisig; our documentation should use it.

  ---

  ### What changed

  - replaced extraction of `external_xpub_n` and `internal_xpub_n` with
    extraction of a single `xpub_n`
  - removed instructions to create and import separate external/internal
    descriptors
  - added instructions to build a multipath `wsh(sortedmulti(...))`
    descriptor and derive checksum with `getdescriptorinfo`
  - checksum field is parsed and appended as the multipath descriptor is not the canonical "desc" output
  - clarified that `importdescriptors` automatically expands multipath
    descriptors into internal and external forms
  - similar changes to the functional test: wallet_multisig_descriptor_psbt.

  ---

  ### Testing

  I have run the updated shell snippets and confirmed the multipath descriptor produces the same `listdescriptors` output after importing as the two descriptor method in bitcoin:master.

  ---

  ### Related issues / PRs

  This tutorial change references the multipath descriptor
  consolidation (see commit / PR referenced in the change). The commit
  message points to bitcoin#22838 as the upstream change that enables
  this behavior.

  ---

  ### Release note (for changelog)

  Documentation: update multisig tutorial and multisig functional test to use multipath descriptors

ACKs for top commit:
  Sjors:
    utACK de7c3587cd
  kannapoix:
    Core review ACK: de7c3587cd
  achow101:
    ACK de7c3587cd
  rkrux:
    crACK de7c3587cd

Tree-SHA512: ca7275d8ad04922b3fa8d2a3084ff96aa2104265f1fc2f749814dd16776351ab692d67e7e643d08052f7492e3eaa1a9a4dfe3470163e57939a49f782d3df511a
2025-11-21 15:05:50 -08:00
Ava Chow
313cdd2bfb Merge bitcoin/bitcoin#33915: test: Retry download in get_previous_releases.py
fad06f3bb4 test: retry download in get_previous_releases.py (MarcoFalke)

Pull request description:

  Hopefully fixes https://github.com/bitcoin/bitcoin/issues/33913 (intermittent download issues)

  If not, the diff there to cache the bins can be considered.

ACKs for top commit:
  davidgumberg:
    ACK fad06f3bb4
  achow101:
    ACK fad06f3bb4
  enirox001:
    ACK fad06f3
  TheCharlatan:
    ACK fad06f3bb4
  janb84:
    ACK fad06f3bb4

Tree-SHA512: 6597e993cc64259e361f65045a938e693b47ff73a7dc22f5a70b6c9890986817849915f90096e3b87684f25152a6d8ae4ec5480c326c32d36273cdb0a33ceb0c
2025-11-21 13:35:03 -08:00
merge-script
6b2d17b132 Merge bitcoin/bitcoin#33888: ci: Re-enable LINT_CI_SANITY_CHECK_COMMIT_SIG
55555db055 doc: Add missing --platform=linux to docker build command (MarcoFalke)
fa0ce4c148 ci: Re-enable LINT_CI_SANITY_CHECK_COMMIT_SIG (MarcoFalke)
faa0973de2 ci: [refactor] Rename CIRRUS_PR env var to LINT_CI_IS_PR (MarcoFalke)
fa1dacaebe ci: Move lint exec snippet to stand-alone py file (MarcoFalke)

Pull request description:

  The sanity check to check the last few merge commit signatures on the main branch was accidentally and silently disabled while moving from the `cirrus-ci.com` platform to the GHA platform.

  So fix that by re-enabling it.

  Also, contains a few other lint cleanup commits.

ACKs for top commit:
  janb84:
    re ACK 55555db055
  willcl-ark:
    ACK 55555db055

Tree-SHA512: e623dc88035ee4d1c6a8efa5fad33c35cface87f54e78c7ebfe5d468d28d8d8097150344d276f90f8ed52a89e61609ce95380476ea0151b50f73ad5919233933
2025-11-20 17:29:57 +00:00
merge-script
29c37651c7 Merge bitcoin/bitcoin#33880: test: Fix race condition in IPC interface block progation test
2578e6fc0f test: Fix race condition in IPC interface block propagation test (Fabian Jahr)

Pull request description:

  CI failed on this condition here: https://github.com/bitcoin/bitcoin/actions/runs/19395398994/job/55494696022?pr=33878#step:9:3983

  The check was added not too long ago in https://github.com/bitcoin/bitcoin/pull/33745 and the fix here switches the check to the node which actually produces the block. There are also some comments added to make the checks easier so understand.

  Closes #33884

ACKs for top commit:
  Sjors:
    re-utACK 2578e6fc0f
  maflcko:
    lgtm ACK 2578e6fc0f

Tree-SHA512: bfb7ae44aede50a00d4096e1a9922f9b8df31ce4242e12863e329d0d1e714d8cb46c852f694c32314e4bd26b524535e3a6967b7c57861a9b00cf09831a950b99
2025-11-20 17:19:26 +00:00
MarcoFalke
fad06f3bb4 test: retry download in get_previous_releases.py 2025-11-20 12:47:03 +01:00
Fabian Jahr
2578e6fc0f test: Fix race condition in IPC interface block propagation test 2025-11-19 23:20:24 +01:00
Ava Chow
53b72372da Merge bitcoin/bitcoin#31734: miniscript: account for all StringType variants in Miniscriptdescriptor::ToString()
28a4fcb03c test: check listdescriptors do not return a mix of hardened derivation marker (pythcoiner)
975783cb79 descriptor: account for all StringType in MiniscriptDescriptor::ToStringHelper() (pythcoiner)

Pull request description:

  In `MiniscriptDescriptor::ToStringHelper()` only the `StringType::Private` variant of the `type` argument was handled. This PR implements serializing w/ all variants of `StringType` & add a functional test for the descriptor triggering the related issue.

  Closes #31694: previously when calling `listdescriptors` RPC on a wallet containing a taproot descriptor w/ a (miniscript) taptree, origins of internal key & taptree were serialized w/ differents hardened derivation markers:
   - origin of the internal key were serialized w/ `StringType::Normalized` type (using `h` as marker)
   - origins of taptree keys were serialized w/ `StringType::Private` type (using `'` as marker)

  Note: Origins in segwit (`wsh()`) miniscript descriptors were also serialized w/ `StringType::Private` type (`'` marker) and are now serialized w/ `StringType::Normalized` type (`h` marker).

ACKs for top commit:
  sipa:
    Code review ACK 28a4fcb03c
  achow101:
    ACK 28a4fcb03c
  rkrux:
    Concept ACK 28a4fcb03c

Tree-SHA512: 15d14000b5951ca69a64a05b9a0b138c48a07b81eaf2fa86b91ac20cc8735533355a787363c64ba88403dd8a56ef5232cba57d34bea80835a0f40774d62fbc2b
2025-11-18 14:32:01 -08:00
Ava Chow
a7f9bbe4c5 Merge bitcoin/bitcoin#32821: rpc: Handle -named argument parsing where '=' character is used
f53dbbc505 test: Add functional tests for named argument parsing (zaidmstrr)
694f04e2bd rpc: Handle -named argument parsing where '=' character is used (zaidmstrr)

Pull request description:

  Addresses [comment](https://github.com/bitcoin/bitcoin/pull/31375#discussion_r2091886628) and [this](https://github.com/bitcoin/bitcoin/pull/31375#discussion_r2092039999).

  The [PR #31375](https://github.com/bitcoin/bitcoin/pull/31375) got merged and enables `-named` by default in the `bitcoin rpc` interface; `bitcoin rpc` corresponds to `bitcoin-cli -named` as it's just a wrapper.  Now, the problem arises when we try to parse the positional paramater which might contain "=" character.  This splits the parameter into two parts first, before the "=" character, which treats this as the parameter name, but the other half is mostly passed as an empty string. Here, the first part of the string is an unknown parameter name; thus, an error is thrown. These types of errors are only applicable to those RPCs which might contain the `=` character as a parameter. Some examples are `finalizepsbt`, `decodepsbt`, `verifymessage` etc.

  This is the one example of the error in `finalizepsbt` RPC:
  ```
  ./bitcoin-cli -named -regtest finalizepsbt cHNidP8BAJoCAAAAAqvNEjSrzRI0q80SNKvNEjSrzRI0q80SNKvNEjSrzRI0AAAAAAD9////NBLNqzQSzas0Es2rNBLNqzQSzas0Es2rNBLNqzQSzasBAAAAAP3///8CoIYBAAAAAAAWABQVQBGVs/sqFAmC8HZ8O+g1htqivkANAwAAAAAAFgAUir7MzgyzDnRMjdkVa7d+Dwr07jsAAAAAAAAAAAA=
  error code: -8
  error message:
  Unknown named parameter cHNidP8BAJoCAAAAAqvNEjSrzRI0q80SNKvNEjSrzRI0q80SNKvNEjSrzRI0AAAAAAD9////NBLNqzQSzas0Es2rNBLNqzQSzas0Es2rNBLNqzQSzasBAAAAAP3///8CoIYBAAAAAAAWABQVQBGVs/sqFAmC8HZ8O+g1htqivkANAwAAAAAAFgAUir7MzgyzDnRMjdkVa7d+Dwr07jsAAAAAAAAAAAA
  ```
  This PR fixes this by updating the `vRPCConvertParams` table that identifies parameters that need special handling in `-named` parameter mode. The parser now recognises these parameters and handles strings with "=" char correctly, preventing them from being incorrectly split as parameter assignments.

ACKs for top commit:
  ryanofsky:
    Code review ACK f53dbbc505. Just applied comment & test suggestions since last review
  kannapoix:
    Code review ACK: f53dbbc505
  achow101:
    ACK f53dbbc505

Tree-SHA512: 1b517144efeff45a4c4256c27a39ddf187f1d6189d133402a45171678214a10ff2925c31edcfd556d67f85bd26d42f63c528b941b68c9880eab443f2c883e681
2025-11-18 14:06:54 -08:00
MarcoFalke
55555db055 doc: Add missing --platform=linux to docker build command
This is required to pick the native arch, similar to how the
CI_IMAGE_PLATFORM is set to linux.
2025-11-18 20:15:55 +01:00
Suhas Daftuar
6c5c44f774 test: add functional test for new cluster mempool RPCs
Co-authored-by: glozow <gloriajzhao@gmail.com>
2025-11-18 11:14:52 -05:00
Suhas Daftuar
21693f031a Expose cluster information via rpc
Co-authored-by: glozow <gloriajzhao@gmail.com>
2025-11-18 11:14:52 -05:00
Suhas Daftuar
cf3ab8e1d0 Stop enforcing descendant size/count limits
Cluster size limits should be enough.
2025-11-18 08:57:51 -05:00
Suhas Daftuar
89ae38f489 test: remove rbf carveout test from mempool_limit.py 2025-11-18 08:57:51 -05:00
Suhas Daftuar
69e1eaa6ed Add test case for cluster size limits to TRUC logic 2025-11-18 08:57:51 -05:00
Suhas Daftuar
9cda64b86c Stop enforcing ancestor size/count limits
The cluster limits should be sufficient.

Co-Authored-By: Gregory Sanders <gsanders87@gmail.com>
2025-11-18 08:57:51 -05:00
Suhas Daftuar
21b5cea588 Use cluster linearization for transaction relay sort order
Previously, transaction batches were first sorted by ancestor count and then
feerate, to ensure transactions are announced in a topologically valid order,
while prioritizing higher feerate transactions. Ancestor count is a crude
topological sort criteria, so replace this with linearization order so that the
highest feerate transactions (as would be observed by the mining algorithm) are
relayed before lower feerate ones, in a topologically valid way.

This also fixes a test that only worked due to the ancestor-count-based sort
order.
2025-11-18 08:53:59 -05:00
Suhas Daftuar
216e693729 Implement new RBF logic for cluster mempool
With a total ordering on mempool transactions, we are now able to calculate a
transaction's mining score at all times. Use this to improve the RBF logic:

- we no longer enforce a "no new unconfirmed parents" rule

- we now require that the mempool's feerate diagram must improve in order
  to accept a replacement

- the topology restrictions for conflicts in the package rbf setting have been
  eliminated

Revert the temporary change to mempool_ephemeral_dust.py that were previously
made due to RBF validation checks being reordered.

Co-authored-by: Gregory Sanders <gsanders87@gmail.com>, glozow <gloriajzhao@gmail.com>
2025-11-18 08:53:59 -05:00
Suhas Daftuar
ff8f115dec policy: Remove CPFP carveout rule
The addition of a cluster size limit makes the CPFP carveout rule useless,
because carveout cannot be used to bypass the cluster size limit. Remove this
policy rule and update tests to no longer rely on the behavior.
2025-11-18 08:53:59 -05:00
Suhas Daftuar
95762e6759 Do not allow mempool clusters to exceed configured limits
Include an adjustment to mempool_tests.cpp due to the additional memory used by
txgraph.

Includes a temporary change to the mempool_ephemeral_dust.py functional test,
due to validation checks being reordered. This change will revert once the RBF
rules are changed in a later commit.
2025-11-18 08:53:58 -05:00
glozow
edb3e7cdf6 [test] rework/delete feature_rbf tests requiring large clusters 2025-11-18 08:53:58 -05:00
Suhas Daftuar
435fd56711 test: update feature_rbf.py replacement test
Preparatory commit to the rbf functional test, before changes are made to the
rbf rules as part of cluster mempool.
2025-11-18 08:53:58 -05:00
Sebastian Falbesoner
0aebdac95d init: completely remove -maxorphantx option 2025-11-13 19:02:39 +01:00
merge-script
dfde31f2ec Merge bitcoin/bitcoin#33864: scripted-diff: fix leftover references to policy/fees.h
b0a3887154 scripted-diff: fix leftover references to `policy/fees.h` (ismaelsadeeq)

Pull request description:

  Fixes #33863

  ryanofsky wrote
  > I still see some references to the src/policy/fees.h file removed by this PR:

  ```
  $ git grep -n policy/fees.h
  src/wallet/rpc/spend.cpp:206: * @param[in]     conf_target       UniValue integer; confirmation target in blocks, values between 1 and 1008 are valid per policy/fees.h;
  test/functional/rpc_estimatefee.py:39:        # max value of 1008 per src/policy/fees.h
  test/functional/rpc_psbt.py:604:                assert_raises_rpc_error(-8, "Invalid conf_target, must be between 1 and 1008",  # max value of 1008 per src/policy/fees.h
  test/functional/wallet_basic.py:337:            assert_raises_rpc_error(-8, "Invalid conf_target, must be between 1 and 1008",  # max value of 1008 per src/policy/fees.h
  test/functional/wallet_fundrawtransaction.py:851:                assert_raises_rpc_error(-8, "Invalid conf_target, must be between 1 and 1008",  # max value of 1008 per src/policy/fees.h
  test/functional/wallet_send.py:315:                expect_error=(-8, "Invalid conf_target, must be between 1 and 1008"))  # max value of 1008 per src/policy/fees.h
  ```

  This is fixed in this PR by running a script that searches for what he greps and replaces it with the right reference.

  ```
  git grep -l "policy\/fees\.h" | xargs sed -i "s/policy\/fees.h/policy\/fees\/block_policy_estimator.h/g"
  ```

ACKs for top commit:
  kevkevinpal:
    ACK [b0a3887](b0a3887154)
  janb84:
    ACK b0a3887154
  rkrux:
    lgtm ACK b0a3887154

Tree-SHA512: e24f2aaf18fcfb0ae047a53ed209135a644ff08f5a8bc162c1522be3f99d7d01d550fc2e73d8db5fec7b748902daf68e61e7a5624f5913b9824feba5641fc78c
2025-11-13 09:48:50 +00:00
ismaelsadeeq
b0a3887154 scripted-diff: fix leftover references to policy/fees.h
-BEGIN VERIFY SCRIPT-
git grep -l "policy\/fees\.h" | xargs sed -i "s/policy\/fees.h/policy\/fees\/block_policy_estimator.h/g"
-END VERIFY SCRIPT-
2025-11-12 17:40:47 +00:00
merge-script
48d4b936e0 Merge bitcoin/bitcoin#33511: init: Fix Ctrl-C shutdown hangs during wait calls
c25a5e670b init: Signal m_tip_block_cv on Ctrl-C (Ryan Ofsky)
6a29f79006 test: Test SIGTERM handling during waitforblockheight call (Ryan Ofsky)

Pull request description:

  Signal `m_tip_block_cv` when Ctrl-C is pressed or `SIGTERM` is received, the same way it is currently signaled when the `stop` RPC is called. This lets RPC calls like `waitforblockheight` and IPC calls like `waitTipChanged` be interrupted, instead of waiting for their original timeouts and delaying shutdown.

  This issue was reported by plebhash in #33463. These hangs have been present since #30409. A similar bug was also fixed previously in Qt in #18452 and this PR simplifies that fix.

ACKs for top commit:
  Sjors:
    tACK c25a5e670b
  TheCharlatan:
    ACK c25a5e670b
  enirox001:
    Concept ACK c25a5e6

Tree-SHA512: 320aaa74fd308e826521c48c9a8aca4bd5f5530064cda2303d251d8e93e50c474bcd0db760ce04921928e73abefe4847aff797ac9ca7c89e74e5051bbed061cd
2025-11-12 10:16:29 -05:00
merge-script
3c3c6adb72 Merge bitcoin/bitcoin#33745: mining: check witness commitment in submitBlock
6eaa00fe20 test: clarify submitBlock() mutates the template (Sjors Provoost)
862bd43283 mining: ensure witness commitment check in submitBlock (Sjors Provoost)
00d1b6ef4b doc: clarify UpdateUncommittedBlockStructures (Sjors Provoost)

Pull request description:

  When an IPC client requests a new block template via the Mining interface, we hold on to its `CBlock`. That way when they call `submitSolution()` we can modify it in place, rather than having to reconstruct the full block like the `submitblock` RPC does.

  Before this commit however we forgot to invalidate `m_checked_witness_commitment`, which we should since the client brings a new coinbase.

  This would cause us to accept an invalid chaintip.

  Fix this and add a test to confirm that we now reject such a block. As a sanity check, we add a second node to the test and confirm that will accept our mined block.

  As first noticed in #33374 the IPC code takes the coinbase as provided, unlike the `submitblock` RPC which calls `UpdateUncommittedBlockStructures()` and adds witness commitment to the coinbase if it was missing.

  Although that could have been an alternative fix, we instead document that IPC clients are expected to provide the full coinbase including witness commitment.

  Patch to produce the original issue:

  ```diff
  diff --git a/src/node/miner.cpp b/src/node/miner.cpp
  index b988e28a3f..28e9048a4d 100644
  --- a/src/node/miner.cpp
  +++ b/src/node/miner.cpp
  @@ -450,15 +450,10 @@ void AddMerkleRootAndCoinbase(CBlock& block, CTransactionRef coinbase, uint32_t
       }
       block.nVersion = version;
       block.nTime = timestamp;
       block.nNonce = nonce;
       block.hashMerkleRoot = BlockMerkleRoot(block);
  -
  -    // Reset cached checks
  -    block.m_checked_witness_commitment = false;
  -    block.m_checked_merkle_root = false;
  -    block.fChecked = false;
   }

   std::unique_ptr<CBlockTemplate> WaitAndCreateNewBlock(ChainstateManager& chainman,
                                                         KernelNotifications& kernel_notifications,
                                                         CTxMemPool* mempool,
  diff --git a/test/functional/interface_ipc.py b/test/functional/interface_ipc.py
  index cce56e3294..bf1b7048ab 100755
  --- a/test/functional/interface_ipc.py
  +++ b/test/functional/interface_ipc.py
  @@ -216,22 +216,22 @@ class IPCInterfaceTest(BitcoinTestFramework):
               assert_equal(res.result, True)

               # The remote template block will be mutated, capture the original:
               remote_block_before = await self.parse_and_deserialize_block(template, ctx)

  -            self.log.debug("Submitted coinbase must include witness")
  +            self.log.debug("Submitted coinbase with missing witness is accepted")
               assert_not_equal(coinbase.serialize_without_witness().hex(), coinbase.serialize().hex())
               res = await template.result.submitSolution(ctx, block.nVersion, block.nTime, block.nNonce, coinbase.serialize_without_witness())
  -            assert_equal(res.result, False)
  +            assert_equal(res.result, True)

               self.log.debug("Even a rejected submitBlock() mutates the template's block")
               # Can be used by clients to download and inspect the (rejected)
               # reconstructed block.
               remote_block_after = await self.parse_and_deserialize_block(template, ctx)
               assert_not_equal(remote_block_before.serialize().hex(), remote_block_after.serialize().hex())

  -            self.log.debug("Submit again, with the witness")
  +            self.log.debug("Submit again, with the witness - does not replace the invalid block")
               res = await template.result.submitSolution(ctx, block.nVersion, block.nTime, block.nNonce, coinbase.serialize())
               assert_equal(res.result, True)

               self.log.debug("Block should propagate")
               assert_equal(self.nodes[1].getchaintips()[0]["height"], current_block_height + 1)
  ```

ACKs for top commit:
  ryanofsky:
    Code review ACK 6eaa00fe20. Just documentation updates and test clarifications since last review, also splitting up a commit.
  TheCharlatan:
    Re-ACK 6eaa00fe20
  ismaelsadeeq:
    Code review and tested ACK   6eaa00fe20

Tree-SHA512: 3a6280345b0290fe8300ebc63c13ad4058d24ceb35b7d7a784b974d5f04f420860ac03a9bf2fc6a799ef3fc55552ce033e879fa369298f976b9a01d72bd55d9e
2025-11-12 10:03:48 -05:00
merge-script
e652b69b8d Merge bitcoin/bitcoin#33003: test: add option to skip large re-org test in feature_block
8810642b57 test: add option to skip large re-org test in feature_block (brunoerg)

Pull request description:

  Fixes #32877

  This PR adds a config flag `--skipreorg` which is used to skip the large re-org test. According to corecheck, `feature_block` is our slowest functional test and primarily because of this large re-org test. However, this test might not be useful for the mutation analysis of some files and could be skipped to save a huge amount of time.

  ```
  time ./build/test/functional/feature_block.py --skipreorg
  ./build/test/functional/feature_block.py --skipreorg  11.38s user 0.33s system 37% cpu 31.422 total
  time ./build/test/functional/feature_block.py
  ./build/test/functional/feature_block.py  25.87s user 3.53s system 56% cpu 52.317 total
  ```

ACKs for top commit:
  maflcko:
    review ACK 8810642b57 🥁
  enirox001:
    tACK 8810642 – Ran tests with/without --skipreorg; saw ~40 % speedup; no regressions.
  theStack:
    Concept and code-review ACK 8810642b57
  glozow:
    lgtm ACK 8810642b57

Tree-SHA512: 4ef38bd32b8ad8ec2b7f30c96d2fe545d920759645ff52f632699f829b64f8d26fe878f3fdd255142235edd0a740a7feb64da8f5a10d0d740ebfa46c43ae60eb
2025-11-12 09:54:08 -05:00
Ava Chow
8c2710b041 Merge bitcoin/bitcoin#32517: rpc: add "ischange: true" to decoded tx outputs in wallet gettransaction response
060bb55508 rpc: add decoded tx details to gettransaction with extra wallet fields (Matthew Zipkin)
ad1c3bdba5 [move only] move DecodeTxDoc() to a common util file for sharing (Matthew Zipkin)
d633db5416 rpc: add "ischange: true" in wallet gettransaction decoded tx output (Matthew Zipkin)

Pull request description:

  This change is motivated by external RBF clients like https://github.com/CardCoins/additive-rbf-batcher/. It saves the user a redundant re-looping of tx outputs, calling `getaddressinfo` on each one, looking for the change output in order to adjust the fee.

  The field `"ischange"` only appears when `gettransaction` is called on a wallet, and is either `true` or not present at all. I chose not to include `ischange: false` because it is confusing to see that on *received* transactions.

  Example of the new field:

  ```
      "vout": [
        {
          "value": 1.00000000,
          "n": 0,
          "scriptPubKey": {
            "asm": "0 5483235e05c76273b3b50af62519738781aff021",
            "desc": "addr(bcrt1q2jpjxhs9ca388va4ptmz2xtns7q6lupppkw7wu)#d42g84j6",
            "hex": "00145483235e05c76273b3b50af62519738781aff021",
            "address": "bcrt1q2jpjxhs9ca388va4ptmz2xtns7q6lupppkw7wu",
            "type": "witness_v0_keyhash"
          }
        },
        {
          "value": 198.99859000,
          "n": 1,
          "scriptPubKey": {
            "asm": "0 870ab1ab58632b05a417d5295f4038500e407592",
            "desc": "addr(bcrt1qsu9tr26cvv4stfqh65547spc2q8yqavj7fnlju)#tgapemkv",
            "hex": "0014870ab1ab58632b05a417d5295f4038500e407592",
            "address": "bcrt1qsu9tr26cvv4stfqh65547spc2q8yqavj7fnlju",
            "type": "witness_v0_keyhash"
          },
          "ischange": true
        }
      ]

  ```

ACKs for top commit:
  furszy:
    ACK [060bb55](060bb55508)
  maflcko:
    review ACK 060bb55508 🌛
  achow101:
    ACK 060bb55508
  rkrux:
    lgtm ACK 060bb55508

Tree-SHA512: aae4854d2bb4e9a7bc1152691ea90e594e8da8a63c9c7fda72a504fb6a7e54ae274ed5fa98d35d270e0829cc8f8d2fd35a5fc9735c252a10aa42cc22828930e7
2025-11-10 08:58:34 -08:00
Ava Chow
1fe851a478 Merge bitcoin/bitcoin#32180: p2p: Advance pindexLastCommonBlock early after connecting blocks
01cc20f330 test: improve coverage for a resolved stalling situation (Martin Zumsande)
9af6daf07e test: remove magic number when checking for blocks that have arrived (Martin Zumsande)
3069d66dca p2p: During block download, adjust pindexLastCommonBlock better (Martin Zumsande)

Pull request description:

  As described in #32179, `pindexLastCommonBlock` is updated later than necessary
  in master.
  In case of a linear chain with no forks, it can be moved forward at the beginning of
  `FindNextBlocksToDownload`, so that the updated value can be used to better estimate `nWindowEnd`.
  This helps the node to request all blocks from peers within the correct 1024-block-window and avoids peers being incorrectly marked as stallers.

  I also changed `p2p_ibd_stalling.py` to cover the situation after a resolved situation, making sure that no additional peers are marked for stalling.

  Fixes #32179

ACKs for top commit:
  Crypt-iQ:
    crACK 01cc20f330
  stringintech:
    re-ACK 01cc20f
  achow101:
    ACK 01cc20f330
  sipa:
    utACK 01cc20f330

Tree-SHA512: a97f7a7ef5ded538ee35576e04b3fbcdd46a6d0189c7ba3abacc6e0d81e800aac5b0c2d2565d0462ef6fd4acc751989f577fd6adfd450171a7d6ab26f437df32
2025-11-10 08:48:49 -08:00
merge-script
f4903dddc9 Merge bitcoin/bitcoin#33433: Bugfix: QA: rpc_bind: Skip nonloopback test if no such address is found
79b4c276e7 Bugfix: QA: rpc_bind: Skip nonloopback test if no such address is found (Luke Dashjr)

Pull request description:

  Without this, I get:

  ```
  2025-09-19T03:14:05.157000Z TestFramework (INFO): PRNG seed is: 3218602557639511064
  2025-09-19T03:14:05.158000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin-test/a
  2025-09-19T03:14:05.158000Z TestFramework (INFO): Check for ipv6
  2025-09-19T03:14:05.158000Z TestFramework (INFO): Check for non-loopback interface
  2025-09-19T03:14:05.158000Z TestFramework (INFO): Bind test for []
  2025-09-19T03:14:05.516000Z TestFramework (INFO): Bind test for []
  2025-09-19T03:14:05.871000Z TestFramework (INFO): Bind test for ['[::1]']
  2025-09-19T03:14:06.227000Z TestFramework (INFO): Bind test for ['127.0.0.1', '[::1]']
  2025-09-19T03:14:06.583000Z TestFramework (INFO): Using interface None for testing
  2025-09-19T03:14:06.583000Z TestFramework (INFO): Bind test for [None]
  2025-09-19T03:14:06.583000Z TestFramework (ERROR): Unexpected exception
  Traceback (most recent call last):
    File "/Bitcoin/bitcoin/workingtree/test/functional/test_framework/test_framework.py", line 135, in main
      self.run_test()
      ~~~~~~~~~~~~~^^
    File "/Bitcoin/bitcoin/workingtree/test/functional/rpc_bind.py", line 126, in run_test
      self._run_nonloopback_tests()
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
    File "/Bitcoin/bitcoin/workingtree/test/functional/rpc_bind.py", line 157, in _run_nonloopback_tests
      self.run_bind_test([self.non_loopback_ip], self.non_loopback_ip, [self.non_loopback_ip],
      ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          [(self.non_loopback_ip, self.defaultport)])
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Bitcoin/bitcoin/workingtree/test/functional/rpc_bind.py", line 38, in run_bind_test
      expected = [(addr_to_hex(addr), port) for (addr, port) in expected]
                   ~~~~~~~~~~~^^^^^^
    File "/Bitcoin/bitcoin/workingtree/test/functional/test_framework/netutil.py", line 132, in addr_to_hex
      if '.' in addr: # IPv4
         ^^^^^^^^^^^
  TypeError: argument of type 'NoneType' is not iterable
  ```

ACKs for top commit:
  maflcko:
    review ACK 79b4c276e7 🏑
  theStack:
    Tested ACK 79b4c276e7

Tree-SHA512: 2a723d9bc5d1d50a8321a4f8a8cac3da3125d373ea71e6cc9d03de07307008f58970e361490d4c34530a6a976cb078b62d0ef09b7fb321ca1cfb9249a70d99a5
2025-11-10 14:52:19 +00:00
merge-script
ddd2afac10 Merge bitcoin/bitcoin#33676: interfaces: enable cancelling running waitNext calls
dcb56fd4cb interfaces: add interruptWait method (ismaelsadeeq)

Pull request description:

  This is an attempt to fix #33575 see the issue for background and the usefulness of this feature.

  This PR uses one of the suggested approaches: adding a new `interruptWaitNext()` method to the mining interface.

  It introduces a new boolean variable, `m_interrupt_wait`, which is set to `false` when the thread starts waiting. The `interruptWaitNext()` method wakes the thread and sets `m_interrupt_wait` to `true`.
  Whenever the thread wakes up, it checks whether the wait was aborted; if so, it simply set ` m_interrupt_wait ` to false and return`nullptr`.

  This PR also adds a functional test for the new method. The test uses `asyncio` to spawn two tasks and attempts to ensure that the wait is executed before the interrupt by using an event monitor. It adds a 0.1-second buffer to ensure the wait has started executing.
  If that buffer elapses without `waitNext` executing, the test will fail because a transaction is created after the buffer.

ACKs for top commit:
  furszy:
    Code ACK dcb56fd4cb
  ryanofsky:
    Code review ACK dcb56fd4cb, just tweaking semantics slightly since last review so if an `interruptWait` call is made shortly after a `waitNext` call it will reliably cause the `waitNext` call to return right away without blocking, even if the `waitNext` call had not begun to execute or wait yet.
  Sjors:
    tACK dcb56fd4cb
  TheCharlatan:
    ACK dcb56fd4cb

Tree-SHA512: a03f049e1f303b174a9e5d125733b6583dfd8effa12e7b6c37bd9b2cff9541100f5f4514e80f89005c44a57d7e47804afe87aa5fdb6831f3b0cd9b01d83e42be
2025-11-10 09:56:27 +00:00
merge-script
3c4bec6223 Merge bitcoin/bitcoin#33782: test: remove obsolete get_{key,multisig} helpers from wallet_util.py
ec8516ceb7 test: remove obsolete `get_{key,multisig}` helpers from wallet_util.py (Sebastian Falbesoner)

Pull request description:

  This small cleanup PR is a late follow-up to #31250 (commit c847dee148). These helpers are unused and wouldn't work anymore, as they call a legacy wallet RPC (`dumpprivkey`). They were only ever used for testing the `importmulti` RPC, which also doesn't exist anymore. Functional tests that need to create key pairs and derive various output script types from them can use `get_generate_key` (introduced in #16528, commit f193ea889d) instead, without involving the node.

ACKs for top commit:
  rkrux:
    crACK ec8516ceb7
  brunoerg:
    code review ACK ec8516ceb7

Tree-SHA512: cab3701f1a8fbcff0eecea4cfdc632ffac226afd2eefe3c9274a84ee1bb71fb231a57cd0876025c714be257a249157b048b67e309b3734442c425d85cf481cf6
2025-11-07 11:46:51 +00:00
Vasil Dimov
2bd155e6ee test: move create_malleated_version() to messages.py for reuse 2025-11-05 16:10:41 +01:00
Sebastian Falbesoner
ec8516ceb7 test: remove obsolete get_{key,multisig} helpers from wallet_util.py
These helpers use a legacy wallet RPC (`dumpprivkey`) and thus don't
work anymore. They were only ever used for testing the `importmulti`
RPC, which also doesn't exist anymore.
2025-11-04 20:01:26 +01:00
merge-script
4da01123df Merge bitcoin/bitcoin#30595: kernel: Introduce C header API
6c7a34f3b0 kernel: Add Purpose section to header documentation (TheCharlatan)
7e9f00bcc1 kernel: Allowing reducing exports (TheCharlatan)
7990463b10 kernel: Add pure kernel bitcoin-chainstate (TheCharlatan)
36ec9a3ea2 Kernel: Add functions for working with outpoints (TheCharlatan)
5eec7fa96a kernel: Add block hash type and block tree utility functions to C header (TheCharlatan)
f5d5d1213c kernel: Add function to read block undo data from disk to C header (TheCharlatan)
09d0f62638 kernel: Add functions to read block from disk to C header (TheCharlatan)
a263a4caf2 kernel: Add function for copying block data to C header (TheCharlatan)
b30e15f432 kernel: Add functions for the block validation state to C header (TheCharlatan)
aa262da7bc kernel: Add validation interface to C header (TheCharlatan)
d27e27758d kernel: Add interrupt function to C header (TheCharlatan)
1976b13be9 kernel: Add import blocks function to C header (TheCharlatan)
a747ca1f51 kernel: Add chainstate load options for in-memory dbs in C header (TheCharlatan)
070e77732c kernel: Add options for reindexing in C header (TheCharlatan)
ad80abc73d kernel: Add block validation to C header (TheCharlatan)
cb1590b05e kernel: Add chainstate loading when instantiating a ChainstateManager (TheCharlatan)
e2c1bd3d71 kernel: Add chainstate manager option for setting worker threads (TheCharlatan)
65571c36a2 kernel: Add chainstate manager object to C header (TheCharlatan)
c62f657ba3 kernel: Add notifications context option to C header (TheCharlatan)
9e1bac4585 kernel: Add chain params context option to C header (TheCharlatan)
337ea860df kernel: Add kernel library context object (TheCharlatan)
28d679bad9 kernel: Add logging to kernel library C header (TheCharlatan)
2cf136dec4 kernel: Introduce initial kernel C header API (TheCharlatan)

Pull request description:

  This is a first attempt at introducing a C header for the libbitcoinkernel library that may be used by external applications for interfacing with Bitcoin Core's validation logic. It currently is limited to operations on blocks. This is a conscious choice, since it already offers a lot of powerful functionality, but sits just on the cusp of still being reviewable scope-wise while giving some pointers on how the rest of the API could look like.

  The current design was informed by the development of some tools using the C header:

  * A re-implementation (part of this pull request) of [bitcoin-chainstate](https://github.com/bitcoin/bitcoin/blob/master/src/bitcoin-chainstate.cpp).
  * A re-implementation of the python [block linearize](https://github.com/bitcoin/bitcoin/tree/master/contrib/linearize) scripts: https://github.com/TheCharlatan/bitcoin/tree/kernelLinearize
  * A silent payment scanner: https://github.com/josibake/silent-payments-scanner
  * An electrs index builder: https://github.com/josibake/electrs/commits/electrs-kernel-integration
  * A rust bitcoin node: https://github.com/TheCharlatan/kernel-node
  * A reindexer: https://github.com/TheCharlatan/bitcoin/tree/kernelApi_Reindexer

  The library has also been used by other developers already:

  * A historical block analysis tool: https://github.com/ismaelsadeeq/mining-analysis
  * A swiftsync hints generator: https://github.com/theStack/swiftsync-hints-gen
  * Fast script validation in floresta: https://github.com/vinteumorg/Floresta/pull/456
  * A swiftsync node implementation: https://github.com/2140-dev/swiftsync/tree/master/node

  Next to the C++ header also made available in this pull request, bindings for other languages are available here:

  * Rust: https://github.com/TheCharlatan/rust-bitcoinkernel
  * Python: https://github.com/stickies-v/py-bitcoinkernel
  * Go: https://github.com/stringintech/go-bitcoinkernel
  * Java: https://github.com/yuvicc/java-bitcoinkernel

  The rust bindings include unit and fuzz tests for the API.

  The header currently exposes logic for enabling the following functionality:
  * Feature-parity with the now deprecated libbitcoin-consensus
  * Optimized sha256 implementations that were not available to previous users of libbitcoin-consensus thanks to a static kernel context
  * Full support for logging as well as control over categories and severity
  * Feature parity with the existing experimental bitcoin-chainstate
  * Traversing the block index as well as using block index entries for reading block and undo data.
  * Running the chainstate in memory
  * Reindexing (both full and chainstate-only)
  * Interrupting long-running functions

  The pull request introduces a new kernel-only test binary that purely relies on the kernel C header and the C++ standard library. This is intentionally done to show its capabilities without relying on other code inside the project. This may be relaxed to include some of the existing utilities, or even be merged into the existing test suite.

  The complete docs for the API as well as some usage examples are hosted on [thecharlatan.ch/kernel-docs](https://thecharlatan.ch/kernel-docs/index.html). The docs are generated from the following repository (which also holds the examples): [github.com/TheCharlatan/kernel-docs](https://github.com/TheCharlatan/kernel-docs).

  #### How can I review this PR?

  Scrutinize the commit messages, run the tests, write your own little applications using the library, let your favorite code sanitizer loose on it, hook it up to your fuzzing infrastructure, profile the difference between the existing bitcoin-chainstate and the bitcoin-chainstate introduced here, be nitty on the documentation, police the C interface, opine on your own API design philosophy.

  To get a feeling for the API, read through the tests, or one of the examples.

  To configure this PR for making the shared library and the bitcoin-chainstate and test_kernel utilities available:
  ```
  cmake -B build -DBUILD_KERNEL_LIB=ON -DBUILD_UTIL_CHAINSTATE=ON
  ```

  Once compiled the library is part of the build artifacts that can be installed with:
  ```
  cmake --install build
  ```

  #### Why a C header (and not a C++ header)

  * Shipping a shared library with a C++ header is hard, because of name mangling and an unstable ABI.
  * Mature and well-supported tooling for integrating C exists for nearly every popular language.
  * C offers a reasonably stable ABI

  Also see https://github.com/bitcoin/bitcoin/pull/30595#issuecomment-2285719575.

  #### What about versioning?

  The header and library are still experimental and I would expect this to remain so for some time, so best not to worry about versioning yet.

  #### Potential future additions

  In future, the C header could be expanded to support (some of these have been roughly implemented):

  * Handling transactions, block headers, coins cache, utxo set, meta data, and the mempool
  * Adapters for an abstract coins store
  * Adapters for an abstract block store
  * Adapters for an abstract block tree store
  * Allocators and buffers for more efficient memory usage
  * An "[io-less](https://sans-io.readthedocs.io/how-to-sans-io.html)" interface
  * Hooks for an external mempool, or external policy rules

  #### Current drawbacks

  * For external applications to read the block index of an existing Bitcoin Core node, Bitcoin Core needs to shut down first, since leveldb does not support reading across multiple processes. Other than migrating away from leveldb, there does not seem to be a solution for this problem. Such a migration is implemented in #32427.
  * The fatal error handling through the notifications is awkward. This is partly improved through #29642.
  * Handling shared pointers in the interfaces is unfortunate. They make ownership and freeing of the resources fuzzy and poison the interfaces with additional types and complexity. However, they seem to be an artifact of the current code that interfaces with the validation engine. The validation engine itself does not seem to make extensive use of these shared pointers.
  * If multiple instances of the same type of objects are used, there is no mechanism for distinguishing the log messages produced by each of them. A potential solution is #30342.
  * The background leveldb compaction thread may not finish in time leading to a non-clean exit. There seems to be nothing we can do about this, outside of patching leveldb.

ACKs for top commit:
  alexanderwiederin:
    re-ACK 6c7a34f3b0
  stringintech:
    re-ACK 6c7a34f
  laanwj:
    Code review ACK 6c7a34f3b0
  ismaelsadeeq:
    reACK 6c7a34f3b0 👾
  fanquake:
    ACK 6c7a34f3b0 - soon we'll be running bitcoin (kernel)

Tree-SHA512: ffe7d4581facb7017d06da8b685b81f4b5e4840576e878bb6845595021730eab808d8f9780ed0eb0d2b57f2647c85dcb36b6325180caaac469eaf339f7258030
2025-11-04 15:38:42 +00:00
merge-script
56329beaee Merge bitcoin/bitcoin#32301: test: cover invalid codesep positions for signature in taproot
81e5c8385b test: cover invalid codesep positions for signature in taproot (Greg Sanders)

Pull request description:

  There is some basic coverage, but I felt like adding some boundary conditions where the only issue is the codesep value would be nice.

ACKs for top commit:
  ajtowns:
    ACK 81e5c8385b
  TheCharlatan:
    ACK 81e5c8385b

Tree-SHA512: de74895c3bb49854987654720ebcefea2f47c4a55ba6ab4a52878f6a9a0bd8b3085afa3485101610327fa8d35c3d074542f58540e126460bd4bea918cb0054ee
2025-11-04 09:56:58 +00:00
TheCharlatan
7990463b10 kernel: Add pure kernel bitcoin-chainstate
Re-write the bitcoin-chainstate utility by only using the kernel C++ API
header instead of internal Bitcoin Core code.
2025-11-04 08:32:13 +01:00
TheCharlatan
65571c36a2 kernel: Add chainstate manager object to C header
This is the main driver class for anything validation related, so expose
it here.

Creating the chainstate manager options will currently also trigger the
creation of their respectively configured directories.

The chainstate manager and block manager options are consolidated into a
single object. The kernel might eventually introduce a separate block
manager object for the purposes of being a light-weight block store
reader.

The chainstate manager will associate with the context with which it was
created for the duration of its lifetime and it keeps it in memory with
a shared pointer.

The tests now also create dedicated temporary directories. This is
similar to the behaviour in the existing unit test framework.

Co-authored-by: stickies-v <stickies-v@protonmail.com>
2025-11-04 08:31:59 +01:00
Ryan Ofsky
f6ec3519a3 init: Require explicit -asmap filename
Currently, if `-asmap` is specified without a filename, bitcoind tries to load
`ip_asn.map` data file.

This change now requires `-asmap=ip_asn.map` or another filename to be
specified explicitly.

The change is intended to make behavior of the option explicit avoid confusion
reported https://github.com/bitcoin/bitcoin/issues/33386 where documentation
specifies a default file which is not actually loaded by default. It was
originally implemented in
https://github.com/bitcoin/bitcoin/pull/33631#issuecomment-3410302383 and
various alternatives are discussed there.

Co-authored-by: Fabian Jahr <fjahr@protonmail.com>
2025-10-31 14:59:58 -04:00
merge-script
832a57673a Merge bitcoin/bitcoin#33749: test: ipc: resolve symlinks in which capnp
51093d6ae1 test: resolve symlinks in which result for capnp (David Gumberg)

Pull request description:

  On Fedora, `/bin/` and `/usr/bin` are symlinked, and on one of my boxes (although I could not reproduce this behavior in a docker container), `/bin` comes before `/usr/bin` in `$PATH`, so `which capnp` reports `/bin/capnp`, and `capnp_dir` is set to `/include`, and the test fails:

  ```console
  $ ./build/test/functional/interface_ipc.py
  2025-10-30T20:43:43.753812Z TestFramework (INFO): PRNG seed is: 8370468257027235753
  2025-10-30T20:43:43.754163Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_b9kjzj2a
  terminate called after throwing an instance of 'kj::ExceptionImpl'
    what():  mp/proxy.capnp:6: failed: Import failed: /capnp/c++.capnp
  Aborted (core dumped)
  ```

  This changes the functional test to resolve any symlinks in the `capnp` binary path reported by `which`.

ACKs for top commit:
  TheCharlatan:
    utACK 51093d6ae1
  ryanofsky:
    Code review ACK 51093d6ae1

Tree-SHA512: 17a3e16c3ef50d19e65c18bd12636f287b41e54fc14629e2eb6efb8f9532af7e0e0d404e4e234eeba92473b7ae18d97144a953d28523670308e78e4c4fbb7137
2025-10-31 16:35:44 +00:00
rustaceanrob
78d4d36730 test: Format strings in *.rs
`format!` strings may contain variables within the string
representation. This is a lint as of a recent `rustc` nightly version.
2025-10-31 13:39:54 +00:00
Sjors Provoost
6eaa00fe20 test: clarify submitBlock() mutates the template
PR #33374 proposed a new Mining IPC method applySolution() which
could be used by clients to obtain the reconstructed block for
inspection, especially in the case of a rejected block.

However it was pointed out during review that submitBlock() modified
the template CBlock in place, so the client can just call getBlock()
and no new method is needed.

This commit adds a test to document that (now intentional) behavior.
2025-10-31 12:01:04 +01:00
Sjors Provoost
862bd43283 mining: ensure witness commitment check in submitBlock
When an IPC client requests a new block template via the Mining interface,
we hold on to its CBlock. That way when they call submitSolution() we can
modify it in place, rather than having to reconstruct the full block like
the submitblock RPC does.

Before this commit however we forgot to invalidate
m_checked_witness_commitment, which we should since the client brings a
new coinbase.

This would cause us to accept an invalid chaintip.

Fix this and add a test to confirm that we now reject such a block.
As a sanity check, we add a second node to the test and confirm that will
accept our mined block.

Note that the IPC code takes the coinbase as provided, unlike the
submitblock RPC which calls UpdateUncommittedBlockStructures() and adds
witness commitment to the coinbase if it was missing.

Although that could have been an alternative fix, we instead document that
IPC clients are expected to provide the full coinbase including witness
commitment.
2025-10-31 11:53:12 +01:00