Avoid two full iterations over all of a chunks' transactions to
recompute the reachable sets, by inlining them into the
dependency-updating loops.
Note that there is no need to do the same for Activate, because the
reachable sets after merging can be computed directly from the input
chunks' reachable sets. Deactivate needs to recompute them, however.
The two calls to UpdateChunk, in Activate and Deactive each, are subtly
different: the top one needs to update the chunk_idx of iterated
transactions, while the bottom one leaves it unchanged. To exploit this
difference, inline the four function calls, getting rid of UpdateChunks.
This is also a preparation for a future improvement that inlines the
recomputation of reachable sets in the same loop in Deactivate.
It suffices to initially only attempt one direction of merges in
MakeTopological(), and only try both directions on chunks that are the
result of other merges.
This means we can iterate over all active dependencies in a
cluster/chunk in O(ntx) time rather than O(ndeps) (*), as the number of
active dependencies in a set of transactions of size is at most ntx-1.
(*) Asymptotically, this is not actually true, as for large transaction
counts, iterating over a BitSet still scales with ntx. In practice
however, where BitSets are represented by a constant number of integers,
it holds.
After a split, if the top part has a dependency on the bottom part, the
first MergeSequence will always perform this merge and then stop. This
is referred to as a self-merge.
We can special case these by detecting self-merges early, and avoiding
the overhead of a full MergeSequence which involves two
PickMergeCandidate calls (a succesful and an unsuccesful one).
Future changes will rely on knowing the chunk indexes of the two created
chunks after a split. It is natural to return this information from
Deactivate, which also simplifies MergeSequence.
Instead of computing the set of reachable transactions inside
PickMergeCandidate, make the information precomputed, and updated in
Activate (by merging the two chunks' reachable sets) and Deactivate (by
recomputing).
This is a small performance gain on itself, but also a preparation for
future optimizations that rely on quickly testing whether dependencies
between chunks exist.
The current process consists of iterating over the transactions of the
chunk one by one, and then for each figuring out which of its
parents/children are in unprocessed chunks.
Simplify this (and speed it up slightly) by splitting this process into
two phases: first determine the union of all parents/children, and then
find which chunks those belong to.
The combined size of TxData::dep_top_idx can be 16 KiB with 64
transactions and SetIdx = uint32_t. Use a smaller type where possible to
reduce memory footprint and improve cache locality of m_tx_data.
Also switch from an std::vector to an std::array, reducing allocation
overhead and indirections.
With the earlier change to pool SetInfo objects, there is little need
for DepData anymore. Use parent/child TxIdxs to refer to dependencies,
and find their top set by having a child TxIdx-indexed vector in each
TxData, rather than a list of dependencies. This makes code for
iterating over dependencies more natural and simpler.
This significantly changes the data structures used in SFL, based on the
observation that the DepData::top_setinfo fields are quite wasteful:
there is one per dependency (up to n^2/4), but we only ever need one per
active dependency (of which there at most n-1). In total, the number of
chunks plus the number of active dependencies is always exactly equal to
the number of transactions, so it makes sense to have a shared pool of
SetInfos, which are used for both chunks and top sets.
To that effect, introduce a separate m_set_info variable, which stores a
SetInfo per transaction. Some of these are used for chunk sets, and some
for active dependencies' top sets. Every activation transforms the
parent's chunk into the top set for the new dependency. Every
deactivation transforms the top set into the new parent chunk.
With indexes into m_set_data (SetIdx) becoming bounded by the number of
transactions, we can use a SetType to represent sets of SetIdxs.
Specifically, an m_chunk_idxs is added which contains all SetIdx
referring to chunks. This leads to a much more natural way of iterating
over chunks.
Also use this opportunity to normalize many variable names.
This is a preparation for the next commit, where chunks will no longer
be identified using a representative transaction, but using a set index.
Reduce the load of line changes by doing this rename ahead of time.
-BEGIN VERIFY SCRIPT-
sed --in-place 's/_rep/_idx/g' src/cluster_linearize.h
-END VERIFY SCRIPT-
This small optimization avoids the need to loop over the parents of each
transaction when initializing the dependency-counting structures inside
GetLinearization().
This splits the chunk_deps variable in LoadLinearization in two, one for
tracking tx dependencies and one for chunk dependencies. This is a
preparation for a later commit, where chunks won't be identified anymore
by a representative transaction in them, but by a separate index. With
that, it seems weird to keep them both in the same structure if they
will be indexed in an unrelated way.
Note that the changes in src/test/util/cluster_linearize.h to the table
of worst observed iteration counts are due to switching to a different
data set, and are unrelated to the changes in this commit.
Since the deterministic ordering change, SpanningForestState holds a
reference to the DepGraph it is linearizing. So this means we do not
need to pass it to SanityCheck() as an argument anymore.
c1355493e2 refactor: fees: split fee rate format from fee estimate mode (ismaelsadeeq)
922ebf96ed refactor: move-only: move `FeeEstimateMode` enum to `util/fees.h` (ismaelsadeeq)
Pull request description:
### Motivation
Part of #34075
- The `FeeEstimateMode` enum was responsible for both selecting the fee estimation algorithm and specifying the fee rate' format.
#### Changes in this PR:
* The `FeeEstimateMode` enum (`UNSET`, `ECONOMICAL`, `CONSERVATIVE`) is moved to a new util/fees.h header.
* A new `FeeRateFormat `enum (`BTC_KVB`, `SAT_VB`) is introduced in `policy/feerate.h` for feerate formatting.
* The `CFeeRate::ToString()` method is updated to use `FeeRateFormat`.
* All relevant function calls have been updated to use the new `FeeRateFormat` enum for formatting and `FeeEstimateMode` for fee estimation mode.
This refactoring separates these unrelated responsibilities to improve code clarity.
ACKs for top commit:
l0rinc:
ACK c1355493e2
furszy:
utACK c1355493e2
musaHaruna:
ACK [c135549](c1355493e2) — reviewed in the context of PR [34075](https://github.com/bitcoin/bitcoin/pull/34075)
willcl-ark:
ACK c1355493e2
Tree-SHA512: 7cbe36350744313d3d688d3fd282a58c441af1818b1e8ad9cddbc911c499a5205f8d4a39c36b21fed60542db1ef763eb69752d141bcef3393bf33c0922018645
CMAKE_COMPILE_WARNING_AS_ERROR is not a cache variable by default in
CMake, so it has no value in the configure summary when not set, and
even when set cannot be toggled in ccmake. Define it as an option() to
make it a cache BOOL with a default of OFF.
eb510f8678 ci: fail fast in test-each-commit script (Lőrinc)
04c4d71008 ci: remove commit count limit from `test-each-commit` (Lőrinc)
Pull request description:
### Problem
`test-each-commit` currently tests only a limited number of ancestor commits in a PR, so failures introduced deeper in the commit stack might be missed.
### Fix
Remove the max-count limit so `test-each-commit` runs the full build + unit + functional test flow on every non-head PR commit, while keeping the PR tip excluded because it is already covered by the normal CI jobs.
It will also stop after the first failure to surface the root cause sooner and keep logs readable when testing ancestor commits.
### Examples
* Example failure 10 commits deep: https://github.com/l0rinc/bitcoin/actions/runs/21390976651/job/61577575033?pr=105 in https://github.com/l0rinc/bitcoin/pull/105
* Example pass with >7 dummy commits: https://github.com/l0rinc/bitcoin/actions/runs/21392557521/job/61595159841?pr=106 in https://github.com/l0rinc/bitcoin/pull/106
---------
Note: this PR has gone through a few iterations, the latest one just extends the existing job.
ACKs for top commit:
maflcko:
lgtm ACK eb510f8678🕓
hebasto:
re-ACK eb510f8678.
willcl-ark:
ACK eb510f8678
Tree-SHA512: 5aadafd32daad610ce882277802c390642dc34f7d5bfa71d4b503ee007942d1ebafce2a3430ea5fd2af6673c83f9aee42450043be4722d7c02407d90920f8bce
211111b804 test: Avoid empty errmsg in JSONRPCException (MarcoFalke)
Pull request description:
It is unclear why the fallback should be an empty message, when it is better to include all rpc_error details that are available.
Also, include the http status.
This allows to revert commit 6354b4fd7f, because it is no longer needed.
Can be tested by running this diff:
```diff
diff --git a/test/functional/wallet_disable.py b/test/functional/wallet_disable.py
index dbcccd4778..9717a2d248 100755
--- a/test/functional/wallet_disable.py
+++ b/test/functional/wallet_disable.py
@@ -18,9 +18,8 @@ class DisableWalletTest (BitcoinTestFramework):
self.extra_args = [["-disablewallet"]]
self.wallet_names = []
- def run_test (self):
- # Make sure wallet is really disabled
- assert_raises_rpc_error(-32601, 'Method not found', self.nodes[0].getwalletinfo)
+ def run_test(self):
+ self.nodes[0].getwalletinfo()
x = self.nodes[0].validateaddress('3J98t1WpEZ73CNmQviecrnyiWrnqRhWNLy')
assert x['isvalid'] == False
x = self.nodes[0].validateaddress('mneYUmWYsuk7kySiURxCi3AGxrAqZxLgPZ')
ACKs for top commit:
ismaelsadeeq:
utACK 211111b804
furszy:
utACK 211111b804
hodlinator:
ACK 211111b804
rkrux:
tACK 211111b804
Tree-SHA512: 92067aaaa61a887398e38f587004ba31070acc6a9dbd003e9e35393e8c049e786177afa43da05d2d653af340b6c69803c4323061d73d417219bcdf37729e4b66
d159b10398 doc: update Windows MSVC build guide to utilize WinGet to install apps (janb84)
Pull request description:
This PR updates the Windows MSVC build guide to bring it more in line with the macOS and Unix build guides.
Currently the guide listed requirements but left users to manually do the download/installation. The macOS and the Unix guide utilize package managers to provide concrete installation commands that users can follow. By introducing `winget`, the Windows MSVC build guide now also provides concrete installation commands.
The changes/commands can be tested on any windows machine, provided it run a Windows 10 (version 1809 (build 17763)) or later (e.g. Windows 11 / Server 2025).
ACKs for top commit:
hodlinator:
re-ACK d159b10398
hebasto:
re-ACK d159b10398.
Tree-SHA512: 9eeee60ba3e50e42362f1a6d8003120868c1f49cf146cc6fe16a6deb53ce29c67e841b8ec80d516fe2e6d6ea4c48d34fb12691151e0fea5568c14377bd3da6fb
Users running on home networks with routers that don't support PCP (Port
Control Protocol) or NAT-PMP port mapping receive frequent warning-level
log messages every few minutes:
"pcp: Mapping failed with result NOT_AUTHORIZED (code 2)"
This is expected behavior for many consumer routers that have PCP
disabled by default, not an actionable error.
Add explicit constants for the NOT_AUTHORIZED result code (value 2)
for both NAT-PMP and PCP protocols. Log the first NOT_AUTHORIZED
failure at warning level for visibility, then downgrade subsequent
occurrences to LogDebug to avoid log noise. Other failure types
continue to warn unconditionally.
Fixes#34114
Co-authored-by: willcl-ark <will@256k1.dev>
It is unclear why the fallback should be an empty message, when it is
better to include all rpc_error details that are available.
Also, include the http status.
This allows to revert commit 6354b4fd7f,
because it is no longer needed.
fa13b13239 ci: [refactor] Use pathlib over os.path (MarcoFalke)
fa2719ab1b ci: [refactor] Move run_unit_tests to ci-windows-cross.py (MarcoFalke)
fa99ba5f14 ci: Set PREVIOUS_RELEASES_DIR env var in ci-windows-cross.py (MarcoFalke)
fa4a1cab6c ci: Move run_functional_tests into ci-windows-cross.py (MarcoFalke)
1111108685 ci: [refactor] Move pyzmq install and get_previous_releases into ci-windows-cross.py (MarcoFalke)
fac9c7bd66 ci: [refactor] Move config.ini rewrite to ci-windows-cross.py (MarcoFalke)
faf7389466 ci: Move check_manifests step to ci-windows-cross.py (MarcoFalke)
fa674d55df ci: [refactor] Move print_version step into ci-windows-cross.py helper (MarcoFalke)
Pull request description:
Currently the ci yaml has a mix of Bash and Pwsh snippets, which is problematic:
* The `shellcheck` tool does not review the Bash
* The ci yaml is not merged with master on re-runs, but the code is, leading to possibly confusing CI errors on re-runs
* The Pwsh isn't reviewed at all by any tool
* It is tedious to run the CI commands locally on Windows
Fix all issues by extracting them into a step-based Python script.
ACKs for top commit:
janb84:
re ACK fa13b13239
hebasto:
ACK fa13b13239, I have reviewed the code and it looks OK.
Tree-SHA512: 23d21d3bfb07e102fe1cc15ba5749d553d9766ae6c4a7648bd77df0705469bd138c76a9a2fdeb4d91d3f889a425b7caf25878ecb2e68b604faf9665f8df4eb6d
c413cf12c5 ci: Split vcpkg tools cache into restore/save (willcl-ark)
Pull request description:
The vcpkg tools cache was using the combined actions/cache action, which by default saves on every run regardless of branch. Split it into the restore/save pattern used by the other caches, so that saves only happen on default branch pushes.
This will have little impact in bitcoin/bitcoin (which uses few branches), but on forks, if you don't update master branch frequently (which saves all caches), then all cache space will eventually be taken up by multiple vckpg tools caches, resulting in bad cache hit rates in all other jobs.
ACKs for top commit:
maflcko:
lgtm ACK c413cf12c5
Tree-SHA512: e28a43b1aa17ce7f0a19d16b98efed0372004d83e4d7e92a126f642599d7e1a94684032a48a3b380b3a7c970c313c92bfe30146b977e33f81b45fe70b49755e3
2cb7e99dee test: also reset CConnman::m_private_broadcast in tests (Vasil Dimov)
91b7c874e2 test: add ConnmanTestMsg convenience method Reset() (Vasil Dimov)
Pull request description:
Member variables of `CConnman::m_private_broadcast` (introduced in
https://github.com/bitcoin/bitcoin/pull/29415) could influence the tests
which creates non-determinism if the same instance of `CConnman` is used
for repeated test iterations.
So, reset the state of `CConnman::m_private_broadcast` from
`ConnmanTestMsg::Reset()`. Currently this affects the fuzz tests
`process_message` and `process_messages`.
Reported in https://github.com/bitcoin/bitcoin/issues/34476#issuecomment-3849088794
ACKs for top commit:
maflcko:
review ACK 2cb7e99dee🚙
Crypt-iQ:
tACK 2cb7e99dee
frankomosh:
Code Review ACK 2cb7e99dee
brunoerg:
code review ACK 2cb7e99dee
Tree-SHA512: 0f4b114542da8dc611689457ce67034c15cbfe409b006b2db72bc74078ee9513f5ce3d0e6e67d37c127cfa0a5170fe72fe3ea45ce2a61d45a358dd11bd1881f8
eafd530d20 kernel: avoid potential duplicate object in shared library/binary (Cory Fields)
24c3b47010 build: add kernel-specific warnings (Cory Fields)
Pull request description:
This is a revival of https://github.com/bitcoin/bitcoin/pull/31807
Introduces the [-Wunique-object-duplication](https://clang.llvm.org/docs/DiagnosticsReference.html#wunique-object-duplication) warning flag available in clang-21 for usage when building the kernel library. It warns of potential duplicate objects in shared libraries. REDUCE_EXPORTS needs to be ON to trigger it.
Though we have a C API now that manages exporting symbols, I think it is prudent to also avoid any duplicate symbols on the internal c++ side in case we ever to decide to expose some of its headers. It also not clear that all linkers would handle these cases correctly even in the current internal usage.
ACKs for top commit:
fanquake:
ACK eafd530d20
hebasto:
ACK eafd530d20.
Tree-SHA512: 81961b50f0268dbe076497e130857f5b4b9151c748d107ec15158d1511dd25bce745e0beeb127b9cea51cb2edd78032735600606a75f7ff8a3fd572acced42e0
Fixes warning and potential bug whereby init_flag may exist in both
libbitcoinkernel as well as a downstream user, as opposed to being shared as
intended.
src/support/lockedpool.h:224:31:
warning: 'init_flag' may be duplicated when built into a shared library: it is mutable, has hidden visibility, and external linkage [-Wunique-object-duplication]
In some cases, we'll want to be more aggressive or care about different things
when building the kernel. In this case, a warning is added for symbols which
may be duplicated between the kernel and downstream users.
This warning was introduced in clang 21, which is not yet the minimum
supported compiler version. REDUCE_EXPORTS needs to be ON to trigger it.
