fa70e23de7 ci: Drop libFuzzer from msan fuzz task (MarcoFalke)
Pull request description:
libFuzzer is mostly unmaintained (https://llvm.org/docs/LibFuzzer.html#status), and it isn't really needed by the CI tasks. While it provides some additional stats like rss or the max input byte size, they are not essential. Dropping libFuzzer here would also drop the "60 seconds sanity check" for empty folders, but I think this is an acceptable price to pay to silence false-positives that were hit for years.
Also, there seems to be a history of intermittent false-positive msan warnings (https://github.com/bitcoin/bitcoin/pull/33600#issuecomment-3391921802).
It is unclear what exactly is causing the false-positives, so just disable libFuzzer in this task for now, to work around them.
ACKs for top commit:
kevkevinpal:
ACK [fa70e23](fa70e23de7)
dergoegge:
ACK fa70e23de7
Tree-SHA512: c3e5958b8378ba30f51d923f97a84dec2ee60af8b9c2a4f13bc8de486a490031468371120e421384aa198ffec591db554e636935ab3c6d4de5e870238f5079f2
fa37153288 util: Abort on failing CHECK_NONFATAL in debug builds (MarcoFalke)
fa0dc4bdff test: Allow testing of check failures (MarcoFalke)
faeb58fe66 refactor: Set G_ABORT_ON_FAILED_ASSUME when G_FUZZING_BUILD (MarcoFalke)
Pull request description:
A failing `CHECK_NONFATAL` will throw an exception. This is fine and even desired in production builds, because the program may catch the exception and give the user a way to easily report the bug upstream.
However, in debug development builds, exceptions for internal bugs are problematic:
* The exception could accidentally be caught and silently ignored
* The exception does not include a full stacktrace, possibly making debugging harder
Fix all issues by turning the exception into an abort in debug builds.
This can be tested by reverting the hunks to `src/rpc/node.cpp` and `test/functional/rpc_misc.py` and then running the functional or fuzz tests.
ACKs for top commit:
achow101:
ACK fa37153288
ryanofsky:
Code review ACK fa37153288, just catching subprocess.CalledProcessError in test fixing up a comment since last review
stickies-v:
ACK fa37153288
Tree-SHA512: 2d892b838ccef6f9b25a066e7c2f6cd6f5acc94aad1d91fce62308983bd3f5c5d724897a76de4e3cc5c3678ddadc87e2ee8c87362965373526038e598dfb0101
cc5dda1de3 headerssync: Make HeadersSyncState more flexible and move constants (Hodlinator)
8fd1c2893e test(headerssync): Test returning of pow_validated_headers behavior (Hodlinator)
7b00643ef5 test(headerssync): headers_sync_chainwork test improvements (Hodlinator)
04eeb9578c doc(test): Improve comments (Hodlinator)
fe896f8faa refactor(test): Store HeadersSyncState on the stack (Hodlinator)
f03686892a refactor(test): Break up headers_sync_state (Hodlinator)
e984618d0b refactor(headerssync): Process spans of headers (Hodlinator)
a4ac9915a9 refactor(headerssync): Extract test constants ahead of breakup into functions (Hodlinator)
Pull request description:
### Background
As part of the release process we often run *contrib/devtools/headerssync-params.py* and increase the values of the constants `HEADER_COMMITMENT_PERIOD` and `REDOWNLOAD_BUFFER_SIZE` in *src/headerssync.cpp* as per *doc/release-process.md* (example: 11a2d3a63e). This helps fine tune the memory consumption per `HeadersSyncState`-instance in the face of malicious peers.
(The `REDOWNLOAD_BUFFER_SIZE`/`HEADER_COMMITMENT_PERIOD` ratio determines how many Headers Sync commitment bits must match between PRESYNC & REDOWNLOAD phases before we start permanently storing headers from a peer. For more details see comments in *src/headerssync.h* and *contrib/devtools/headerssync-params.py*).
### Problem: Not feeding back headers until completing sync
During v30 release process #33274 made `REDOWNLOAD_BUFFER_SIZE` exceed the `target_blocks` constant used to control the length of chains generated for testing Headers Sync (`15000`, *headers_sync_chainwork_tests.cpp*).
The `HeadersSyncState::m_redownloaded_headers`-buffer now does not reach the `REDOWNLOAD_BUFFER_SIZE`-threshold during those unit tests. As a consequence `HeadersSyncState::PopHeadersReadyForAcceptance()` will not start feeding back headers until the PoW threshold has been met. While this will not cause the unit test to start failing on master, it means we have gone from testing behavior that resembles mainnet (way more than `REDOWNLOAD_BUFFER_SIZE` headers to reach the PoW limit), to behavior that is not possible/expected there.
### Solution
Avoid testing this unrealistic condition of completing Headers Sync before reaching `REDOWNLOAD_BUFFER_SIZE` by making tests able to define their own values through the new `HeadersSyncParams` instead of having them hard-coded for all chains & tests.
### Commits
* First 6 commits refactor and improve the unit tests in order to clarify latter changes.
* We then add checks for the behavior around the `REDOWNLOAD_BUFFER_SIZE` threshold.
* The main change: we extract the section from *headerssync.cpp* containing the constants to *kernel/chainparams.cpp*, making `HeadersSyncState` no longer hard-coded to mainnet.
### Notes
This PR used to be called "headerssync: Preempt unrealistic unit test behavior".
ACKs for top commit:
l0rinc:
reACK cc5dda1de3
marcofleon:
code review ACK cc5dda1de3
danielabrozzoni:
reACK cc5dda1de3
Tree-SHA512: ccc824dcbbb8ad5ae98c3bf5808b38467aac0230739898a758c9b939eecd74f982df088fa0ba81cc1c1732f19a607b135a6e9577bb9fcf7f8570567ce92f66e6
d0e1bbad01 test: repeat block malleability test with relayable block over P2P (Musa Haruna)
Pull request description:
This PR adds a functional test to repeat the existing malleability check for oversized coinbase witness nonce size using a block that is small enough to be relayed over the P2P network.
This addresses the TODO in test_block_malleability by ensuring behavior is consistent between submitblock RPC and P2P relay.
ACKs for top commit:
maflcko:
lgtm ACK d0e1bbad01
janb84:
re ACK d0e1bbad01
glozow:
utACK d0e1bbad01
Tree-SHA512: 05aec4fade5af8043f40274a8d2f3cf3f540acd038138975bdefbbbc81e105792d6d2588256a2ee5ddb1e05b37fe2d0b3d287160d2dbe86e1aac7cfa9cc02116
faa9d10c84 refactor: Construct g_verify_flag_names on first use (MarcoFalke)
Pull request description:
The current usage of the `g_verify_flag_names` map seems fine and I can not see a static initialization order fiasco here.
However, it seems brittle to hope this remains the case in the future. Also, it triggers a msan false-positive in the fuzz CI task. (C.f https://github.com/bitcoin-core/qa-assets/actions/runs/18352815555/job/52413137315?pr=241#step:7:5245)
So just apply the "Construct on first use" idiom.
ACKs for top commit:
kevkevinpal:
ACK [faa9d10](faa9d10c84)
ajtowns:
ACK faa9d10c84
janb84:
lgtm ACK faa9d10c84
stickies-v:
ACK faa9d10c84
Tree-SHA512: 6685dfc91c99a8245722e07fac99a7a6d58586c30964be7ccd74a176dfbf00c6255c8594621e2909640763924f51d3efd4ce65ed65eaeeb1d05c2fd01fe63604
8f7673257a miner: fix empty mempool case for waitNext() (Sjors Provoost)
Pull request description:
Block template fees are calculated by looping over `new_tmpl->vTxFees` and return (early) once the `fee_threshold` is exceeded.
This left an edge case when the mempool is empty, which this commit fixes and adds a test for.
Also update `test/functional/interface_ipc.py` to reflect the new behavior,
Fixes https://github.com/Sjors/sv2-tp/issues/9
ACKs for top commit:
optout21:
ACK 8f7673257a
cedwies:
tACK 8f76732
sipa:
utACK 8f7673257a
zaidmstrr:
Concept ACK [8f76732](8f7673257a)
Tree-SHA512: ef200fe95e96f810e425283bc37f945c4bf5efa16f4b74820b8a07968f30c5146bca213a372124be84b48beead5dfd35f2b5d10d188d0a465f847ebab61de10a
e9cd45e3d3 test: set number of RPC server threads to 2 (furszy)
Pull request description:
The default `-rpcthreads` value spawns 16 HTTP server threads for each node.
Running the functional test suite with default `rpcthreads` can exhaust file
descriptors or hit other resource limits very easily (more when tests are run
in parallel).
Furthermore, having 16 threads is unnecessary since they are mostly idle. We
run RPC calls on a single RPC connection and wait for it result synchronously.
There is (almost) never two RPC calls occurring concurrently.
Because of this, the threads are mostly idle, so we can safely limit the number
of them to two.
Note for reviewers:
I checked this does not introduce any timing regression but would be good
to double-check it on your end too. We could add another thread if needed.
Just the 16 threads default value is too high and unnecessary.
ACKs for top commit:
maflcko:
lgtm ACK e9cd45e3d3
l0rinc:
ACK e9cd45e3d3
kevkevinpal:
ACK [e9cd45e](e9cd45e3d3)
andrewtoth:
ACK e9cd45e3d3
Tree-SHA512: a777286f4a890fb87f5df72cd2ccfdc628657206a4b3e995044e5a0d12987b8c78a7cf7d684cc4e92605aa782aaeebc44e9f754752c3a524152fac94fa30f4b5
The default `-rpcthreads` value spawns 16 HTTP server threads for each node.
Running the functional test suite with default `rpcthreads` can exhaust file
descriptors or hit other resource limits very easily.
Moreover, having 16 threads is unnecessary since they are mostly idle. We
run RPC calls on a single RPC connection and wait for it result synchronously.
There is (almost) never two RPC calls occurring concurrently.
Because of this, the threads are mostly idle, so we can safely limit the number
of them to two.
fabe0e07de ci: Only write docker build images to Cirrus cache (MarcoFalke)
fab64a5d6f ci: Move buildx command to python script (MarcoFalke)
fa72a2bd5c ci: Remove unused MAYBE_CPUSET (MarcoFalke)
Pull request description:
The `DOCKER_BUILD_CACHE_ARG` env var holds the options on how to use cache providers. Storing the image layers is useful for the Cirrus cache provider, because it offers 10GB per runner (https://cirrus-runners.app/setup/#speeding-up-the-cache). The cached image layers can help to avoid issues when the upstream package manager infra (apt native, apt llvm, pip, apk, git clone, ...) has outages or network issues.
However, on the GitHub Actions cache provider, a *total* cache of 10GB is offered for the whole repo. This cache must be shared with the depends cache, and the ccache, as well as the previous releases cache. So it is already full and trying to put the docker build layers into it will lead to an overflow.
Fix it by only writing to the docker cache on Cirrus.
Also, `DOCKER_BUILD_CACHE_ARG` requires a `shellcheck disable=SC2086` on the full build command. Fix that as well by using `shlex.split` from Python on just this variable.
ACKs for top commit:
m3dwards:
ACK fabe0e07de
cedwies:
reACK fabe0e0
l0rinc:
Code review ACK fabe0e07de
willcl-ark:
ACK fabe0e07de
Tree-SHA512: 4f471f080007fdd0c3bc97b0cfe0e9c0457e5029a7ccde1d784d30eb4752e5eb309cd4b122b182bce31f1b986c8a9f3e9a49da1768bedbb2b1f64f70183680ba
9610b0d1e2 randomenv: Fix MinGW dllimport warning for `environ` (Lőrinc)
Pull request description:
Related to https://github.com/bitcoin/bitcoin/pull/33550#issuecomment-3378978210
Extends 7703884 to guard environ declaration on all Windows builds, not just MSVC.
In the `mingw-w64` headers (used by `llvm-mingw`), `environ` is defined as a macro which expands through [`_environ`](cdb052f1d4/mingw-w64-headers/crt/stdlib.h (L262-L264)) to `(* __p__environ())`, a call to a `dllimport` function, causing the same inconsistent linkage warning as MSVC.
Use `WIN32` instead of `_MSC_VER` to match the platform-specific guards already used throughout the file.
The warning occurs with `llvm-mingw` (both `UCRT` and `MSVCRT` variants as tested by Hebasto), but not with the `mingw-w64` toolchain currently used in CI (as mentioned by fanquake).
----
The error was reproduced by adding a temporary [nightly build](https://github.com/l0rinc/bitcoin-core-nightly/pull/4) pointing to https://github.com/l0rinc/bitcoin/pull/45. On `master` the failure can be seen in https://github.com/l0rinc/bitcoin-core-nightly/pull/2
before:
https://github.com/l0rinc/bitcoin-core-nightly/actions/runs/18327936488/job/52196728885?pr=2
<details>
<summary>Details</summary>
```
/home/runner/work/bitcoin-core-nightly/bitcoin-core-nightly/src/randomenv.cpp:61:15: warning: '__p__environ' redeclared without 'dllimport' attribute: previous 'dllimport' ignored [-Winconsistent-dllimport]
61 | extern char** environ; // NOLINT(readability-redundant-declaration): Necessary on some platforms
| ^
/home/runner/work/bitcoin-core-nightly/bitcoin-core-nightly/llvm_mingw_toolchain/aarch64-w64-mingw32/include/stdlib.h:656:17: note: expanded from macro 'environ'
656 | #define environ _environ
| ^
/home/runner/work/bitcoin-core-nightly/bitcoin-core-nightly/llvm_mingw_toolchain/aarch64-w64-mingw32/include/stdlib.h:225:21: note: expanded from macro '_environ'
225 | #define _environ (* __p__environ())
| ^
/home/runner/work/bitcoin-core-nightly/bitcoin-core-nightly/llvm_mingw_toolchain/aarch64-w64-mingw32/include/stdlib.h:221:27: note: previous declaration is here
221 | _CRTIMP char ***__cdecl __p__environ(void);
| ^
/home/runner/work/bitcoin-core-nightly/bitcoin-core-nightly/llvm_mingw_toolchain/aarch64-w64-mingw32/include/stdlib.h:221:3: note: previous attribute is here
221 | _CRTIMP char ***__cdecl __p__environ(void);
| ^
/home/runner/work/bitcoin-core-nightly/bitcoin-core-nightly/llvm_mingw_toolchain/aarch64-w64-mingw32/include/_mingw.h:52:40: note: expanded from macro '_CRTIMP'
52 | # define _CRTIMP __attribute__ ((__dllimport__))
| ^
1 warning generated.
```
</details>
after:
https://github.com/l0rinc/bitcoin-core-nightly/actions/runs/18329616268/job/52201940831?pr=4
<details>
<summary>Details</summary>
```
[ 28%] Building CXX object src/util/CMakeFiles/bitcoin_util.dir/__/randomenv.cpp.obj
```
</details>
Note that there are some other remaining warnings in the logs that will be fixed in separate PRs
ACKs for top commit:
sipa:
utACK 9610b0d1e2 if this makes the compilers happy
laanwj:
Code review ACK 9610b0d1e2
hebasto:
re-ACK 9610b0d1e2.
Tree-SHA512: a9e39d288b663ed24cbbbae228850e6f02d417d8781a3ac3d0b3db0b7ff734bbd62fddb9f57b8f77daab4e9c016ff66906ebc5fb2de7635ef539ef7f4dc2eaba
fa20275db3 test: Use unassigned p2p_port instead of hardcoded 60000 in p2p_i2p_ports.py (MarcoFalke)
Pull request description:
The goal is to fix https://github.com/bitcoin/bitcoin/issues/30030.
The root cause it unclear. However, hard-coding the port to 60000 does not seem ideal anyway. This could break in an unlikely setting where so many functional tests are run, such that the port is occupied. Also, it could fail when `TEST_RUNNER_PORT_MIN` is set sufficiently high. (This is purely theoretical, as I don't think anyone would run a command like this, but on current master it fails, and on this pull it passes: `TEST_RUNNER_PORT_MIN=60000 ./bld-cmake/test/functional/p2p_i2p_ports.py --portseed=0`)
So fix those issues (and hopefully also 30030) by using an unoccupied p2p_port.
The logic is similar to the `extra_port()` logic in the `feature_bind_extra.py` test.
ACKs for top commit:
laanwj:
Code review ACK fa20275db3
mzumsande:
ACK fa20275db3
Tree-SHA512: ac5487ca195db9ca746b78b8add91d0b9ef59cc3be0cdb7fbd9f76d42549eea68a61c32b4f5a162e01f3777959110f9f8d56ff05af6a13a9f61ea5be5b7d8631
c864a4c194 Simplify fs::path by dropping filename() and make_preferred() overloads (Ryan Ofsky)
b0113afd44 Fix windows libc++ fs::path fstream compile errors (Ryan Ofsky)
Pull request description:
Drop support for passing `fs::path` directly to `std::ifstream` and `std::ofstream` constructors and `open()` functions, because as reported by hebasto in https://github.com/bitcoin/bitcoin/issues/33545, after https://wg21.link/lwg3430 there is no way this can continue to work in windows builds, and there are already compile errors compiling for windows with newer versions of libc++.
Instead, add an `fs::path::std_path()` method that returns `std::filesystem::path` references and use it where needed.
ACKs for top commit:
hebasto:
ACK c864a4c194.
l0rinc:
Code review ACK c864a4c194
maflcko:
re-ACK c864a4c194 🌥
Tree-SHA512: d22372692ab86244e2b2caf4c5e9c9acbd9ba38df5411606b75e428474eabead152fc7ca1afe0bb0df6b818351211a70487e94b40a17b68db5aa757604a0ddf6
This has a few benefits:
* The shellcheck SC2086 warning is disabled for the whole command, but
is only needed for the DOCKER_BUILD_CACHE_ARG env var. So in Python,
only pass this one env var to shlex.split() for proper word splitting.
* Future logic improvements can be implemented in Python.
The comments are moved, which can be checked via the git options:
--color-moved=dimmed-zebra --color-moved-ws=ignore-all-space
The option is currently unused. If it is used again in the future, it
could trivially be added back.
Also, the logic is just a single undocumented python command one-liner.
So remove it for now.
4b41f99d57 build: Move CMAKE_SKIP_INSTALL_RPATH from CMake to Guix script (Henry Romp)
Pull request description:
Remove `CMAKE_SKIP_INSTALL_RPATH` from CMakeLists.txt and add `CMAKE_SKIP_RPATH` to the Guix build script. This keeps build-environment-specific settings in the build scripts rather than hardcoded in the CMake configuration.
ACKs for top commit:
purpleKarrot:
ACK 4b41f99d57
janb84:
re ACK 4b41f99d57
Tree-SHA512: 74d6af382476d731f10f9833978d670e9981c160ba306d0e9d4b1ad1e9b9960b8d03a3b9b608e234edb1c0c2c7a2b4f9f606a2a7887b7a153792159e71ae9b21
fa75ef4328 test: Move export_env_build_path to util.py (MarcoFalke)
fa9f495308 test: Move get_binary_paths and Binaries to util.py (MarcoFalke)
Pull request description:
Having the binary related utils sit in the test_framework.py is fine. However, they are mostly stand-alone utils, which may be used externally.
So move them to utils.py, to allow easier external use. The diff is trivial and can be reviewed via the git options `--color-moved=dimmed-zebra --color-moved-ws=ignore-all-space`.
ACKs for top commit:
kevkevinpal:
ACK [fa75ef4](fa75ef4328)
Sjors:
lgtm ACK fa75ef4328
yuvicc:
Code review ACK fa75ef4328
janb84:
ACK fa75ef4328
musaHaruna:
Code Review ACK [fa75ef4](fa75ef4328)
enirox001:
ACK [fa75ef4](fa75ef4328)
Tree-SHA512: f382118484cb5495e8888214437e72c81727d54f97b3c09dfd996faab6cb6643c4c2d816b89ab82de73fc091c36ed7b8744c7d34a443b6adc415eb06697ef6ea
3cbf7cb3e6 Squashed 'src/secp256k1/' changes from b9313c6e1a..d543c0d917 (fanquake)
Pull request description:
Updates the subtree to d543c0d917
Related to #33284.
ACKs for top commit:
hebasto:
ACK 879c21045e.
janb84:
ACK 879c21045e
Tree-SHA512: 1802cd84959b5c935170792f458651f30431fe8340ead7966ff381c1c0c3a9f6c21bbb8dd96a07482ffed49642ded49e80b61802e688b8351956b111dffd5a78
Remove CMAKE_SKIP_INSTALL_RPATH from CMakeLists.txt and add CMAKE_SKIP_RPATH to the Guix build script. This keeps build-environment-specific settings in the build scripts rather than hardcoded in the CMake configuration.
3d22282564 [doc] correct topology requirements in submitpackage helptext (glozow)
Pull request description:
This doc is outdated since #31385. Also made it explicit that a singleton is ok.
Can be backported to 30.x, but doesn't need to be backported earlier ("if any" covers #31096).
ACKs for top commit:
janb84:
ACK 3d22282564
instagibbs:
ACK 3d22282564
Tree-SHA512: 95e40630a5b2a571029c0657c20a5e2a1cf1789913b868cee314c1a9fcb9a09fccdd3c87f3f15a8eb95c5ff9b83f8adee0661f86619bf21965866b6f6a76dfd0
f21162d819 Squashed 'src/leveldb/' changes from aba469ad6a..cad64b151d (fanquake)
Pull request description:
Rather than continue to close PRs/"Send these upstream" i.e: #33638, #33148, #22664, #13781; just fix the typos.
Includes https://github.com/bitcoin-core/leveldb-subtree/pull/57.
ACKs for top commit:
l0rinc:
ACK 54ffe3de5b
cedwies:
ACK 54ffe3d
stickies-v:
ACK 54ffe3de5b
Tree-SHA512: cc4d758ee95a1943f14e800472dfef24d5598a1dfafede32300821bc27e02a80ae97ea12ee87643b395b204262c7bc28e64d421a3d375d46bef7782381fd4362
9b43428c96 TxGraph: change m_excluded_clusters (Greg Sanders)
Pull request description:
Change BlockBuilderImpl's m_excluded_clusters to unordered set since ordering is not used.
Change the set to a set of sequence numbers for a modest stability increase under fuzz testing.
ACKs for top commit:
sipa:
ACK 9b43428c96
marcofleon:
tACK 9b43428c96
glozow:
ACK 9b43428c96
Tree-SHA512: 140a492af93f3eff756847a8168aab2624bb7df407f177dde6f3b07e9db2d0ced6b125e2b126f4957ccd054272056bedf74f9f0e64a80d90c16fd94e0fa86a44
24d861da78 coins: only adjust `cachedCoinsUsage` on `EmplaceCoinInternalDANGER` insert (Lőrinc)
d7c9d6c291 coins: fix `cachedCoinsUsage` accounting to prevent underflow (Lőrinc)
39cf8bb3d0 refactor: remove redundant usage tracking from `CoinsViewCacheCursor` (Lőrinc)
67cff8bec9 refactor: assert newly-created parent cache entry has zero memory usage (Lőrinc)
Pull request description:
### Summary
This PR fixes `cachedCoinsUsage` accounting bugs in `CCoinsViewCache` that caused UBSan `unsigned-integer-overflow` violations during testing. The issues stemmed from incorrect decrement timing in `AddCoin()`, unconditional reset in `Flush()` on failure, and incorrect increment in `EmplaceCoinInternalDANGER()` when insertion fails.
### Problems Fixed
**1. `AddCoin()` underflow on exception**
- Previously decremented `cachedCoinsUsage` *before* the `possible_overwrite` validation
- If validation threw, the map entry remained unchanged but counter was decremented
- This corrupted accounting and later caused underflow
- **Impact**: Test-only in current codebase, but unsound accounting that could affect future changes
**2. `Flush()` accounting drift on failure**
- Unconditionally reset `cachedCoinsUsage` to 0, even when `BatchWrite()` failed
- Left the map populated while the counter read zero
- **Impact**: Test-only (production `BatchWrite()` returns `true`), but broke accounting consistency
**3. Cursor redundant usage tracking**
- `CoinsViewCacheCursor::NextAndMaybeErase()` subtracted usage when erasing spent entries
- However, `SpendCoin()` already decremented and cleared the `scriptPubKey`, leaving `DynamicMemoryUsage()` at 0
- **Impact**: Redundant code that obscured actual accounting behavior
**4. `EmplaceCoinInternalDANGER()` double-counting**
- Incremented `cachedCoinsUsage` even when `try_emplace` did not insert (duplicate key)
- Inflated the counter on duplicate attempts
- **Impact**: Mostly test-reachable (AssumeUTXO doesn't overwrite in production), but incorrect accounting
### Testing
To reproduce the historical UBSan failures on the referenced baseline and to verify the fix, run:
```
MAKEJOBS="-j$(nproc)" FILE_ENV="./ci/test/00_setup_env_native_fuzz.sh" ./ci/test_run_all.sh
```
The change was tested with the related unit and fuzz test, and asserted before/after each `cachedCoinsUsage` change (in production code and fuzz) that the calculations are still correct by recalculating them from scratch.
<details>
<summary>Details</summary>
```C++
bool CCoinsViewCache::CacheUsageValid() const
{
size_t actual{0};
for (auto& entry : cacheCoins | std::views::values) actual += entry.coin.DynamicMemoryUsage();
return actual == cachedCoinsUsage;
}
```
or
```patch
diff --git a/src/coins.cpp b/src/coins.cpp
--- a/src/coins.cpp(revision fd3b1a7f4bb2ac527f23d4eb4cfa40a3215906e5)
+++ b/src/coins.cpp(revision 872a05633bfdbd06ad82190d7fe34b42d13ebfe9)
@@ -96,6 +96,7 @@
fresh = !it->second.IsDirty();
}
if (!inserted) {
+ Assert(cachedCoinsUsage >= it->second.coin.DynamicMemoryUsage());
cachedCoinsUsage -= it->second.coin.DynamicMemoryUsage();
}
it->second.coin = std::move(coin);
@@ -133,6 +134,7 @@
bool CCoinsViewCache::SpendCoin(const COutPoint &outpoint, Coin* moveout) {
CCoinsMap::iterator it = FetchCoin(outpoint);
if (it == cacheCoins.end()) return false;
+ Assert(cachedCoinsUsage >= it->second.coin.DynamicMemoryUsage());
cachedCoinsUsage -= it->second.coin.DynamicMemoryUsage();
TRACEPOINT(utxocache, spent,
outpoint.hash.data(),
@@ -226,10 +228,12 @@
if (itUs->second.IsFresh() && it->second.coin.IsSpent()) {
// The grandparent cache does not have an entry, and the coin
// has been spent. We can just delete it from the parent cache.
+ Assert(cachedCoinsUsage >= itUs->second.coin.DynamicMemoryUsage());
cachedCoinsUsage -= itUs->second.coin.DynamicMemoryUsage();
cacheCoins.erase(itUs);
} else {
// A normal modification.
+ Assert(cachedCoinsUsage >= itUs->second.coin.DynamicMemoryUsage());
cachedCoinsUsage -= itUs->second.coin.DynamicMemoryUsage();
if (cursor.WillErase(*it)) {
// Since this entry will be erased,
@@ -279,6 +283,7 @@
{
CCoinsMap::iterator it = cacheCoins.find(hash);
if (it != cacheCoins.end() && !it->second.IsDirty() && !it->second.IsFresh()) {
+ Assert(cachedCoinsUsage >= it->second.coin.DynamicMemoryUsage());
cachedCoinsUsage -= it->second.coin.DynamicMemoryUsage();
TRACEPOINT(utxocache, uncache,
hash.hash.data(),
```
</details>
ACKs for top commit:
optout21:
reACK 24d861da78
andrewtoth:
ACK 24d861da78
sipa:
ACK 24d861da78
w0xlt:
ACK 24d861da78
Tree-SHA512: ff1b756b46220f278ab6c850626a0f376bed64389ef7f66a95c994e1c7cceec1d1843d2b24e8deabe10e2bdade2a274d9654ac60eb2b9bf471a71db8a2ff496c
444409ff2b ci: Reduce Alpine musl task to md runner size (MarcoFalke)
fa6b2e9efe ci: Turn centos config into alpine musl config (MarcoFalke)
Pull request description:
Fixes https://github.com/bitcoin/bitcoin/issues/33437
Historically, the centos task was added to add CI coverage for old packages and 32-bit depends builds, but both are now covered by different tasks.
The CentOS task aligns with Ubuntu/Debian CI tasks in terms of libc usage, but (slightly) differs in package naming and update philosophy. I am not aware of the task ever discovering a centos-related issue, so it seems fine to recycle it into an Alpine Linux task.
The main difference would be that musl libc is now used. Also, busybox is used in Alpine, so in theory the busybox install could be removed from the arm CI task in the future.
Packaging considerations: All packages should roughly be the same (gcc remains at version 14, python remains at version 3.12, etc). Also, all packages are from the Alpine main track, coming with 2 years of support. The only exception is the py3-pip package (https://pkgs.alpinelinux.org/packages?name=py3-pip&branch=v3.22&repo=&arch=riscv64) from the community track, however, I don't expect any issues arising from that.
ACKs for top commit:
janb84:
reACK 444409ff2b
willcl-ark:
ACK 444409ff2b
Tree-SHA512: fd1a1da0fd766591e44a57dbdb84f9b3b47ca92113a429bba139ee5fef54714b8fe509c321e7b3a470c29b4af7d9eab9786e1660b9effb862ecea52824f458aa
3a10d700bc test: P2SH sig ops are only counted with `SCRIPT_VERIFY_P2SH` flag (brunoerg)
Pull request description:
This PR adds a test case for `GetTransactionSigOpCost` to check that P2SH sig ops are only counted when `SCRIPT_VERIFY_P2SH` flag is set.
Kills the following [mutant](https://corecheck.dev/mutation/src/consensus/tx_verify.cpp#L150):
```diff
diff --git a/src/consensus/tx_verify.cpp b/src/consensus/tx_verify.cpp
index 9d09872597..cc7cdaaf8f 100644
--- a/src/consensus/tx_verify.cpp
+++ b/src/consensus/tx_verify.cpp
@@ -147,7 +147,7 @@ int64_t GetTransactionSigOpCost(const CTransaction& tx, const CCoinsViewCache& i
if (tx.IsCoinBase())
return nSigOps;
- if (flags & SCRIPT_VERIFY_P2SH) {
+ if (1==1) {
nSigOps += GetP2SHSigOpCount(tx, inputs) * WITNESS_SCALE_FACTOR;
}
```
ACKs for top commit:
l0rinc:
Tested ACK 3a10d700bc
maflcko:
re-lgtm ACK 3a10d700bc
instagibbs:
ACK 3a10d700bc
janb84:
tested ACK 3a10d700bc
Tree-SHA512: f560b4f9f2ce5c5fdd0a86e7e1f8ea27a8c6fda0327a6186a0c21e2c06ef13beeb017686db1688cace68812a01701abe46e8e1a095afefc6f2aed6ed96ba8288
d543c0d917 Merge bitcoin-core/secp256k1#1734: Introduce (mini) unit test framework
f44c1ebd96 Merge bitcoin-core/secp256k1#1719: ci: DRY workflow using anchors
a44a339384 Merge bitcoin-core/secp256k1#1750: ci: Use clang-snapshot in "MSan" job
15d014804e ci: Drop default for `inputs.command` in `run-in-docker-action`
1decc49a1f ci: Use YAML anchor and aliases for repeated "CI script" steps
dff1bc107d ci, refactor: Generalize use of `matrix.configuration.env_vars`
4b644da199 ci: Use YAML anchor and aliases for repeated "Print logs" steps
a889cd93df ci: Bump `actions/checkout` version
574c2f3080 ci: Use YAML anchor and aliases for repeated "Checkout" steps
53585f93b7 ci: Use clang-snapshot in "MSan" job
6894c964f3 Fix Clang 21+ `-Wuninitialized-const-pointer` warning when using MSan
2b7337f63a Merge bitcoin-core/secp256k1#1756: ci: Fix image caching and apply other improvements
f163c35897 ci: Set `DEBIAN_FRONTEND=noninteractive`
70ae177ca0 ci: Bump `docker/build-push-action` version
b2a95a420f ci: Drop `tags` input for `docker/build-push-action`
122014edb3 ci: Add `scope` parameter to `cache-{to,from}` options
2f4546ce56 test: add --log option to display tests execution
95b9953ea4 test: Add option to display all available tests
953f7b0088 test: support running specific tests/modules targets
0302c1a3d7 test: add --help for command-line options
9ec3bfe22d test: adapt modules to the new test infrastructure
48789dafc2 test: introduce (mini) unit test framework
baa265429f Merge bitcoin-core/secp256k1#1727: docs: Clarify that callback can be called more than once
4d90585fea docs: Improve API docs of _context_set_illegal_callback
895f53d1cf docs: Clarify that callback can be called more than once
de6af6ae35 Merge bitcoin-core/secp256k1#1748: bench: improve context creation in ECDH benchmark
5817885153 Merge bitcoin-core/secp256k1#1749: build: Fix warnings in x86_64 assembly check
ab560078aa build: Fix warnings in x86_64 assembly check
10dab907e7 Merge bitcoin-core/secp256k1#1741: doc: clarify API doc of `secp256k1_ecdsa_recover` return value
dfe284ed2d bench: improve context creation in ECDH benchmark
7321bdf27b doc: clarify API doc of `secp256k1_ecdsa_recover` return value
b475654302 Merge bitcoin-core/secp256k1#1745: test: introduce group order byte-array constant for deduplication
9cce703863 refactor: move 'gettime_i64()' to tests_common.h
0c91c56041 test: introduce group order byte-array constant for deduplication
88be4e8d86 Merge bitcoin-core/secp256k1#1735: musig: Invalidate secnonce in secp256k1_musig_partial_sign
36e76952cb Merge bitcoin-core/secp256k1#1738: check-abi: remove support for obsolete CMake library output location (src/libsecp256k1.so)
399b582a5f Split memclear into two versions
4985ac0f89 Merge bitcoin-core/secp256k1#1737: doc: mention ctx requirement for `_ellswift_create` (not secp256k1_context_static)
7ebaa134a7 check-abi: remove support for obsolete CMake library output location (src/libsecp256k1.so)
806de38bfc doc: mention ctx requirement for `_ellswift_create` (not secp256k1_context_static)
03fb60ad2e Merge bitcoin-core/secp256k1#1681: doc: Recommend clang-cl when building on Windows
d93380fb35 Merge bitcoin-core/secp256k1#1731: schnorrsig: Securely clear buf containing k or its negation
8113671f80 Merge bitcoin-core/secp256k1#1729: hash: Use size_t instead of int for RFC6979 outlen copy
325d65a8cf Rename and clear var containing k or -k
960ba5f9c6 Use size_t instead of int for RFC6979 outlen copy
737912430d ci: Add more tests for clang-cl
7379a5bed3 doc: Recommend clang-cl when building on Windows
f36afb8b3d Merge bitcoin-core/secp256k1#1725: tests: refactor tagged hash verification
5153cf1c91 tests: refactor tagged hash tests
d2dcf52091 Merge bitcoin-core/secp256k1#1726: docs: fix broken link to Tromer's cache.pdf paper
489a43d1bf docs: fix broken link to eprint cache.pdf paper
d599714147 Merge bitcoin-core/secp256k1#1722: docs: Exclude modules' `bench_impl.h` headers from coverage report
0458def51e doc: Add `--gcov-ignore-parse-errors=all` option to `gcovr` invocations
1aecce5936 doc: Add `--merge-mode-functions=separate` option to `gcovr` invocations
106a7cbf41 doc: Exclude modules' `bench_impl.h` headers from coverage report
a9e955d3ea autotools, docs: Adjust help string for `--enable-coverage` option
e523e4f90e Merge bitcoin-core/secp256k1#1720: chore(ci): Fix typo in Dockerfile comment
24ba8ff168 chore(ci): Fix typo in Dockerfile comment
74b8068c5d Merge bitcoin-core/secp256k1#1717: test: update wycheproof test vectors
c25c3c8a88 test: update wycheproof test vectors
20e3b44746 Merge bitcoin-core/secp256k1#1688: cmake: Avoid contaminating parent project's cache with `BUILD_SHARED_LIBS`
2c076d907a Merge bitcoin-core/secp256k1#1711: tests: update Wycheproof
7b07b22957 cmake: Avoid contaminating parent project's cache with BUILD_SHARED_LIBS
5433648ca0 Fix typos and spellings
9ea54c69b7 tests: update Wycheproof files
git-subtree-dir: src/secp256k1
git-subtree-split: d543c0d917a76a201578948701cc30ef336e0fe6
e4c04f7759 ci: add libcpp hardening flags to macOS fuzz job (fanquake)
Pull request description:
Follows up to https://github.com/bitcoin/bitcoin/pull/33425#issuecomment-3323149107.
ACKs for top commit:
maflcko:
lgtm ACK e4c04f7759. The qa-assets repo has a libc++ debug run, so this isn't required, but it seems fast enough to not hurt.
Tree-SHA512: 6c0dc90528ca867df49027eebf2d1c417a7395f9f94779076ace48e3e4b21771e7d99e8b3ed75ca56da87153418a446075429aa6b9ec5cd4b6b8cb5b0c25c1d7
ac599c4a9c test: Test MuSig2 in the wallet (Ava Chow)
68ef954c4c wallet: Keep secnonces in DescriptorScriptPubKeyMan (Ava Chow)
4a273edda0 sign: Create MuSig2 signatures for known MuSig2 aggregate keys (Ava Chow)
258db93889 sign: Add CreateMuSig2AggregateSig (Ava Chow)
bf69442b3f sign: Add CreateMuSig2PartialSig (Ava Chow)
512b17fc56 sign: Add CreateMuSig2Nonce (Ava Chow)
82ea67c607 musig: Add MuSig2AggregatePubkeys variant that validates the aggregate (Ava Chow)
d99a081679 psbt: MuSig2 data in Fill/FromSignatureData (Ava Chow)
4d8b4f5336 signingprovider: Add musig2 secnonces (Ava Chow)
c06a1dc86f Add MuSig2SecNonce class for secure allocation of musig nonces (Ava Chow)
9baff05e49 sign: Include taproot output key's KeyOriginInfo in sigdata (Ava Chow)
4b24bfeab9 pubkey: Return tweaks from BIP32 derivation (Ava Chow)
f14876213a musig: Move synthetic xpub construction to its own function (Ava Chow)
fb8720f1e0 sign: Refactor Schnorr sighash computation out of CreateSchnorrSig (Ava Chow)
a4cfddda64 tests: Clarify why musig derivation adds a pubkey and xpub (Ava Chow)
39a63bf2e7 descriptors: Add a doxygen comment for has_hardened output_parameter (Ava Chow)
2320184d0e descriptors: Fix meaning of any_key_parsed (Ava Chow)
Pull request description:
This PR implements MuSig2 signing so that the wallet can receive and spend from imported `musig(0` descriptors.
The libsecp musig module is enabled so that it can be used for all of the MuSig2 cryptography.
Secnonces are handled in a separate class which holds the libsecp secnonce object in a `secure_unique_ptr`. Since secnonces must not be used, this class has no serialization and will only live in memory. A restart of the software will require a restart of the MuSig2 signing process.
ACKs for top commit:
fjahr:
tACK ac599c4a9c
rkrux:
lgtm tACK ac599c4a9c
theStack:
Code-review ACK ac599c4a9c🗝️
Tree-SHA512: 626b9adc42ed2403e2f4405321eb9ce009a829c07d968e95ab288fe4940b195b0af35ca279a4a7fa51af76e55382bad6f63a23bca14a84140559b3c667e7041e
0626b90f50 multiprocess: align our logging with libmultiprocess's (Cory Fields)
9d068225ee multiprocess: update multiprocess EventLoop construction to use options (Cory Fields)
Pull request description:
This fixes https://github.com/bitcoin-core/libmultiprocess/issues/215 on Core's side. ~It depends on https://github.com/bitcoin-core/libmultiprocess/pull/220 being merged upstream, and a PR to update our subtree. I've included a subtree merge from my repo here for now, but will rebase on top of the merge from upstream once it's in.~ Edit: Rebased on top of #33518.
For context: before https://github.com/bitcoin-core/libmultiprocess/pull/220, libmultiprocess serializesd every log message parameter, even if that message was ultimately going to be discarded. The upstream PR accomplished 2 main things:
- Creates logging categories, similar to Core's
- Using macros, avoids serializing parameters for disabled log levels.
That allowed the expensive serialization to be skipped, but the default log level is `Trace`. This PR updates our usage of libmultiprocess options to honor our log categories and levels.
Because of the substantial unnecessary overhead (see the [flamegraphs](https://github.com/bitcoin-core/libmultiprocess/issues/215). Logging accounts for 50% of my application's cpu time, and nearly 10% of bitcoin-node's, both of which go to ~0% once fixed), it'd be a shame to ship the first multiprocess binaries without this fixed. So I propose that we also backport this (and the required libmultiprocess subtree merge) ~to v30. Sorry about the timing~ :(
Edit: Didn't make it for v30, but it would still make sense to backport for a v30.1.
ACKs for top commit:
Sjors:
ACK 0626b90f50
TheCharlatan:
ACK 0626b90f50
sipa:
utACK 0626b90f50
Tree-SHA512: 70b63b62d1f6de547f4d4775538d7bcaf32f57d8a72c5b26762b57755810c8be6942d9dfebab43cf1c1d8d025a555f72a48e9ebf3d84f8d40d6592ca801cda5d
Change BlockBuilderImpl's m_excluded_clusters to unordered
set since ordering is not used.
Change the set to a set of sequence numbers for a modest
stability increase under fuzz testing.
