0c1b29a05777256c5ee686fff60f281dfeae289c ci: use GCC 13 for some jobs (fanquake)
cbc65b3ad5ad573844f9841199e1b0817f6c648a guix: use GCC 13.3.0 for base toolchain. (fanquake)
Pull request description:
Switch release builds to using GCC 13.3.0: https://gcc.gnu.org/gcc-13/, which landed in Guix in: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=750148ce1ea6c65a7c14424546db0078161f7e17.
Does not solve the cross-arch non-determinism for `powerpc64le-linux-gnu` builds.
ACKs for top commit:
achow101:
ACK 0c1b29a05777256c5ee686fff60f281dfeae289c
hebasto:
ACK 0c1b29a05777256c5ee686fff60f281dfeae289c.
TheCharlatan:
Re-ACK 0c1b29a05777
Tree-SHA512: eb3f091278d371166eb1df4718b6d0d68b09db65291d563dddd581964f2b488f901e4ba43831a699e2d0fd053d6e9038a307cbea78d5597da77699c34b440ea6
9b033bebb18dfd609c02736292f37cc6589fcc8d cmake: rename Kernel component to bitcoinkernel for consistency (Cory Fields)
2e0c92558e96b43351312d8fd403a20b9795acd4 cmake: add and use install_binary_component (Cory Fields)
0264c5d86c74c25407f0a07326d6f8722665fe74 cmake: use per-target components for bitcoin-qt and bitcoin-gui (Cory Fields)
fb0546b1c5ebb858605bef4c9fa001782e0ab213 ci: don't try to install for a fuzz build (Cory Fields)
Pull request description:
This makes it possible to build/install only the desired binaries regardless of the configuration.
For consistency, the component names match the binary names. `Kernel` and `GUI` have been renamed.
Additionally it fixes#31762 by installing only the manpages for the configured targets (and includes them in the component installs for each).
Also fixes#31745.
Alternative to #31765 which is (imo) more correct/thorough.
Can be tested using (for ex):
```bash
$ cmake -B build
$ cmake --build build -t bitcoind -t bitcoin-cli
$ cmake --install build --component bitcoind
$ cmake --install build --component bitcoin-cli
```
ACKs for top commit:
hebasto:
ACK 9b033bebb18dfd609c02736292f37cc6589fcc8d.
TheCharlatan:
Re-ACK 9b033bebb18dfd609c02736292f37cc6589fcc8d
stickies-v:
re-ACK 9b033bebb18dfd609c02736292f37cc6589fcc8d
Tree-SHA512: fd4818e76f190dbeafbf0c246b466f829771902c9d6d7111ed917093b811c8a5536a4a45e20708f73e7f581d6cb77c8e61cfa69e065788dcf0886792f553a355
fa952acdb6ef1b07b7927202d1373fa7479fd5e4 ci: Skip read-write of default env vars (MarcoFalke)
Pull request description:
If they remain unset, they use the default anyway. Except for `USER`, but this seems unused anyway.
Can be checked via:
```
sh-5.2# touch /tmp/empty_env
sh-5.2# podman run --rm --env-file /tmp/empty_env 'ubuntu:24.04' env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
container=podman
HOME=/root
HOSTNAME=19ece5c9e052
ACKs for top commit:
0xB10C:
ACK fa952acdb6ef1b07b7927202d1373fa7479fd5e4
Prabhat1308:
utACK [fa952ac](fa952acdb6)
Tree-SHA512: fe0c173b23cfda3025306303a44ffe32ecc57c2e0e1a2376594696f9887ed22f5105da84e898e790041bf15a4aa42a365fba016710ad269d439dda691977be90
Currently the manpages are installed, but that is a bug. An upcoming commit
will avoid installing manpages for targets that aren't configured, which
removes the "install" target for fuzz builds.
fa5a02bcfa2f3769859332fddb8954d6217de7fc ci: Use clang-20 for sanitizer tasks (MarcoFalke)
Pull request description:
A new clang version generally comes with bugfixes, new (sanitizer) features, and deprecations.
Upgrade the sanitizer tasks to use the new version.
This was also suggested in https://github.com/bitcoin/bitcoin/pull/31691#issuecomment-2602517116
ACKs for top commit:
fanquake:
ACK fa5a02bcfa2f3769859332fddb8954d6217de7fc - tested 20 in some other infra, we just needed to fix the same deprecation warnings we'd seen, in cryptofuzz: 09ca550c3e.
Tree-SHA512: 6114d790b5d7145eb5f019e02da6c2c833342707ad67fb9f9c09506001afbef0c9b9beee7e51321f17f12ea692509d6428e6072ad105dba51e4d54cd057621cd
faf8fc5487d409eeff7b7b260eabb6929a7b7a5f lint: Call lint_commit_msg from test_runner (MarcoFalke)
fa99728b0c8b3cac7056fa554fab7a8a4624a2de lint: Move commit range printing to test_runner (MarcoFalke)
fa673cf3449f4e71501814bf99c2e2bbb49b8fcb lint: Call lint_scripted_diff from test_runner (MarcoFalke)
Pull request description:
The lint `commit-script-check.sh` can not be called from the test_runner at all and must be called manually. Also, some checks require `COMMIT_RANGE` to be set.
Fix all issues by moving two lint checks into the test_runner. Also, the proper commit range is passed to the checks by the test_runner, so that the user no longer has to do it.
ACKs for top commit:
kevkevinpal:
reACK [faf8fc5](faf8fc5487)
willcl-ark:
tACK faf8fc5487d409eeff7b7b260eabb6929a7b7a5f
Tree-SHA512: 78018adc618d997508c226c9eee0a4fada3899cdfd91587132ab1c0389aea69127bafc3a900e90e30aca2c6bae9dcd6e6188ef287e91413bc63ee66fb078b1af
8888ee4403fa62972c49e18752695d15fd32c0b0 ci: Allow build dir on CI host (MarcoFalke)
Pull request description:
This is required to pass cross builds on to a different machine after the build.
See for example https://github.com/bitcoin/bitcoin/pull/31176, but this pull will also allow someone to implement it outside this repo.
ACKs for top commit:
davidgumberg:
lgtm ACK 8888ee4403
hebasto:
re-ACK 8888ee4403fa62972c49e18752695d15fd32c0b0.
Tree-SHA512: a1e2c32bc1b95efbd0b48287ac5b49e0e1bacbf5a5800845be5352bbdd3e17fa478e90348b2e94e95cf3ae863cdf75ab444089376588f6f8eec438f73a4b5b97
e87429a2d0f23eb59526d335844fa5ff5b50b21f ci: optionally use local docker build cache (0xb10c)
Pull request description:
By setting `DANGER_DOCKER_BUILD_CACHE_HOST_DIR`, the task-specific docker images built during the CI run can be cached. This allows, for example, ephemeral CI runners to reuse the docker images (or layers of it) from earlier runs, by persisting the image cache before the ephemeral CI runner is shut down. The cache keyed by `CONTAINER_NAME`.
As `--cache-to` doesn't remove old cache files, the existing cache is removed after a successful `docker build` and the newly cached image is moved to it's location to avoid the cache from growing indefinitely with old, unused layers.
When `--cache-from` doesn't find the directory, the cached version is a cache-miss, or the cache can't be imported for whatever other reason, it warns and `docker build` continues by building the docker image.
This feature is opt-in. The documentation for the docker build cache of `type=local` can be found on https://docs.docker.com/build/cache/backends/local/
This replaces https://github.com/bitcoin/bitcoin/pull/31377 - some of the discussion there might provide more context.
ACKs for top commit:
maflcko:
I haven't tested this, and it looks harmless and is easy to revert, if needed. So lgtm ACK e87429a2d0f23eb59526d335844fa5ff5b50b21f
achow101:
ACK e87429a2d0f23eb59526d335844fa5ff5b50b21f
TheCharlatan:
tACK e87429a2d0f23eb59526d335844fa5ff5b50b21f
willcl-ark:
ACK e87429a2d0f23eb59526d335844fa5ff5b50b21f
Tree-SHA512: 0887c395dee2e2020394933246d4c1bfb6dde7165219cbe93eccfe01379e05c75dce8920b6edd7df07364c703fcee7be4fba8fa45fd0e0e89da9e24759f67a71
fa8ade300f421dcc3b0cd956ab03a50a5ae80646 refactor: Avoid GCC false positive error (MarcoFalke)
fa40807fa830ee9724b5cfeef263263aaa6ce5d9 ci: Enable DEBUG=1 for one GCC-12+ build to catch 117966 regressions (MarcoFalke)
Pull request description:
It is possible that someone accidentally removes the workaround in fa9e0489f57968945d54ef56b275f51540f3e5e4, or more likely that someone accidentally adds new code without the workaround.
Avoid this by adding a temporary CI check.
This can be tested by reverting the workaround and observing a failure.
ACKs for top commit:
hebasto:
ACK fa8ade300f421dcc3b0cd956ab03a50a5ae80646, I've tested locally on Ubuntu 24.04.
Tree-SHA512: 7ee1538fd5304a5ab91ac8c7619a573548d7e0345592a1e9d38b3b73729e09e7c77a9ee703d64cf02a8218de3148376d7836e294abb939aa7533034ba36dfb6c
Allowing to call the check from the test_runner allows for consistent
error messages and better UX by having a single test_runner for all
checks.
This requires the env var to be set for now. The next commit makes the
commit range optional.
faaabfaea768deb7767c489d32fd2097fd180872 ci: Bump centos stream 10 (MarcoFalke)
Pull request description:
This is a follow-up to fa47baa03bcfcf44fb2ed05f009a32d32f860c45, which bumped the gcc version to avoid a warning bloat in the CI log. However, it is also required to bump python3, see https://github.com/bitcoin/bitcoin/issues/31476#issue-2735206340
> This will uncover an issue in the centos task that the correct python version is missing. I guess this should be fixed by installing and activating an acceptable python version.
Instead of bumping the packages individually in centos stream 9, just bump to stream 10.
ACKs for top commit:
fanquake:
ACK faaabfaea768deb7767c489d32fd2097fd180872
Tree-SHA512: a564ff3a2a0dc4d39874e87540e67072f293bbed82c8eca22266fcadc16c5571e0e41d38576a63e466b64d13f7e3acbd95be10cf2420de33127aa420eca3b928
fabefd9915802d53d8c83ae66e5cb259196d9422 ci: Turn CentOS task into native one (MarcoFalke)
Pull request description:
Cross-compiling to `i686-pc-linux-gnu` on CentOS in the CI is mostly redundant with the `ci/test/00_setup_env_i686_multiprocess.sh` task (albeit it using clang):
35bf426e02/ci/test/00_setup_env_i686_multiprocess.sh (L9-L12)
One task seems sufficient as a sanity check, given that there seems to be no real demand for this architecture anyway.
Turning the task into a native one makes it possible to run the task natively on aarch64 or any other supported architecture.
Also, remove the install of the `lbzip2` package, which is unused since commit a46065e36cf868265c909dc5edf29dc17be53c1f
Also, remove the `CONFIG_SHELL` env var, which is unused since the cmake migration. (`CONFIG_SHELL` in depends is still kept).
ACKs for top commit:
davidgumberg:
ACK fabefd9915
hebasto:
ACK fabefd9915802d53d8c83ae66e5cb259196d9422, tested locally on Ubuntu 24.10.
Tree-SHA512: 5a7b3131b379d11ef602e5821165861e9bdf61d605014bf8fcb33b8e12d8823450798af2d3289b96f7559dfa47b839bf939ddc0b3725efecfeac7ae570a981e7
According to the description for pkg-config, "pkgconf is a
replacement for pkg-config, providing additional functionality
while also maintaining compatibility. This package only provides
a dependency link to the pkgconf package to help with package
upgrades. It can be safely removed."
Thus several scripts and markdown files are updated.
For the task `MSan, depends (Cirrus CI)` we build a custom libc++ for
which we already use `-DLIBCXX_HARDENING_MODE=debug`. Compile it also
with `_LIBCPP_ABI_BOUNDED_*` to enable further checks.
Docs at: https://libcxx.llvm.org/Hardening.html#abi-options
By setting DANGER_DOCKER_BUILD_CACHE_HOST_DIR, the task-specific
docker images built during the CI run can be cached. This allows,
for example, ephemeral CI runners to reuse the docker images (or
layers of it) from earlier runs, by persisting the image cache
before the ephemeral CI runner is shut down. The cache keyed by
`CONTAINER_NAME`.
As --cache-to doesn't remove old cache files, the existing cache
is removed after a successful `docker build` and the newly cached
image is moved to it's location to avoid the cache from growing
indefinitly with old, unused layers.
When --cache-from doesn't find the directory, the cached version is
a cache-miss, or the cache can't be imported for whatever other reason,
it warns and `docker build` continues by building the docker image.
This feature is opt-in. The documentation for the cache type=local
can be found https://docs.docker.com/build/cache/backends/local/
This replaces https://github.com/bitcoin/bitcoin/pull/31377
31e59d94c67b58f24dd701fc7ccfee7d79f311cb iwyu: Drop backported mapping (Hennadii Stepanov)
fe9bc5abef3dcbe0f6395c3233f9d122579cd1f0 ci: Update Clang in "tidy" job (Hennadii Stepanov)
Pull request description:
This PR switches to the latest [IWYU 0.23](https://github.com/include-what-you-use/include-what-you-use/releases/tag/0.23), which is compatible with Clang 19.
New "bugprone-use-after-move" and "modernize-use-starts-ends-with" warnings that emerged have been addressed.
ACKs for top commit:
maflcko:
lgtm ACK 31e59d94c67b58f24dd701fc7ccfee7d79f311cb
l0rinc:
ACK 31e59d94c67b58f24dd701fc7ccfee7d79f311cb
theuni:
ACK 31e59d94c67b58f24dd701fc7ccfee7d79f311cb
Tree-SHA512: ae0ca150673e1bfa78664f2ef35dbc965094b32374cafeeae390c6d368c28169a7f7790debe9a6eeb5efc39c9a468f5032d92f30cc4032b09d8265f6a75de882
This change switches to the latest IWYU 0.23, which is compatible with
Clang 19.
Fixed new "modernize-use-starts-ends-with" warnings.
The new "bugprone-use-after-move" warning in `result_tests.cpp` is a
false positive caused by a bug in Boost.Test versions < 1.87. This has
been addressed by introducing a local variable.
See upstream references:
- Issue: https://github.com/boostorg/test/issues/343
- Fix: https://github.com/boostorg/test/pull/348
Co-authored-by: MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz>
Fixes: #31044
This MLC update includes a change which will ignore files being ignored
by git, and help avoid false-positives when linting in this repo.
fa5e7064597fc51366f082e3d07a4591e576db38 ci: Skip broken Wine64 tests by default (MarcoFalke)
Pull request description:
I don't think the unit tests run in Wine after the Windows cross-compilation have ever shown a true positive since the MSVC task was added. However, they are a source of frequent false-positives.
Thus, disable them by default for now. Anyone can still enable them by setting `RUN_UNIT_TESTS=true`.
A follow-up could run them on real Windows, see https://github.com/bitcoin/bitcoin/pull/31176.
Conceptually there are many other nightly tasks, which rarely find issues and are not run by default, like the valgrind or s390x tasks. So putting the Wine unit tests in the same bucket should be fine.
ACKs for top commit:
hebasto:
ACK fa5e7064597fc51366f082e3d07a4591e576db38, to avoid false-positives.
willcl-ark:
ACK fa5e7064597fc51366f082e3d07a4591e576db38
Tree-SHA512: 6bd54470e4d5ce18923c5d724aba0dbf475d053d7097d3f87e822a455cc537b6ce5f0dfcc8ccd0719c12c5d0c8fc7355a6c84185a6c9b5d484d98aee763d0c49
b031b7910d62819443813b91705211c466933a53 [ci] Split out native fuzz jobs for macOS and windows (dergoegge)
Pull request description:
Split out two new CI jobs (for native macOS and windows) that run the fuzz tests on the qa-assets input corpora.
In both jobs the fuzz binary is built with `-DBUILD_FOR_FUZZING` to enable `Assume` assertions as well as `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`.
ACKs for top commit:
maflcko:
re-lgtm ACK b031b7910d62819443813b91705211c466933a53
achow101:
ACK b031b7910d62819443813b91705211c466933a53
hebasto:
ACK b031b7910d62819443813b91705211c466933a53.
Tree-SHA512: 7d0dc5a9cb299f6f4596dd9a5526b6aaf82efc6eba308bdc9d8b0a45f79dea87204fb6cd4b2ea2a1bd952466b2e958d64021999296d110d7a83c1667f4de51fe
This is mentioned in the developer notes, but isn't present in
`TSAN_OPTIONS`, resulting in:
```bash
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=60508)
Cycle in lock order graph: M0 (0xffff98e02208) => M1 (0xffff98e0cbe8) => M2 (0xffff98e0cd98) => M0
<snip>
Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
```
Add it, for (potentially) more informative output, when failures
occur. Checked that adding does output more information.
fabe90c8242aa45a8b9925347ca6fc11d5852ffd ci: Use clang-19 from apt.llvm.org (MarcoFalke)
Pull request description:
A new clang version generally comes with bugfixes, new sanitizer features, deprecations, as well as new features.
Upgrade the sanitizer tasks to use the new version.
ACKs for top commit:
TheCharlatan:
ACK fabe90c8242aa45a8b9925347ca6fc11d5852ffd
hebasto:
ACK fabe90c8242aa45a8b9925347ca6fc11d5852ffd, I have reviewed the code and it looks OK.
Tree-SHA512: 637452e3cbc8ad97a3650976e2dbf4bcd032f2c454e12b48514eb5d252b3e106448674fe2d3bf45d11c0782874250d7a79e34fbb0aaa990499a156fcafd50329
40e5f26a3ff77e50df808f6f850c617aec2df203 mapport: remove dead code in DispatchMapPort (Antoine Poinsot)
38fdf7c1fb1946820236c319ad44c7bcbf0c6a98 mapport: drop outdated comments (Antoine Poinsot)
b7b24352906f1dba64826e7a093069b5bfc504dc doc: add release note for #31130 (Antoine Poinsot)
1b6dec98da3025c19951daf209347cecf1f0c6ab depends: drop miniupnpc (Antoine Poinsot)
953533d0214819a05d36672d295821ef06ced8d6 doc: remove mentions of UPnP (Antoine Poinsot)
94ad614482f4f1f9d207509a209badbc2fb5700d ci: remove UPnP options (Antoine Poinsot)
a9598e5eaab861fd6e6ce279f1282a83eec407d6 build: drop miniupnpc dependency (Antoine Poinsot)
a5fcfb7385c10d83a294cb2bb2248d06b2ab931e interfaces: remove now unused 'use_upnp' arg from 'mapPort' (Antoine Poinsot)
038bbe7b20074cc2201585dcc631e81b9e1e306c daemon: remove UPnP support (Antoine Poinsot)
844770b05ebc34789dc46d70cd6398089539c915 qt: remove UPnP settings (Antoine Poinsot)
Pull request description:
This PR removes UPnP IGD support and drops our [miniupnp](https://github.com/miniupnp/miniupnp) dependency.
Miniupnpc is a C library (somewhat) maintained by a single person which had several vulnerabilities in the past (a couple dozens are listed [here](https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=miniupnp)), some of which directly affected our software ([RCE in 2015](https://bitcoincore.org/en/2024/07/03/disclose_upnp_rce/), [OOM in 2020](https://bitcoincore.org/en/2024/07/31/disclose-upnp-oom/)).
The main purpose of this functionality is to have more (non-data-center) reachable nodes on the network. For a non-technical user running Bitcoin Core at home, the software would automatically open a port on their router to receive incoming connections. This way, users not able to manually open a port on their router would still provide the network with more resources and enhance its diversity.
However, due to past vulnerabilities (and a worry about unknown future ones) in miniupnpc this feature was disabled by default in https://github.com/bitcoin/bitcoin/pull/6795. Having it disabled by default kills (most of?) the purpose of having this functionality in the first place: someone technical enough to understand the `-upnp` startup option or the "enable UPnP" setting is most likely able to open a port on his box in the first place.
In addition, laanwj implemented PCP with a NAT-PMP fallback directly in Bitcoin Core in https://github.com/bitcoin/bitcoin/pull/30043. If we ever want to re-enable automatic NAT traversal by default in Bitcoin Core, this is the best option (and in my opinion the only sane one). The NAT-PMP fallback makes it so compatibility shouldn't be (much of) an issue.
On balance, i believe that keeping this functionality and this barely maintained C dependency has higher costs than benefits. Therefore i propose that we get rid of it.
ACKs for top commit:
jarolrod:
ACK 40e5f26a3f
1440000bytes:
Code Review ACK 40e5f26a3f
laanwj:
Code review ACK 40e5f26a3ff77e50df808f6f850c617aec2df203
i-am-yuvi:
Tested ACK 40e5f26a3ff77e50df808f6f850c617aec2df203
Tree-SHA512: 9ea48662775510f5ec6de7af65790f7c8d211603398e9d8c634a86387be81b28081419a95b4d6680d3d7fe6a9f16cec99f16516548201dc7e49781909899a657
fa71bedf8609f06618aa85342ea6f5c4d2c5fea0 ci: Approximate MAKEJOBS in image build phase (MarcoFalke)
Pull request description:
The `MAKEJOBS` env var is the default in image builds, which is fine, because it is only relevant when building msan (or iwyu) and only differs when setting MAKEJOBS to something other than `nproc` (currently used as an approximation).
So the normal workflow of `MAKEJOBS="-j$(nproc)" FILE_ENV="./ci/test/00_setup_env_native_msan.sh" ./ci/test_run_all.sh` already works today.
However, `MAKEJOBS="-j1" FILE_ENV="./ci/test/00_setup_env_native_msan.sh" ./ci/test_run_all.sh` does not.
This is hard to fix, because making the env var a build arg means that changing it (and only it) requires a new (expensive and redundant) build.
So add an option `HAVE_CGROUP_CPUSET`, which can be set to approximate `MAKEJOBS` a bit. Can be tested via:
`HAVE_CGROUP_CPUSET=yo MAKEJOBS="-j_something" FILE_ENV="./ci/test/00_setup_env_native_msan.sh" ./ci/test_run_all.sh`
ACKs for top commit:
fanquake:
ACK fa71bedf8609f06618aa85342ea6f5c4d2c5fea0
Tree-SHA512: 43ef194c71d726f4cfa3fe08a5894c7872150f37da7e4fa0c2d89e4572bc63acadb5dae3286a5e5cc14a8ce3e1ebcc14571f1a3541e8db2d18d2f7503764a2f3
