75cf9708a0 ci: add one more routable address to the VMs (docker containers) (Vasil Dimov)
1b93983bf5 test: make feature_bind_port_(discover|externalip).py auto-detect the skip condition (Vasil Dimov)
Pull request description:
`feature_bind_port_discover.py` and `feature_bind_port_externalip.py` require a routable address on the machine to run. Since that was not predictably available on CI, those tests required a manual setting up of IP addresses (e.g. using `ifconfig`) and then running the tests with a command line option telling them that the addresses are set up. The tests were not run in CI and [got rot](https://github.com/bitcoin/bitcoin/issues/31293#issuecomment-2497792487).
Change that to auto-detect, from the tests, whether the needed IP addresses are present and if yes, run the test, otherwise skip it. Also change the CI to configure the needed addresses when running the functional tests. This way the tests will be run regularly on CI.
Fixes: https://github.com/bitcoin/bitcoin/issues/31336
ACKs for top commit:
willcl-ark:
ACK 75cf9708a0
frankomosh:
Tested ACK 75cf9708a0. Built from source.
ryanofsky:
Code review ACK 75cf9708a0. Tested locally with and without the special addresses, and the detection seems to work well.
Tree-SHA512: 252911a37a06764f644a1a83c808f5255ac3bc74919426afa5d082c59e1ea924196354735f229d381cb5aff2340e001c2240bbadc8b5f27e5321fb4cfaef0fdb
2424e52836 lint: doc: detail lint tool install methods (will)
5fefa5a654 Don't pin Python patch version (Sjors Provoost)
fd15b55c2e lint: use requirements.txt (will)
5f4d3383da lint: switch to ruff for formatting and linting (will)
a53b81ce4e lint: switch to uv for python management in linter (will)
Pull request description:
Modernise our lint tooling by:
\- Replacing pyenv + pip with [uv](https://docs.astral.sh/uv/) for better Python environment and dependency management
\- Move uv ruff and ty to install via `COPY --from` multi-stage Docker image imports
\- Moving ruff lint rules from hardcoded Rust array (in lint_py.rs) into a top-level ruff.toml
\- Extracting all remaining pip dependencies into dedicated ci/lint/requirements.txt
Extra rationale:
`COPY --from` pulls pre-built binaries from upstream images instead of compiling/downloading at runtime. Containerfile layer optimisations reduce rebuild frequency further.
Pinning tool versions in the dockerfile makes it more excplicit and easier to find.
The tradeoff we make here is that there is no longer a single install script to install tooling on a local machine. However I think this is OK, as it currently only works for `apt`-based OSes anyway, and I don't think running the linter outside of the container is such a valuable use-case as it is with some of the other CI jobs.
ACKs for top commit:
maflcko:
review ACK 2424e52836🗿
sedited:
ACK 2424e52836
Tree-SHA512: 32ef989c1e241cebe5f13da10abd23f6f63306591fd1f81880d688b886082bca17987591dc592c41fbb72278eba57b3cc6e786de7cfa80eb490ab34465d0119b
d084bc88be doc: clarify IWYU workflow (Lőrinc)
7c7cec4567 ci: update IWYU patch reference (Lőrinc)
Pull request description:
### Problem
This was prompted by https://github.com/bitcoin/bitcoin/pull/34435#discussion_r3123255248, where it was not clear to me how (and where) exceptional IWYU cases should be documented.
### Fix
This PR documents the IWYU CI wrapper as the reproducible local entrypoint.
The developer notes now recommend reducing suspected IWYU false positives to a minimal upstream reproducer, treat `IWYU pragma` as a narrow workarounds, and ask for nearby rationale comments on non-obvious IWYU pragma use. An example comment was also added.
The IWYU patch comment is also updated to point at the current `clang_22` include picker reference.
### Reproducer
Create a dummy commit on top that adds an unused include, then run the command from the developer notes.
Without the dummy commit, the command should pass.
<details><summary>IWYU demo commit</summary>
```diff
diff --git a/src/kernel/bitcoinkernel.cpp b/src/kernel/bitcoinkernel.cpp
--- a/src/kernel/bitcoinkernel.cpp(revision c92b329e7b7d49476b5977d26c24d7c4982c6024)
+++ b/src/kernel/bitcoinkernel.cpp(revision ad2c5ba2ba69156e77061c1e6c098b725c28f322)
@@ -43,6 +43,7 @@
#include <functional>
#include <list>
#include <memory>
+#include <vector>
#include <span>
#include <stdexcept>
#include <string>
```
</details>
> [!NOTE]
> After repeated failing runs, `docker container rm -f ci_native_iwyu` may be needed because the local CI wrapper can leave the detached container running when the inner test command fails.
ACKs for top commit:
hebasto:
ACK d084bc88be.
sedited:
ACK d084bc88be
Tree-SHA512: 0aac42d468a1fdfa9f4a3856372e05fb43ec9f0973aeb3a4194fff948fc61e8e72e3b280cde10e74b8da88b6cff93962b3a7f7390eb042113ef92aa6b51d6d8f
a49bc1e24e ci: add --extended when using --usecli (Pol Espinasa)
5603ae0ffa test: fix send_batch_request to pass callables when using --usecli (Pol Espinasa)
Pull request description:
**First commit**
This commit fixes an error that made feature_index_prune.py fail if --usecli was used.
The test was failing because `node.batch(data)` was called with `data` being a dict. This worked in the normal scenario because `AuthServiceProxy.batch()` expects a list of petitions in the form of a dict. But `TestNodeCLI.batch()` expects callables and not a dict. When `--usecli` is used the test fails with an error `TypeError: 'dict' object is not callable`.
This is fixed by using `get_request()` which returns a lambda function if `--usecli` is used and returns a dict if not.
The `assert` is also changed because before this PR, the requests, constructed by hand were not specifiying the json-rpc version. By default if no version is specified we use version 1.0 which always returns `error: none` if there is no error. However, in version 2.0, it does not include an error key if there is no error. By using `get_request()` the requests are done in version 2.0 so there's no key error.
**Second commit**
The current CI doesn't cover the case of running all tests with --extended if using --usecli.
This led to a test failing (feature_index_prune.py) if run with --usecli and the CI not catching it.
See https://github.com/bitcoin/bitcoin/pull/34991 for context.
This commit improves the CI test coverage by also running now all functional tests (--extended) with the flag --usecli.
<details>
<summary>First "wrong" approach</summary>
Fixes two bugs that make `feature_index_prune.py` fail if `--usecli` is used.
1. Makes `TestNodeCLI.batch()` response equivalent to what a JSON-RPC response would look like by adding `error=None` in the response. The lack of `error` in the response was giving a `KeyError` message.
2. Makes `send_batch_request()` compatible with --usecli. Before the PR it was passing dicts to `node.batch()`, but `TestNodeCLI.batch()` expects callables, not dicts.
</details>
ACKs for top commit:
Bicaru20:
Re-ACK a49bc1e24e.
sedited:
Re-ACK a49bc1e24e
Tree-SHA512: 75fca26cf120638ced1fe38e86d8e3efa7addb6d97fc801e34783efd2cf6417f4ded2ec6247b6dcbcdb3cf4f48c4858f0932cbaa3e836a973d53581e75470a3f
Also explicitly specify which addresses from the docker network to
assign to the VM.
With `1.1.1.5` and `1111:1111::5` set on the machine, the tests
`feature_bind_port_discover.py` and
`feature_bind_port_externalip.py` will run.
65379bb8d0 ci: add FreeBSD cross CI job (fanquake)
f44191f163 depends: build qrencode for Freebsd (fanquake)
7f7018738e depends: FreeBSD cross with Clang (fanquake)
6464f14081 depends: disable inotify in Freebsd Qt build (fanquake)
Pull request description:
Alternative to #33562, which was adding a native FreeBSD job; however that had issues with permissions/caching, as well as potential determinism issues. This adds a FreeBSD cross job using Linux and Clang.
Would close#33438. The same changes here could also be used to produce FreeBSD binaries out of Guix.
ACKs for top commit:
hebasto:
ACK 65379bb8d0. I've cross-compiled on Ubuntu 25.10 for FreeBSD 14.4 and 15.0. The former binaries (`bitcoind`, `test_bitcoin` and `bitcoin-qt`) were tested on FreeBSD 14.4 locally.
Tree-SHA512: 52a3edaa56fe40ca901416cb9e1af04a84505526edfa7309bfa40024baa7d3b1a05303659553d9fbcf1f49d4e3d42b415a1e2523d448b22724d1415a49331259
3129d4a693 ci: Rename `TIDY_LLVM_V` to `IWYU_LLVM_V` in IWYU-specific code (Hennadii Stepanov)
0b489886f8 ci: Upgrade IWYU to 0.26 compatible with Clang 22 (Hennadii Stepanov)
Pull request description:
This PR upgrades IWYU to [0.26](https://github.com/include-what-you-use/include-what-you-use/releases/tag/0.26) and removes mapping workarounds for issues that have been fixed upstream.
ACKs for top commit:
fanquake:
ACK 3129d4a693
Tree-SHA512: 9a926e489573d040423461c039ecda7636beb70e8214a02c1c594cbd5b89d26331455a9872c38993fa5ee6d27fdfefc2375dedc369721b2933411542a57a3884
fa70b9ebaa ci: Temporarily use clang in valgrind tasks (MarcoFalke)
faf3ef4ee7 ci: Clarify why valgrind task has gui disabled (MarcoFalke)
fadb77169b test: Scale feature_dbcrash.py timeout with factor (MarcoFalke)
Pull request description:
valgrind currently does not work on GCC compiled executables, due to an upstream bug. https://bugs.kde.org/show_bug.cgi?id=472329
So temporarily switch to clang, so that a long term solution can be figured out in the meantime.
ACKs for top commit:
l0rinc:
ACK fa70b9ebaa
fanquake:
ACK fa70b9ebaa - also checked that it doesn't currently work under aarch64.
Tree-SHA512: 2e7c7a709311efa7bf29c3f9b1db60886b189b2d2bfebb516062163d65f0d7e8de3b6fc21c94cd62f6bd7e786e9c36fba55c4bae956b849851eb8b08e772c03e
valgrind currently does not work on GCC -O2 compiled executables, which
contain std::optional use, due to an upstream bug. See
https://bugs.kde.org/show_bug.cgi?id=472329
One workaround could be to use -O1. However, that seems brittle, as
variantions of the bug were seen with -O1 as well.
So temporarily use clang in the valgrind CI tasks, because this also
allows to drop a false-positive suppression for:
-DCMAKE_CXX_FLAGS='-Wno-error=array-bounds'
Also, update the comment in contrib/valgrind.supp to mention the
background:
* GCC -O2 wasn't tested with the suppressions file, due to the mentioned
bug.
* Clang-17 (or later) on aarch64 wasn't tested due to bug
https://github.com/bitcoin/bitcoin/issues/29635 and the minimum
supported clang version is clang-17 right now.
* GUI isn't tested, because it requires a debug build, see the prior
commit.
This means the only tested config right now is the one mentioned in the
suppression file.
A build with system libs (or with a normal depends build) will fail
with:
```sh
$ valgrind --exit-on-first-error=yes --error-exitcode=1 --quiet ./bld-cmake/bin/test_bitcoin-qt
Detected locale "C" with character encoding "ANSI_X3.4-1968", which is not UTF-8.
Qt depends on a UTF-8 locale, and has switched to "C.UTF-8" instead.
If this causes problems, reconfigure your locale. See the locale(1) manual
for more information.
********* Start testing of AppTests *********
Config: Using QtTest library 6.10.2, Qt 6.10.2 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 15.2.0), ubuntu 26.04
PASS : AppTests::initTestCase()
QINFO : AppTests::appTests() Backing up GUI settings to "/tmp/test_common bitcoin/60d474ffae390f81657d/regtest/guisettings.ini.bak"
==18007== Conditional jump or move depends on uninitialised value(s)
==18007== at 0x12655E26: ???
==18007== by 0xCB28E7F: ???
==18007==
==18007==
==18007== Exit program on first error (--exit-on-first-error=yes)
```
A DEBUG=1 depends build would work, but that seems tedious for
questionable benefit.
Replace the hardcoded x86_64 binary name with $(uname --machine) so the
correct binary is downloaded when building the lint container, where at
all possible.
fa90b21430 test: Remove unused feature_segwit.py functions (MarcoFalke)
fa6b05c96f test: Remove unused CUSTOM_._COUNT (MarcoFalke)
fa7bac94d8 test: Remove unused wait_for_addr, firstAddrnServices, on_addr (MarcoFalke)
fa388a3585 test: Remove unused self.p2p_conn_index = 1 (MarcoFalke)
fa803710e2 test: Remove unused AddressType (MarcoFalke)
fab5072ce1 ci: Remove vulture (MarcoFalke)
Pull request description:
Currently, `vulture` is run with `--min-confidence=100`, which reduces its checks to dead code after control statements, which is nice, but not really a common nor severe issue. See the discussion in https://github.com/bitcoin/bitcoin/issues/34810#issuecomment-4045927137 and commit 5c005363a8, which had to remove dead code manually.
Reducing the confidence has shown to be too brittle/tedious in the past, so remove the tool for now from CI.
Of course, removing this from CI does not prevent anyone from running it locally and removing dead code.
Fixes https://github.com/bitcoin/bitcoin/issues/34810
ACKs for top commit:
fanquake:
ACK fa90b21430
willcl-ark:
ACK fa90b21430
Tree-SHA512: 6a5998470dae3a17baec29b70b02333f4cd9b81bc4c6a05b56085ff1ba527ed7bdeccd17b09d9ad785ae03c97982ee1f3147e4df3bd537c66b02e9a44d0e5f15
d03e3be246 ci: check macos bundle structure and codesigning (fanquake)
66d80d57b4 macdeploy: use plugins dir to find plugins (fanquake)
ab137cbfe2 macdeploy: subprocess out to zip rather than shutil.make_archive (fanquake)
Pull request description:
Fix bundle format.
Add a CI check that codesigning works.
Fixes#34744.
ACKs for top commit:
Sjors:
tACK d03e3be246
hebasto:
ACK d03e3be246, tested on macOS Tahoe 26.3.1.
sedited:
ACK d03e3be246
Tree-SHA512: 5a7db896952edf338ff4fe8c934f1e1c992642850a99d5fafbb1212c6979601b3b72b6f3af880fb6f6ac8759cd4102e9f01792abb05410ceaf36cbffaec48e47
fafdb8f635 ci: Allow running iwyu ci in worktree (MarcoFalke)
fab73e213d ci: Reject unsafe execution of shell scripts (MarcoFalke)
Pull request description:
Currently, the iwyu CI fails to run in a git-worktree, or git-archive. This is due to the use of `git diff`.
Fix this by force-initializing a dummy git repo with a single dummy commit.
It may be possible to detect when `git diff` is not available in the directory, and only apply the fallback when needed, but the git history is not needed and it is easier to unconditionally apply the git init.
ACKs for top commit:
willcl-ark:
reACK fafdb8f635
hebasto:
ACK fafdb8f635, I have reviewed the code and it looks OK. Tested on Fedora 43.
sedited:
ACK fafdb8f635
Tree-SHA512: 572f1e2b9e215c2804095382498abb5b8636e3a49d5ba2a736b975e06afa2881d815b854a8a593d0f187c7c6b55034688e11f46d6814edfe7c29505197e80b18
9f3752c437 ci: use latest versions of lint deps (fanquake)
Pull request description:
Use the latest available versions, except for LIEF, which is changed with Guix.
ACKs for top commit:
hebasto:
ACK 9f3752c437, I've verified the releases against https://pypi.org and https://github.com/becheran/mlc.
Tree-SHA512: e6ed79bb7dc8601ed0708eb7b53cbf4cf843b69829c073c41e9d97be690b4b2bf9ea5ecf250e05cbacba4ad35df06aa3e2cb2ff319145a34e1a7831cf182ec21
24699fec84 doc: Add initial asmap data documentation (Fabian Jahr)
bab085d282 ci: Use without embedded asmap build option in one ci job (Fabian Jahr)
e53934422a doc: Expand documentation on asmap feature and tooling (Fabian Jahr)
6244212a55 init, net: Implement usage of binary-embedded asmap data (Fabian Jahr)
6202b50fb9 build: Generate ip_asn.dat.h during build process (Fabian Jahr)
634cd60dc8 build: Add embedded asmap data (Fabian Jahr)
Pull request description:
This is the final in a series of PRs that implement the necessary changes for embedding of asmap data into the binary. This last part add the initial asmap data, implements the build changes and adds further documentation.
Currently an asmap file needs to be acquired by there user from some location or the user needs to generate one themselves. Then they need to move the file to the right place in datadir or pass the path to the file as `-asmap=PATH` in order to use the asmap feature. The change here allows for builds to embed asmap data into the bitcoind binary which makes it possible to use the feature without handling of the asmap file by the user. If the user starts bitcoind with `-asmap` the embedded data will be used for bucketing of nodes.
The data lives in the repository at `src/node/data/ip_asn.dat` and can be replaced with a new version at any time. The idea is that the data should be updated with every release. By default the data at that location is embedded into the binary but there is also a build option to prevent this (`-DWITH_EMBEDDED_ASMAP=OFF`). In this case the original behavior of the `-asmap` option is maintained.
ACKs for top commit:
achow101:
ACK 24699fec84
sipa:
ACK 24699fec84
hodlinator:
ACK 24699fec84
Tree-SHA512: c2e33dbeea387efdfd3d415432bf8fa64de80f272c1207015ea53b85bb77f5c29f1dae5644513a23c844a98fb0a4bb257bf765f38b15bfc4c41984f0315b4c6a
The shell scripts are inherently unsafe, because they will install new
software packages, modify global configuration settings, write to the
root / or $HOME, and possibly modify the git repo.
The only safe way to run them is through the CI system itself, that is
the ci_exec python function.
The ci_exec funtion ensures that the user has set up a sandbox
externally and set DANGER_RUN_CI_ON_HOST=1 at their own risk, or that a
sandbox was set up with the given container_id, in which case it is safe
to set DANGER_RUN_CI_ON_HOST=1 for that sandbox.
Also, it is safe to set DANGER_RUN_CI_ON_HOST=1 when building the
sandbox image in ci/test_imagefile.
Then, the two shell scripts can reject early if unsafe execution is
detected.
faba426b3b lint: Flatten lint image entry points (MarcoFalke)
1111fff91c lint: Add missing --platform=linux to docker build command (MarcoFalke)
Pull request description:
Two fixups to the lint container:
* Add a missing `--platform=linux` to avoid running a non-native arch, like s390x, which happens with podman if such a container was most recently used.
* Flatten the entry points to remove the bash-based one:
Previously, an additional entry point into the container that spawned a bash was supported. The bash had an alias `lint` to run all lint scripts. However, such a use-case seems limited (because it only runs inside the container), inflexible (because it only allows running all lint scripts), and possibly brittle (because it can miss re-building the image when the cache is stale). So remove it and just offer the single entry point via the `./ci/lint.py` script.
If there is a use-case to skip the image building, it should be trivial to add an env var setting the the lint Python script like `DANGER_SKIP_IMAGE_RE_BUILD=1` (or so) in the future.
ACKs for top commit:
willcl-ark:
ACK faba426b3b
Tree-SHA512: 9afda16723c215602c6c42fa3a286d1828c887c8f6ff9512c8ec162ec8997789695f0c464d389cae94e67acf8b5e0f1a55e2ee0d60131a2eee091cf281f91514
d79249d279 ci: add chimera Linux LTO CI job (fanquake)
Pull request description:
Adds a CI config based on using [Chimera Linux](https://chimera-linux.org/). This might be interesting for any of the following:
* Chimera is based on LLVM & musl libc - we test both of these in isolation, but not together.
* No GNU components. I don't think we have an existing Linux CI job that doesn't have a gcc/stdlibc++ install. This exercises the depends logic for a fully LLVM/Clang/lld only build, including building the native tools (related to #33902).
* We don't currently have a job with LTO enabled (here using CMakes `CMAKE_INTERPROCEDURAL_OPTIMIZATION`, which is `-flto=thin` for LLVM/Clang). I think this is worth having generally (we do use LTO in some other places, like oss-fuzz). If runtime is too much of an issue, then it could also be dropped. (Chimera itself is also compiled with LTO).
QT in depends doesn't build (#32744), so is excluded for now.
Chimera has pointed out at least a few quirks, i.e #34390, #34408 and https://github.com/bitcoin/bitcoin/pull/29963#discussion_r2707922298.
ACKs for top commit:
maflcko:
lgtm ACK d79249d279
hebasto:
ACK d79249d279.
Tree-SHA512: 1174a7462bf2e7433a2c27a6cf398e94b05db42bb414629c71cf9f9a297ca269e173ae1b7517b30510b494b4397f918eef706d3c75c4286767c5557aeb6db4c7
b65a3d8009 iwyu: Fix patch to prefer `<cstdint>` (Hennadii Stepanov)
Pull request description:
The goal of the [patch](https://github.com/bitcoin/bitcoin/blob/master/ci/test/01_iwyu.patch) is to suggest C++ headers rather than their C counterparts. However, for fixed width integer types, the patched IWYU currently suggests `<cinttypes>` where `<cstdint>` is sufficient.
This PR fixes this behavior.
ACKs for top commit:
maflcko:
lgtm ACK b65a3d8009
furszy:
utACK b65a3d8009
willcl-ark:
utACK b65a3d8009
Tree-SHA512: 695efdd44b92a642401738572e49c8b6591aa4463d387107fdf3d2f7c9c4b39f4097cb82413752caf9e8890dcca7246a894e562a1dd17023b05a7e455705beac
-Werror is added to the previous releases job, given it runs on Ubuntu
22.04, which uses an older CMake.
`--compile-no-warning-as-error` can be used, if needed, in future, to
suppress the `CMAKE_COMPILE_WARNING_AS_ERROR` behaviour from a CI
config.
CMAKE_COMPILE_WARNING_AS_ERROR was added to CMake in 3.24.
See https://cmake.org/cmake/help/latest/prop_tgt/COMPILE_WARNING_AS_ERROR.html.
Co-authored-by: willcl-ark <will8clark@gmail.com>
The goal of the patch is to suggest C++ headers rather than their C
counterparts. However, for fixed width integer types, the patched IWYU
currently suggests `<cinttypes>` where `<cstdint>` is sufficient.
This change fixes this behavior.
