726e196ef2 ci: inline runner selection (will)
Pull request description:
We are currently sometimes backlogged on waiting for runner selection.
Selecting Warp or GitHub-hosted runners directly from the repository context avoids serializing all CI jobs behind a metadata job. This keeps forks on public runners while allowing upstream jobs to schedule immediately on the intended runner labels.
This will make the changes needed for "a fork also wanting to use warp runners" slightly larger, but I think that number of people is zero.
ACKs for top commit:
m3dwards:
ACK 726e196ef2
Tree-SHA512: 14f3f22fbc53b3f65fce48bd08aef8db8d60a37409fd8b4b5b128073cf81408f7d3135c1e78fe3f93af8c88b3ee8c1b770e445b1ee7d5650ea91b329bc7ff777
Speeds of 1MB/s and 15 minute cached docker image pulls during builds
are not uncommon.
Warp runners provide a local GitHub Actions cache protocol proxy for
Docker layer cache traffic. Point BuildKit's gha cache backend at that
proxy on Warp runners so cached image layers do not have to be fetched
from GitHub's slower cache service.
Add a default for provider so other users (e.g. qa-assets) don't have to
update this unless they use custome runners.
We are currently sometimes backlogged on waiting for runner selection.
Selecting Warp or GitHub-hosted runners directly from the repository context avoids serializing all CI jobs behind a metadata job. This keeps forks on public runners while allowing upstream jobs to schedule immediately on the intended runner labels.
The GHA cache is very slow, taking on the order of minutes to save and
restore from.
Use WarpBuild's cache instead as this is in the same region and much
faster.
WarpBuild cache action does not auto-fallback to GHA if not being run on
Warp. To allow fork runs to fallback to GHA caching, whilst minimising
duplication in the action files, create new "interal" actions which
perform the switching logic, and use these in the (renamed) cache|save
actions.
Without this we would need the `if` logic in our prvious actions, 4
times in each of save and restore.
Plumb the provider through into the action, as a composite action can't
read `env` (`GITHUB_OUTPUT`) from previous steps.
4bdd46ace3 ci: switch runners from cirrus to warpbuild (will)
Pull request description:
As cirrus is closing down, switch to warpbuild runners.
Switch runner and provider names over. We now use GHA cache, so we don't need to switch that over here.
ACKs for top commit:
m3dwards:
ACK 4bdd46ace3
maflcko:
review ACK 4bdd46ace3🤾
hebasto:
ACK 4bdd46ace3.
Tree-SHA512: 47ed28a6cb7ab10a973af6aa24f4f7a632f59ed17e189ae4f658de37069d763c92cc0e32769693568db6d0e5d2543abcb77bb0977f0b3f296d80a254d6bb3833
a154c05d49 cmake: Check dependencies after build option interaction (Hennadii Stepanov)
Pull request description:
At present, `CMakeLists.txt` interleaves configuration-option handling with dependency discovery. As a result, unnecessary checks may be performed. For example:
```
$ cmake -B build --preset dev-mode -DBUILD_FOR_FUZZING=ON
<snip>
-- Found PkgConfig: /usr/bin/pkg-config (found version "2.3.0")
-- Found ZeroMQ: /usr/lib64 (found suitable version "4.3.5", minimum required is "4.0.0")
-- Performing Test HAVE_USDT_H
-- Performing Test HAVE_USDT_H - Success
-- Found USDT: /usr/include
-- Found QRencode: /usr/lib64/libqrencode.so (found version "4.1.1")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Performing Test HAVE_STDATOMIC
-- Performing Test HAVE_STDATOMIC - Success
-- Found WrapAtomic: TRUE
-- Found OpenGL: /usr/lib64/libOpenGL.so
-- Found WrapOpenGL: TRUE
-- Found WrapVulkanHeaders: /usr/include
-- Found Qt: /usr/lib64/cmake/Qt6 (found suitable version "6.10.1", minimum required is "6.2")
CMake Warning at CMakeLists.txt:206 (message):
BUILD_FOR_FUZZING=ON will disable all other targets and force
BUILD_FUZZ_BINARY=ON.
<snip>
```
This PR restructures the code to ensure that all dependencies are checked only after the final build option interaction.
ACKs for top commit:
fanquake:
ACK a154c05d49
sedited:
Re-ACK a154c05d49
Tree-SHA512: f863316770a1e27e6c08736932f73f297add326943e9c32c721f7e771f38b0fdc651abfed61add5ee4924cfbf2afff91fde005d41d1fd239d401184b6caf5453
7777a92a92 ci: Use path with spaces on windows as well (MarcoFalke)
fac6c4270d ci: Put space and non-ASCII char in scratch dir (MarcoFalke)
fa38759823 ci: Require $FILE_ENV (MarcoFalke)
Pull request description:
It seems unlikely that many users have a space in their paths, but it seems a use-case worth enough to be tested by CI, so that it does not have to be done manually. Ref https://github.com/bitcoin/bitcoin/pull/33929#discussion_r2590523065 / https://mirror.b10c.me/bitcoin-bitcoin/33929/#discussion_r2590523065
So do that here, and also add a non-ASCII char while touching.
Also, fix all tests that are broken and assume no space exists in paths.
ACKs for top commit:
hebasto:
ACK 7777a92a92.
sedited:
ACK 7777a92a92
Tree-SHA512: eceb1f6c932c6966cdca8ca8df750081ec5134db5e5f558f7d955716409117bec7c8585d75865e2c98bc1ae7394f3ce64dff87bcebe1e68591afaeef1831d6dd
c03107acf5 ci: switch to GitHub cache for all runners (willcl-ark)
Pull request description:
Cirrus is winding down, and github now offers more than 10GB cache.
Switch to GH cache for all runner-types. Simplify configure-docker action.
ACKs for top commit:
maflcko:
review ACK c03107acf5🚴
Tree-SHA512: b6111c7559a86eed8488a4b0775df812a303a99eed5c80297593936e61d1d5e2ce72fe2fa615625816672414e6947ac4d93b9fd2925522fba06417ea4711ce79
d61053d97b build: Drop libevent from bitcoin-cli link libraries (Fabian Jahr)
798d051c80 cli: Remove libevent usage (Fabian Jahr)
376e7ef07c util: Expose IOErrorIsPermanent in sock header (Fabian Jahr)
5d562430de netbase: Add timeout parameter to ConnectDirectly (Fabian Jahr)
a988ac592f cli: Add HTTPResponseHeaders class for parsing response headers (Fabian Jahr)
c471c5085b common: Add unused UrlEncode function (Fabian Jahr)
9687ef1bd9 ci: Tolerate unused free functions in intermediate commits (Fabian Jahr)
Pull request description:
Part of the effort to remove the libevent dependency altogether, see #31194
This takes the parsing logic from the [`HTTPHeaders` class](d549f01caa) from #32061 and puts it into `bitcoin-cli` as a small `HTTPResponseHeaders` class with a comment to revisit potentially sharing this code somehow. This decoupled the two pulls which seems like the most sensible way to deal with this since the actual overlap is very small compared to the impact of each of the pulls which should ideally not block each other.
Otherwise the change itself replaces the libevent-based HTTP client with a simple synchronous implementation which uses the `Sock` class directly.
ACKs for top commit:
hodlinator:
re-ACK d61053d97b
theStack:
re-ACK d61053d97b
w0xlt:
ACK d61053d97b
Tree-SHA512: a3580a45faf540ee844aac8cb1dc056a89e8e11b45781d2807baa4736d5c0934284c6066206101b6984111a48a186d67845545d07639b623cb35ccc2d85d3ab2
Cirrus is winding down and github now offers more than 10GB cache.
Switch to GH cache for all runner-types. Simplify docker build arg
construction, and reduce the number of needed action permissions.
When bigger changes are split across multiple commits, intermediate
commits may introduce unused functions. Do not check for this error
in intermediate commits.
The new cache is keyed with the hash of 'vcpkg.json', which reduces
cache storage consumption compared to keying by run ID.
The `vcpkg/downloads/tools` subdirectory is excluded to further save
space.
Using a non-default target triplet introduced a regression because
packages with `"host": true` in their vcpkg configurations were still
picking up the default `x64-windows` triplet, effectively building both
release and debug packages.
Fix this regression by setting `VCPKG_HOST_TRIPLET` explicitly.
65379bb8d0 ci: add FreeBSD cross CI job (fanquake)
f44191f163 depends: build qrencode for Freebsd (fanquake)
7f7018738e depends: FreeBSD cross with Clang (fanquake)
6464f14081 depends: disable inotify in Freebsd Qt build (fanquake)
Pull request description:
Alternative to #33562, which was adding a native FreeBSD job; however that had issues with permissions/caching, as well as potential determinism issues. This adds a FreeBSD cross job using Linux and Clang.
Would close#33438. The same changes here could also be used to produce FreeBSD binaries out of Guix.
ACKs for top commit:
hebasto:
ACK 65379bb8d0. I've cross-compiled on Ubuntu 25.10 for FreeBSD 14.4 and 15.0. The former binaries (`bitcoind`, `test_bitcoin` and `bitcoin-qt`) were tested on FreeBSD 14.4 locally.
Tree-SHA512: 52a3edaa56fe40ca901416cb9e1af04a84505526edfa7309bfa40024baa7d3b1a05303659553d9fbcf1f49d4e3d42b415a1e2523d448b22724d1415a49331259
The GHA Windows image contains two vcpkg installations: one vendored by
VS (`VCPKG_ROOT`) and an independent one (`VCPKG_INSTALLATION_ROOT`).
Our build documentation recommends using the VS-vendored instance to
minimize user effort. While this option previously failed to build in
the CI, forcing us to use the independent installation, it is now
working correctly.
This change removes the no-longer-needed workaround.
111864ac30 qa: Avoid duplicating output in case the diff is the same (Hodlinator)
c2e28d455a ci: Enable `wallet_multiwallet.py` in "Windows, test cross-built" job (Hodlinator)
850a80c199 qa: Disable parts of the test when running under Windows or root (Hodlinator)
fb803e3c79 qa: Test scanning errors individually (Hodlinator)
ed43ce57cc qa: Check for platform-independent part of error message (Hodlinator)
64a098a9b6 refactor(qa): Break apart ginormous run_test() (Hodlinator)
bb1aff7ed7 move-only(qa): Move wallet creation check down to others (Hodlinator)
d1a4ddb58e refactor(qa): Lift out functions to outer scopes (Hodlinator)
c811e47367 scripted-diff: self.nodes[0] => node (Hodlinator)
73cf858911 refactor(qa): Remove unused option (Hodlinator)
Pull request description:
Makes the functional test compatible with *Linux->Windows cross-built executables*.
Main parts:
* Commit "qa: Check for platform-independent part of error message" switches to match on platform-independent part of error message.
* Commit "qa: Test scanning errors individually" disentangles code causing the same error message substring, based on #31410.
* Commit "qa: Disable parts of the test when running under Windows or root" enables the test to be run on Windows, based in part on https://github.com/bitcoin/bitcoin/pull/31410#issuecomment-3554721014.
Also:
* Removes unused option in wallet_multiwallet.py.
* Breaks apart wallet_multiwallet.py's `run_test()` into smaller test functions.
* Improves `assert_equal()` output for dicts.
Fixes#31409.
ACKs for top commit:
achow101:
ACK 111864ac30
janb84:
re ACK 111864ac30
w0xlt:
reACK 111864ac30
Tree-SHA512: 4e3ff92588ac9f2611fc963ce62097b6c0dd4d4eb8da7952c72619c7b554ff3cae5163fe1886d4d9bbd7af1acca5b846411e7f5b46f9bddb08719b61108efbba
89386e700e kernel: Use fs:: namespace and unicode path in kernel tests (sedited)
Pull request description:
Add support for unicode characters in paths to the kernel tests by using our fs:: wrappers for std::filesystem calls and adding the windows application manifest to the binary. This exercises their handling through the kernel API.
ACKs for top commit:
hebasto:
ACK 89386e700e.
w0xlt:
ACK 89386e700e
Tree-SHA512: 7b541f482d84a66c89eec63aea0e7f7626bbbd62082ad7a7fb2c7a517296c291a6ff301c628e5e9e1d7b850ead89005141481a2bfd06d8a9081622e32f7340cc
Add support for unicode characters in paths to the kernel tests by using
our fs:: wrappers for std::filesystem calls and adding the windows
application manifest to the binary. This exercises their handling
through the kernel API.
fa7612f253 ci: Download script_assets_test.json for Windows CI (MarcoFalke)
7777a13306 test: Move Fetching-print to download_from_url util (MarcoFalke)
faf96286ce test: move-only download_from_url to stand-alone util file (MarcoFalke)
fa0cc1c5a4 test: [doc] Remove outdated comment (MarcoFalke)
Pull request description:
Fixes https://github.com/bitcoin/bitcoin/issues/34670 by adding a new `download_script_assets` Python helper and calling it.
ACKs for top commit:
hebasto:
re-ACK fa7612f253.
janb84:
re ACK fa7612f253
hodlinator:
utACK fa7612f253
Tree-SHA512: 73c2cb3a31f231174566fb880b82de92734b1679ef000f8d793d774b7e5f5a7b8c7994a3998ca78821115bdc80c16aada69cf596e92c083cf9b9a95c7cee16ea
fa36adeb71 ci: [refactor] Drop last use of pwsh (MarcoFalke)
fae31b1e2f ci: [refactor] Move github_import_vs_env to python script (MarcoFalke)
Pull request description:
The use of pwsh was a bit confusing and inconsistent around the exit code. See also https://github.com/bitcoin/bitcoin/pull/32672
I think it is fine to drop it and purely use Bash/Python.
This also moves a bit of code from the github yaml to the python script.
ACKs for top commit:
m3dwards:
Looks good! re-ACK fa36adeb71
hodlinator:
re-ACK fa36adeb71
Tree-SHA512: 78edffc60c58c476b0acca5224150169d154b0b818114844a04295af9ba19b7cdf1fb2afb738f6cafd6172f0f477d546018ebf95061eb5bd8bbb35e065a129d4
The standard and fuzz matrix jobs share the same github.job value
(windows-native-dll), so both try to save the vcpkg tools cache with the
same key.
Since the tools are identical across build types, let them share a
single cache entry by restricting the save to the standard job only.
eb510f8678 ci: fail fast in test-each-commit script (Lőrinc)
04c4d71008 ci: remove commit count limit from `test-each-commit` (Lőrinc)
Pull request description:
### Problem
`test-each-commit` currently tests only a limited number of ancestor commits in a PR, so failures introduced deeper in the commit stack might be missed.
### Fix
Remove the max-count limit so `test-each-commit` runs the full build + unit + functional test flow on every non-head PR commit, while keeping the PR tip excluded because it is already covered by the normal CI jobs.
It will also stop after the first failure to surface the root cause sooner and keep logs readable when testing ancestor commits.
### Examples
* Example failure 10 commits deep: https://github.com/l0rinc/bitcoin/actions/runs/21390976651/job/61577575033?pr=105 in https://github.com/l0rinc/bitcoin/pull/105
* Example pass with >7 dummy commits: https://github.com/l0rinc/bitcoin/actions/runs/21392557521/job/61595159841?pr=106 in https://github.com/l0rinc/bitcoin/pull/106
---------
Note: this PR has gone through a few iterations, the latest one just extends the existing job.
ACKs for top commit:
maflcko:
lgtm ACK eb510f8678🕓
hebasto:
re-ACK eb510f8678.
willcl-ark:
ACK eb510f8678
Tree-SHA512: 5aadafd32daad610ce882277802c390642dc34f7d5bfa71d4b503ee007942d1ebafce2a3430ea5fd2af6673c83f9aee42450043be4722d7c02407d90920f8bce
fa13b13239 ci: [refactor] Use pathlib over os.path (MarcoFalke)
fa2719ab1b ci: [refactor] Move run_unit_tests to ci-windows-cross.py (MarcoFalke)
fa99ba5f14 ci: Set PREVIOUS_RELEASES_DIR env var in ci-windows-cross.py (MarcoFalke)
fa4a1cab6c ci: Move run_functional_tests into ci-windows-cross.py (MarcoFalke)
1111108685 ci: [refactor] Move pyzmq install and get_previous_releases into ci-windows-cross.py (MarcoFalke)
fac9c7bd66 ci: [refactor] Move config.ini rewrite to ci-windows-cross.py (MarcoFalke)
faf7389466 ci: Move check_manifests step to ci-windows-cross.py (MarcoFalke)
fa674d55df ci: [refactor] Move print_version step into ci-windows-cross.py helper (MarcoFalke)
Pull request description:
Currently the ci yaml has a mix of Bash and Pwsh snippets, which is problematic:
* The `shellcheck` tool does not review the Bash
* The ci yaml is not merged with master on re-runs, but the code is, leading to possibly confusing CI errors on re-runs
* The Pwsh isn't reviewed at all by any tool
* It is tedious to run the CI commands locally on Windows
Fix all issues by extracting them into a step-based Python script.
ACKs for top commit:
janb84:
re ACK fa13b13239
hebasto:
ACK fa13b13239, I have reviewed the code and it looks OK.
Tree-SHA512: 23d21d3bfb07e102fe1cc15ba5749d553d9766ae6c4a7648bd77df0705469bd138c76a9a2fdeb4d91d3f889a425b7caf25878ecb2e68b604faf9665f8df4eb6d
The vcpkg tools cache was using the combined actions/cache action,
which saves on every run regardless of branch. Split it into the
restore/save pattern used by the other caches, so that saves only
happen on default branch pushes.
