edde96376a cmake: Respect user-provided configuration-specific flags (Hennadii Stepanov)
Pull request description:
This PR addresses [this](https://github.com/bitcoin/bitcoin/issues/31491#issuecomment-2542140874) comment:
> I suppose that should only happen if the `-O3` isn't coming from an explicitly set `CMAKE_CXX_FLAGS_RELEASE`.
With this PR:
```
$ cmake -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS_RELEASE="-O3"
<snip>
C++ compiler flags .................... -O3 -std=c++20 -fPIC -fno-extended-identifiers -fdebug-prefix-map=/home/hebasto/dev/bitcoin/src=. -fmacro-prefix-map=/home/hebasto/dev/bitcoin/src=. -fstack-reuse=none -Wall -Wextra -Wformat -Wformat-security -Wvla -Wredundant-decls -Wdate-time -Wduplicated-branches -Wduplicated-cond -Wlogical-op -Woverloaded-virtual -Wsuggest-override -Wimplicit-fallthrough -Wunreachable-code -Wbidi-chars=any -Wundef -Wno-unused-parameter -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -Wstack-protector -fstack-protector-all -fcf-protection=full -fstack-clash-protection
Linker flags .......................... -O3 -fstack-reuse=none -fstack-protector-all -fcf-protection=full -fstack-clash-protection -Wl,-z,relro -Wl,-z,now -Wl,-z,separate-code -fPIE -pie
```
and
```
$ cmake -B build -DCMAKE_BUILD_TYPE=Release
<snip>
C++ compiler flags .................... -O2 -std=c++20 -fPIC -fno-extended-identifiers -fdebug-prefix-map=/home/hebasto/dev/bitcoin/src=. -fmacro-prefix-map=/home/hebasto/dev/bitcoin/src=. -fstack-reuse=none -Wall -Wextra -Wformat -Wformat-security -Wvla -Wredundant-decls -Wdate-time -Wduplicated-branches -Wduplicated-cond -Wlogical-op -Woverloaded-virtual -Wsuggest-override -Wimplicit-fallthrough -Wunreachable-code -Wbidi-chars=any -Wundef -Wno-unused-parameter -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -Wstack-protector -fstack-protector-all -fcf-protection=full -fstack-clash-protection
Linker flags .......................... -O2 -fstack-reuse=none -fstack-protector-all -fcf-protection=full -fstack-clash-protection -Wl,-z,relro -Wl,-z,now -Wl,-z,separate-code -fPIE -pie
```
When calling `cmake` repeatedly using the same build directory, each newly provided `CMAKE_CXX_FLAGS_RELEASE` value will be accommodated. In such a scenario, if the user wishes to revert to the build system defaults, they should unset the `CMAKE_CXX_FLAGS_RELEASE` variable by passing `-UCMAKE_CXX_FLAGS_RELEASE` to `cmake`.
---
This PR does not aim to resolve _all_ issues mentioned in https://github.com/bitcoin/bitcoin/issues/31491.
ACKs for top commit:
purpleKarrot:
ACK edde96376a
janb84:
ACK [edde963](edde96376a)
ryanofsky:
Code review ACK edde96376a
Tree-SHA512: 1fbc879bd02cf0be726ced490f65985e728f0686ccb3a32cd38787b56377aa666e1965448e5069515abc814df49a0083c8000bc3f6f322f5f395695638168fb6
When ENABLE_IPC option is on, build with libmultiprocess subtree and
`add_subdirectory(src/ipc/libmultiprocess)` instead of external package
and `find_package(Libmultiprocess)` by default.
Behavior can be toggled with `WITH_EXTERNAL_LIBMULTIPROCESS` option. Using a
subtree should be more convenient for most bitcoin developers, but using an
external package is more convenient for developing in the libmultiprocess
repository.
The `WITH_EXTERNAL_LIBMULTIPROCESS` option is also used to avoid needing to
changing the depends build here. But in later commits, the depends build is
switched to use the add_subdirectory build as well.
Co-authored-by: Cory Fields <cory-nospam-@coryfields.com>
Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
52ac17757e cmake: Add `NO_CACHE_IF_FAILED` option for checking linker flags (Hennadii Stepanov)
Pull request description:
Use it for checking `-fsanitize`.
This change improves the user experience when the configuration step fails due to a missing library. Now, there is no need to manually clean the CMake cache after installing the required library.
Addresses [this](https://github.com/bitcoin/bitcoin/issues/31942#issuecomment-2703801270) comment from https://github.com/bitcoin/bitcoin/issues/31942.
ACKs for top commit:
fanquake:
ACK 52ac17757e
Tree-SHA512: 4004110585413792faa01551cf5a5b3b0de7f213c7a1dd333647107741f84abf626fd0ed067fc17e4c5a523de549432738d3752facf25d1e3dab240be8d13d03
1f9b2e150c cmake: Require `zip` only for `deploy` target (Hennadii Stepanov)
0aeff29951 cmake: Check for `makensis` tool before using it (Hennadii Stepanov)
Pull request description:
For `x86_64-w64-mingw32` and `*-apple-darwin` targets, the optional `deploy` target requires dedicated tools: `makensis` and `zip`, respectively.
This PR introduces a uniform checks for those tools when attempting to build the `deploy` target, ensuring they are not required for configuring and building any other targets.
Here is an example of workflow for `x86_64-w64-mingw32`:
```
$ # `nsis` is not installed
$ cmake -B build -G "GNU Makefiles" --toolchain depends/x86_64-w64-mingw32/toolchain.cmake
$ cmake --build build -j $(nproc)
$ cmake --build build -t deploy
Error: NSIS not found.
Please install NSIS and/or ensure that its executable is accessible to the find_program() command—
for example, by setting the MAKENSIS_EXECUTABLE variable or another relevant CMake variable.
Then re-run cmake to regenerate the build system.
Built target deploy
$ sudo apt install nsis
$ cmake -B build
$ cmake --build build -t deploy
...
[100%] Generating bitcoin-win64-setup.exe
[100%] Built target deploy
```
Fixes https://github.com/bitcoin/bitcoin/issues/32018.
ACKs for top commit:
hodlinator:
re-ACK 1f9b2e150c
fanquake:
ACK 1f9b2e150c
Tree-SHA512: 5e2bd28a13bd8fa7c4ba8cf1756d200a4651afe83c463d76ece10027cca343e124eff97012a5368028f761df60f420ab891106b4e33b50045051d57c7464ff98
36b6f36ac4 build: require sqlite when building the wallet (Sjors Provoost)
Pull request description:
Require that sqlite is available in order to compile the wallet. Removes instances of `USE_SQLITE` since it is no longer possible to not have sqlite available.
The `NO_SQLITE` option is dropped from depends.
This is another step towards dropping the legacy wallet, extracted from #31250.
ACKs for top commit:
m3dwards:
ACK 36b6f36ac4
davidgumberg:
crACK 36b6f36ac4
hebasto:
re-ACK 36b6f36ac4.
Tree-SHA512: 870a0135671c80c4f28602119eb8637a1ed43b51b1673bfe88425782fb62ec6ef0f3d6baf0d5984d6a243779b0f63423fd4c4dc324ef87bffba13d63e05ad793
Require that sqlite is available in order to compile the wallet. Removes
instances of USE_SQLITE since it is no longer possible to not have
sqlite available.
The NO_SQLITE option is dropped from depends.
Co-authored-by: Ava Chow <github@achow101.com>
Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
Use it for checking `-fsanitize`.
This change improves the user experience when the configuration step
fails due to a missing library. Now, there is no need to manually clean
the CMake cache after installing the required library.
ecf54a32ed cmake: Add support for builtin `codegen` target (Hennadii Stepanov)
a8c78a0574 cmake: Revamp handling of data files (Hennadii Stepanov)
Pull request description:
This PR leverages the approach from the https://github.com/chaincodelabs/libmultiprocess project and introduces a new functions `target_json_data_sources()` and `target_raw_data_sources()`, which minimize the amount of code required to assign to assign a `*.json` or `*.raw` data file to the `test_bitcoin`, `bench_bitcoin` or `unitester` targets.
As requested in https://github.com/bitcoin/bitcoin/pull/30901#issuecomment-2654622689, the `codegen` build target is now supported, if available:
```
$ cmake --version
cmake version 3.31.5
CMake suite maintained and supported by Kitware (kitware.com/cmake).
$ cmake -G "Ninja" -B build
$ cmake --build build --target codegen
```
ACKs for top commit:
fjahr:
re-ACK ecf54a32ed
Sjors:
re-tACK ecf54a32ed
theuni:
ACK ecf54a32ed
Tree-SHA512: bab92df6b81c47d9d97ba8db37470a6d7aa435d5578afe40df7154885eda55afc59f0bf20dc9db3b2fd88ceb9a0319b9678f9e9af01e7afd4851ec3a79f3f402
2c4b229c90 cmake: Introduce `FUZZ_LIBS` (Hennadii Stepanov)
ea929c0848 scripted-diff: Rename CMake helper module (Hennadii Stepanov)
8d238c1dfd cmake: Delete `check_cxx_source_links*` macros (Hennadii Stepanov)
71bf8294a9 cmake: Convert `check_cxx_source_compiles_with_flags` to a function (Hennadii Stepanov)
88ee6800c9 cmake: Delete `check_cxx_source_links_with_flags` macro (Hennadii Stepanov)
09e8fd25b1 build: Don't override CMake's default try_compile target (Hennadii Stepanov)
Pull request description:
This was requested in https://github.com/bitcoin/bitcoin/pull/31359#issuecomment-2515287092.
From https://github.com/bitcoin/bitcoin/pull/31359#issuecomment-2511246212:
> (Almost?) every CMake check internally uses the [`try_compile()`](https://cmake.org/cmake/help/latest/command/try_compile.html) command, whose behaviour, in turn, depends on the [`CMAKE_TRY_COMPILE_TARGET_TYPE`](https://cmake.org/cmake/help/latest/variable/CMAKE_TRY_COMPILE_TARGET_TYPE.html) variable:
>
> 1. The default value, `EXECUTABLE`, enables both compiler and linker checks.
>
> 2. The `STATIC_LIBRARY` value enables only compiler checks.
>
>
> To mimic Autotools' behaviour, we [disabled](d3f42fa08f/cmake/module/CheckSourceCompilesAndLinks.cmake (L9-L10)) linker checks by setting `CMAKE_TRY_COMPILE_TARGET_TYPE` to `STATIC_LIBRARY` globally (perhaps not the best design). This effectively separates the entire CMake script into regions where `CMAKE_TRY_COMPILE_TARGET_TYPE` is:
>
> * unset
>
> * set to `STATIC_LIBRARY`
>
> * set to `EXECUTABLE`
From https://github.com/bitcoin/bitcoin/pull/31359#issuecomment-2515287092:
> > This seems very fragile and unintuitive, and the fact that this could silently break at any point is not documented in any way. I don't think other bad design decisions should lead to us having to write even more boilerplate code to fix things that should "just work" (minus the upstream bugs).
>
> Agreed. I forgot that we set `CMAKE_TRY_COMPILE_TARGET_TYPE` globally. And even worse, it's buried in a module. If that upsets CMake internal tests, I think we should undo that.
This PR ensures that `CMAKE_TRY_COMPILE_TARGET_TYPE` is modified only within local scopes.
Additionally, the `FUZZ_LIBS` variable has been introduced to handle additional libraries required for linking, rather than link options, in certain build environment, such as OSS-Fuzz.
ACKs for top commit:
TheCharlatan:
Re-ACK 2c4b229c90
theuni:
utACK 2c4b229c90
Tree-SHA512: f72ffa8f50f216fc1a2f8027ba8ddfd4acd42b94ff6c1cb2138f2da51eb8f945660e97d3c247d7f3f7ec8dfebbccec3ab84347d6ae2e3f8a40f3d7aa8b14cde9
9e4a4b4832 cmake: Check `-Wno-*` compiler options for `leveldb` target (Hennadii Stepanov)
Pull request description:
Otherwise, https://cirrus-ci.com/task/4830737755537408:
```
At global scope:
cc1plus: note: unrecognized command-line option ‘-Wno-conditional-uninitialized’ may have been intended to silence earlier diagnostics
```
ACKs for top commit:
TheCharlatan:
ACK 9e4a4b4832
Tree-SHA512: 05553c80399180e01d45c3f02074ca0ce620011b29b03bef5433b87c9d88fd281fb6bf0203fc6fff590f3780c182a3fab8307002536b6350e03748420c346602
CMake assumes the default value internally, so overriding this causes
problems. The minimal speedup of skipping the linker isn't worth the
complexity of setting it to static.
113a7a363f build: remove ENABLE_HARDENING cond from check-security (fanquake)
Pull request description:
This check is only used in release builds, where hardening should always be enabled. I can't think of a reason we'd want to silently skip these checks if hardening was inadvertently disabled.
ACKs for top commit:
maflcko:
lgtm ACK 113a7a363f
TheCharlatan:
ACK 113a7a363f
hebasto:
ACK 113a7a363f.
Tree-SHA512: 46217e8ab238e23374d758b12e5b6bdc22353d8bf272aa0d2260cdea023b5b80aba972dccaa0a4fb8da21c8c665991848f7fd79966d20ac2489d499c68d95639
9cf746d663 cmake: add optional source files to crc32c directly (Daniel Pfeifer)
9c7823c5b5 cmake: add optional source files to bitcoin_crypto directly (Daniel Pfeifer)
Pull request description:
Avoid having many static libraries by adding the optional sources to the target `bitcoin_crypto` directly.
Set the necessary compile options at the source file level, rather than the target level.
fixes: #31697
ACKs for top commit:
s373nZ:
ACK 9cf746d663
hebasto:
re-ACK 9cf746d663.
TheCharlatan:
ACK 9cf746d663
Tree-SHA512: 04b468ccbd284d63fc83b382177bb8183b325369835c3b92e555e159955c73d71712a63a2e556f8da68a1232ac07d3845e11f1057c50666843db91db98fca979
This check is only used in release builds, where hardening should always
be enabled. I can't think of a reason we'd want to silently skip these
checks if hardening was inadvertently disabled.
81c174e318 cmake: Refer to the configure log instead of printing PIE test error (Hennadii Stepanov)
65a0920ca6 cmake: Add `CheckLinkerSupportsPIE` module (Hennadii Stepanov)
Pull request description:
This new module is a wrapper around CMake's `CheckPIESupported` module that fixes an upstream bug.
See: https://gitlab.kitware.com/cmake/cmake/-/issues/26463.
Fixes https://github.com/bitcoin/bitcoin/issues/30771.
ACKs for top commit:
theuni:
utACK 81c174e318.
vasild:
ACK 81c174e318
Tree-SHA512: 77d7022238551a4e69c59d1fe6b78975bb552cbbed5339459853d7ebf0086813036081f464fed230be330b3bd7d6cf8590b536b064028d2f786d6ae40f342f95
9b033bebb1 cmake: rename Kernel component to bitcoinkernel for consistency (Cory Fields)
2e0c92558e cmake: add and use install_binary_component (Cory Fields)
0264c5d86c cmake: use per-target components for bitcoin-qt and bitcoin-gui (Cory Fields)
fb0546b1c5 ci: don't try to install for a fuzz build (Cory Fields)
Pull request description:
This makes it possible to build/install only the desired binaries regardless of the configuration.
For consistency, the component names match the binary names. `Kernel` and `GUI` have been renamed.
Additionally it fixes#31762 by installing only the manpages for the configured targets (and includes them in the component installs for each).
Also fixes#31745.
Alternative to #31765 which is (imo) more correct/thorough.
Can be tested using (for ex):
```bash
$ cmake -B build
$ cmake --build build -t bitcoind -t bitcoin-cli
$ cmake --install build --component bitcoind
$ cmake --install build --component bitcoin-cli
```
ACKs for top commit:
hebasto:
ACK 9b033bebb1.
TheCharlatan:
Re-ACK 9b033bebb1
stickies-v:
re-ACK 9b033bebb1
Tree-SHA512: fd4818e76f190dbeafbf0c246b466f829771902c9d6d7111ed917093b811c8a5536a4a45e20708f73e7f581d6cb77c8e61cfa69e065788dcf0886792f553a355
e3c0152769 cmake: Copy `cov_tool_wrapper.sh.in` to the build tree (Hennadii Stepanov)
Pull request description:
This PR ensures that `cov_tool_wrapper.sh.in` is available when invoking the `Coverage.cmake` script from any directory.
Here is an example of usage on Ubuntu 24.10 with the default GCC 14.2.0:
```
$ cmake -B build -DCMAKE_BUILD_TYPE=Coverage -DCMAKE_CXX_FLAGS="-fprofile-update=atomic" -DCMAKE_C_FLAGS="-fprofile-update=atomic"
$ cmake --build build -j $(nproc)
$ cd ..
$ cmake -DJOBS=$(nproc) -DLCOV_OPTS="--ignore-errors inconsistent,inconsistent" -P bitcoin/build/Coverage.cmake
```
Fixes https://github.com/bitcoin/bitcoin/issues/31638.
ACKs for top commit:
theuni:
utACK e3c0152769
Tree-SHA512: ccfc8e893567f199d9b05ea3916cac06fca89c5892cc7492d5251c331c35408222fd918ed08017515e2dfca10970ae3f633b3917bfb7037db539559e71d7f711
12fa9511b5 build: simplify dependency graph (Cory Fields)
c4e498300c build: avoid unnecessary dependencies on generated headers (Cory Fields)
Pull request description:
These changes speed up my build (default config/options/targets) by roughly 10%. I suspect the difference may be more significant in other build configs.
Before:
> $ time cmake --build build -j24
> real3m26.932s
After:
> $ time cmake --build build -j24
> real3m7.556s
Generally they allow for jobservers (either `make -jX` or `ninja`) to be better utilized. This can be verified using `top` while building and looking at the number of compiles running at any given time before/after these changes. Before, it's easy to observe periods of stalling when only one or two compiles are happening. After these changes, the compiler process count should mostly match the number of jobs given (`-jX`) until it falls off at the end.
---
The first commit sets [DEPENDS_EXPLICIT_ONLY](https://cmake.org/cmake/help/latest/command/add_custom_command.html#command:add_custom_command) for commands which generate our test header files. Without this option, `test_bitcoin`'s generated headers won't be built until all of its other dependencies have been built. This introduces a significant stall in the build, though currently only Ninja benefits from this being set, and only CMake >= 3.27 understands it.
Example from a generated `build.ninja`:
Before:
> \# Custom command for src/test/data/base58_encode_decode.json.h
>
> build src/test/data/base58_encode_decode.json.h | ${cmake_ninja_workdir}src/test/data/base58_encode_decode.json.h: CUSTOM_COMMAND /home/cory/dev/bitcoin/src/test/data/base58_encode_decode.json /home/cory/dev/bitcoin/cmake/script/GenerateHeaderFromJson.cmake || libcrc32c.a libcrc32c_sse42.a libleveldb.a libminisketch.a minisketch_clmul src/bitcoin_clientversion src/crypto/libbitcoin_crypto.a src/crypto/libbitcoin_crypto_avx2.a src/crypto/libbitcoin_crypto_sse41.a src/crypto/libbitcoin_crypto_x86_shani.a src/generate_build_info src/libbitcoin_cli.a src/libbitcoin_common.a src/libbitcoin_consensus.a src/libbitcoin_node.a src/secp256k1/src/libsecp256k1.a src/secp256k1/src/secp256k1_precomputed src/test/util/libtest_util.a src/univalue/libunivalue.a src/util/libbitcoin_util.a src/wallet/libbitcoin_wallet.a src/zmq/libbitcoin_zmq.a
After:
> \# Custom command for src/test/data/base58_encode_decode.json.h
>
> build src/test/data/base58_encode_decode.json.h | ${cmake_ninja_workdir}src/test/data/base58_encode_decode.json.h: CUSTOM_COMMAND /home/cory/dev/bitcoin/src/test/data/base58_encode_decode.json /home/cory/dev/bitcoin/cmake/script/GenerateHeaderFromJson.cmake
---
The second commit is more significant. It sets [CMAKE_OPTIMIZE_DEPENDENCIES](https://cmake.org/cmake/help/latest/prop_tgt/OPTIMIZE_DEPENDENCIES.html) globally, which allows the objects of static libs to be built in parallel when one lib depends on the other. This can be set as a per-lib property, ~but I don't see any need for that as we don't currently have any edge-cases where this wouldn't be ok. If those should arise, we could always disable on a per-lib basis~.
Edit: turns out this triggers an [upstream bug](https://gitlab.kitware.com/cmake/cmake/-/issues/24058), which I guess can be considered an edge-case until fixed in CMake. I've added 2 per-lib opt-outs as a result.
Example:
Before:
> \# Link the static library src/libbitcoin_cli.a
>
> build src/libbitcoin_cli.a: CXX_STATIC_LIBRARY_LINKER__bitcoin_cli_RelWithDebInfo src/CMakeFiles/bitcoin_cli.dir/compat/stdin.cpp.o src/CMakeFiles/bitcoin_cli.dir/rpc/client.cpp.o || src/univalue/libunivalue.a
After:
> \# Link the static library src/libbitcoin_cli.a
>
> build src/libbitcoin_cli.a: CXX_STATIC_LIBRARY_LINKER__bitcoin_cli_RelWithDebInfo src/CMakeFiles/bitcoin_cli.dir/compat/stdin.cpp.o src/CMakeFiles/bitcoin_cli.dir/rpc/client.cpp.o
>
ACKs for top commit:
l0rinc:
utACK 12fa9511b5
hebasto:
ACK 12fa9511b5.
Tree-SHA512: f85f507e70cdc06acd07542161d9f9b8edf9ba866f08c8ef17aaaed770fa11530a27521c4413456d863463a6e77d4d6983fa623a64e17bbd602c2bc70aacc112
56a9b847bb build: set build type and per-build-type flags as early as possible (Cory Fields)
f605f7a9c2 build: refactor: set debug definitions in main CMakeLists (Cory Fields)
Pull request description:
This ensures that most compiler tests are not run with the wrong build type's flags. The initial c++ checks are an exception to that because many internal CMake variables are unset until a language is selected, so it's problematic to change our build type before that.
The difference can be seen in `build/CMakeFiles/CMakeConfigureLog.yaml`. Before, `Debug` was used for many of the earlly checks. After this PR, it's only the first 2 checks.
ACKs for top commit:
hebasto:
ACK 56a9b847bb.
Tree-SHA512: 87947352d6d4fd08554515822cb13634ed3be33fcda2af817c22ef943b1d0856ceb39311ddc01b8221397528fdc09f630dc7e74fc92f5a4a073f09c4ae493596
With the exception of the first c++ checks, this ensures that compiler tests
are never run with the wrong build type's flags.
Co-Authored-By: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
These scripts are becoming more of nuisance, than a value-add;
particularly since we've been building releases using Guix. Adding new
(release bin) tests can be harder, because it requires constructing a
failing test, which is becoming less easy e.g trying to disable a
feature or protection that has been built into the compiler/toolchain by
default.
In the pre-Guix days, these were valuable to sanity-check the environment,
because we were pulling that pre-built from Ubuntu, with little control.
At this point, it's less clear what these scripts are (sanity) checking.
Note that these also weren't completely ported to CMake (#31698), see
also #31715 which contains other fixes that would be needed for these
test-tests, to accomodate future changes.
This prevents the generation of these headers from also depending on the
dependencies of the libs/binaries which consume them.
Specifically, this prevents generated test headers (such as
test/data/base58_encode_decode.json.h) from depending on the
dependencies of test_bitcoin (libcrc32c.a libcrc32c_sse42.a libleveldb.a)
Note that this is currently only relevant for Ninja.
For more detail, see:
https://cmake.org/cmake/help/latest/command/add_custom_command.html
Use character literals instead of integer hex values (i.e. `'\x5b','\x0a', ...` instead of `0x5b, 0x0a, ...`) for generated headers.
This avoids C++11 narrowing warnings in a more concise way than using explicit char casts.
Extra whitespace is also removed between elements for brevity.
