This should fix the macOS-cross build on Cirrus CI containers.
Locally this was already working, because the SDK was cached in
/ci_container_base/ in the image, which is also the folder used for a
later CI run.
However, on Cirrus CI, when using an image *and* a custom BASE_ROOT_DIR,
the SDK will not be found in /ci_base_install/, nor in BASE_ROOT_DIR.
Fix this by normalizing *all* folders to /ci_container_base/.
faaba770e11352ddf6414b9855f4baa46a967580 Sort includes in compat.h (MarcoFalke)
fa91a23d63dcbf222bd70d05ed49cf1f8e4edce0 remove unused limits.h include in compat.h (MarcoFalke)
fa32af22b323e7c58b6b20af6517f4795a72cdc5 Replace LocaleIndependentAtoi with ToIntegral (MarcoFalke)
faab76c1c01e6d3fff8ac1bc71baeecd8846dc32 iwyu on torcontrol (MarcoFalke)
fa0a60dd93e4485c1f62ffcc87fa9e6b195ce795 Remove unused boost signals2 from torcontrol (MarcoFalke)
Pull request description:
Remove unused boost, and other includes, and other legacy functions from torcontrol.
ACKs for top commit:
TheCharlatan:
Re-ACK faaba770e11352ddf6414b9855f4baa46a967580
achow101:
ACK faaba770e11352ddf6414b9855f4baa46a967580
dergoegge:
utACK faaba770e11352ddf6414b9855f4baa46a967580
Tree-SHA512: 440f8d3ae9c3cf4dcc368e35b29459b5fcec8c6d233e8f9be3a854e7624b8633d6ccdde10cb0c6f74f86278e06557c4e9e24de30c3c692826237939265c6160a
9658d0dc17c270138523c41a982425e276b24271 ci: Run "macOS native x86_64" job on GitHub Actions (Hennadii Stepanov)
Pull request description:
From https://github.com/bitcoin/bitcoin/issues/28098:
> Thus, someone would have to sponsor an amount of roughly 5kUSD/mo for those two tasks.
> If the goal is to stay on a free plan, I think the only option is GitHub Actions CI.
---
**IMPORTANT NOTE**. We currently ship macOS release binaries for both architectures: `x86_64` and `arm64`. If this PR gets merged, only `x86_64` architecture will be tested on CI, which implies some [drawbacks](https://github.com/bitcoin/bitcoin/pull/28187#issuecomment-1658077549).
However, it has never been the case that our CI tested both architectures simultaneously. And we hope that GitHub Actions will soon host macOS `arm64` runners.
Historically, we moved from `x86_64` to `arm64` in https://github.com/bitcoin/bitcoin/pull/26388 less than a year ago.
---
Security concerns:
- https://github.com/bitcoin/bitcoin/issues/28098#issuecomment-1651432106
- https://github.com/bitcoin/bitcoin/issues/28098#issuecomment-1651688197
`GITHUB_TOKEN` permissions (from the build log in my personal repo):
```
2023-07-27T07:30:17.8313534Z ##[group]GITHUB_TOKEN Permissions
2023-07-27T07:30:17.8314113Z Contents: read
2023-07-27T07:30:17.8314608Z Metadata: read
2023-07-27T07:30:17.8314957Z Packages: read
2023-07-27T07:30:17.8315233Z ##[endgroup]
```
Comparison of resources:
| Resource | Current, Cirrus CI | Suggested, GitHub Actions |
|---|:-:|:-:|
| CPU | 4 | 4 \*\* |
| RAM, GB | 8 | 14 |
**\*\* NOTE**: However, [docs](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources) are mentioning:
> 3-core CPU (x86_64)
ACKs for top commit:
MarcoFalke:
re-ACK 9658d0dc17c270138523c41a982425e276b24271 🏂
achow101:
ACK 9658d0dc17c270138523c41a982425e276b24271
jarolrod:
ACK 9658d0dc17c270138523c41a982425e276b24271
Tree-SHA512: 6123e68e6784cdf4e53c3e77b435709261db21f09091af2c22e667d3816a305fffb9d617297a5bc1bda18aaba84a6e210cec6a75c52afa7746a3780a67b69865
fa776e61cd64a5ffd9a4be589ab8efeb5421861a Add importmempool RPC (MarcoFalke)
fa20d734a29ba50cd19b78cb4fe39a2d826131b7 refactor: Add and use kernel::ImportMempoolOptions (MarcoFalke)
fa8866990dba7817427977bfe834efdb17114d37 doc: Clarify the getmempoolinfo.loaded RPC field documentation (MarcoFalke)
6888886cecf6665da70b3dc3772b3c12ef06ad76 Remove Chainstate::LoadMempool (MarcoFalke)
Pull request description:
Currently it is possible to import a mempool by placing it in the datadir and starting the node. However this has many issues:
* Users aren't expected to fiddle with the datadir, possibly corrupting it
* An existing mempool file in the datadir may be overwritten
* The node needs to be restarted
* Importing an untrusted file this way is dangerous, because it can corrupt the mempool
Fix all issues by adding a new RPC.
ACKs for top commit:
ajtowns:
utACK fa776e61cd64a5ffd9a4be589ab8efeb5421861a
achow101:
ACK fa776e61cd64a5ffd9a4be589ab8efeb5421861a
glozow:
reACK fa776e61cd64a5ffd9a4be589ab8efeb5421861a
Tree-SHA512: fcb1a92d6460839283c546c47a2d930c363ac1013c4c50dc5215ddf9fe5e51921d23fe0abfae0a5a7631983cfc7e2fff3788b70f95937d0a989a203be4d67546
fafa17c00b87c685200198979ed13018e55e474a ci: Use hard-coded root path for CI containers (MarcoFalke)
fa084f5ba5dad58a55b5a0a0e9d839c54e9a9238 ci: Only create folders when needed (MarcoFalke)
fab27127f4794b57a00328bdf8e779d9db07a2f5 ci: Drop BASE_SCRATCH_DIR from LIBCXX_DIR (MarcoFalke)
Pull request description:
Currently the CI system will fail if the git folder that holds the Bitcoin Core source is moved from one location to another.
Fix this by using a single hard-coded root path *inside* the CI system containers.
Steps to test:
* Run the CI system: `MAKEJOBS="-j$(nproc)" FILE_ENV="./ci/test/00_setup_env_win64.sh" ./ci/test_run_all.sh`
* Move the git folder: `pwd && cd .. && mv bitcoin_core_folder_1 bitcoin_core_folder_2 && cd ./bitcoin_core_folder_2 && pwd`
* Run the CI system again: (same cmd as above)
On master (error):
```
STRIPPROG="x86_64-w64-mingw32-strip" /bin/bash /bitcoin_core_folder_2/ci/scratch/build/bitcoin-x86_64-w64-mingw32/build-aux/install-sh -c -s ./src/qt/bitcoin-qt.exe ./release
/bitcoin_core_folder_2/ci/scratch/build/bitcoin-x86_64-w64-mingw32/build-aux/install-sh: ./src/qt/bitcoin-qt.exe does not exist.
make: *** [Makefile:1258: bitcoin-25.99.0-win64-setup.exe] Error 1
```
On this pull: (pass).
ACKs for top commit:
fanquake:
ACK fafa17c00b - somewhat tested. MSAN changes are the same as what we did for tidy.
Tree-SHA512: 2ce693a3773c70fcfca062c2a6f0e5a16b94960b34a6145d10cee1a28f79154829d59d014465ccbb80e1cb9dcd5aa043729cee9afd2c4175b05e9bc945364b79
c2a87bd302585c242d0d66db72c6ce6c60f81b78 ci: Drop no longer needed `macos_sdk_cache` (Hennadii Stepanov)
Pull request description:
It has been cached in the Docker image since https://github.com/bitcoin/bitcoin/pull/27028.
ACKs for top commit:
MarcoFalke:
lgtm ACK c2a87bd302585c242d0d66db72c6ce6c60f81b78
Tree-SHA512: fba888d132910f9600db0acccf633400e699f7d5ca802ef109a792546f60f3a04791c503b43d156c8debc3bd0bce2ad911f6209eabc47cd63fce7996a6ae3cfc
93cb8f03807dcd3297b7fe18b6c901a8d2a01596 refactor: add missing headers for BIP324 ciphersuite (stratospher)
d22d5d925c000bf25ad2410ca66c4c21eea75004 crypto: BIP324 ciphersuite follow-up (stratospher)
Pull request description:
follow-up to #28008.
* move `dummy_tag` variable in FSChaCha20Poly1305 crypto_tests outside of the loop to be reused every time
* use easy to read `cipher.last()` in `AEADChaCha20Poly1305::Decrypt()`
* comment for initiator in `BIP324Cipher::Initialize()`
* systematically damage ciphertext with bit positions in bip324_tests
* use 4095 max bytes for `aad` in bip324 fuzz test
ACKs for top commit:
fanquake:
ACK 93cb8f03807dcd3297b7fe18b6c901a8d2a01596 - thanks for following up here.
Tree-SHA512: 361f3e226d3168fdef69a2eebe6092cfc04ba14ce009420222e762698001eaf8be69a1138dab0be237964509c2b96a41a0b4db5c1df43ef75062f143c5aa741a
e417c988f61bf9d3948d5c8e169626922fe6e24c fuzz: coins_view: remove an incorrect assertion (Antoine Poinsot)
c5f6b1db56f67f529377bfb61f58c0a8c17b0127 fuzz: coins_view: correct an incorrect assertion (Antoine Poinsot)
Pull request description:
The `coins_view` fuzz target would assert in two places that the cache is consistent with the backend. But it's never the case (that's the whole point of using a cache).
The only reason this didn't result in a crash was that we would never actually hit these assertions. I ran into this while introducing a new target with an in-memory `CCoinsViewDB` as the backend view (see https://github.com/bitcoin/bitcoin/pull/28216) which made the code paths with those assertions actually reachable.
ACKs for top commit:
dergoegge:
Code review ACK e417c988f61bf9d3948d5c8e169626922fe6e24c
Tree-SHA512: 5847bb2744a2f2831dace62d32b79cc491bf54e2af4ce425411d245d566622d9aff816d9be5ec8e830d10851c13f2500bf4f0c004d88b4d7cca1d483ef8960a6
* PERSISTENT_WORKER_TEMPLATE_ENV is not needed at all, because
RESTART_CI_DOCKER_BEFORE_RUN is already set on the persistent worker.
* PERSISTENT_WORKER_TEMPLATE can be replaced by pinning the
previous_releases task to a type of worker. This should make the CI
performance more consistent.
CTxDestination is really our internal representation of an address and
doesn't really have anything to do with standard script types, so move
them to their own file.
Replaces the constructor in CScriptID that converts a ScriptHash with a
function ToScriptID that does the same. This prepares for a move of
CScriptID to avoid a circular dependency.
5364dd8666ae1afa35536f9b4fc0170eeaf37888 test: locked_wallet, skip default fee estimation (furszy)
Pull request description:
Coming from https://github.com/bitcoin/bitcoin/pull/28139#discussion_r1284563239.
No test case in this file is meant to exercise fee estimation. All default wallets have a
custom tx fee set [here](b7138252ac/test/functional/wallet_fundrawtransaction.py (L100)). The only one missing is the one created for `locked_wallet`.
ACKs for top commit:
theStack:
ACK 5364dd8666ae1afa35536f9b4fc0170eeaf37888
Tree-SHA512: 514c02708081d18330d759d10e306cee16c6350de243c68f0973777d2582f5d81968a237393c1f59aba245297e03f3f98d3ae5249a042469d0d016255f568719
Have each TestNode keep track of the last timestamp it called
setmocktime with, and add a bumpmocktime() function to bump by a
number of seconds. Makes it easy to fast forward n seconds without
keeping track of what the last timestamp was.
bb3263d3e3d9f9d4db86dde679f469e7278bf737 bitcoin-tidy: fix macOS build (Cory Fields)
Pull request description:
[LLVM uses these options](https://github.com/llvm/llvm-project/blob/main/llvm/cmake/modules/HandleLLVMOptions.cmake#L178) for building as well, so there's precedent.
Also fix the shared library extension which was incorrectly being set to dylib.
Thanks to jonatack for reporting and debugging.
ACKs for top commit:
jonatack:
ACK bb3263d3e3d9f9d4db86dde679f469e7278bf737 tested with arm64 macos 13.5, llvm 16.0.6 and cmake 3.27.2
Tree-SHA512: de7bfd497f38f1565a14d217d0b057cbfa788bdda702b5942b7f0b55947ae5e1c05af13e7d6a073ed036bc4db57035868f180034508b6e084ab9b901a5baaf2f
follow-up to #28008.
* move `dummy_tag` variable in FSChaCha20Poly1305 crypto_tests
outside of the loop to be reused every time
* use easy to read `cipher.last()` in `AEADChaCha20Poly1305::Decrypt()`
* comment for initiator in `BIP324Cipher::Initialize()`
* systematically damage ciphertext with bit positions in bip324_tests
* use 4095 max bytes for aad in bip324 fuzz test
d82bb90a5b9dc1fd48b10514bdcd8f425aced256 doc: use llvm-config for bitcoin-tidy example (fanquake)
Pull request description:
An LLVM installation will have `llvm-config` available to query for info. Ask it for the `--cmakedir`, and use that in our bitcoin-tidy example, rather than listing multiple different (potential) paths per distro/OS etc.
ACKs for top commit:
theuni:
ACK d82bb90a5b9dc1fd48b10514bdcd8f425aced256.
jonatack:
ACK d82bb90a5b9dc1fd48b10514bdcd8f425aced256
TheCharlatan:
Nice, Re-ACK d82bb90a5b9dc1fd48b10514bdcd8f425aced256
Tree-SHA512: e07e979231f8f000deafce0751bed4b73ff0eff995bec49e90f579c9051cf5859dac5e49554b8219d33b00c81192db979eed98fee1c643a9205ea8babfce2c5d
An LLVM installation will have `llvm-config` available to query for
info. Ask it for the `--cmakedir`, and use that in our bitcoin-tidy
example, rather than listing multiple different (potential) paths per
distro/OS etc.
1c7582ead6e1119899922041c1af2b4169b0bc74 tests: add decryption test to bip324_tests (Pieter Wuille)
990f0f8da92a2d11828a7c05ca93bf0520b2a95e Add BIP324Cipher, encapsulating key agreement, derivation, and stream/AEAD ciphers (Pieter Wuille)
c91cedf281e5207fb5fd2ca81feec9760f7c2ed0 crypto: support split plaintext in ChaCha20Poly1305 Encrypt/Decrypt (Pieter Wuille)
af2b44c76e5de8ce880381e5535ead37ab0b3ba9 bench: add benchmark for FSChaCha20Poly1305 (Pieter Wuille)
aa8cee93342ee857931afec9af3ff5dbd8ce7749 crypto: add FSChaCha20Poly1305, rekeying wrapper around ChaCha20Poly1305 (Pieter Wuille)
0fee267792eb8cbdd48ad78f1712420b5d8d905b crypto: add FSChaCha20, a rekeying wrapper around ChaCha20 (Pieter Wuille)
9ff0768bdcca06836ccc673eacfa648e801930cb crypto: add the ChaCha20Poly1305 AEAD as specified in RFC8439 (Pieter Wuille)
9fd085a1a49d317abcaf1492b71c48bf1a1b3007 crypto: remove outdated variant of ChaCha20Poly1305 AEAD (Pieter Wuille)
Pull request description:
Depends on #27985 and #27993, based on and partially replaces #25361, part of #27634. Draft while dependencies are not merged.
This adds implementations of:
* The ChaCha20Poly1305 AEAD from [RFC8439 section 2.8](https://datatracker.ietf.org/doc/html/rfc8439#section-2.8), including test vectors.
* The FSChaCha20 stream cipher as specified in [BIP324](https://github.com/bitcoin/bips/blob/master/bip-0324.mediawiki#rekeying-wrappers-fschacha20poly1305-and-fschacha20), a rekeying wrapper around ChaCha20.
* The FSChaCha20Poly1305 AEAD as specified in [BIP324](https://github.com/bitcoin/bips/blob/master/bip-0324.mediawiki#rekeying-wrappers-fschacha20poly1305-and-fschacha20), a rekeying wrapper around ChaCha20Poly1305.
* A BIP324Cipher class that encapsulates key agreement, key derivation, and stream ciphers and AEADs for [BIP324 packet encoding](https://github.com/bitcoin/bips/blob/master/bip-0324.mediawiki#overall-packet-encryption-and-decryption-pseudocode).
The ChaCha20Poly1305 and FSChaCha20Poly1305 implementations are new, taking advance of the improvements in #27993.
ACKs for top commit:
jamesob:
reACK 1c7582e
theStack:
ACK 1c7582ead6e1119899922041c1af2b4169b0bc74
stratospher:
tested ACK 1c7582e.
Tree-SHA512: 06728b4b95b21c5b732ed08faf40e94d0583f9d86ff4db3b92dd519dcd9fbfa0f310bc66ef1e59c9e49dd844ba8c5ac06e2001762a804fb5aa97027816045a46
547fa52443cbb5e8ccfee993486f5ced8cdbb33b net processing: clamp -blockreconstructionextratxn to uint32_t bounds (stickies-v)
e451d1e3c66350017da195335f428a96fdc7d840 net processing: clamp -maxorphantx to uint32_t bounds (stickies-v)
aa89e04e07ca9ff51b1d7d310a11821c6ad963cf doc: document PeerManager::Options members (stickies-v)
Pull request description:
Avoid out-of-bounds user input for `PeerManager::Options` by safely clamping `-maxorphantx` and `-blockreconstructionextratxn`, and avoid platform-specific behaviour by changing `PeerManager::Options::max_extra_txs` from `size_t` to a `uint32_t`. Addresses https://github.com/bitcoin/bitcoin/pull/27499#pullrequestreview-1544114932.
Also documents all `PeerManager::Options` members, addressing https://github.com/bitcoin/bitcoin/pull/27499#discussion_r1272302469.
ACKs for top commit:
dergoegge:
Code review ACK 547fa52443cbb5e8ccfee993486f5ced8cdbb33b
glozow:
reACK 547fa52443cbb5e8ccfee993486f5ced8cdbb33b
Tree-SHA512: 042d47b35bb8a7b29ef3dadd4c0c5d26f13a8f174f33687855d603c19f8de0fcbbda94418453331e149885412d4edd5f402d640d938f6d94b4dcf54e2fdbbcc9
fad0b67c212dcb8a16fcbda5a74acc959ed4e284 ci: Use qemu-user through container engine (MarcoFalke)
Pull request description:
Currently the CI containers always run on the host architecture, and only wrap `bitcoind` into `qemu-user` when needed. This has many issues:
* The `i386` tasks can not be run on non-x86 hosts.
* `config.guess` isn't present when building the CI image, which is fine. But it prints a warning, see https://github.com/bitcoin/bitcoin/pull/27739#pullrequestreview-1446580353
* The python tests are run on the host architecture, making it harder to find architecture specific bugs. See for example https://github.com/bitcoin/bitcoin/pull/27529#issuecomment-1521387810
* All modern container engines support automatic dispatch to qemu-user, so it seems redundant to re-invent the wheel.
Fix all issues by:
* removing `HOST` from `ci/test/00_setup_env.sh`.
* removing `QEMU_USER_CMD` and `ci/test/wrap-qemu.sh`.
* removing `DPKG_ADD_ARCH` where possible, and pruning `PACKAGES` where possible.
* specifying the architecture in `CI_IMAGE_NAME_TAG` to be used by the container engine.
ACKs for top commit:
fanquake:
ACK fad0b67c212dcb8a16fcbda5a74acc959ed4e284 - this seems ok to me, and removes complexity from our CI system.
Tree-SHA512: 85e79f9f570e292d70a629d112fd4a6e6217d96226a1b665ed13485f616d84720ad2126b7d4b22fc603049f72fa7f2163b56a6bc276319fcd8b0496304ea4157
d8f1222ac50f089a0af29eaf8ce0555bad8366ef refactor: Correct dbwrapper key naming (TheCharlatan)
be8f159ac59b9e700cbd3314ed71ebf39bd5b67a build: Remove leveldb from BITCOIN_INCLUDES (TheCharlatan)
c95b37d641b1eed4a62d55ca5342a6ed8c7a1ce7 refactor: Move CDBWrapper leveldb members to their own context struct (TheCharlatan)
c534a615e93452a5f509aaf5f68c600391a98d6a refactor: Split dbwrapper CDBWrapper::EstimateSize implementation (TheCharlatan)
586448888b72f7c87db4dcd30fc4e4044afae13b refactor: Move HandleError to dbwrapper implementation (TheCharlatan)
dede0eef7adb7413f62f5abd68cac8e01635ba4a refactor: Split dbwrapper CDBWrapper::Exists implementation (TheCharlatan)
a5c2eb57484314b04ec94523d14e0ef0c6c46d4f refactor: Fix logging.h includes (TheCharlatan)
84058e0eed9c05bc30984b39131e88ad1425628f refactor: Split dbwrapper CDBWrapper::Read implementation (TheCharlatan)
e4af2408f2ac59788567b6fc8cb3a68fc43da9fe refactor: Pimpl leveldb::Iterator for CDBIterator (TheCharlatan)
ef941ff1281e76308c3e746e592375bec023e9e4 refactor: Split dbwrapper CDBIterator::GetValue implementation (TheCharlatan)
b7a1ab5cb4e60230f62c94efb3a10d07c9af4883 refactor: Split dbwrapper CDBIterator::GetKey implementation (TheCharlatan)
d7437908cdf242626263ba9d5541addcddadc594 refactor: Split dbwrapper CDBIterator::Seek implementation (TheCharlatan)
ea8135de7e617259cda3fc7b1c8e7569d454fd57 refactor: Pimpl leveldb::batch for CDBBatch (TheCharlatan)
b9870c920dc475ec759eaf7339ea42aecba92138 refactor: Split dbwrapper CDBatch::Erase implementation (TheCharlatan)
532ee812a499e13b123af6b8415d8de1f3804f0f refactor: Split dbwrapper CDBBatch::Write implementation (TheCharlatan)
afc534df9adbf5599b286b5dc3531a4b9ac2d056 refactor: Wrap DestroyDB in dbwrapper helper (TheCharlatan)
Pull request description:
Leveldb headers are currently included in the `dbwrapper.h` file and thus available to many of Bitcoin Core's source files. However, leveldb-specific functionality should be abstracted by the `dbwrapper` and does not need to be available to the rest of the code. Having leveldb included in a widely-used header such as `dbwrapper.h` bloats the entire project's header tree.
The `dbwrapper` is a key component of the libbitcoinkernel library. Future users of this library would not want to contend with having the leveldb headers exposed and potentially polluting their project's namespace.
For these reasons, the leveldb headers are removed from the `dbwrapper` by moving leveldb-specific code to the implementation file and creating a [pimpl](https://en.cppreference.com/w/cpp/language/pimpl) where leveldb member variables are indispensable. As a final step, the leveldb include flags are removed from the `BITCOIN_INCLUDES` and moved to places where the dbwrapper is compiled.
---
This pull request is part of the [libbitcoinkernel project](https://github.com/bitcoin/bitcoin/issues/27587), and more specifically its stage 1 step 3 "Decouple most non-consensus headers from libbitcoinkernel".
ACKs for top commit:
stickies-v:
re-ACK d8f1222ac5
MarcoFalke:
ACK d8f1222ac50f089a0af29eaf8ce0555bad8366ef 🔠
Tree-SHA512: 0f58309be165af0162e648233451cd80fda88726fc10c0da7bfe4ec2ffa9afe63fbf7ffae9493698d3f39653b4ad870c372eee652ecc90ab1c29d86c387070f3
11a499eb4d57f2c5eabd7955bcc7e419364b3194 doc: remove Fedora libdb4-*-devel install docs (fanquake)
Pull request description:
These are no-longer installable on any recent Fedora (last working version was 32).
Remove the install instructions, and consolidate this section to be the same as the
Ubuntu & Debian BDB install instructions.
ACKs for top commit:
MarcoFalke:
lgtm ACK 11a499eb4d57f2c5eabd7955bcc7e419364b3194
Tree-SHA512: 11e3c92d6dcf475a6f5529a2e41dc9f79eeae8f8d3600087ce5ae083264f999782a2c04a4c4c70073e96d4053daa23037a344224197ee5f15a3d635172c201e2
