52149b7a2c2b48ed4a4c0900c74cda4bb52a1ea5 contrib: Fix clang-format-diff.py lint errors (TheCharlatan)
008e81e025d64c33e1e71b5a2fe63dfdf6b31437 contrib: Latest clang-format-diff.py script (TheCharlatan)
Pull request description:
This was taken from 900bb318b5/clang/tools/clang-format/clang-format-diff.py and is useful for systems where clang tools are shipped with a version suffix.
ACKs for top commit:
maflcko:
lgtm ACK 52149b7a2c2b48ed4a4c0900c74cda4bb52a1ea5 🌱
Tree-SHA512: cad720b283650e36c5b3ec597278112899ae6413a42c79b6296c58242000f32ae6ab7ed732a3a0f30f1f2586d7e3c0d1ef856a4821e28272bae6f428ed9497e9
We assume to be using python3, so don't check for it. This removes a
type error on the line `from io import BytesIO as StringIO`.
Specify the encoding as "utf8" when opening a file.
5335e454c0889c8a1bb05aa09435883322133974 contrib: add macho branch protection check (fanquake)
Pull request description:
Followup to https://github.com/bitcoin/bitcoin/pull/28459. Add a sanity check that `bti` instructions are present in the arm macho binary, similar to our x86_64 check for control flow.
Could do something similar for aarch64 linux in future, and maybe could use https://github.com/lief-project/LIEF/issues/975.
ACKs for top commit:
TheCharlatan:
ACK 5335e454c0889c8a1bb05aa09435883322133974
Tree-SHA512: 6cc8721209fe07fe07f0524ef6f114004e2b98844f73d31ff16547f7055c7cb4a5609480058c45ede21b457b2dea5357f1475eaa5063ea1f9772aa260f49039b
This was take from
900bb318b5/clang/tools/clang-format/clang-format-diff.py
Updating it introduces some new options. For example specifying the
clang-format binary, which is useful for systems where clang tools are
shipped with a version suffix.
1f8450f066724dfbb5c5bc4060843e2f3340ed88 doc: upgrade Bitcoin Core license to 2024 (22388o⚡️)
Pull request description:
See https://github.com/bitcoin/bitcoin/pull/26748.
Cherry-picked these commits from 22388o and then squashed them.
ACKs for top commit:
fanquake:
ACK 1f8450f066724dfbb5c5bc4060843e2f3340ed88
Tree-SHA512: 6d12f24a6c7cd421f7d975d7e445de4583144a4d0902d4e68e7648395074ca804e3ee585b2d3f307d193690e2ed2f9fbd8e7938f8fb8af263888b8701993782a
a395218d8cdffba375006590edca58c844126b16 ci, iwyu: Drop backported mappings (Hennadii Stepanov)
Pull request description:
See https://github.com/include-what-you-use/include-what-you-use/pull/1026.
Split from https://github.com/bitcoin/bitcoin/pull/27710 as a non-controversial change.
ACKs for top commit:
fanquake:
ACK a395218d8cdffba375006590edca58c844126b16
Tree-SHA512: ae7955a99396ab4f62ab7c989dba59c26448837f0ba4436bf3fddebe4099b5d3c03492e22a8497104c6afcceede1bb1b81f9d71c7c7e43692e6d70dcdfc11e7c
4fdd836db92e789c98b9e68398ca931a968cc9c3 Use hardened runtime on macOS release builds. (Mark Friedenbach)
Pull request description:
The Apple notary service requires submitted app bundles to be configured to use the hardened runtime libraries. This is configured at signing time, and supported by the signapple tool Bitcoin Core uses for reproduceable signed binaries. We simply need to pass "--hardened-runtime" when the signature is created. Once attached to the bundle, the resulting codesigned binary can be successfully submitted to the Apple binary notarization service by any Apple Developer.
This partially resolves#15774. The release maintainer, or any authorized Apple Developer, will need to run `xcrun notarytool` to prevent gatekeeper warnings on macOS. Using `xcrun staple` to generate a binary that doesn't call home on first launch would be bonus, but at least this would massively improve the user experience.
ACKs for top commit:
fanquake:
ACK 4fdd836db92e789c98b9e68398ca931a968cc9c3 - we can move ahead with this, and figure out notarisation / stapling as a followup.
Tree-SHA512: 7b8ba50030fb230d44bd63d12ed082537e8eaaa61396114c5df715f8dd6772fd8d84b00dc819f88d9a463996c2170a84981fce1bde7f7999b4bdb914fbcdfdac
The Apple notary service requires submitted app bundles to be configured to use the hardened runtime libraries. This is configured at signing time, and supported by the signapple tool Bitcoin Core uses for reproduceable signed binaries. We simply need to pass "--hardened-runtime" when the signature is created. Once attached to the bundle, the resulting codesigned binary can be successfully submitted to the Apple binary notarization service by any Apple Developer.
ff896d25819da1c1e80591595c922fb093942645 contrib: drop GCC MAX_VERSION to 4.3.0 in symbol-check (fanquake)
Pull request description:
Reflect the actual symbols used, i.e:
```bash
bitcoind: symbol __bswapsi2 from unsupported version GCC_4.3.0(7)
```
ACKs for top commit:
TheCharlatan:
ACK ff896d25819da1c1e80591595c922fb093942645
Tree-SHA512: b38ff8f4dd78d2d1c9063c53544dc4f240c3043f142e1581f7ba42f088a509293f6f17cc402c60ac82bff3b36668866b87e0e9e4d10d929484bb4c7a3e654f25
11b7269d83a56f919f9dddb7f7c72a96d428627f script: Enhance validations in utxo_snapshot.sh (pablomartin4btc)
Pull request description:
This PR resolves#27841 and some more:
- Ensure that the snapshot height is higher than the pruned block height when the node is pruned (Suggested by @Sjors [here](https://github.com/bitcoin/bitcoin/pull/28553#issuecomment-1804941396)).
- Validate the correctness of the file path and check if the file already exists (@hazeycode's [#27845](https://github.com/bitcoin/bitcoin/pull/27845)).
- Make network activity disablement optional for the user (Suggested by @Sjors [here](https://github.com/bitcoin/bitcoin/pull/16899#discussion_r342735815) and [here](https://github.com/bitcoin/bitcoin/pull/16899#issuecomment-536520911)).
- Ensure the `reconsiderblock` command is triggered on exit (@hazeycode's same PR as above), even in the case of user interruption (Ctrl-C).
In order to perform some testing please follow the instructions in the description of previous @hazeycode's PR #27845.
ACKs for top commit:
Sjors:
tACK 11b7269d83a56f919f9dddb7f7c72a96d428627f
ryanofsky:
Code review ACK 11b7269d83a56f919f9dddb7f7c72a96d428627f
Tree-SHA512: 2b699894c6f732ad5104f5a2bcf5dc86ed31edcc9d664690cab55b94a8ab00e2ca5bde901ee1d63acddca7ea80ad1734d8cfe78f9c02f8470f264fe93a2af759
f95af98128f17002bf137a48441167020f3ef9bb guix: default ssp for Windows GCC (fanquake)
95d55b96c2cfd2a0d5a246d4a9eff9d0744ba223 guix: remove ssp workaround from Windows GCC (fanquake)
8f43302a0a1bb79129933e4cc174bf8d8d59ec15 build: remove explicit libssp linking from Windows build (fanquake)
Pull request description:
I was expecting this to fail to compile somewhere, maybe in the CI, but that doesn't seem to be the case?
Seems workable given the SSP related changes in the newer mingw-w64 headers (which are in Guix):
> Implement some of the stack protector functions/variables so -lssp is now optional when _FORTIFY_SOURCE or -fstack-protector-strong is used.
However I think this would still be broken in some older environments, so we might have to wait for a compiler bump, or similar. The optional -lssp also seems to work when using older headers, which doesn't make sense.
Would fix#28104.
ACKs for top commit:
hebasto:
ACK f95af98128f17002bf137a48441167020f3ef9bb, I've verified binaries from `bitcoin-f95af98128f1-win64.zip` on Windows 11 Pro 23H2.
TheCharlatan:
ACK f95af98128f17002bf137a48441167020f3ef9bb
Tree-SHA512: 71169ec513cfe692dfa7741d2bf37b45da05627c0af1cbd50cf8c3c04cc21c4bf88f3284532bddc1e3e648391ec78dbaca5170987a13c21ac204a7bcaf27f349
fa552e8a4e4fc61b01eb36387bdf86b27c6893c3 doc: Simplify guix install doc, after 1.4 release (MarcoFalke)
Pull request description:
Now that 1.4 is out (for a while), remove the recommendation to build a random commit.
ACKs for top commit:
fanquake:
ACK fa552e8a4e4fc61b01eb36387bdf86b27c6893c3
hebasto:
ACK fa552e8a4e4fc61b01eb36387bdf86b27c6893c3.
Tree-SHA512: f5642df201ff0e2af8a7ae9660a66920ddbb5f522b3e921f6f4aa7c411ced23afa91bdfe43b943ac012228eebbaad3396df505d00aa8f721a4358f03fda9d8e3
a478c817b2f62b7334b36e331a2e37fe8380c754 test: replace `Callable`/`Iterable` with their `collections.abc` alternative (PEP 585) (stickies-v)
4b9afb18e6b9e16d7b299820f3a1382986a451d4 scripted-diff: use PEP 585 built-in collection types for verify-binary script (Sebastian Falbesoner)
d516cf83ed2da86dfefb395cd46f8a894907b88e test: use built-in collection types for type hints (Python 3.9 / PEP 585) (Sebastian Falbesoner)
Pull request description:
With Python 3.9 / [PEP 585](https://peps.python.org/pep-0585/), [type hinting has become a little less awkward](https://docs.python.org/3.9/whatsnew/3.9.html#type-hinting-generics-in-standard-collections), as for collection types one doesn't need to import the corresponding capitalized types (`Dict`, `List`, `Set`, `Tuple`, ...) anymore, but can use the built-in types directly (see https://peps.python.org/pep-0585/#implementation for the full list).
This PR applies the replacement for all Python scripts (i.e. in the contrib and test folders) for the basic types, i.e.:
- typing.Dict -> dict
- typing.List -> list
- typing.Set -> set
- typing.Tuple -> tuple
For an additional check, I ran mypy 1.6.1 on both master and the PR branch via
```
$ mypy --ignore-missing-imports --explicit-package-bases $(git ls-files "*.py")
```
and verified that the output is identical -- (from the 22 identified problems, most look like false-positives, it's probably worth it to go deeper here and address them in a follow-up though).
ACKs for top commit:
stickies-v:
ACK a478c817b2f62b7334b36e331a2e37fe8380c754
fanquake:
ACK a478c817b2f62b7334b36e331a2e37fe8380c754
Tree-SHA512: 6948c905f6abd644d84f09fcb3661d7edb2742e8f2b28560008697d251d77a61a1146ab4b070e65b0d27acede7a5256703da7bf6eb1c7c3a897755478c76c6e8
Running the miner under python >= 3.12 causes a SyntaxWarning. The problem was
already present in previous versions, but it only triggered a
DeprecationWarning, which was not shown by default.
The change is useful for future-proofing the code base, since future python
versions will start to exit with a runtime exception (see the reference given
later).
Command to see the warning at runtime under python3.11 (DeprecationWarning,
needs "-Walways"):
$ python3.11 -Walways ./contrib/signet/miner
<BASE>/contrib/signet/miner:33: DeprecationWarning: invalid escape sequence '\d'
RE_MULTIMINER = re.compile("^(\d+)(-(\d+))?/(\d+)$")
2023-11-15 16:02:49 ERROR Must specify command
Command to see the warning at runtime under python3.12 (SyntaxWarning, no
modifiers needed):
$ python3.12 ./contrib/signet/miner
<BASE>/contrib/signet/miner:33: SyntaxWarning: invalid escape sequence '\d'
RE_MULTIMINER = re.compile("^(\d+)(-(\d+))?/(\d+)$")
2023-11-15 16:03:00 ERROR Must specify command
Reference ( https://docs.python.org/3.8/library/re.html ):
Regular expressions use the backslash character ('\') [...]. This collides
with Python’s usage of the same character for the same purpose in string
literals; [...]
Also, please note that any invalid escape sequences in Python’s usage of the
backslash in string literals now generate a DeprecationWarning and in the
future this will become a SyntaxError.
The solution is to use Python’s raw string notation for regular expression
patterns;
f718a74b124c723548f5d1961ef4e3aa15c33847 guix: remove python-macholib (fanquake)
d3cbff16c2734f310ccc532a4ad8eaf8a21b010d guix: update signapple (fanquake)
Pull request description:
Update to the latest signapple, which includes https://github.com/achow101/signapple/pull/13.
Drop python-macholib and python-altgraph.
ACKs for top commit:
Sjors:
ACK f718a74b124c723548f5d1961ef4e3aa15c33847
Tree-SHA512: 199b2108f2f063b6b0fb5354ac79a30b46e848c923ebe7d02f7d7d3f08749817a1f6b4c14d21658fd2f2d68f8be1698e1999edf7e2366b1cae3bf2709a665e30
3c61c60b90db1b6a77b3804784430fcd57b447b6 build: Add an old hack to remove bind_at_load from libtool. (Cory Fields)
45257601da4cf38f081d9af2c18cd0be7174bcef build: remove -bind_at_load usage (fanquake)
Pull request description:
This is deprecated on macOS:
```bash
ld: warning: -bind_at_load is deprecated on macOS
```
and likely redundant anyways, given the behaviour of dyld3.
Unfortunately libtool is still injecting a `-bind_at_load`, because it's version check is broken:
```bash
# Don't allow lazy linking, it breaks C++ global constructors
# But is supposedly fixed on 10.4 or later (yay!).
if test CXX = "$tagname"; then
case ${MACOSX_DEPLOYMENT_TARGET-10.0} in
10.[0123])
func_append compile_command " $wl-bind_at_load"
func_append finalize_command " $wl-bind_at_load"
;;
esac
fi
```
so this adds another change to strip them out at the end of configure.
Note that anywhere the ld64 warnings are being emitted, we are already not adding this flag to our hardened ldflags, because of `-Wl,-fatal_warnings`.
ACKs for top commit:
theuni:
utACK 3c61c60b90db1b6a77b3804784430fcd57b447b6.
hebasto:
ACK 3c61c60b90db1b6a77b3804784430fcd57b447b6, tested on macOS Sonoma 14.1.1 (23B81, Apple M1) and Ubuntu 23.10 (cross-compiling for macOS). Also I've verified the actual diff in the `libtool` script.
Tree-SHA512: 98e6a095dc2d2409f8ec3b9d462e0db3643d7873d7903a12f8acd664829e7e84e797638556fa42ca8ebc1003f13a38fe9bb8a2a50cecfa991155da818574bf08
- Ensure that the snapshot height is higher than the pruned block height when the node is pruned.
- Validate the correctness of the file path and check if the file already exists.
- Make network activity disablement optional for the user.
- Ensure the reconsiderblock command is triggered on exit, even in the case of user interruption (Ctrl-C).
Co-authored-by: Chris Heyes <22148308+hazeycode@users.noreply.github.com>
Co-authored-by: Sjors Provoost <sjors@sprovoost.nl>
This is deprecated on macOS:
```bash
ld: warning: -bind_at_load is deprecated on macOS
```
and likely redundant anyways, given the behaviour of dyld3.
Unfortunately libtool is still injecting a `-bind_at_load`:
```bash
# Don't allow lazy linking, it breaks C++ global constructors
# But is supposedly fixed on 10.4 or later (yay!).
if test CXX = "$tagname"; then
case ${MACOSX_DEPLOYMENT_TARGET-10.0} in
10.[0123])
func_append compile_command " $wl-bind_at_load"
func_append finalize_command " $wl-bind_at_load"
;;
esac
fi
```
so this doesn't remove all the warnings, but removes us as a potential
source of them.
Note that anywhere the ld64 warnings are being emitted, we are already
not adding this flag to our hardened ldflags, because of `-Wl,-fatal_warnings`.
79539fbfbf4d09a8b4861ddcba5b194297bc1b65 guix: update signapple (fanquake)
Pull request description:
Fixes#28449, and removes the need to boostrap Rust, by avoiding the `python-requests` dependency.
Comparing a `--no-substitutes` build of this PR, to master, signapple requires ~1350 _less_ packages to boostrap:
Master derivation - https://gist.github.com/fanquake/dbf69a62c9a78b7ae8c183a160e6d58d
PR derivation - https://gist.github.com/fanquake/0aa2d8eddaba861ba489ed3d936f727d
ACKs for top commit:
achow101:
ACK 79539fbfbf4d09a8b4861ddcba5b194297bc1b65
Tree-SHA512: 341ddcae27e53c31d114465cb5173573dcc9e1c0874ee160715630f686da6f69255f6080ec0181ffeffc26efbdb545599d667784b1cd17dfa7e3da0998ec9bd6
The zip for codesigned MacOS distribution needs to have all files have
the same timestamp. These files also need to be included in the zip as
zip is not automatically recursive. We use the same pattern for zip as
is done for the other zip files produced by guix.
Since Python 3.9, type hinting has become a little less awkward, as for
collection types one doesn't need to import the corresponding
capitalized types (`Dict`, `List`, `Set`, `Tuple`, ...) anymore, but can
use the built-in types directly. [1] [2]
This commit applies the replacement for all Python scripts (i.e. in the
contrib and test folders) for the basic types:
- typing.Dict -> dict
- typing.List -> list
- typing.Set -> set
- typing.Tuple -> tuple
[1] https://docs.python.org/3.9/whatsnew/3.9.html#type-hinting-generics-in-standard-collections
[2] https://peps.python.org/pep-0585/#implementation for a list of type
348e79f7c6030c9bcbec6bd25e246a779081152a lint: Include test_utxo_snapshots in lint_shell (Fabian Jahr)
Pull request description:
jamesob excluded `test_utxo_snapshots.sh` from the shell linter with this explanation: "Add the script to the shellcheck exception list since the quoted variables rule needs to be violated in order to get bitcoind to pick up on $EARLY_IBD_FLAGS." However, macrofake pointed out that single lines can be excluded from linting.
This fixes one fixable rule violation, excludes the rest of the offending lines from the linter and then removes the exclusion of the `test_utxo_snapshots.sh` file. Also adds documentation.
ACKs for top commit:
Empact:
ACK 348e79f7c6030c9bcbec6bd25e246a779081152a
maflcko:
lgtm ACK 348e79f7c6030c9bcbec6bd25e246a779081152a
pablomartin4btc:
tACK 348e79f7c6030c9bcbec6bd25e246a779081152a
Tree-SHA512: a904cc1cc3c94488dfbd39ea69a3ef17259f991708a797009001669448fef81eed086ecbce1ec433988d88baef293849698e2e0eb86a969b949cc7ef93af7b4b
092daa2f9524e371ee6b505519d9b722a2c6361a contrib: add test for macOS linker version to symbol-check (fanquake)
cefbf0bc20dfe932240877b3096300c8c241ad61 depends: update LD64_VERSION to 711 (fanquake)
Pull request description:
I forgot to do this in 7d5815293e.
Add a test so it's impossible to forget.
ACKs for top commit:
TheCharlatan:
utACK 092daa2f9524e371ee6b505519d9b722a2c6361a
achow101:
ACK 092daa2f9524e371ee6b505519d9b722a2c6361a
jarolrod:
ACK 092daa2f9524e371ee6b505519d9b722a2c6361a
hebasto:
ACK 092daa2f9524e371ee6b505519d9b722a2c6361a.
laanwj:
ACK 092daa2f9524e371ee6b505519d9b722a2c6361a
Tree-SHA512: 37f0bdfd6607a7760eabe5efe279532ba0c59c0915161e08d5e3b9a0b7705839d62537d6e17406062f6a0a1db5407575da7cd671e9cb916e422e77f5649c6e2b
