fa1da3d4bfc0511a89f5b19d5a4d89e55ff7ccde test: Add basic addr relay test (MarcoFalke)
fa1793c1c44a3f75a09f9c636467b8274c541bdd net: Pass connman const when relaying address (MarcoFalke)
fa47a0b003f53708b6d5df1ed4e7f8a7c68aa3ac net: Make addr relay mockable (MarcoFalke)
Pull request description:
As usual:
* Switch to std::chrono time to be type-safe and mockable
* Add basic test that relies on mocktime to add code coverage
ACKs for top commit:
naumenkogs:
utACK fa1da3d
promag:
ACK fa1da3d4bfc0511a89f5b19d5a4d89e55ff7ccde (fabe56e44b6f683e24e37246a7a8851190947cb3 before https://github.com/bitcoin/bitcoin/pull/18454#issuecomment-607866453), fa5bf23d527a450e72c2bf13d013e5393b664ca3 was dropped since last review.
Tree-SHA512: 0552bf8fcbe375baa3cab62acd8c23b2994efa47daff818ad1116d0ffaa0b9e520dc1bca2bbc68369b25584e85e54861fe6fd0968de4f503b95439c099df9bd7
478c11dde326e2ff0480c14f76f9f6b52a7bdfd0 Correct scripted-diff example link (Yahia Chiheb)
Pull request description:
ACKs for top commit:
fanquake:
ACK 478c11dde326e2ff0480c14f76f9f6b52a7bdfd0
Tree-SHA512: 3bc741a79db9bd7abb17ef11f697b768565ec01303a5823ee6a7d8dfa6e888a99a15e9eda69f97a912abc3fd56a54f698f9a580596511bc9bcf62a6870b273f6
5ca90f8b598978437340bb8467f527b9edfb2bbf scripts: add MACHO lazy bindings check to security-check.py (fanquake)
Pull request description:
This is a slightly belated follow up to #17686 and some discussion with Cory. It's not entirely clear if we should make this change due to the way the macOS dynamic loader appears to work. However I'm opening this for some discussion. Also related to #17768.
#### Issue:
[`LD64`](https://opensource.apple.com/source/ld64/) doesn't set the [MH_BINDATLOAD](https://opensource.apple.com/source/xnu/xnu-6153.11.26/EXTERNAL_HEADERS/mach-o/loader.h.auto.html) bit in the header of MACHO executables, when building with `-bind_at_load`. This is in contradiction to the [documentation](https://opensource.apple.com/source/ld64/ld64-450.3/doc/man/man1/ld.1.auto.html):
```bash
-bind_at_load
Sets a bit in the mach header of the resulting binary which tells dyld to
bind all symbols when the binary is loaded, rather than lazily.
```
The [`ld` in Apples cctools](https://opensource.apple.com/source/cctools/cctools-927.0.2/ld/layout.c.auto.html) does set the bit, however the [cctools-port](https://github.com/tpoechtrager/cctools-port/) that we use for release builds, bundles `LD64`.
However; even if the linker hasn't set that bit, the dynamic loader ([`dyld`](https://opensource.apple.com/source/dyld/)) doesn't seem to ever check for it, and from what I understand, it looks at a different part of the header when determining whether to lazily load symbols.
Note that our release binaries are currently working as expected, and no lazy loading occurs.
#### Example:
Using a small program, we can observe the behaviour of the dynamic loader.
Conducted using:
```bash
clang++ --version
Apple clang version 11.0.0 (clang-1100.0.33.17)
Target: x86_64-apple-darwin18.7.0
ld -v
@(#)PROGRAM:ld PROJECT:ld64-530
BUILD 18:57:17 Dec 13 2019
LTO support using: LLVM version 11.0.0, (clang-1100.0.33.17) (static support for 23, runtime is 23)
TAPI support using: Apple TAPI version 11.0.0 (tapi-1100.0.11)
```
```cpp
#include <iostream>
int main() {
std::cout << "Hello World!\n";
return 0;
}
```
Compile and check the MACHO header:
```bash
clang++ test.cpp -o test
otool -vh test
...
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
MH_MAGIC_64 X86_64 ALL LIB64 EXECUTE 16 1424 NOUNDEFS DYLDLINK TWOLEVEL WEAK_DEFINES BINDS_TO_WEAK PIE
# Run and dump dynamic loader bindings:
DYLD_PRINT_BINDINGS=1 DYLD_PRINT_TO_FILE=no_bind.txt ./test
Hello World!
```
Recompile with `-bind_at_load`. Note still no `BINDATLOAD` flag:
```bash
clang++ test.cpp -o test -Wl,-bind_at_load
otool -vh test
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
MH_MAGIC_64 X86_64 ALL LIB64 EXECUTE 16 1424 NOUNDEFS DYLDLINK TWOLEVEL WEAK_DEFINES BINDS_TO_WEAK PIE
...
DYLD_PRINT_BINDINGS=1 DYLD_PRINT_TO_FILE=bind.txt ./test
Hello World!
```
If we diff the outputs, you can see that `dyld` doesn't perform any lazy bindings when the binary is compiled with `-bind_at_load`, even if the `BINDATLOAD` flag is not set:
```diff
@@ -1,11 +1,27 @@
+dyld: bind: test:0x103EDF030 = libc++.1.dylib:__ZNKSt3__16locale9use_facetERNS0_2idE, *0x103EDF030 = 0x7FFF70C9FA58
+dyld: bind: test:0x103EDF038 = libc++.1.dylib:__ZNKSt3__18ios_base6getlocEv, *0x103EDF038 = 0x7FFF70CA12C2
+dyld: bind: test:0x103EDF068 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryC1ERS3_, *0x103EDF068 = 0x7FFF70CA12B6
+dyld: bind: test:0x103EDF070 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryD1Ev, *0x103EDF070 = 0x7FFF70CA1528
+dyld: bind: test:0x103EDF080 = libc++.1.dylib:__ZNSt3__16localeD1Ev, *0x103EDF080 = 0x7FFF70C9FAE6
<trim>
-dyld: lazy bind: test:0x10D4AC0C8 = libsystem_platform.dylib:_strlen, *0x10D4AC0C8 = 0x7FFF73C5C6E0
-dyld: lazy bind: test:0x10D4AC068 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryC1ERS3_, *0x10D4AC068 = 0x7FFF70CA12B6
-dyld: lazy bind: test:0x10D4AC038 = libc++.1.dylib:__ZNKSt3__18ios_base6getlocEv, *0x10D4AC038 = 0x7FFF70CA12C2
-dyld: lazy bind: test:0x10D4AC030 = libc++.1.dylib:__ZNKSt3__16locale9use_facetERNS0_2idE, *0x10D4AC030 = 0x7FFF70C9FA58
-dyld: lazy bind: test:0x10D4AC080 = libc++.1.dylib:__ZNSt3__16localeD1Ev, *0x10D4AC080 = 0x7FFF70C9FAE6
-dyld: lazy bind: test:0x10D4AC070 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryD1Ev, *0x10D4AC070 = 0x7FFF70CA1528
```
Note: `dyld` also has a `DYLD_BIND_AT_LAUNCH=1` environment variable, that when set, will force any lazy bindings to be non-lazy:
```bash
dyld: forced lazy bind: test:0x10BEC8068 = libc++.1.dylib:__ZNSt3__113basic_ostream
```
#### Thoughts:
After looking at the dyld source, I can't find any checks for `MH_BINDATLOAD`. You can see the flags it does check for, such as MH_PIE or MH_BIND_TO_WEAK [here](https://opensource.apple.com/source/dyld/dyld-732.8/src/ImageLoaderMachO.cpp.auto.html).
It seems that the lazy binding of any symbols depends on whether or not [lazy_bind_size](https://opensource.apple.com/source/xnu/xnu-6153.11.26/EXTERNAL_HEADERS/mach-o/loader.h.auto.html) from the `LC_DYLD_INFO_ONLY` load command is > 0. Which was mentioned in [#17686](https://github.com/bitcoin/bitcoin/pull/17686#issue-350216254).
#### Changes:
This PR is one of [Corys commits](7b6ba26178), that I've rebased and modified to make build. I've also included an addition to the `security-check.py` script to check for the flag.
However, given the above, I'm not entirely sure this patch is the correct approach. If the linker no-longer inserts it, and the dynamic loader doesn't look for it, there might be little benefit to setting it. Or, maybe this is an oversight from Apple and needs some upstream discussion. Looking for some thoughts / Concept ACK/NACK.
One alternate approach we could take is to drop the patch and modify security-check.py to look for `lazy_bind_size` == 0 in the `LC_DYLD_INFO_ONLY` load command, using `otool -l`.
ACKs for top commit:
theuni:
ACK 5ca90f8b598978437340bb8467f527b9edfb2bbf
Tree-SHA512: 444022ea9d19ed74dd06dc2ab3857a9c23fbc2f6475364e8552d761b712d684b3a7114d144f20de42328d1a99403b48667ba96885121392affb2e05b834b6e1c
b1d24d1d031a2b2ce67bf846bafa1c3a499b7553 Reorder the test instructions by number (Pieter Wuille)
c2ccadc26a04358b11539097c1aadb8d11b85c21 Merge and generalize case 3 and case 6 (Pieter Wuille)
402ad5aaca9509d45d861d77eb6431d6e1944f91 Only run sanity check once at the end (Pieter Wuille)
eda8309bfc6a8c94f0b7c076d1cccc86c1011cbc Assert immediately rather than caching failure (Pieter Wuille)
55608455cbed4234f26f62ed9ff500fe5dbc21c4 Make a fuzzer-based copy of the prevector randomized test (Pieter Wuille)
Pull request description:
The current prevector test effectively randomly generates a number of operations to perform on a prevector and a normal vector, and checks consistency between the two.
By converting this into a fuzzer the operations can be targetted rather than random.
ACKs for top commit:
MarcoFalke:
ACK b1d24d1d031a2b2ce67bf846bafa1c3a499b7553 🍬
Tree-SHA512: 2b5c62abcd5fee94f42db03400531484d98c59e7f4308e0e683c61aabcd9ce42f85c5d058d2d5e7f8221124f71d2112b6a5f3c80e5d0fdae265a70647747e92f
faede1b293354560317b67f0b4e6874dcac6ef41 test: Properly raise FailedToStartError when rpc shutdown before warmup finished (MarcoFalke)
Pull request description:
Should fix issues such as https://travis-ci.org/github/bitcoin/bitcoin/jobs/671910152#L7034
Top commit has no ACKs.
Tree-SHA512: ac659f29c5ec91985c916b734e24911cbf4e2c5c4b1f1891a7e6c2d2511ec285167550fb03848eee4a7a3cbc9f8cdb0c766f4e881d9e44368c7415d007006368
6136a96cdf631f824fb89cf2720422787439acc9 ci: Rename RUN_CI_ON_HOST to DANGER_RUN_CI_ON_HOST (Hennadii Stepanov)
97ba77aa8e58c3c7ff767e6ca804d0e46370e5dc ci: Add native s390x (Hennadii Stepanov)
Pull request description:
Unlike the Docker wrapped solution (#17591) this PR suggests running on host system directly.
This approach makes builds quick and stable (see: #18106).
The excerpt from the Travis log:
```
...
Running on host system without docker wrapper
...
Byte Order: Big Endian
...
```
ACKs for top commit:
MarcoFalke:
ACK 6136a96cdf631f824fb89cf2720422787439acc9
Tree-SHA512: 1b591de13e38d10a35217e1de11cbd648a359d18d16eed166fac18ea5788b58cc9fc6d407086ed342b99e57e479efd951a0ea693710177e500eb116316b9a788
fa7af33b4cb12315de86315c163393bdb50fad75 ci: Run unit tests sequential once (MarcoFalke)
fa68a3e7640fc00c47f03fa565357679bdafb4b2 appveyor: Enable minimal unit test logging to aid debugging (MarcoFalke)
Pull request description:
Fixes#16976
Top commit has no ACKs.
Tree-SHA512: 1f1ee8776a67afa8c1c5a16ef170c9975b6a486c087c8eba12e97d23382befd1c2801622ec70ca8e4cd1fbedce1dec46be67677ceaf07f35f1d3f3bead0200f0
Previously, a default match-everything bloom filter was set for every peer,
i.e. even before receiving a 'filterload' message and after receiving a
'filterclear' message code branches checking for the existence of the filter
by testing the pointer "pfilter" were _always_ executed.
cdfb8e7afa7648405dd6b957f47b1c7ab566a076 tests: Add fuzzing harness for HTTPRequest, libevent's evhttp and related functions (practicalswift)
Pull request description:
Add fuzzing harness for `HTTPRequest`, `libevent`'s `evhttp` and related functions.
ACKs for top commit:
laanwj:
ACK cdfb8e7afa7648405dd6b957f47b1c7ab566a076
Tree-SHA512: da481afed5eb3232d3f3d0583094e56050e6234223dfcb356d8567fe0616336eb1b78c5e6821325fc9767e385e5dfaf3c96f0d35ffdb67f18d74f9a9a9464e24
7777e3624fabe4718675b2be8b088697b7ad4d0d scripted-diff: Replace strCommand with msg_type (MarcoFalke)
Pull request description:
Receiving a message is not a command, but simply a message of some type
ACKs for top commit:
promag:
ACK 7777e3624fabe4718675b2be8b088697b7ad4d0d.
naumenkogs:
ACK 7777e36
practicalswift:
ACK 7777e3624fabe4718675b2be8b088697b7ad4d0d -- I've always thought the `strCommand` name is confusing :)
theStack:
ACK 7777e36
Tree-SHA512: 662bac579064c621191916274314b85111cfb4df488f00893ceb16def1c47af4b2a0f34cd7349722099b5a9d23160edb8eb999841f1d64af3e0da02e4870b4bf
283bd72156959f420f13acc7a34e513ca3446025 tests: Add coverage of {,Incremental}DynamicUsage(const std::set<X, Y>& s) to existing fuzzer (practicalswift)
bf76000493082da05bf7258a5038e16fa76cd143 tests: Add fuzzing harness for classes/functions in cuckoocache.h (practicalswift)
57890b2555ca347373109052f6789c23f46bc594 tests: Add fuzzing harness for classes/functions in checkqueue.h (practicalswift)
2df5701e902effa93834d9520690cbaca7e504f3 tests: Add coverage of GetVirtualTransactionSize(...) to existing fuzzer (practicalswift)
7b9a2dc86426926038b2f49d3d4ce4cb64dcd14b tests: Add fuzzing harness for AdditionOverflow(...) (practicalswift)
44fb2a596b4a1aa70253c4145c35be6de68da22a tests: Add fuzzing harness for FeeFilterRounder (practicalswift)
Pull request description:
Includes:
```
tests: Add fuzzing harness for FeeFilterRounder
tests: Add fuzzing harness for classes/functions in checkqueue.h
tests: Add fuzzing harness for classes/functions in cuckoocache.h
tests: Add coverage of {,Incremental}DynamicUsage(const std::set<X, Y>& s) to existing fuzzer
tests: Add coverage of GetVirtualTransactionSize(...) to existing fuzzer
tests: Add fuzzing harness for AdditionOverflow(...)
```
See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core.
ACKs for top commit:
MarcoFalke:
ACK 283bd72156959f420f13acc7a34e513ca3446025
Tree-SHA512: 2361edfb5c47741b22d9fb996836c5250c5a26bc5e956039ea6a0c55ba2d36c78f241d66f85bc02f5b85b9b83d5fde56a5c4702b9d1b7ac4a9a3ae391ca79eaa
13d2a33537a403ac47a989be92109d3214375b6a Fix unregister_all_during_call cleanup (Russell Yanofsky)
Pull request description:
Use `TestingSetup` fixture to fix `unregister_all_during_call` test not calling `UnregisterBackgroundSignalScheduler`, which could trigger an assert in `RegisterBackgroundSignalScheduler` when called in later tests
Failure reported by fanquake https://github.com/bitcoin/bitcoin/pull/18551#issuecomment-610974251
ACKs for top commit:
MarcoFalke:
ACK 13d2a33537a403ac47a989be92109d3214375b6a if appveyor unit tests pass
Tree-SHA512: d2ec8ff14c54d97903af50031abfac1f38ec1c3aabc90371cfd5b79481fa69d3d77f339bfdf7d2178fd85e83402f72eda7cf4d339e5bbfa7e6e1a68836643b93
da0842dcd44f8c9c9b167917fac0949b4978c3b0 build: Update ax_boost_mase.m4 to the latest serial (Hennadii Stepanov)
Pull request description:
Picked from the upstream 90814f1895Fix#17010.
This PR is [alternative](https://github.com/bitcoin/bitcoin/issues/17010#issuecomment-610651736) to #18501.
ACKs for top commit:
laanwj:
ACK da0842dcd44f8c9c9b167917fac0949b4978c3b0
Tree-SHA512: 5e43e12c524e4ea6b967c9be02c81a75948eac6cf55b819e3339222a2e3414731581d40af3524ad865abae7c5247c190448ebf2aa5e0d9a338edb501cc23ba38
Use TestingSetup fixture to fix unregister_all_during_call test not calling
UnregisterBackgroundSignalScheduler, which could trigger an assert in
RegisterBackgroundSignalScheduler when called in later tests
Failure reported by fanquake <fanquake@gmail.com>
https://github.com/bitcoin/bitcoin/pull/18551#issuecomment-610974251
2276339a176f83ffe8ceefb3e41ecca8601aa13b Add test for UnregisterAllValidationInterfaces bug (Russell Yanofsky)
3c61abbbc847d725f30d169278d84655571407c1 Do not clear validationinterface entries being executed (Pieter Wuille)
Pull request description:
The previous code for MainSignalsInstance::Clear would decrement the reference
count of every interface, including ones that were already Unregister()ed but
still being executed.
This fixes the issue pointed out here: https://github.com/bitcoin/bitcoin/pull/18524/files#r404395685 . It's not currently observable.
ACKs for top commit:
jonasschnelli:
utACK 2276339a176f83ffe8ceefb3e41ecca8601aa13b - reviewed code and test (thanks @ryanofsky for adding the test).
MarcoFalke:
ACK 2276339a176f83ffe8ceefb3e41ecca8601aa13b 🎎
ryanofsky:
Code review ACK 2276339a176f83ffe8ceefb3e41ecca8601aa13b. No change to bugfix, just rebased and new test commit added since last review
Tree-SHA512: c1d68e7c681a45c6cadc84e407c2266bcb4b12d34264e1232a61c4eadb74b551231c5a3b1d041de39f507aef4dfa7d4589b8bfe1833f069c739c6270d2a05dbe
The libtool unsorted 'find' determinism issue seemed to have been solved
in gcc-9's git: d41cd173e23ebea7c758644d6ad6e0fde1c2e3a6 or SVN: r262451
Furthermore, it seems that Ubuntu Focal 20.04 LTS is going to ship with
gcc 9 and mingw-w64 7, which will match what we have now.
-----
A note on this:
Careful observers will see that previously I stated that all released
versions of gcc were bootstrapped with a libtool 2.2.7a, meaning that
they all had the unsorted 'find' determinism issue first resolved in
libtool 2.2.7b.
However, I was mistaken, gcc's ltmain.sh CLAIMS it was generated by
libtool 2.2.7a, but it was in fact edited manually. It seems that gcc
maintains their own versions of ltmain.sh and libtool.m4, and only
sometimes backports patches from upstream.
Quite confusing.
This is no longer needed after 3bef7c22 in the mingw-w64 git repository,
which is first included in mingw-w64 v7.0.0.
As of the previous bump to our Guix time machine, we now use mingw-w64
v7.0.0.
Most of the mingw-w64 toolchain changes have now been upstreamed, we can
point to a commit that exists upstream.
NOTE: I'm not changing the URL yet until we see that Guix upstream will
accept all my patches for macOS.
-----
The Guix tree that's referred to by this commit contains the following
changes relevant to our mingw-w64 build:
b066c25026
Adds a PACKAGES-WITH-*PATCHES procedure which we can use in the future
to apply patches to packages if those patches are not considered
appropriate to upstream Guix
4719b71572
Adds mingw-w64 (the libc itself) reproducibility patches, taken from
debian.
79825bee07 + 401d28e433 + c1c50cb5b0
Add mingw-w64 specific binutils patches, taken from debian.
Specifically, the "Make DLL import libraries reproducible" patch made
libbitcoinconsensus.dll.a build reproducibly. The followup commits
were hotfixes for my mistakes.
0f864175dc
Bumps mingw-w64 to v7.0.0. This is the first release that enables
secure APIs by default (which we need), and gains _FORTIFY_SOURCE
support. This will also be what Ubuntu Focal 20.04 LTS releases with.
cdf00cf75d
Bumps NSIS to v3.05. This is the first release that includes a fix for
a reproducibility bug found by some of the electrum developers. See
details here: https://sourceforge.net/p/nsis/bugs/1230/
The previous code for MainSignalsInstance::Clear would decrement the reference
count of every interface, including ones that were already Unregister()ed but
still being executed.
fa1a92224dd78de817d15bcda35a8310254e1a54 rpc: Avoid initialization-order-fiasco on static CRPCCommand tables (MarcoFalke)
Pull request description:
Currently the fiasco is only theoretical because all content of the table are compile-time constants. However, the fiasco materializes should they ever become run-time constants (e.g. #18531).
ACKs for top commit:
promag:
ACK fa1a92224dd78de817d15bcda35a8310254e1a54.
practicalswift:
ACK fa1a92224dd78de817d15bcda35a8310254e1a54 -- fiasco bad :)
Tree-SHA512: cccadb0ad56194599b74f04264d74c34fa865958580a850efc6474bbdc56f30cadce6b2e9a6ad5472ff46c3f4c793366acd8090fad409a45b25d961f2d89da19
7a2ecf16df938dd95d3130a46082def7a02338eb Wallet: Change IsMine check in CWallet::DelAddressBook from assert to failure (Luke Dashjr)
2952c46b923042f2de801f319e03ed5c4c4eb735 Wallet: Replace CAddressBookData.name with GetLabel() method (Luke Dashjr)
d7092c392e10889cd7a080b3d22ed6446a59b87a QA: Test that change doesn't turn into non-change when spent in an avoid-reuse wallet (Luke Dashjr)
Pull request description:
Follow-up to #18192, not strictly necessary for 0.20
ACKs for top commit:
MarcoFalke:
re-ACK 7a2ecf16df, only change is adding an assert_equal in the test 🔰
jnewbery:
utACK 7a2ecf16df938dd95d3130a46082def7a02338eb
Tree-SHA512: e0933ee40f705b751697dc27249e1868ed4874254b174ebdd0a7150125d8c818402e66df2371718c7eeb90e67ee2317215fb260aa9b9d7b9b45ee436de2988ff
56fe839e4e6d837b95b920a524570d11f7c3c5ce qt: Fix Window -> Minimize menu item (Hennadii Stepanov)
Pull request description:
Now Window -> Minimize menu item is broken on Linux.
Steps to reproduce:
1. start `bitcoin-qt`
2. activate Window -> Minimize menu item with a keyboard (not by a shortcut) or a mouse
**Expected behavior**
The main window gets minimized.
**Actual behavior**
The main window still unchanged. Even worse: the menu widget becomes a separate window:
This PR does not touch the macOS specific code as `qApp->focusWindow()` seems work on macOS flawlessly.
ACKs for top commit:
promag:
Tested ACK 56fe839e4e6d837b95b920a524570d11f7c3c5ce on bionic with qt 5.9.5.
Tree-SHA512: 3582e44ba181d859f5994b9cddc6ce1b60aa1db520a31dd3a0684336c79d558d7410ce7a1ab5b0860c6431b54d8acc3aa16e399717b4c70839861e6b6c4290c0
