In the functional tests, we often compare dicts with assert_equal, but the
output makes it very hard to tell exactly which entry in the dicts don't
match when there are a lot of entries and only minor differences. Change
the output to make it clearer.
Make sure that v2 handshake is complete before comparing getpeerinfo
outputs so that `transport_protocol_type` isn't stuck at 'detecting'.
- on the python side, this is ensured by default
`wait_for_handshake = True` inside `add_p2p_connection()`.
- on the c++ side, add a wait_until statement till
`transport_protocol_type = v2` so that v2 handshake is complete.
Co-Authored-By: Martin Zumsande <mzumsande@gmail.com>
d8087adc7ebd4ea05dd3843e5a92e8115fd7bbcc [test] IsBlockMutated unit tests (dergoegge)
1ed2c9829700054526156297552bb47230406098 Add transaction_identifier::size to allow Span conversion (dergoegge)
1ec6bbeb8d27d31647d1433ccb87b362f6d81f90 [validation] Cache merkle root and witness commitment checks (dergoegge)
5bf4f5ba32da4627f152b54d266df9b2aa930457 [test] Add regression test for #27608 (dergoegge)
49257c0304828a185c273fcb99742c54bbef0c8e [net processing] Don't process mutated blocks (dergoegge)
2d8495e0800f5332748cd50eaad801ff77671bba [validation] Merkle root malleation should be caught by IsBlockMutated (dergoegge)
66abce1d98115e41f394bc4f4f52341960ddc839 [validation] Introduce IsBlockMutated (dergoegge)
e7669e1343aec4298fd30d539847963e6fa5619c [refactor] Cleanup merkle root checks (dergoegge)
95bddb930aa72edd40fdff52cf447202995b0dce [validation] Isolate merkle root checks (dergoegge)
Pull request description:
This PR proposes to check for mutated blocks early as a defense-in-depth mitigation against attacks leveraging mutated blocks.
We introduce `IsBlockMutated` which catches all known forms of block malleation and use it to do an early mutation check whenever we receive a `block` message.
We have observed attacks that abused mutated blocks in the past, which could have been prevented by simply not processing mutated blocks (e.g. https://github.com/bitcoin/bitcoin/pull/27608 for which a regression test is included in this PR).
ACKs for top commit:
achow101:
ACK d8087adc7ebd4ea05dd3843e5a92e8115fd7bbcc
maflcko:
ACK d8087adc7ebd4ea05dd3843e5a92e8115fd7bbcc 🏄
fjahr:
Code review ACK d8087adc7ebd4ea05dd3843e5a92e8115fd7bbcc
sr-gi:
Code review ACK d8087adc7e
Tree-SHA512: 618ff4ea7f168e10f07504d3651290efbb1bb2ab3b838ffff3527c028caf6c52dedad18d04d3dbc627977479710930e200f2dfae18a08f627efe7e64a57e535f
1484998b6b08c93714325952bd94dd8a2de446ae ci: print python version on win64 native job (Max Edwards)
Pull request description:
Adds python version output to the Win64 Native CI job on Github Actions. Also clarifies that one of the versions already printed is the VCToolsVersion.
Before:
After:
Should the individual python test runners print the python version instead or also?
ACKs for top commit:
hebasto:
ACK 1484998b6b08c93714325952bd94dd8a2de446ae.
Tree-SHA512: 6d084ff4a667156fa8797450de83bbcf596ddd3b2fa8ec04c1ca9a532a6fec716817b66da34db4ea0184bd802ef613e2b8f6142be9a511c5397785cfbfede0c3
51bc1c7126d6e130bc40c529fb71ae6486da0492 test: Remove Windows-specific code from `system_tests/run_command` (Hennadii Stepanov)
Pull request description:
The removed code has been dead since https://github.com/bitcoin/bitcoin/pull/28967.
Required as a precondition for replacing Boost.Process with [cpp-subprocess](https://github.com/bitcoin/bitcoin/pull/28981) to make diff for this code meaningful and reviewable.
The plan is to reintroduce Windows-specific code in this test simultaneously with enabling Windows support in cpp-subprocess.
ACKs for top commit:
Sjors:
utACK 51bc1c7126d6e130bc40c529fb71ae6486da0492
theStack:
Code-review ACK 51bc1c7126d6e130bc40c529fb71ae6486da0492
Tree-SHA512: 0e3875c4dc20564332555633daf2227223b10dc3d052557635eced2734575d1e0252fb19e46ea6e6c47a15c51c345f70b6d437e33435abcd0e4fcf29edb50887
ad7584d8b60119ca3717117a1eb6a16d753c5d74 serialization: replace char-is-int8_t autoconf detection with c++20 concept (Cory Fields)
Pull request description:
Doesn't depend on #29263, but it's really only relevant after that one's merged.
This removes the only remaining autoconf macro in our serialization code (after #29263), so it can now be used trivially and safely out-of-tree.
~Our code does not currently contain any concepts, but couldn't find any discussion or docs about avoiding them. I guess we'll see if this blows up our c-i.~
Edit: Ignore this. ajtowns pointed out that we're already using a few concepts.
This was introduced in #13580. Please check my logic on this as I'm unable to test on a SmartOS system. Even better would be a confirmation from someone who can build there.
ACKs for top commit:
Empact:
Code review ACK ad7584d8b6
Tree-SHA512: 1faf65c900700efb1cf3092c607a2230321b393cb2f029fbfb94bc8e50df1dabd7a9e4b91e3b34f0d2f3471aaf18ee7e56d91869db5c5f4bae84da95443e1120
b052b2d1f2b220582a933eb5fa6a28144bed07d8 build: remove -Wdocumentation conditional (fanquake)
Pull request description:
Now that `--enable-suppress-external-warnings` is on by default, we can drop it. CIs are all already building with this flag.
ACKs for top commit:
Empact:
Code review ACK b052b2d1f2
theuni:
utACK b052b2d1f2b220582a933eb5fa6a28144bed07d8
Tree-SHA512: 8b55f366dfeece082090fb87de67d8811967f4c89987a346431b2deb73c3c94401b59ec98bb1cbf790e18894f3d4c4aebb57cbc5fbf931c1046bf40239bc7a58
During shutdown, already queue events dispatched from the backend such
'numConnectionsChanged' and 'networkActiveChanged' could try to access
the clientModel object, which might not exist because we manually delete
it inside 'BitcoinApplication::requestShutdown()'.
These replace our platform-specific mess in favor of c++20 endian detection
via std::endian and internal byteswap functions when necessary.
They no longer rely on autoconf detection.
Rather than a complicated set of tests to decide which bswap functions to
use, always prefer the compiler built-ins when available.
These builtins and fallbacks can all be removed once we're using c++23, which
adds std::byteswap.
This code has been dead since https://github.com/bitcoin/bitcoin/pull/28967.
Required as a precondition for replacing Boost.Process with
cpp-subprocess to make diff for this code meaningful and reviewable.
The plan is to reintroduce Windows-specific code in this test
simultaneously with enabling Windows support in cpp-subprocess.
We preemptively perform a block mutation check before further processing
a block message (similar to early sanity checks on other messsage
types). The main reasons for this change are as follows:
- `CBlock::GetHash()` is a foot-gun without a prior mutation check, as
the hash returned only commits to the header but not to the actual
transactions (`CBlock::vtx`) contained in the block.
- We have observed attacks that abused mutated blocks in the past, which
could have been prevented by simply not processing mutated blocks
(e.g. https://github.com/bitcoin/bitcoin/pull/27608).
fccfdb25b2c337bbd8b283c27493f10d5e02b5d4 doc: Update OpenBSD build docs to 7.4 (Jesse Barton)
Pull request description:
Updated OpenBSD Build doc for 7.4 after testing all build options. No issues on my end.
Also added a note about referring to depends/README.md for detailed instructions on required dependencies.
This was added in reference to a conversation in #29443
ACKs for top commit:
fanquake:
ACK fccfdb25b2c337bbd8b283c27493f10d5e02b5d4
theStack:
lgtm ACK fccfdb25b2c337bbd8b283c27493f10d5e02b5d4
Tree-SHA512: be6d22b605140b37a71e11c5bbed54f60655832d78cd3cb221eddc77c7621a65c0d71baf436f90819be536d9b5dbf1a0b2c82b6b23d62356addc495403f2ba35
bf5662c678455fb47c402f8520215726ddfe7a94 test: enable v2 for python p2p depending on global --v2transport flag (Martin Zumsande)
6e9e39da434f8dafacee4cba068daf499bdb4cc2 test: Don't use v2transport when it's too slow. (Martin Zumsande)
87549c8f89fe8c9f404b74c5a40b5ee54c5a966d test: enable p2p_invalid_messages.py with v2transport (Martin Zumsande)
5fc9db504b9ac784019e7e8c215c31abfccb62b6 test: enable p2p_sendtxrcncl.py with v2transport (Martin Zumsande)
Pull request description:
#24748 added v2 transport to the python `P2PConnection`, but so far each test that wants to make use of it needs to enable it on an individual basis.
This PR changes it so that if the test suite is run with `--v2transport` option, v2 is used in each test by default, not only for connections between two bitcoind instances as before, but also wherever `P2PConnection` is used. Individual tests can override this global option.
To do that, a few tests need to be adjusted.
In addition, I added a commit to always use v1 in a few select subtests that send a large number of large messages (e.g. large reorgs). These tests don't have a fundamental problem with v2 but become very slow due to the unoptimised python ChaCha20 implementation (~30 minutes on my computer, so probably not suitable to be run in the CI).
As a result, `python3 test_runner.py --v2transport` should succeed and use `v2` everywhere (unless v1 is chosen explicitly).
[Edit]: To make the "test each commit" CI pass, several test fixes were squashed into the last commit, which actually enables v2 p2p for `P2PConnection`. I have an unsquashed version at https://github.com/mzumsande/bitcoin/tree/202401_bip324_alltests_unsquashed, in case that helps with review.
ACKs for top commit:
fjahr:
tACK bf5662c678455fb47c402f8520215726ddfe7a94
vasild:
ACK bf5662c678455fb47c402f8520215726ddfe7a94
stratospher:
reACK bf5662c6.
theStack:
Tested ACK bf5662c678455fb47c402f8520215726ddfe7a94
Tree-SHA512: 4f5a08248ba8a755f7d0f48deb2b79bef03292345cacb7deef01be955481093800e4e56ff218ea56734eef5de1fb3ab0f04657447ea27d393bb536539d7b289d
fa3a4102ef0ae06d8930d7a7b567759e2a5b5fde fuzz: Set -rss_limit_mb=8000 for generate as well (MarcoFalke)
fa4e396e1da8e5b04a5f906b95017b969ea37bae fuzz: Generate with random libFuzzer settings (MarcoFalke)
Pull request description:
Sometimes a libFuzzer setting like `-use_value_profile=1` helps [0], sometimes it hurts [1].
[0] https://github.com/bitcoin/bitcoin/pull/20789#issuecomment-752961937
[1] https://github.com/bitcoin/bitcoin/pull/27888#issuecomment-1645976254
By picking a random value, it is ensured that at least some of the runs will have the beneficial configuration set.
Also, set `-max_total_time` to prevent slow fuzz targets from getting a larger time share, or possibly peg to a single core for a long time and block the python script from exiting for a long time. This can be improved in the future. For example, the python script can exit after some time (https://github.com/bitcoin/bitcoin/pull/20752#discussion_r549248791). Alternatively, it can measure if coverage progress was made and run for less time if no progress has been made recently anyway, so that more time can be spent on targets that are new or still make progress.
ACKs for top commit:
murchandamus:
utACK fa3a4102ef0ae06d8930d7a7b567759e2a5b5fde
dergoegge:
utACK fa3a4102ef0ae06d8930d7a7b567759e2a5b5fde
brunoerg:
light ACK fa3a4102ef0ae06d8930d7a7b567759e2a5b5fde
Tree-SHA512: bfd04a76ca09aec612397bae5f3f263a608faa7087697169bd4c506c8195c4d2dd84ddc7fcd3ebbc75771eab618fad840af819114968ca3668fc730092376768
d2fe90571e98e02617682561ea82acb1e2647827 test: Drop `x` modifier in `fsbridge::fopen` call for mingw builds (Hennadii Stepanov)
Pull request description:
The MinGW-w64 toolchain links executables to the old msvcrt C Runtime Library that does not support the `x` modifier for the [`_wfopen()`](https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/fopen-wfopen?view=msvc-170) function.
Fixes https://github.com/bitcoin/bitcoin/issues/29014.
ACKs for top commit:
maflcko:
ACK d2fe90571e98e02617682561ea82acb1e2647827
fanquake:
ACK d2fe90571e98e02617682561ea82acb1e2647827 - the plan here should still be to migrate to the newer windows runtime.
Tree-SHA512: 0269b66531e58c093ecda3a3e355a20ee8274e165d7e010f8f125881b3c8d4cfe801abdca4605d81efd3b2dbe9a81896968971f6f53da7f6c6093b76b47c5bc9
Tested and used all build options on OpenBSD 7.4 with no issues.
Added a note about referring to depends/README.md for detailed instructions on required dependencies.
This was added in reference to a conversation in #29443
