On default legacy wallets, the backup filename starts with an "-" due
to the wallet name being empty. This is inconvenient for systems who
treat what follows the initial "-" character as flags.
Refactors the helper functions in the test to provide more
fine-grained control over RPC requests and responses than
the usual AuthProxy methods.
No change in test behavior, the same RPC requests are made.
53ffd5a410186109b9a56c5922768905e168acb3 docs: Fix broken reference to CI setup in test/lint/README.md (naiyoma)
Pull request description:
The current [reference](https://github.com/bitcoin/bitcoin/blob/master/test/ci/lint/04_install.sh
) for CI setup in /test/lint#readme returns a 404.
ACKs for top commit:
fanquake:
ACK 53ffd5a410186109b9a56c5922768905e168acb3
Tree-SHA512: 813c19a145f09e7da11963598b70dc438acba784eb722e509d6af59dc3af8f5da97628c454bed2b03cc919689603e070796de2db8d784d9162ae34e7b85a77d9
e67ab174c9c04bba7a10724b7e694dd57f732177 test: fix flaky wallet_send functional test (Max Edwards)
3c49e6967050cfc367b3c826c9eac86a48528af5 test: fix weight estimates in functional tests (Max Edwards)
Pull request description:
Fixes: https://github.com/bitcoin/bitcoin/issues/25164
The wallet_send functional test has been flaky due to a slightly overestimated weight calculation. This PR makes the weight calculation more accurate, although occasionally, due to how ECDSA signatures can be different lengths it might slightly over estimate. The assertion in the test can handle this slight variation and so should continue passing.
Update:
Because the signature can be shorter that is used in the weight estimation or the final transaction the estimate could be both slightly smaller or slightly larger.
ACKs for top commit:
achow101:
ACK e67ab174c9c04bba7a10724b7e694dd57f732177
S3RK:
Code review ACK e67ab174c9c04bba7a10724b7e694dd57f732177
Tree-SHA512: 3bf73b355309dce860fa1520afb8461e94268e4bcf0e92a8273c279b41b058c44472cf59daafa15a515529b50bd665b5d498bbe4d934f2315dbe810a05bc73f9
a1fbde0ef7cf6c94d4a5181f8ceb327096713160 p2p: Don't consider blocks mutated if they don't connect to known prev block (Greg Sanders)
Pull request description:
Followup to https://github.com/bitcoin/bitcoin/pull/29412 to revert some of the behavior change that was likely unintentional.
Based on comments from https://github.com/bitcoin/bitcoin/pull/29412#discussion_r1507499192
ACKs for top commit:
0xB10C:
utACK a1fbde0ef7cf6c94d4a5181f8ceb327096713160
dergoegge:
Code review ACK a1fbde0ef7cf6c94d4a5181f8ceb327096713160
Sjors:
ACK a1fbde0ef7cf6c94d4a5181f8ceb327096713160
sr-gi:
tACK a1fbde0ef7
Tree-SHA512: be6204c8cc57b271d55c1d02a5c77d03a37738d91cb5ac164f483b0bab3991c24679c5ff02fbaa52bf37ee625874b63f4c9f7b39ad6fd5f3a25386567a0942e4
Rather than asserting that the exact fees are the same, check the fee rate rounded to nearest interger. This will account for small differences in fees caused by variability in ECDSA signature lengths.
Updates the weight estimate to be more accurate by removing byte buffers and calculating the length of the count of scriptWitnesses rather than just using the count itself.
a8c3454ba137dfac20b3c89bc558374de0524114 test: speedup bip324_cipher.py unit test (Sebastian Falbesoner)
Pull request description:
Executing the unit tests for the bip324_cipher.py module currently takes quite long (>60 seconds on my older notebook). Most time here is spent in empty plaintext/ciphertext encryption/decryption loops in `test_fschacha20poly1305aead`:
9eeee7caa3/test/functional/test_framework/crypto/bip324_cipher.py (L193-L194)9eeee7caa3/test/functional/test_framework/crypto/bip324_cipher.py (L198-L199)
Their sole purpose is increasing the FSChaCha20Poly1305 packet counter in order to trigger rekeying, i.e. the actual encryption/decryption is not relevant, as the result is thrown away. This commit speeds up the tests by supporting to pass "None" as plaintext/ciphertext, indicating to the routines that no actual encryption/decryption should be done.
The approach here is a bit hacky, a cleaner alternative would probably be to introduce a special `seek`/`skip_packets` method and not touch the encrypt/decrypt routines, but that seemed overkill to me only for speeding up a unit test. Open for suggestions.
master branch:
```
$ python3 -m unittest ./test/functional/test_framework/crypto/bip324_cipher.py
..
----------------------------------------------------------------------
Ran 2 tests in 64.658s
```
PR branch:
```
$ python3 -m unittest ./test/functional/test_framework/crypto/bip324_cipher.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.822s
```
ACKs for top commit:
delta1:
Concept ACK a8c3454
epiccurious:
Tested ACK a8c3454ba137dfac20b3c89bc558374de0524114.
achow101:
ACK a8c3454ba137dfac20b3c89bc558374de0524114
marcofleon:
ACK a8c3454ba137dfac20b3c89bc558374de0524114. The comments at the top of `bip324_cipher.py` specify that this should only be used for testing, so I think this optimization makes sense in that context.
cbergqvist:
ACK a8c3454!
stratospher:
ACK a8c3454. I think it's worth it because of the significant speedup in the unit test.
Tree-SHA512: 737dd805a850be6e035aa3c6d9e2c5b5b5e89ddc564f84a045c37e0238fef6419912de7c902139b64914abdd647c649fe02a694f1a5e1741d7d4459c041caccc
e073f1dfda7a2a2cb2be9fe2a1d576f122596021 test: make sure keypool sizes do not change on `getrawchangeaddress`/`getnewaddress` failures (UdjinM6)
367bb7a80cc71130995672c853d4a6e0134721d6 wallet: Avoid updating `ReserveDestination::nIndex` when `GetReservedDestination` fails (UdjinM6)
Pull request description:
I think the expected behaviour of `getrawchangeaddress` and `getnewaddress` RPCs is that their failure should not affect keypool in any way. At least that's how legacy wallets work, you can confirm this behaviour by running `wallet_keypool.py --legacy-wallet` on master with e073f1dfda7a2a2cb2be9fe2a1d576f122596021 applied on top. However running `wallet_keypool.py --descriptors` on the same commit results in the following failure:
```
File "/path/to/bitcoin/test/functional/test_framework/test_framework.py", line 131, in main
self.run_test()
File "/path/to/bitcoin/test/functional/wallet_keypool.py", line 114, in run_test
assert_equal(kp_size_before, kp_size_after)
File "/path/to/bitcoin/test/functional/test_framework/util.py", line 57, in assert_equal
raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
AssertionError: not([18, 24] == [19, 24])
```
This happens because we pass `nIndex` (which is a class member) into `GetReservedDestination` and since it's passed by reference we get an updated value back, so `nIndex` won't be equal `-1` anymore, no matter if the function failed or succeeded. This means that `ReturnDestination` (called by dtor of `ReserveDestination`) will try to return something we did not actually reserve.
The fix is to simply use a temporary variable instead of a class member and only update `nIndex` when `op_address` actually has value, basically do it the same way we do for other class members (`address` and `fInternal`) already.
ACKs for top commit:
achow101:
ACK e073f1dfda7a2a2cb2be9fe2a1d576f122596021
josibake:
ACK e073f1dfda
Tree-SHA512: 1128288a60dd4d8f306ef6f7ac66cdfeae3c9cc35c66ecada2d78fa61ac759f2a757b70fc3976ba8b5081200942b58dfabc184c01ccf911af40ba8c145344651
0487f91a2046c0d6f91ccaedeb00fbefa635c66d test: Fix intermittent failure in rpc_net.py --v2transport (stratospher)
Pull request description:
Fixes#29508.
Make sure that v2 handshake is complete before comparing getpeerinfo outputs so that `transport_protocol_type` isn't stuck at 'detecting'.
This is done by adding a wait_until statement till `transport_protocol_type = v2` so that bitcoind waits until the v2 handshake is complete. (on the python side, this is ensured by default since `wait_for_handshake = True` inside `add_p2p_connection()`)
ACKs for top commit:
Sjors:
ACK 0487f91a2046c0d6f91ccaedeb00fbefa635c66d
mzumsande:
Code Review ACK 0487f91a2046c0d6f91ccaedeb00fbefa635c66d
achow101:
ACK 0487f91a2046c0d6f91ccaedeb00fbefa635c66d
vasild:
ACK 0487f91a2046c0d6f91ccaedeb00fbefa635c66d
Tree-SHA512: 44dd646a61cd38da243f527df7321e22d1821c2b090be43673027746098caf450c6671708ed731ba257952df6b5886e64c9c2f9686a82f6ef0f25780b7a87d3d
This option beats the --v2transport option and is meant to be used in
test_runner.py.
It applies these to a few tests that are particulary interesting
in terms of the transport type.
This ensures that these tests arei always run with both v1 and v2, irrespective of
whether the global --v2transport test_runner option is set or not.
In the functional tests, we often compare dicts with assert_equal, but the
output makes it very hard to tell exactly which entry in the dicts don't
match when there are a lot of entries and only minor differences. Change
the output to make it clearer.
Make sure that v2 handshake is complete before comparing getpeerinfo
outputs so that `transport_protocol_type` isn't stuck at 'detecting'.
- on the python side, this is ensured by default
`wait_for_handshake = True` inside `add_p2p_connection()`.
- on the c++ side, add a wait_until statement till
`transport_protocol_type = v2` so that v2 handshake is complete.
Co-Authored-By: Martin Zumsande <mzumsande@gmail.com>
d8087adc7ebd4ea05dd3843e5a92e8115fd7bbcc [test] IsBlockMutated unit tests (dergoegge)
1ed2c9829700054526156297552bb47230406098 Add transaction_identifier::size to allow Span conversion (dergoegge)
1ec6bbeb8d27d31647d1433ccb87b362f6d81f90 [validation] Cache merkle root and witness commitment checks (dergoegge)
5bf4f5ba32da4627f152b54d266df9b2aa930457 [test] Add regression test for #27608 (dergoegge)
49257c0304828a185c273fcb99742c54bbef0c8e [net processing] Don't process mutated blocks (dergoegge)
2d8495e0800f5332748cd50eaad801ff77671bba [validation] Merkle root malleation should be caught by IsBlockMutated (dergoegge)
66abce1d98115e41f394bc4f4f52341960ddc839 [validation] Introduce IsBlockMutated (dergoegge)
e7669e1343aec4298fd30d539847963e6fa5619c [refactor] Cleanup merkle root checks (dergoegge)
95bddb930aa72edd40fdff52cf447202995b0dce [validation] Isolate merkle root checks (dergoegge)
Pull request description:
This PR proposes to check for mutated blocks early as a defense-in-depth mitigation against attacks leveraging mutated blocks.
We introduce `IsBlockMutated` which catches all known forms of block malleation and use it to do an early mutation check whenever we receive a `block` message.
We have observed attacks that abused mutated blocks in the past, which could have been prevented by simply not processing mutated blocks (e.g. https://github.com/bitcoin/bitcoin/pull/27608 for which a regression test is included in this PR).
ACKs for top commit:
achow101:
ACK d8087adc7ebd4ea05dd3843e5a92e8115fd7bbcc
maflcko:
ACK d8087adc7ebd4ea05dd3843e5a92e8115fd7bbcc 🏄
fjahr:
Code review ACK d8087adc7ebd4ea05dd3843e5a92e8115fd7bbcc
sr-gi:
Code review ACK d8087adc7e
Tree-SHA512: 618ff4ea7f168e10f07504d3651290efbb1bb2ab3b838ffff3527c028caf6c52dedad18d04d3dbc627977479710930e200f2dfae18a08f627efe7e64a57e535f
Turn the `std::vector` to `std::array` because it is cheaper and
allows us to have the number of the messages as a compile time
constant: `ALL_NET_MESSAGE_TYPES.size()` which can be used in
future code to build other `std::array`s with that size.
bf5662c678455fb47c402f8520215726ddfe7a94 test: enable v2 for python p2p depending on global --v2transport flag (Martin Zumsande)
6e9e39da434f8dafacee4cba068daf499bdb4cc2 test: Don't use v2transport when it's too slow. (Martin Zumsande)
87549c8f89fe8c9f404b74c5a40b5ee54c5a966d test: enable p2p_invalid_messages.py with v2transport (Martin Zumsande)
5fc9db504b9ac784019e7e8c215c31abfccb62b6 test: enable p2p_sendtxrcncl.py with v2transport (Martin Zumsande)
Pull request description:
#24748 added v2 transport to the python `P2PConnection`, but so far each test that wants to make use of it needs to enable it on an individual basis.
This PR changes it so that if the test suite is run with `--v2transport` option, v2 is used in each test by default, not only for connections between two bitcoind instances as before, but also wherever `P2PConnection` is used. Individual tests can override this global option.
To do that, a few tests need to be adjusted.
In addition, I added a commit to always use v1 in a few select subtests that send a large number of large messages (e.g. large reorgs). These tests don't have a fundamental problem with v2 but become very slow due to the unoptimised python ChaCha20 implementation (~30 minutes on my computer, so probably not suitable to be run in the CI).
As a result, `python3 test_runner.py --v2transport` should succeed and use `v2` everywhere (unless v1 is chosen explicitly).
[Edit]: To make the "test each commit" CI pass, several test fixes were squashed into the last commit, which actually enables v2 p2p for `P2PConnection`. I have an unsquashed version at https://github.com/mzumsande/bitcoin/tree/202401_bip324_alltests_unsquashed, in case that helps with review.
ACKs for top commit:
fjahr:
tACK bf5662c678455fb47c402f8520215726ddfe7a94
vasild:
ACK bf5662c678455fb47c402f8520215726ddfe7a94
stratospher:
reACK bf5662c6.
theStack:
Tested ACK bf5662c678455fb47c402f8520215726ddfe7a94
Tree-SHA512: 4f5a08248ba8a755f7d0f48deb2b79bef03292345cacb7deef01be955481093800e4e56ff218ea56734eef5de1fb3ab0f04657447ea27d393bb536539d7b289d
fa3a4102ef0ae06d8930d7a7b567759e2a5b5fde fuzz: Set -rss_limit_mb=8000 for generate as well (MarcoFalke)
fa4e396e1da8e5b04a5f906b95017b969ea37bae fuzz: Generate with random libFuzzer settings (MarcoFalke)
Pull request description:
Sometimes a libFuzzer setting like `-use_value_profile=1` helps [0], sometimes it hurts [1].
[0] https://github.com/bitcoin/bitcoin/pull/20789#issuecomment-752961937
[1] https://github.com/bitcoin/bitcoin/pull/27888#issuecomment-1645976254
By picking a random value, it is ensured that at least some of the runs will have the beneficial configuration set.
Also, set `-max_total_time` to prevent slow fuzz targets from getting a larger time share, or possibly peg to a single core for a long time and block the python script from exiting for a long time. This can be improved in the future. For example, the python script can exit after some time (https://github.com/bitcoin/bitcoin/pull/20752#discussion_r549248791). Alternatively, it can measure if coverage progress was made and run for less time if no progress has been made recently anyway, so that more time can be spent on targets that are new or still make progress.
ACKs for top commit:
murchandamus:
utACK fa3a4102ef0ae06d8930d7a7b567759e2a5b5fde
dergoegge:
utACK fa3a4102ef0ae06d8930d7a7b567759e2a5b5fde
brunoerg:
light ACK fa3a4102ef0ae06d8930d7a7b567759e2a5b5fde
Tree-SHA512: bfd04a76ca09aec612397bae5f3f263a608faa7087697169bd4c506c8195c4d2dd84ddc7fcd3ebbc75771eab618fad840af819114968ca3668fc730092376768
5f240ab2e89fb20286fbaf9a1f00346bb1cad5a1 test: Add option to skip unit tests for the test runner (Martin Zumsande)
Pull request description:
In the python `test_runner`, it's possible to disable specific functional tests (or just enable a few specific ones), but the unit tests for the python test framework cannot be skipped.
Add this option (`--skipunit` or `-u`), it would save some time for devs not interested in running those every time.
ACKs for top commit:
fjahr:
re-ACK 5f240ab2e89fb20286fbaf9a1f00346bb1cad5a1
tdb3:
Code review and tested ACK 5f240ab2e89fb20286fbaf9a1f00346bb1cad5a1
stratospher:
tested ACK 5f240ab.
Tree-SHA512: f7c9cfefc18a6510e24ca4601309b40fdf4180a4c5fe592be9cf7607be6541784b283c46c8d6e60740ff3eba83025dd5d0db36e55bf8bad1404b38120859e113
fa58ae74ea67485495dbc2d003adfbca68d6869b refactor: Add missing include for USE_BDB, USE_SQLITE to bench/wallet_ismine.cpp (MarcoFalke)
fa31908ea848488ff842f1b9fce6235bb8855ec7 lint: Check for missing or redundant bitcoin-config.h includes (MarcoFalke)
fa63b0e351dee4782ee19ad46603957ef8d020eb lint: Make lint error easier to spot in output (MarcoFalke)
fa770fd368e32950dd727e111a5d66e1dbb93716 doc: Add missing RUST_BACKTRACE=1 (MarcoFalke)
fa100512677587b4e84a8be2235cf6d49b6a0134 lint: Add get_subtrees() helper (MarcoFalke)
Pull request description:
Missing `bitcoin-config.h` includes are problematic, because the build could silently pass, but produce an unintended result. For example, a slower fallback algorithm could be picked, even though `bitcoin-config.h` indicates that a faster feature is available and should be used.
As the build succeeds silently, this problem is not possible to detect with iwyu.
Thus, fix this by using a linter based on grepping the source code.
ACKs for top commit:
theuni:
Weak ACK fa58ae74ea67485495dbc2d003adfbca68d6869b.
TheCharlatan:
ACK fa58ae74ea67485495dbc2d003adfbca68d6869b
hebasto:
ACK fa58ae74ea67485495dbc2d003adfbca68d6869b, tested on Ubuntu 23.10 -- it catches bugs properly. I didn't review rust code changes.
Tree-SHA512: cf4346f81ea5b8c215da6004cb2403d1aaf569589613c305d8ba00329b82b3841da94fe1a69815ce15f2edecbef9b031758ec9b6433564976190e3cf91ec8181
fab15723b0518acbb1015e64df47dcac0187e92f test: Fix SegwitV0SignatureMsg nLockTime signedness (MarcoFalke)
Pull request description:
It is unsigned in Bitcoin Core, so the tests should match it:
5b8990a1f3/src/script/interpreter.cpp (L1611)
The bug was introduced when the code was written in 330b0f31ee5719d94f9e52dfc83c5d82168241f9.
(Lowercase `i` means signed, see https://docs.python.org/3/library/struct.html#format-characters)
ACKs for top commit:
epiccurious:
Tested ACK fab15723b0518acbb1015e64df47dcac0187e92f.
Eunovo:
Tested ACK fab15723b0
achow101:
ACK fab15723b0518acbb1015e64df47dcac0187e92f
Tree-SHA512: 68cb8582f6af22e6abb2fc9d6770277501baa0f9873e2e8d47699e87096fc5d4b9de45fa07199757b6e945c99d4c4ea95f01478322f2c093ef95cf5e0c78522b
Adds test in mempool_accept to check if a negative maxfeerate is inputed
into check_mempool_result, asserts "Amount out of range" error message
and -3 error code
8d20602e555dfe026b421363ee32cfca17c674d8 test, assumeutxo: Add test to ensure failure when mempool not empty (Hernan Marino)
Pull request description:
Add a test to ensure that loadtxoutset fails when the node's mempool is not empty, as suggested by maflcko here: https://github.com/bitcoin/bitcoin/pull/27596#discussion_r1344713537
ACKs for top commit:
maflcko:
re-ACK 8d20602e555dfe026b421363ee32cfca17c674d8
BrandonOdiwuor:
ACK 8d20602e555dfe026b421363ee32cfca17c674d8
Tree-SHA512: 97c9668c0f38897934bf0d326515d898d4e682ff219deba9d751b35125b5cf33d51c9df116a74117ecf0394f28995a3d0cae1266b1e5acb4365ff4f309ce3f6c
