Whenever a non-topological permutation is encountered, fast forward to the
last permutation with the same non-topological prefix, skipping over
potentially many permutations that are non-topological for the same reason.
With that, increase the checking of all permutations to clusters of size 8
instead of 7.
The separates the existing fuzz test into:
* clusterlin_linearize: establishes the correctness of Linearize() using the
simpler SimpleLinearize() function.
* clusterlin_simple_linearize: establishes the correctness of SimpleLinearize() by
comparing with all valid linearizations computed by
std::next_permutation.
rather than combining the first two into a single fuzz test.
This separates the existing fuzz test into:
* clusterlin_search_finder: establishes SearchCandidateFinder's correctness using the
simpler SimpleCandidateFinder.
* clusterlin_simple_finder: establishes SimpleCandidateFinder's correctness using the
(even) simpler ExhaustiveCandidateFinder.
rather than trying to do both at once.
Only count the number of actual new subsets added. If the queue contains
a work item that completely covers a component, no transaction can be added
to it without creating a disconnected component. In this case, also don't
count it as an iteration.
With this, the number of iterations performed by SimpleCandidateFinder is
bounded by the number of distinct connected topologically-valid subsets of
the cluster.
Additionally this commit gives each test its
own function.
The assert_submitblock helper is absorbed into
assert_template.
Review hint:
git show --color-moved=dimmed-zebra
Comments are expanded.
Return BlockValidationState instead of passing a reference.
Lock Chainman mutex instead of cs_main.
Remove redundant chainparams and pindexPrev arguments.
Drop defaults for checking proof-of-work and merkle root.
The ContextualCheckBlockHeader check is moved to after CheckBlock,
which is more similar to normal validation where context-free checks
are done first.
Validation failure reasons are no longer printed through LogError(),
since it depends on the caller whether this implies an actual bug
in the node, or an externally sourced block that happens to be invalid.
When called from getblocktemplate, via BlockAssembler::CreateNewBlock(),
this method already throws an std::runtime_error if validation fails.
Additionally it moves the inconclusive-not-best-prevblk check from RPC
code to TestBlockValidity.
There is no behavior change when callling getblocktemplate with proposal.
Previously this would return a BIP22ValidationResult which can throw for
state.IsError(). But CheckBlock() and the functions it calls only use
state.IsValid().
The final assert is changed into Assume, with a LogError.
Co-authored-by: <Ryan Ofsky <ryan@ofsky.org>
ce90f0c99f rpc, wallet, refactor: Remove non-descriptor errors (pablomartin4btc)
573bcd75d7 wallet, refactor: Remove unused SetupGeneration (pablomartin4btc)
5431f2dc21 wallet, refactor: Remove Legacy warnings and errors (pablomartin4btc)
Pull request description:
Remove dead code due to legacy wallet support removal.
These changes have no impact on functionality. They are transparent to the end user, as legacy wallets can't be created or loaded anymore, so these checks are no longer reached. The legacy-to-descriptor wallet migration flow is not affected either, as these removals are not part of its process.
ACKs for top commit:
achow101:
ACK ce90f0c99f
rkrux:
utACK ce90f0c99f
Tree-SHA512: 9229ad9dda9ff1dece73b5b15a20d69c6ab1ff2c75b2ec430ddbbaeb3467f6a850f53df527bcb4a8114ccbf1aa9c794462d71a8d516aed6f9a9da74edae16feb
0def84d407 test: Verify parent_desc in RPCs (Ava Chow)
2554cee988 test: Enable default wallet for wallet_descriptor.py (Ava Chow)
3fc9d9f241 wallet, rpc: Push the normalized parent descriptor (Ava Chow)
Pull request description:
Instead of prividing the descriptor string as stored in the db, use the normalized descriptor as is done for getaddressinfo's parent_desc field.
Split from #32489
ACKs for top commit:
Sjors:
re-utACK 0def84d407
rkrux:
ACK 0def84d407
w0xlt:
reACK 0def84d407
Tree-SHA512: 575c5b545d6f0aa7e135696b7a55c004e754fca4dd35dd9cf71b0b45b49a2e86e7b20570e768534d587005953bb893645379ec1ba4f98cfd26811f9c2f17de2d
130a922980 wallet, interfaces: Use BERKELEY_RO in isEncrypted (Ava Chow)
Pull request description:
The GUI uses `WalletLoader::isEncrypted()` to detect whether a wallet file is encrypted so that it knows whether to prompt for a passphrase when migrating a legacy wallet. However, legacy wallets need to be opened with `options.require_format = BERKELEY_RO`. Since this wasn't being provided, following #28710, encrypted legacy wallets could not be migrated.
This fixes the issue by detecting when a wallet file is for a legacy wallet, and re-attempting with `options.require_format = BERKELEY_RO` in that case.
Depends on #32449 for `DatabaseStatus::FAILED_LEGACY_DISABLED`
ACKs for top commit:
davidgumberg:
Tested ACK 130a922980
furszy:
utACK 130a922980
pablomartin4btc:
tACK 130a922980
w0xlt:
Code review ACK 130a922980
rkrux:
utACK 130a922980
Tree-SHA512: aa70defc3b5f41635333a4d83c46ecdb5cd3cb129d590b4c0fe7a5f16e8aeaba1592f932ead242ed5f84524b146d87319154f4a1820bb34d9e80f63d24fc6b20
206bc05e62 test: remove unnecessary m_best_header setting hack in feature_assumeutxo.py (Sebastian Falbesoner)
Pull request description:
The mentioned bug (issue #26245) has been fixed in PR #30666, so the workaround is not needed anymore.
ACKs for top commit:
fjahr:
ACK 206bc05e62
janb84:
ACK 206bc05e62
mzumsande:
utACK 206bc05e62
Tree-SHA512: 92ad6c09c80318a02e00d8c2a4ca1ec8a6caa6aa310916ff705871c91fe5c960c15ed394ead8ce9c7466b592dba176aa87a635b3228ddc93f57082124f04613f
61ea5f348d fuzz: doc: add info about `afl-system-config` for macOS (brunoerg)
Pull request description:
`afl-system-config` adjusts the shared memory segment size limits and configures kernel parameters for better fuzzing performance. Since macOS has more conservative values on shared memory, it's necessary to run `afl-system-config`, or manually adjust the values to fuzz with AFL++.
e.g.:
```sh
kern.sysv.shmmax: 524288000
kern.sysv.shmmin: 1
kern.sysv.shmseg: 48
kern.sysv.shmall: 131072000
```
ACKs for top commit:
janb84:
reACK [61ea5f3](61ea5f348d)
w0xlt:
ACK 61ea5f348d
Crypt-iQ:
ACK 61ea5f3
Tree-SHA512: 8fd9fcb397a6b22e344ac37d3d18052ce8b6b32a5118ab3554c49116d7dd8769cdcbad41f0784d9ab525d4284a82e3d7a89b2c66966b725bf3781d9389d7454d
Seems to have been broken since conversion from Boost in #20744. The std::filesystem iteration aborts upon failure while Boost might have allowed skipping over faulty entries.
d7c37906e7 build: patch cmake min version on freetype (josibake)
Pull request description:
## Problem
While doing a depends build with CMake 4.0.1, I got the following error:
```
Extracting freetype...
/root/bitcoin/depends/sources/freetype-2.11.0.tar.xz: OK
Preprocessing freetype...
Configuring freetype...
CMake Error at CMakeLists.txt:100 (cmake_minimum_required):
Compatibility with CMake < 3.5 has been removed from CMake.
Update the VERSION argument <min> value. Or, use the <min>...<max> syntax
to tell CMake that the project requires at least <min> but has been updated
to work with policies introduced by <max> or earlier.
Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.
-- Configuring incomplete, errors occurred!
make: *** [funcs.mk:343: /root/bitcoin/depends/x86_64-pc-linux-gnu/.freetype_stamp_configured] Error 1
make: Leaving directory '/root/bitcoin/depends'
```
.. which led me to https://cmake.org/cmake/help/latest/release/4.0.html#deprecated-and-removed-features, which states compatibility with CMake versions less than 3.5 has been removed in 4.0.
## Fix
Based on the suggestion from the error message (and from reading the CMake docs), I added `-DCMAKE_POLICY_VERSION_MINIMUM=3.22`. I picked `3.22` (as opposed to 3.5) since that is the minimum version of CMake we specify in `doc/dependencies.md`. Would be nice if there was a way to pipe the min version in as a variable (since presumably we'd want to update this to be in lock step with the minimum CMake version of the whole project), but I couldn't think of a simple way to do this. Open to suggestions on a more robust way to do this if this is deemed too brittle.
ACKs for top commit:
fanquake:
ACK d7c37906e7
hebasto:
ACK d7c37906e7. Tested on Ubuntu 25.04 with both cmake 4.0.2 and the default cmake 3.31.6.
Tree-SHA512: fb664ec73bfffc504f1dcc9076072307f443d056d14325de41c4a29f3ee4077f1922e79b5895b49e7354f45ad6a35be4973c153c2baf3376df6c0d209efc9c54
Dropped the default expected_hash parameter from `ReadBlock()`.
In `blockmanager_flush_block_file` tests, we pass {} since the tests would already fail at PoW validation for corrupted blocks.
In `ChainstateManager::LoadExternalBlockFile`, we pass {} when processing child blocks because their hashes aren't known beforehand.
Switch to the index-aware `ReadBlock()` overload in `ComputeFilter` so that filter creation will abort if the stored block header hash doesn't match the expected one.
In the `readwriteblock` benchmark, pass the expected hash to `ReadBlock()` to match the new signature without affecting benchmark performance.
The non-recent-block code path in `ProcessGetBlockData` already has `inv.hash` available (equaling `pindex->GetBlockHash()`).
Pass it to `ReadBlock()` and assert that the on-disk header matches the requested hash.
The `GETBLOCKTXN` message handler in `ProcessMessage` receives `req.blockhash` from the peer (equaling `pindex->GetBlockHash()`).
Pass this hash to `ReadBlock()` for verification and assert that the index lookup matches.
Co-authored-by: TheCharlatan <seb.kung@gmail.com>
Co-authored-by: Hodlinator <172445034+hodlinator@users.noreply.github.com>
The catchup loop in the outbound eviction functional test currently has
a small flaw, as the contained waiting for a `getheaders` message just
waits for any such message instead of one with the intended block hash.
The reason is that the `prev_prev_hash` variable is set incorrectly,
since the `tip_header` instance is not updated and its field `.hash` is
None. Fix that by updating `tip_header` and use the correct field -- we
want the tip header's previous hash (`.hashPrevBlock`).
029ba1a21d index: remove CBlockIndex access from CustomAppend() (furszy)
91b7ab6c69 refactor: index, simplify CopyHeightIndexToHashIndex to process single block (furszy)
6f1392cc42 indexes, refactor: Remove remaining CBlockIndex* uses in index Rewind methods (Ryan Ofsky)
0a248708dc indexes, refactor: Stop requiring CBlockIndex type to call IsBIP30Unspendable (Ryan Ofsky)
331a25cb16 test: indexes, avoid creating threads when sync runs synchronously (furszy)
Pull request description:
Combining common refactors from #24230 and #26966, aiming to move both efforts forward while reducing their size and review burden.
Broadly, #24230 focuses on enabling indexes to run in a separate process, and #26966 aims to parallelize the indexes initial synchronization process. A shared prerequisite for both is ensuring that only the base index class interacts with the node’s chain internals - child index classes should instead operate solely through chain events.
This PR moves disk read lookups from child index classes to the base index class. It also includes a few documentation improvements and a test-only code cleanup.
ACKs for top commit:
maflcko:
review ACK 029ba1a21d👡
achow101:
ACK 029ba1a21d
TheCharlatan:
Re-ACK 029ba1a21d
davidgumberg:
ACK 029ba1a21d
mzumsande:
Code Review ACK 029ba1a21d
Tree-SHA512: f073af407fc86f228cb47a32c7bcf2241551cc89ff32059317eb81d5b86fd5fda35f228d2567e0aedbc9fd6826291f5fee05619db35ba44108421ae04d11e6fb
IsValid() also returns false for blocks that have not been
validated yet up to the default validity level of BLOCK_VALID_TRANSACTIONS but
are not marked as invalid - e.g. if we only know the header.
Here, we specifically want to filter for invalid blocks.
Also removes the default arg from IsValid() which is now unused outside
of tests, to prevent this kind of misuse for the future.
Co-authored-by: TheCharlatan <seb.kung@gmail.com>
when we reconsiderblock, previously only block and it's
descendants were considered as chain tip candidates/inserted into
setBlockIndexCandidates
ex: on this chain, with block 4 invalidated
1 -> 2 -> 3 -> 4 -> 5 -> 6 -> header 7
blocks 4, 5, 6, header 7 have BLOCK_FAILED_* flags set
previously:
- if we reconsiderblock header 7, the chain would have all the
BLOCK_FAILED_* flags cleared but would report chain tip as block 3.
- after restart, it reports correct chain tip block 6.
now:
- if we reconsiderblock header 7, the correct chain tip block 6 is
reported since ancestors are also considered as chain tip
candidates/inserted into setBlockIndexCandidates.
Co-authored-by: Martin Zumsande <mzumsande@gmail.com>
When we call reconsiderblock for some block, ResetBlockFailureFlags puts the descendants of that block
into setBlockIndexCandidates (if they meet the criteria, i.e. have more work than the tip etc.)
We also clear the failure flags of the ancestors, but we never put any of those into setBlockIndexCandidates
this is wrong and could lead to failures in CheckBlockIndex().
fa9ca13f35 refactor: Sort includes of touched source files (MarcoFalke)
facb152697 scripted-diff: Bump copyright headers after include changes (MarcoFalke)
fae71d30f7 clang-tidy: Apply modernize-deprecated-headers (MarcoFalke)
Pull request description:
Bitcoin Core is written in C++, so it is confusing to sometimes use the deprecated C headers (with the `.h` extension). For example, it is less clear whether `string.h` refers to the file in this repo or the cstring stdlib header (https://github.com/bitcoin/bitcoin/pull/31308#discussion_r2121492797).
The check is currently disabled for headers, to exclude subtree headers.
ACKs for top commit:
l0rinc:
ACK fa9ca13f35
achow101:
ACK fa9ca13f35
janb84:
ACK fa9ca13f35
stickies-v:
ACK fa9ca13f35
Tree-SHA512: 6639608308c598d612e24435aa519afe92d71b955874b87e527245291fb874b67f3ab95d3a0a5125c6adce5eb41c0d62f6ca488fbbfd60a94f2063d734173f4d
4ef6253017 test: avoid unneeded (w)txid hex -> integer conversions (Sebastian Falbesoner)
472f3770ae scripted-diff: test: rename CTransaction `.getwtxid()` -> `wtxid_hex` for consistency (Sebastian Falbesoner)
81af4334e8 test: rename CTransaction `.sha256` -> `.txid_int` for consistency (Sebastian Falbesoner)
ce83924237 test: rename CTransaction `.rehash()`/`.hash` -> `.txid_hex` for consistency (Sebastian Falbesoner)
e9cdaefb0a test: introduce and use CTransaction `.wtxid_int` property (Sebastian Falbesoner)
9b3dce24a3 test: remove bare CTransaction `.rehash()`/`.calc_sha256()` calls (Sebastian Falbesoner)
a2724e3ea3 test: remove txid caching in CTransaction class (Sebastian Falbesoner)
Pull request description:
In the functional test framework, determining a (w)txid for a `CTransaction` instance is currently rather confusing and footgunny due to inconsistent naming/interfaces (see table below) and statefulness involved. This PR aims to improve that by:
* removing the (w)txid caching mechanism, in order to avoid the need to call additional rehashing functions (`.rehash()`/`.calculate_sha256()`, see first two commits and https://github.com/bitcoin/bitcoin/pull/32050#discussion_r1993286997). This change in theory decreases the performance, as the involved serialization and hashing involved might be called more often than previously, but I couldn't find a functional test where this leads to a measurable run-time increase on my machine.
* introduce consistent naming that shows the type of the returned txid, i.e. hex string vs. test-framework-internal representation [currently integers] (see remaining commits)
Summary table showing (w)txid determaination before/after this PR:
| Task | master | PR |
|:-----------------------|:-----------------------|:-------------|
| get TXID (hex string) | `.rehash()` / `.hash`[1] | `.txid_hex` |
| get TXID (integer) | `.sha256`[1] | `.txid_int` |
| get WTXID (hex string) | `.getwtxid()` | `.wtxid_hex` |
| get WTXID (integer) | `.calc_sha256(True)` | `.wtxid_int` |
Unfortunately, most renames can't be done with a scripted-diff, as the property names (`.hash`, `.sha256`) are also used for blocks and other message types. The PR is rather invasive and touches a lot of files, but I think it's worth to do it, also to make life easier for new contributors. Future tasks like e.g. doing the same overhaul for block (header) objects or getting rid of the integer representation (see https://github.com/bitcoin/bitcoin/pull/32050) become easier should become easier after this one.
[1] = returned value might be out-of-date, if rehashing function wasn't called after modification
ACKs for top commit:
maflcko:
re-ACK 4ef6253017🏈
achow101:
ACK 4ef6253017
marcofleon:
code review ACK 4ef6253017
Tree-SHA512: 4b472c31d169966b6f6878911a8404d25bf3e503b6e8ef30f36a7415d21ad4bc1265083af2d3ead6edfcd9fac9ccb0a8be57e1b0739ad431b836413070d7d583
f6b782f3aa doc: Improve m_best_header documentation (Martin Zumsande)
ee673b9aa0 validation: remove m_failed_blocks (Martin Zumsande)
ed764ea2b4 validation: Add more checks to CheckBlockIndex() (Martin Zumsande)
9a70883002 validation: in invalidateblock, calculate m_best_header right away (Martin Zumsande)
8e39f2d20d validation: in invalidateblock, mark children as invalid right away (Martin Zumsande)
4c29326183 validation: cache all headers with enough PoW in invalidateblock (Martin Zumsande)
15fa5b5a90 validation: call InvalidBlockFound also from AcceptBlock (Martin Zumsande)
Pull request description:
Some fields in validation are set opportunistically by "best effort":
- The `BLOCK_FAILED_CHILD` status (which means that the block index has an invalid predecessor)
- `m_best_header` (the most-work header not known to be invalid).
This means that there are known situations in which these fields are not set when they should be, or set to wrong values. This is tolerated because the fields are not used for anything consensus-critical and triggering these situations involved creating invalid blocks with valid PoW header, so would have a cost attached. Also, having stricter guarantees for these fields requires iterating over the entire block index, which has some DoS potential, especially with any header above the checkpoint being accepted int he past (see e.g. #11531).
However, there are reasons to change this now:
- RPCs use these fields and can report wrong results
- There is the constant possibility that someone could add code that expects these fields to be correct, especially because it is not well documented that these fields cannot always be relied upon.
- DoS concerns have become less of an issue after #25717 - now an attacker would need to invest much more work because they can't fork off the last checkpoint anymore
This PR continues the work from #30666 to ensure that `BLOCK_FAILED_CHILD` status and `m_best_header` are always correct:
- it adds a call to `InvalidChainFound()` in `AcceptBlock()`.
- it adds checks for `BLOCK_FAILED_CHILD` and `m_best_header` to `CheckBlockIndex()`. In order to be able to do this, the existing cache in the RPC-only `InvalidateBlock()` is adjusted to handle these as well. These are performance optimizations with the goal of avoiding having a call of `InvalidChainFound()` / looping over the block index after each disconnected block.
I also wrote a fuzz test to find possible edge cases violating `CheckBlockIndex`, which I will PR separately soon.
- it removes the `m_failed_blocks` set, which was a heuristic necessary when we couldn't be sure if a given block index had an invalid predecessor or not. Now that we have that guarantee, the set is no longer needed.
ACKs for top commit:
stickies-v:
re-ACK f6b782f3aa
achow101:
reACK f6b782f3aa
ryanofsky:
Code review ACK f6b782f3aa with only minor code & comment updates
TheCharlatan:
Re-ACK f6b782f3aa
Tree-SHA512: 1bee324216eeee6af401abdb683abd098b18212833f9600dbc0a46244e634cb0e6f2a320c937a5675a12af7ec4a7d10fabc1db9e9bc0d9d0712e6e6ca72d084f
fa0b766f43 test: Remove intermittent and presumed fixed tsan race suppressions (MarcoFalke)
fa4b659dcd test: Explain how to reproduce zmq:: upstream race (MarcoFalke)
Pull request description:
An explanation makes it easier to reproduce, if needed.
ACKs for top commit:
fanquake:
ACK fa0b766f43
Tree-SHA512: 4857cc1e2c97e3d8c194fd12d0bb2a3293136c51ae1b89e0320161d1b8f22ef5122519e099288e52e42bb828ee4a56bfdfbe80717d95178748b76dd7209e12db
8713e8060d depends: fix SHA256SUM command on OpenBSD (use GNU mode output) (Sebastian Falbesoner)
2d938720bd depends: add patch to fix capnp build on OpenBSD (Sebastian Falbesoner)
Pull request description:
This PR fixes the multiprocess depends build for OpenBSD by applying upstream patch https://github.com/capnproto/capnproto/pull/2308 and switching the SHA256SUM command to output hash sums in the expected format (the default is BSD format [1], but we need GNU format [2], see commit message for details). Note that the hashing issue is only prevailing for packages defining the `$(package)_local_dir` variable (introduced in 5d105fb8c3, part of #31741), where the following line of the `fetch_local_dir_sha256` function leads to the wrong output:
ae024137bd/depends/funcs.mk (L57)
The first commit can be replaced with a simple capnp version bump once this is available in a release.
Tested on OpenBSD 7.7 (x86_64) via
```
$ gmake -C depends MULTIPROCESS=1 NO_BOOST=1 NO_LIBEVENT=1 NO_QT=1 NO_QR=1 NO_WALLET=1 NO_ZMQ=1 NO_USDT=1
```
[1] example output: `SHA256 (/home/thestack/.vimrc) = 6ba69d100e8c5ca0488ded6293d4e5f740a6a5d5ace96cbcf0599c18d27389e4`
[2] example output: `6ba69d100e8c5ca0488ded6293d4e5f740a6a5d5ace96cbcf0599c18d27389e4 /home/thestack/.vimrc`
ACKs for top commit:
Sjors:
ACK 8713e8060d
hebasto:
ACK 8713e8060d.
fanquake:
ACK 8713e8060d
Tree-SHA512: 178b8b41156e1f1eea101849110167d2636c3093b6a68c88a91a994f0750831aa02e415eb2793c522682c92cb3085de025300e0e2dee894e112dd7e1f495cc08
Both are rational numbers. Client software should only use them to
display information to humans. Followup calculations should use the
underlying values such as target.
Therefore it's not necessary to test the handling of these floating
point values. Round them down to avoid spurious test failures.
Fixes#32515
