BlockManager is a large data structure, and cs_main is not required to
take its address or access every part of it. Individual BlockManager
fields and methods which do require cs_main like m_block_index and
LookupBlockIndex are already annotated separately, and these other
annotations describe locking requirements more accurately and do a
better job enforcing thread safety.
Since cs_main is not needed to access the address of the m_block object,
this commit drops cs_main LOCK calls which were added pointlessly to
satisfy this annotation in the past.
Co-authored-by: Carl Dong <contact@carldong.me>
This is a refactor and safe to do because:
* UnloadBlockIndex calls ChainstateManager::Unload, which calls
BlockManager::Unload
* Only unit tests call Unload directly
fa7efc915b87ec56ca1cc0bad7d8f79591bfa099 Fixup style of moved code (MarcoFalke)
fade2a44f4aabc64185031dbf4c70d875ece6740 Move BlockManager to node/blockstorage (MarcoFalke)
Pull request description:
`BlockManager` is responsible for reading and writing block(headers). So move it to the existing `blockstorage` module in `node`. Also, move validation code unrelated to block-storage out from `BlockManager`.
ACKs for top commit:
ryanofsky:
Code review obvious ACK fa7efc915b87ec56ca1cc0bad7d8f79591bfa099
Tree-SHA512: 0197943d818e5f59e743b07fbb92e7661bff90081127a41e35e5692ce49d6f6a7872448670b0da282f7714580a45c8d93e571a67177c8b5f785ce9edefe834c5
b4adc5ad6769e4a5a6179dfff271cd4c9dc47a5b [bugfix] update lockpoints correctly during reorg (glozow)
b6002b07a36f0d58dc6becd04bfcf78599056b7c MOVEONLY: update_lock_points to txmempool.h (glozow)
Pull request description:
I introduced a bug in #22677 (sorry! 😅)
Mempool entries cache `LockPoints`, containing the first height/blockhash/`CBlockIndex*` at which the transaction becomes valid. During a reorg, we re-check timelocks on all mempool entries using `CheckSequenceLocks(useExistingLockPoints=false)` and remove any now-invalid entries. `CheckSequenceLocks()` also mutates the `LockPoints` passed in, and we update valid entries' `LockPoints` using `update_lock_points`. Thus, `update_lock_points(lp)` needs to be called right after `CheckSequenceLocks(lp)`, otherwise we lose the data in `lp`. I incorrectly assumed they could be called in separate loops.
The incorrect behavior introduced is: if we have a reorg in which a timelocked mempool transaction is still valid but becomes valid at a different block, the cached `LockPoints` will be incorrect.
This PR fixes the bug, adds a test, and adds an assertion at the end of `removeForReorg()` to check that all mempool entries' lockpoints are valid. You can reproduce the bug by running the test added in the [test] commit on the code before the [bugfix] commit.
ACKs for top commit:
jnewbery:
ACK b4adc5ad6769e4a5a6179dfff271cd4c9dc47a5b
vasild:
ACK b4adc5ad6769e4a5a6179dfff271cd4c9dc47a5b
mzumsande:
Code Review ACK b4adc5ad6769e4a5a6179dfff271cd4c9dc47a5b
hebasto:
ACK b4adc5ad6769e4a5a6179dfff271cd4c9dc47a5b
MarcoFalke:
re-ACK b4adc5ad6769e4a5a6179dfff271cd4c9dc47a5b 🏁
Tree-SHA512: 16b59f6ff8140d0229079ca1c6b04f2f4a00a2e49931275150e4f3fe5ac4ec109698b083fa6b223ba9511f328271cc1ab081263669d5da020af7fee83c13e401
fadd73037e266edb844f0972e82e9213171ef214 refactor: Remove implicit-integer-sign-change suppressions in validation.cpp (MarcoFalke)
Pull request description:
A file-wide suppression is problematic because it will wave through future violations, potentially bugs.
Fix that by using per-statement casts.
ACKs for top commit:
shaavan:
ACK fadd73037e266edb844f0972e82e9213171ef214
theStack:
Code-review ACK fadd73037e266edb844f0972e82e9213171ef214
Tree-SHA512: a8a05613be35382b92d7970f958a4e8f4332432056eaa9d72f6719495134b93aaaeea692899d9035654d0e0cf56bcd759671eeeacfd0535582c0ea048ab58a56
fa1a51cbc1c50a6d3adcad5ccea4c6067f89f7d3 doc: testnet3 was not reset and is doing BIP30 checks again (MarcoFalke)
Pull request description:
ACKs for top commit:
theStack:
ACK fa1a51cbc1c50a6d3adcad5ccea4c6067f89f7d3
Tree-SHA512: 793eccda583a3edb056b142c36a09a5c867f61d90b96e15e6643417d62eb651eb2f3429c5f245bdb062d18ab9bb05b5048c0888aa5a492cb7bb21a2f3f52324e
Previously, the `utxocache:flush` tracepoint was in the wrong scope and
reached every time `CChainState::FlushStateToDisk` was called, even when
there was no flushing of the cache. The tracepoint is now properly scoped
and will be reached during a full flush.
Inside the scope, the `fDoFullFlush` value will always be `true`, so it
doesn't need to be logged separately. Hence, it's dropped from the
tracepoint arguments.
fab6d6b2d154893ab422dda87f3535d42c3e06f4 Move pindexBestInvalid to ChainstateManager (MarcoFalke)
facd2137eceacb95e1f71c87ddc704d752b37272 Move m_failed_blocks to ChainstateManager (MarcoFalke)
fa47b5c100f81c65c15b5a6afaf6c91bc0861264 Move AcceptBlockHeader to ChainstateManager (MarcoFalke)
fa3d62cf7b3501a056b34c5458c14d2fe6a55bd7 Move FindForkInGlobalIndex from BlockManager to CChainState (MarcoFalke)
Pull request description:
Move globals or members of the wrong class to the right class.
ACKs for top commit:
naumenkogs:
ACK fab6d6b2d154893ab422dda87f3535d42c3e06f4
Sjors:
ACK fab6d6b2d154893ab422dda87f3535d42c3e06f4
shaavan:
ACK fab6d6b2d154893ab422dda87f3535d42c3e06f4
Tree-SHA512: 926cbdfa22838517497bacb79ed5f521f64117c2aacf96a0176f62831b4713314a32abc0213df5ee067edf63e4a4300f752a26006d36e5aab415bb91209a271f
The member is unrelated to block storage (BlockManager). It is related
to validation.
Fix the confusion by moving it.
Can be reviewed with
--color-moved=dimmed-zebra --color-moved-ws=ignore-all-space
The helper was moved in commit b026e318c39f59a06e29f1b25c7f577e01b25ccb,
which also mentioned that it could be moved to CChainState. So do that,
as the functionality is not block-storage related.
This also allows to drop one function argument.
2283b9cd1ee0fbd1e8ebc61673b1fe7596199a24 test: add tests for LoadBlockIndex when using multiple chainstates (James O'Beirne)
0fd599a51a700c028d6f7ed8067d2d9f6e6a04a5 validation: have LoadBlockIndex account for snapshot use (James O'Beirne)
d0c6e61f5dd3b6af818459d9d03b7ba316c5a3f7 validation: don't modify genesis during snapshot load (James O'Beirne)
Pull request description:
This is part of the [assumeutxo project](https://github.com/bitcoin/bitcoin/projects/11) (parent PR: #15606)
---
Currently, `BlockManager::LoadBlockIndex` adds all blocks that have downloaded transactions to the active chain state's `setBlockIndexCandidates` set, ignoring the background chain state.
This PR changes ChainstateManager::LoadBlockIndex to update `setBlockIndexCandidates` in the background chain, not just the active chain. In the active chain, the same blocks are added as before. In the background chain, only blocks that have actually been validated, not blocks marked assumed-valid are added so the background chain will continue to download and validate assumed-valid blocks.
ACKs for top commit:
MarcoFalke:
Concept ACK 2283b9cd1ee0fbd1e8ebc61673b1fe7596199a24 🤽
Sjors:
utACK 2283b9cd1ee0fbd1e8ebc61673b1fe7596199a24
Tree-SHA512: 7c9a80802df4722d85d12b78d2e7f628ac5f11cb8be66913d5c3230339bd1220c6723805509d4460826a17d1dc04b0ae172eb7d09ac0ea5dc5e41d77975cbd5e
Ensure that blocks past the snapshot base block (i.e. the end of the
assumed-valid region of the chain) are not included in
setBlockIndexCandidates for the background validation chainstate. These
blocks, while fully validated and lacking the BLOCK_ASSUMED_VALID flag,
*rely* on blocks which are assumed-valid, and so shouldn't be added to
the IBD chainstate.
Co-authored-by: Russ Yanofsky <russ@yanofsky.org>
Avoid modifying the genesis block index entry during snapshot load. This
is because, in a future change that fixes LoadBlockIndex for UTXO
snapshots, we detect block index entries that are reliant on
assumed-valid ancestors and treat them specially.
Since the genesis block doesn't have BLOCK_VALID_SCRIPTS, it would be
erroneously marked BLOCK_ASSUMED_VALID during snapshot load if we didn't
skip it here. This would cause a "setBlockIndexCandidates() empty"
assertion to be tripped since all block index entries would be marked
assume-valid due to genesis, which is never re-validated.
There's probably no good reason to modify the genesis block index entry
during snapshot load anyway...
It would make for sense for the TODO to be done in PR 17487
(or noted in the review feedback for a follow-up),
no need to continue maintaining the TODO in the codebase.
Use the `LOG_TIME_MILLIS_WITH_CATEGORY_MSG_ONCE` macro to improve the
logging of snapshot persistance and no longer manually track the duration.
before
[snapshot] flushing coins cache (0 MB)... done (0.00ms)
[snapshot] flushing snapshot chainstate to disk (0 MB)... done (0.00ms)
after
FlushSnapshotToDisk: flushing coins cache (0 MB) started
FlushSnapshotToDisk: completed (0.00ms)
FlushSnapshotToDisk: saving snapshot chainstate (0 MB) started
FlushSnapshotToDisk: completed (0.00ms)
The logging can be observed in the output of
./src/test/test_bitcoin -t validation_chainstate_tests -- DEBUG_LOG_OUT
7f15eff2ddd86034e84a19413e1a42883987f660 style-only: Remove redundant scope in *Chainstate (Carl Dong)
89bec827fdea1cedd560be85201f04e0a78aa48d Collapse the 2 cs_main locks in LoadChainstate (Carl Dong)
3b1584b794499158e0df07bd1bee1b803c568614 Remove all #include // for * comments (Carl Dong)
9a5a5a3d08b2c130ab9147914739ff3583b0dc84 test/setup: Use LoadChainstate (Carl Dong)
c541da0d62eaf5e96eca00d7508899f98bdfc1bc node/chainstate: Add options for in-memory DBs (Carl Dong)
ceb979034184345a662be4e3b327a573fbb31c63 node/caches: Remove intermediate variables (Carl Dong)
ac4bf138b849a8544798f3891d6623803040c265 node/caches: Extract cache calculation logic (Carl Dong)
15f2e33bb3d1ad3bc997f6a84956337f46495091 validation: VerifyDB only needs Consensus::Params (Carl Dong)
4da9c076d1cf12728730bb1f7e8906d4e9bfaba5 node/chainstate: Decouple from ShutdownRequested (Carl Dong)
05441c2dc5f60e2025476d8ec94c9025032d118c node/chainstate: Decouple from GetTime (Carl Dong)
2414ebc18b8bebf79c47e58a4293d0fc6420a811 init: Delay RPC block notif until warmup finished (Carl Dong)
8d466a8504bfb81ce8699d650aa72ec9cc8b0a54 Move -checkblocks LogPrintf to AppInitMain (Carl Dong)
aad8d597890c3707ae96fdb2b9fadc270ca574dd node/chainstate: Reduce coupling of LogPrintf (Carl Dong)
b345979a2b03b671c0984edd7e48e0baec2e2f34 node/chainstate: Decouple from concept of uiInterface (Carl Dong)
ca7c0b934db68acdc410e3a82f1ed898382da2e5 Split off VerifyLoadedChainstate (Carl Dong)
adf4912d77496b9a243476c5944528f95641f14d node/chainstate: Remove do/while loop (Carl Dong)
975235ca0a8f5bcf9df880698b3b0d4bbde9f7fb Move init logistics message for BAD_GENESIS_BLOCK to init.cpp (Carl Dong)
8715658983a0a07c56513acd8ded8dfc59c5c169 Move mempool nullptr Assert out of LoadChainstate (Carl Dong)
9162a4f93ef5aeb57fe11a6e09f5881cf431f5e6 node/chainstate: Decouple from concept of NodeContext (Carl Dong)
c7a5c46e6fd6d6ff46ca7a65fc3f0fff3cbdb24e node/chainstate: Decouple from ArgsManager (Carl Dong)
ae9121f958a4124ea6238cad0c3f2acb8b9eb4bb node/chainstate: Decouple from stringy errors (Carl Dong)
cbac28b72f5b831f6f84b7628f73e85627af3d94 node/chainstate: Decouple from GetTimeMillis (Carl Dong)
cb64af9635a9553e335f2dc0b1cca20c6bbd0933 node: Extract chainstate loading sequence (Carl Dong)
Pull request description:
This PR:
1. Coalesce the Chainstate loading sequence between `AppInitMain` and `*TestingSetup` (which makes it more tested)
2. Makes the Chainstate loading sequence reusable in preparation for future work extracting out our consensus engine.
Code-wise, this PR:
1. Extracts `AppInitMain`'s Chainstate loading sequence into a `::LoadChainstateSequence` function
2. Makes this `::LoadChainstateSequence` function reusable by
1. Decoupling it from various concepts (`ArgsManager`, `uiInterface`, etc)
2. Making it report errors using an `enum` rather than by setting a `bilingual_str`
3. Makes `*TestingSetup` use this new `::LoadChainstateSequence`
Reviewers: Aside from commentary, I've also included `git diff` flags of interest in the commit messages which I hope will aid review!
ACKs for top commit:
ryanofsky:
Code review ACK 7f15eff2ddd86034e84a19413e1a42883987f660. Thanks for updates!
MarcoFalke:
review ACK 7f15eff2ddd86034e84a19413e1a42883987f66 💳
Tree-SHA512: fb9a6cbd1c511a52b477c62a5e68e53a8be5dec2fff0e44a279966afb91efbab44bf1fe7c6b1519f8464ecc25f42dd4bae8e1efbf55ee91fc90fa0b92e3a83e2
Often when we're updating mempool entries we update entries that we
ultimately end up removing the updated entries shortly thereafter. This
patch makes it so that we filter for such entries a bit earlier in
processing, which yields a mild improvement for these cases, and is
negligible overhead otherwise.
dce8c4c38111556ca480aa0e63c46b71f66b508f rpc: getblockfrompeer (Sjors Provoost)
b884ababc29ce963826d8a4327ed6a5e629ff175 rpc: move Ensure* helpers to server_util.h (Sjors Provoost)
Pull request description:
This adds an RPC method to fetch a block directly from a peer. This can used to fetch stale blocks with lower proof of work that are normally ignored by the node (`headers-only` in `getchaintips`).
Usage:
```
bitcoin-cli getblockfrompeer HASH peer_n
```
Closes#20155
Limitations:
* you have to specify which peer to fetch the block from
* the node must already have the header
ACKs for top commit:
jnewbery:
ACK dce8c4c38111556ca480aa0e63c46b71f66b508f
fjahr:
re-ACK dce8c4c38111556ca480aa0e63c46b71f66b508f
Tree-SHA512: 843ba2b7a308f640770d624d0aa3265fdc5c6ea48e8db32269b96a082b7420f7953d1d8d1ef2e6529392c7172dded9d15639fbc9c24e7bfa5cfb79e13a5498c8
f1f10c0514fe81318c8b064f9dad0e2f9a2cd037 Remove CTxMemPool params from ATMP (lsilva01)
Pull request description:
Remove `CTxMemPool` parameter from `AcceptToMemoryPool` function, as suggested in https://github.com/bitcoin/bitcoin/pull/23437#issuecomment-962536149 .
This requires that `CChainState` has access to `MockedTxPool` in `tx_pool.cpp` as mentioned https://github.com/bitcoin/bitcoin/pull/23173#discussion_r731895386. So the `MockedTxPool` is attributed to `CChainState::m_mempool` before calling `AcceptToMemoryPool`.
Requires #23437.
ACKs for top commit:
jnewbery:
utACK f1f10c0514fe81318c8b064f9dad0e2f9a2cd037
MarcoFalke:
review ACK f1f10c0514fe81318c8b064f9dad0e2f9a2cd037 🔙
Tree-SHA512: 2a4885f4645014fc1fa98bb1090f13721c1a0796bc0021b9cb43bc8cc13920b6eaf057d1f5ed796e0a110e7813e41fe0196334ce7c80d1231fc057a9a3bdf349
During a reorg, we re-check timelocks on all mempool entries using
CheckSequenceLocks(useExistingLockPoints=false) and remove any
now-invalid entries. CheckSequenceLocks() also mutates the LockPoints
passed in, and we update valid entries' LockPoints using
update_lock_points. Thus, update_lock_points(lp) needs to be called
right after CheckSequenceLocks(lp), otherwise we lose the data in lp.
commit bedf246 introduced a bug by separating those two loops.
fa3942fc4c66d7624bd3578d1e7f4a6a7721c11a Remove GetSpendHeight (MarcoFalke)
Pull request description:
It is unclear what the goal of the helper is, as the caller already
knows the spend height before calling the helper.
Also, in case the coins view is corrupted, LookupBlockIndex will return
nullptr. Dereferencing a nullptr is UB.
Fix both issues by removing it. Also, add a sanity check, which aborts
if the coins view is corrupted.
ACKs for top commit:
laanwj:
Code review ACK fa3942fc4c66d7624bd3578d1e7f4a6a7721c11a
ryanofsky:
Code review ACK fa3942fc4c66d7624bd3578d1e7f4a6a7721c11a. I'm not aware of cases where coins GetBestBlock could be different from active chain tip, and asset seems sufficient to guarantee PR doesn't change behavior if that doesn't happen.
Tree-SHA512: 29f65d72e116ec5a4509e0947ceeaa5bb6b7dfd5d174d3c7945cb15fa266d590c4f8b48e6385de74ef7d7c84ebd2255de902ad9c87c24955348a91b12e5bffd5
It is unclear what the goal of the helper is, as the caller already
knows the spend height before calling the helper.
Also, in case the coins view is corrupted, LookupBlockIndex will return
nullptr. Dereferencing a nullptr is UB.
Fix both issues by removing it. Also, add a sanity check, which aborts
if the coins view is corrupted.
a64078e38563ef3ac5e5ec20c07569441c87eeee Break validation <-> txmempool circular dependency (glozow)
64e4963c635ec3a73a5fa3f32f6ec08e70609f60 [mempool] always assert coin spent (glozow)
bb9078ed51159fa162484f16993313ed6cf980e3 [refactor] put finality and maturity checking into a lambda (glozow)
bedf246f1e2497a3641093c6e8fa11fb34dddac4 [mempool] only update lockpoints for non-removed entries (glozow)
1b3a11e126b258fba975ed7c452221608f2c5472 MOVEONLY: TestLockPointValidity to txmempool (glozow)
Pull request description:
Remove 2 circular dependencies: validation - txmempool and validation - policy/rbf - txmempool
Validation should depend on txmempool (e.g. `CChainstateManager` has a mempool and we often need to know what's in our mempool to validate transactions), but txmempool is a data structure that shouldn't really need to know about chain state.
- Changes `removeForReorg()` to be parameterized by a callable that returns true/false (i.e. whether the transaction should be removed due to being now immature or nonfinal) instead of a `CChainState`. The mempool really shouldn't need to know about coinbase maturity or lockpoints, it just needs to know which entries to remove.
ACKs for top commit:
laanwj:
Code review ACK a64078e38563ef3ac5e5ec20c07569441c87eeee
mjdietzx:
reACK a64078e38563ef3ac5e5ec20c07569441c87eeee
theStack:
re-ACK a64078e38563ef3ac5e5ec20c07569441c87eeee
Tree-SHA512: f75995200569c09dfb8ddc09729da66ddb32167ff1e8a7e72f105ec062d2d6a9a390e6b4a2a115e7ad8ad3525f891ee1503f3cd2bed11773abcaf7c3230b1136
faad05c6d2ad1840c26f6fdea7d358bf9ba6efea Crash debug builds when mempool ConsensusScriptChecks fails (MarcoFalke)
Pull request description:
Currently a bug in the function might sneak around our testing infrastructure.
Fix that by turning bugs into crashes during tests.
ACKs for top commit:
glozow:
utACK faad05c6d2ad1840c26f6fdea7d358bf9ba6efea, there's something seriously wrong with the code if this returns false, good to throw in debug mode
Tree-SHA512: dfea1cd9ce3f1c303f49cca1417cd5c77c6ed12849aaff7b6ab1b6060f2f0c9cf5d4689017355d11f66639bab35823f65f848e6979042fa875181509dfd5d3d7
No behavior change.
Parameterize removeForReorg using a CChain and callable that
encapsulates validation logic. The mempool shouldn't need to know a
bunch of details about coinbase maturity and lock finality. Instead,
just pass in a callable function that says true/false. Breaks circular
dependency by removing txmempool's dependency on validation.
As node operators are free to set their mempool policies however they
please, it's possible for package transaction(s) to already be in the
mempool. We definitely don't want to reject the entire package in that
case (as that could be a censorship vector).
We should still return the successful result to the caller, so add
another result type to MempoolAcceptResult.
