mirror of
https://github.com/bitcoin/bitcoin.git
synced 2026-09-12 21:52:53 +02:00
c90c23d388
25bed560betest: add forward-compat functional test for txindex (sedited)703304ed8cdoc: add release notes for txindex disk usage and downgrading (Andrew Toth)8e5320a2d2tests: cover txindex hash prefix collisions and legacy fallback (Andrew Toth)b75efa19batxindex: skip bloom filters and legacy lookups for new databases (Andrew Toth)004d7c098ctxindex: hash key prefixes and pack block positions (Andrew Toth)5a255970fdrefactor: move txindex db constants and legacy key to txindex_key.h (Andrew Toth)327660134ctxindex: pass the full block to DB::WriteTxs (Andrew Toth)42771e7998txindex: use a new block locator for downgrade safety (Andrew Toth)4b08baed72txindex: return optional tx and block hash from FindTx (Andrew Toth) Pull request description: The current txindex uses the full 32-byte txid as keys, which takes up about 66 GB of disk space today on mainnet. Using a 5-byte key prefix instead drops the disk usage to 26 GB - cutting the size to less than half. Using the full 32-bytes is unnecessary since a 5-byte salted siphash will produce collisions in about 1 in 1.1 trillion. Some collisions will occur, but the penalty is just an extra disk read, deserialization and hash. The tx position can be appended to the key instead of used as a value, and a LevelDB iterator can seek to the prefix and then scan for the correct tx. This is an almost identical approach to `txospenderindex`. Also instead of storing the file position of the block, we can store only the sequence of the connected block and offset of the transaction in the block. This can be packed into a 6-byte key suffix using 3-byte representations of the sequence and offset in the block. The block file can be recovered by the CBlockIndex that is already in memory. The sequence is mapped to the block hash in the db, so we can lookup the block hash to find the CBlockIndex during reads. If a tx is not found with this method, we fallback to looking up the legacy entry. With this method a user with an existing db can opt to erase the `indexes/txindex` folder and reindex, or keep the current index and new entries will be appended with the smaller footprint. The time to index was faster on my machine with this method, 1h19m vs current 1h50m. Lookups are roughly the same, around 0.2ms per lookup with `getrawtransaction`. When testing on mainnet, I got 894,549 2-way collisions, 395 3-way collision, and 1 4-way collision that worst case could cause an extra 3 false positives when reading. ACKs for top commit: l0rinc: diff reACK25bed560besedited: ACK25bed560beajtowns: ACK25bed560beTree-SHA512: a25c79ca7e722e2f372b65f5fc11c8b194ad49f2240b4881c7e606306aabbd3604aede3f1c33606b467486affac3a3f503638f513c896935cebbc02709cb60d8
src/node/
The src/node/ directory contains code that needs to access node state
(state in CChain, CBlockIndex, CCoinsView, CTxMemPool, and similar
classes).
Code in src/node/ is meant to be segregated from code in
src/wallet/ and src/qt/, to ensure wallet and GUI
code changes don't interfere with node operation, to allow wallet and GUI code
to run in separate processes, and to perhaps eventually allow wallet and GUI
code to be maintained in separate source repositories.
As a rule of thumb, code in one of the src/node/,
src/wallet/, or src/qt/ directories should avoid
calling code in the other directories directly, and only invoke it indirectly
through the more limited src/interfaces/ classes.
This directory is at the moment
sparsely populated. Eventually more substantial files like
src/validation.cpp and
src/txmempool.cpp might be moved there.