321984705d
d5104cfbaeprevector: store `P2WSH`/`P2TR`/`P2PK` scripts inline (Lőrinc)52121506b2test: assert `CScript` allocation characteristics (Lőrinc)65ac7f6d4drefactor: modernize `CScriptBase` definition (Lőrinc)756da2a994refactor: extract `STATIC_SIZE` constant to prevector (Lőrinc) Pull request description: This change is part of [[IBD] - Tracking PR for speeding up Initial Block Download](https://github.com/bitcoin/bitcoin/pull/32043) ### Summary The current `prevector` size of 28 bytes (chosen to fill the `sizeof(CScript)` aligned size) was introduced in 2015 (https://github.com/bitcoin/bitcoin/pull/6914) before `SegWit` and `TapRoot`. However, the increasingly common `P2WSH` and `P2TR` scripts are both 34 bytes, and are forced to use heap (re)allocation rather than efficient inline storage. The core trade-off of this change is to eliminate heap allocations for common 34-36 byte scripts at the cost of increasing the base memory footprint of all `CScript` objects by 8 bytes (while still respecting peak memory usage defined by `-dbcache`). ### Context Increasing the `prevector` size allows these scripts to be stored inline, avoiding heap allocations, reducing potential memory fragmentation, and improving performance during cache flushes. Massif analysis confirms a lower stable memory usage after flushing, suggesting the elimination of heap allocations outweighs the larger base size for common workloads. Due to memory alignment, increasing the prevector size to 36 bytes doesn't change the overall `sizeof(CScript)` compared to an increase to 34 bytes, allowing us to include `P2PK` scripts as well at no additional memory cost. <details> <summary>Massif measurements</summary> > dbcache=440 Massif before, with a heap threshold of `28`: ```bash MB 744.1^# |#: ::::::@: ::::::: :@:: @::::::::::::::@@ |#: ::::::@::::: ::: :@:::@:::::: :: ::::@ |#: ::::::@::::: ::: :@:::@:::::: :: ::::@ |#: ::::::@::::: ::: : :@:::@:::::: :: ::::@ |#: ::::::@::::: ::: : :@:::@:::::: :: ::::@ |#: ::::::@::::: ::: : :@:::@:::::: :: ::::@ |#::::::::@::::: ::: : :@:::@:::::: :: ::::@ |#::::::::@::::: ::: :::@:::@:::::: :: ::::@ |#::::::::@::::: ::: :::@:::@:::::: :: ::::@ |#::::::::@::::: ::: :::@:::@:::::: :: ::::@ |#::::::::@::::: ::: :::@:::@:::::: :: ::::@ |#::::::::@::::: :::::::@:::@:::::: :: ::::@ |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@:::: |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@:::: |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@:::: |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@:::: |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@:::: |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@:::: |#::::::::@::::: :::::::@:::@:::::: :: ::::@ :::::@:::::@:::::@:::::@:::: 0 +----------------------------------------------------------------------->h 0 1.805 ``` and after, with a heap threshold of `36`: ```bash MB 744.2^ : |# : ::::::::::: : : :: ::: @@:::::: :: : |# : :::: :::::: : : :: ::: @ :: :: : : |# : :::: ::::::: : :@:: ::: @ :: :: : ::: |# : :::: ::::::: : :@:: ::: @ :: :: : : : |# : :::: ::::::: : :@:: ::: @ :: :: : : : |# : :::: ::::::: : :@:: ::: @ :: :: : : : |# :: :::: ::::::: : :@:: ::: @ :: :: : : : |# :: :::: ::::::: : :@:: ::::@ :: :: : : : |#:::: :::: ::::::: :::@:: ::::@ :: :: : : : |#: ::::::: ::::::: :::@:: ::::@ :: :: @: : : |#: ::::::: ::::::: :::@:::::::@ :: :: @: : : |#: ::::::: ::::::::::::@:::::::@ :: :: @: : : |#: ::::::: :::::::: :::@:::::::@ :: :: @: : : |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@:: |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@:: |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@:: |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@:: |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@:: |#: ::::::: :::::::: :::@:::::::@ :: :: @: : :::@:::@::::@::::::@:::::@:: 0 +----------------------------------------------------------------------->h 0 1.618 ``` --- > for `dbcache=4500`: Massif before, with a heap threshold of `28`: ```bash GB 4.565^ :: | ##: @@::: :::: :@:::: :::: :::: | # : @ :: ::: :@: :: : :: ::: | # : @ :: ::::: :@: :: : :: ::: | # : @ :: : ::: :@: :: @: :: ::: | # : @ :: : ::: :@: :: @: :: ::: | # : @ :: : ::: :@: :: @: :: ::: | # : @ :: : ::: :@: :: @: :: ::: | # : ::@ :: : ::: :@: :: @: :: ::: | # : : @ :: : ::: :@: :: @: :: ::: | # : : @ :: : ::: :@: :: @: :::::: | # : : @ :: : ::: :@: :: @: :::::: | # : : @ :: : ::: :@: :: @: :::::: | # : : @ :: : ::: ::@: :: @: :::::: | # : : @ :: : ::: ::@: :: @: :::::: | # : : @ :: : ::: ::@: :: @: :::::: | # : : @ :: : ::: ::@: :: @: :::::: @:: | # : : @ :: : ::: ::@: :: @: :::::: @: | # : : @ :: : ::: ::@: :::@: :::::: @: | # : : @ :: : ::: ::@: :::@: :::::: @: :::::::::::::::::::::::::::::@::: 0 +----------------------------------------------------------------------->h 0 1.500 ``` and after, with a heap threshold of `36`: ``` GB 4.640^ : | ##:: ::::: :::: ::::::@ :::: | # :: : ::: :::: :: :::@ :::: | # :: :: ::: :::: :: :::@ :::: | # :: :: ::: ::::: :: :::@ :::: | # :: :: ::: ::::: :: :::@ :::: | # :: :: ::: ::::: :: :::@ :::: | # :: :: ::: ::::: :: :::@ :::: | # :: :: ::: ::::: :: :::@ :::: :@@ | # :: :: ::: ::::: ::: :::@ :::::::@ | # :: :: ::: ::::: ::: :::@ ::::: :@ | # :: :: ::: ::::: ::: :::@::::::: :@ | # ::::: ::: ::::: ::: :::@: ::::: :@ | # ::::: ::: ::::: ::: :::@: ::::: :@ | # ::::: :::::::::: ::: :::@: ::::: :@ | # ::::: :::: ::::: ::: :::@: ::::: :@ | # ::::: :::: ::::: ::: :::@: ::::: :@ | # ::::: :::: ::::::::: :::@: ::::: :@ | # ::::: :::: ::::::::: :::@: ::::: :@ | # ::::: :::: ::::::::: :::@: ::::: :@ ::::::@:::@:::@::::@:::::@::::@:: 0 +----------------------------------------------------------------------->h 0 1.360 ``` </details> ### Benchmarks and Memory Performance benchmarks for `AssumeUTXO` load and flush show: - Small dbcache (450MB): ~1-3% performance improvement (despite more frequent flushes) - Large dbcache (4500MB): ~6-8% performance improvement due to fewer heap allocations (and basically the number of flushes) - Very large dbcache (4500MB): ~5-6% performance improvement due to fewer heap allocations (and memory limit not being reached, so there's no memory penalty) Full IBD and `-reindex-chainstate` with also show an overall ~3-4% speedup (both for smaller and larger dbcache values). We haven't investigated using different `prevector` sizes based on script type, though this could be explored in the future if needed. ### Historical explanation for the speedup (by [Anthony Towns](https://github.com/bitcoin/bitcoin/pull/32279#issuecomment-3111757079)) > I think the tradeoff is something like: > > * spends of p2pk, p2sh, p2pkh coins -- these cost 8 more bytes > * spends of p2wpkh -- these cost 16 more bytes (sPK and scriptSig didn't need an allocation) > * spends of p2wsh and p2tr -- these cost ~48 fewer bytes (save 64 byte allocation on 64bit system, lose 8 bytes for both scriptSig and sPK) > * spends of nested p2wsh -- presumably save ~96 bytes, since the scriptSig would save an allocation, but I'm bundling it in the previous section > > Based on mainnet.observer stats for 2025-05-08, p2wpkh is about 55% of txs, p2tr is about 28%, p2pkh about 13%, p2wsh about 4% and the rest is noise, maybe? Those numbers net out to a saving of ~5.5 bytes per input. If p2wpkh rose from 55% to 80% and p2tr dropped to 20%, that would net to wasting ~3.2 bytes per input. ACKs for top commit: maflcko: review ACKd5104cfbae🐺 achow101: reACKd5104cfbaejonatack: Review ACKd5104cfbaeandrewtoth: ACKd5104cfbaeTree-SHA512: 7c5271ebaf4f6d91dc4b679ecbde4b7d0467579f072289f30da988a17c38a552d0b8cdf0e9c001739975dd019894c35e541908571527916cec56e04a8e214ae2
Bitcoin Core integration/staging tree
For an immediately usable, binary version of the Bitcoin Core software, see https://bitcoincore.org/en/download/.
What is Bitcoin Core?
Bitcoin Core connects to the Bitcoin peer-to-peer network to download and fully validate blocks and transactions. It also includes a wallet and graphical user interface, which can be optionally built.
Further information about Bitcoin Core is available in the doc folder.
License
Bitcoin Core is released under the terms of the MIT license. See COPYING for more information or see https://opensource.org/license/MIT.
Development Process
The master branch is regularly built (see doc/build-*.md for instructions) and tested, but it is not guaranteed to be
completely stable. Tags are created
regularly from release branches to indicate new official, stable release versions of Bitcoin Core.
The https://github.com/bitcoin-core/gui repository is used exclusively for the development of the GUI. Its master branch is identical in all monotree repositories. Release branches and tags do not exist, so please do not fork that repository unless it is for development reasons.
The contribution workflow is described in CONTRIBUTING.md and useful hints for developers can be found in doc/developer-notes.md.
Testing
Testing and code review is the bottleneck for development; we get more pull requests than we can review and test on short notice. Please be patient and help out by testing other people's pull requests, and remember this is a security-critical project where any mistake might cost people lots of money.
Automated Testing
Developers are strongly encouraged to write unit tests for new code, and to
submit new unit tests for old code. Unit tests can be compiled and run
(assuming they weren't disabled during the generation of the build system) with: ctest. Further details on running
and extending unit tests can be found in /src/test/README.md.
There are also regression and integration tests, written
in Python.
These tests can be run (if the test dependencies are installed) with: build/test/functional/test_runner.py
(assuming build is your build directory).
The CI (Continuous Integration) systems make sure that every pull request is tested on Windows, Linux, and macOS. The CI must pass on all commits before merge to avoid unrelated CI failures on new pull requests.
Manual Quality Assurance (QA) Testing
Changes should be tested by somebody other than the developer who wrote the code. This is especially important for large or high-risk changes. It is useful to add a test plan to the pull request description if testing the changes is not straightforward.
Translations
Changes to translations as well as new translations can be submitted to Bitcoin Core's Transifex page.
Translations are periodically pulled from Transifex and merged into the git repository. See the translation process for details on how this works.
Important: We do not accept translation changes as GitHub pull requests because the next pull from Transifex would automatically overwrite them again.