Commit Graph

1019 Commits

Author SHA1 Message Date
MarcoFalke
face8123fd log: [refactor] Use info level for init logs
This refactor does not change behavior.
2025-07-25 09:50:50 +02:00
MarcoFalke
fa183761cb log: Remove function name from init logs
It is redundant with -logsourcelocations and the log messages are
clearer without it.

Also, remove a double-space.

Also, add braces around `if` touched in the next commit.

This tiny behavior change requires a test fixup.
2025-07-25 09:50:24 +02:00
merge-script
7129c9ea8e Merge bitcoin/bitcoin#32827: mempool: Avoid needless vtx iteration during IBD
249889bee6 orphanage: avoid vtx iteration when no orphans (furszy)
41ad2be434 mempool: Avoid expensive loop in `removeForBlock` during IBD (Lőrinc)

Pull request description:

  During Initial Block Download, the mempool is usually empty, but `CTxMemPool::removeForBlock` is still called for every connected block where we:
  * iterate over every transaction in the block even though none will be found in the empty `mapTx`, always leaving `txs_removed_for_block` empty...
  * which is pre-allocated regardless with `40 bytes * vtx.size()`, even though it will always remain empty.

  Similarly to https://github.com/bitcoin/bitcoin/pull/32730#discussion_r2140691354, this change introduces a minor performance & memory optimization by only executing the loop if any of the affected mempool maps have any contents. The second commit is cherry-picked from there since it's related to this change as well.

ACKs for top commit:
  optout21:
    ACK 249889bee6
  glozow:
    ACK 249889bee6
  ismaelsadeeq:
    reACK 249889bee6

Tree-SHA512: 80d06ff1515164529cdc3ad21db3041bb5b2a1d4b72ba9e6884cdf40c5f1477fee7479944b8bca32a6f0bf27c4e5501fccd085f6041a2dbb101438629cfb9e4b
2025-07-21 11:01:12 -04:00
Ava Chow
5878f35446 Merge bitcoin/bitcoin#31144: [IBD] multi-byte block obfuscation
248b6a27c3 optimization: peel align-head and unroll body to 64 bytes (Lőrinc)
e7114fc6dc optimization: migrate fixed-size obfuscation from `std::vector<std::byte>` to `uint64_t` (Lőrinc)
478d40afc6 refactor: encapsulate `vector`/`array` keys into `Obfuscation` (Lőrinc)
377aab8e5a refactor: move `util::Xor` to `Obfuscation().Xor` (Lőrinc)
fa5d296e3b refactor: prepare mempool_persist for obfuscation key change (Lőrinc)
6bbf2d9311 refactor: prepare `DBWrapper` for obfuscation key change (Lőrinc)
0b8bec8aa6 scripted-diff: unify xor-vs-obfuscation nomenclature (Lőrinc)
972697976c bench: make ObfuscationBench more representative (Lőrinc)
618a30e326 test: compare util::Xor with randomized inputs against simple impl (Lőrinc)
a5141cd39e test: make sure dbwrapper obfuscation key is never obfuscated (Lőrinc)
54ab0bd64c refactor: commit to 8 byte obfuscation keys (Lőrinc)
7aa557a37b random: add fixed-size `std::array` generation (Lőrinc)

Pull request description:

  This change is part of [[IBD] - Tracking PR for speeding up Initial Block Download](https://github.com/bitcoin/bitcoin/pull/32043)

  ### Summary

  Current block obfuscations are done byte-by-byte, this PR batches them to 64 bit primitives to speed up obfuscating bigger memory batches.
  This is especially relevant now that https://github.com/bitcoin/bitcoin/pull/31551 was merged, having bigger obfuscatable chunks.

  Since this obfuscation is optional, the speedup measured here depends on whether it's a [random value](https://github.com/bitcoin/bitcoin/pull/31144#issuecomment-2523295114) or [completely turned off](https://github.com/bitcoin/bitcoin/pull/31144#issuecomment-2519764142) (i.e. XOR-ing with 0).

  ### Changes in testing, benchmarking and implementation

  * Added new tests comparing randomized inputs against a trivial implementation and performing roundtrip checks with random chunks.
  * Migrated `std::vector<std::byte>(8)` keys to plain `uint64_t`;
  * Process unaligned bytes separately and unroll body to 64 bytes.

  ### Assembly

  Memory alignment is enforced by a small peel-loop (`std::memcpy` is optimized out on tested platform), with an `std::assume_aligned<8>` check, see the Godbolt listing at https://godbolt.org/z/59EMv7h6Y for details

  <details>
  <summary>Details</summary>

  Target & Compiler | Stride (per hot-loop iter) | Main operation(s) in loop | Effective XORs / iter
  -- | -- | -- | --
  Clang x86-64 (trunk) | 64 bytes | 4 × movdqu → pxor → store | 8 × 64-bit
  GCC x86-64 (trunk) | 64 bytes | 4 × movdqu/pxor sequence, enabled by 8-way unroll | 8 × 64-bit
  GCC RV32 (trunk) | 8 bytes | copy 8 B to temp → 2 × 32-bit XOR → copy back | 1 × 64-bit (as 2 × 32-bit)
  GCC s390x (big-endian 14.2) | 64 bytes | 8 × XC (mem-mem 8-B XOR) with key cached on stack | 8 × 64-bit

  </details>

  ### Endianness

  The only endianness issue was with bit rotation, intended to realign the key if obfuscation halted before full key consumption.
  Elsewhere, memory is read, processed, and written back in the same endianness, preserving byte order.
  Since CI lacks a big-endian machine, testing was done locally via Docker.
  <details>
  <summary>Details</summary>

  ```bash
  brew install podman pigz
  softwareupdate --install-rosetta
  podman machine init
  podman machine start
  docker run --platform linux/s390x -it ubuntu:latest /bin/bash
    apt update && apt install -y git build-essential cmake ccache pkg-config libevent-dev libboost-dev libssl-dev libsqlite3-dev python3 && \
    cd /mnt && git clone --depth=1 https://github.com/bitcoin/bitcoin.git && cd bitcoin && git remote add l0rinc https://github.com/l0rinc/bitcoin.git && git fetch --all && git checkout l0rinc/optimize-xor && \
    cmake -B build && cmake --build build --target test_bitcoin -j$(nproc) && \
    ./build/bin/test_bitcoin --run_test=streams_tests
  ```

  </details>

  ### Measurements (micro benchmarks and full IBDs)

  > cmake -B build -DBUILD_BENCH=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc/clang -DCMAKE_CXX_COMPILER=g++/clang++ && \
    cmake --build build -j$(nproc) && \
    build/bin/bench_bitcoin -filter='ObfuscationBench' -min-time=5000

  <details>
  <summary>GNU 14.2.0</summary>

  > Before:

  |             ns/byte |              byte/s |    err% |        ins/byte |        cyc/byte |    IPC |       bra/byte |   miss% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
  |                0.84 |    1,184,138,235.64 |    0.0% |            9.01 |            3.03 |  2.971 |           1.00 |    0.1% |      5.50 | `ObfuscationBench`

  > After (first optimizing commit):

  |             ns/byte |              byte/s |    err% |        ins/byte |        cyc/byte |    IPC |       bra/byte |   miss% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
  |                0.04 |   28,365,698,819.44 |    0.0% |            0.34 |            0.13 |  2.714 |           0.07 |    0.0% |      5.33 | `ObfuscationBench`

  > and (second optimizing commit):

  |             ns/byte |              byte/s |    err% |        ins/byte |        cyc/byte |    IPC |       bra/byte |   miss% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
  |                0.03 |   32,464,658,919.11 |    0.0% |            0.50 |            0.11 |  4.474 |           0.08 |    0.0% |      5.29 | `ObfuscationBench`

  </details>

  <details>
  <summary>Clang 20.1.7</summary>

  > Before:

  |             ns/byte |              byte/s |    err% |        ins/byte |        cyc/byte |    IPC |       bra/byte |   miss% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
  |                0.89 |    1,124,087,330.23 |    0.1% |            6.52 |            3.20 |  2.041 |           0.50 |    0.2% |      5.50 | `ObfuscationBench`

  > After (first optimizing commit):

  |             ns/byte |              byte/s |    err% |        ins/byte |        cyc/byte |    IPC |       bra/byte |   miss% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
  |                0.08 |   13,012,464,203.00 |    0.0% |            0.65 |            0.28 |  2.338 |           0.13 |    0.8% |      5.50 | `ObfuscationBench`

  > and (second optimizing commit):

  |             ns/byte |              byte/s |    err% |        ins/byte |        cyc/byte |    IPC |       bra/byte |   miss% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
  |                0.02 |   41,231,547,045.17 |    0.0% |            0.30 |            0.09 |  3.463 |           0.02 |    0.0% |      5.47 | `ObfuscationBench`

  </details>

  i.e. 27.4x faster obfuscation with GCC, 36.7x faster with Clang

  For other benchmark speedups see  https://corecheck.dev/bitcoin/bitcoin/pulls/31144

  ------

  Running an IBD until 888888 blocks reveals a 4% speedup.

  <details>
  <summary>Details</summary>

  SSD:

  ```bash
  COMMITS="8324a00bd4a6a5291c841f2d01162d8a014ddb02 5ddfd31b4158a89b0007cfb2be970c03d9278525"; \
  STOP_HEIGHT=888888; DBCACHE=1000; \
  CC=gcc; CXX=g++; \
  BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
  (for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \
  hyperfine \
    --sort 'command' \
    --runs 1 \
    --export-json "$BASE_DIR/ibd-${COMMITS// /-}-$STOP_HEIGHT-$DBCACHE-$CC.json" \
    --parameter-list COMMIT ${COMMITS// /,} \
    --prepare "killall bitcoind; rm -rf $DATA_DIR/*; git checkout {COMMIT}; git clean -fxd; git reset --hard; \
      cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && \
      cmake --build build -j$(nproc) --target bitcoind && \
      ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; sleep 100" \
    --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
    "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=$DBCACHE -blocksonly -printtoconsole=0"
  ```

  > 8324a00bd4 test: Compare util::Xor with randomized inputs against simple impl
  > 5ddfd31b41 optimization: Xor 64 bits together instead of byte-by-byte

  ```python
  Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=1000 -blocksonly -printtoconsole=0 (COMMIT = 8324a00bd4a6a5291c841f2d01162d8a014ddb02)
    Time (abs ≡):        25033.413 s               [User: 33953.984 s, System: 2613.604 s]

  Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=1000 -blocksonly -printtoconsole=0 (COMMIT = 5ddfd31b4158a89b0007cfb2be970c03d9278525)
    Time (abs ≡):        24110.710 s               [User: 33389.536 s, System: 2660.292 s]

  Relative speed comparison
          1.04          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=1000 -blocksonly -printtoconsole=0 (COMMIT = 8324a00bd4a6a5291c841f2d01162d8a014ddb02)
          1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=1000 -blocksonly -printtoconsole=0 (COMMIT = 5ddfd31b4158a89b0007cfb2be970c03d9278525)
  ```

  > HDD:

  ```bash
  COMMITS="71eb6eaa740ad0b28737e90e59b89a8e951d90d9 46854038e7984b599d25640de26d4680e62caba7"; \
  STOP_HEIGHT=888888; DBCACHE=4500; \
  CC=gcc; CXX=g++; \
  BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \
  (for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \
  hyperfine \
    --sort 'command' \
    --runs 2 \
    --export-json "$BASE_DIR/ibd-${COMMITS// /-}-$STOP_HEIGHT-$DBCACHE-$CC.json" \
    --parameter-list COMMIT ${COMMITS// /,} \
    --prepare "killall bitcoind; rm -rf $DATA_DIR/*; git checkout {COMMIT}; git clean -fxd; git reset --hard; \
      cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && cmake --build build -j$(nproc) --target bitcoind && \
      ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; sleep 100" \
    --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \
    "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=$DBCACHE -blocksonly -printtoconsole=0"
  ```

  > 71eb6eaa74 test: compare util::Xor with randomized inputs against simple impl
  > 46854038e7 optimization: migrate fixed-size obfuscation from `std::vector<std::byte>` to `uint64_t`

  ```python
  Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=4500 -blocksonly -printtoconsole=0 (COMMIT = 71eb6eaa740ad0b28737e90e59b89a8e951d90d9)
    Time (mean ± σ):     37676.293 s ± 83.100 s    [User: 36900.535 s, System: 2220.382 s]
    Range (min … max):   37617.533 s … 37735.053 s    2 runs

  Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=4500 -blocksonly -printtoconsole=0 (COMMIT = 46854038e7984b599d25640de26d4680e62caba7)
    Time (mean ± σ):     36181.287 s ± 195.248 s    [User: 34962.822 s, System: 1988.614 s]
    Range (min … max):   36043.226 s … 36319.349 s    2 runs

  Relative speed comparison
          1.04 ±  0.01  COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=4500 -blocksonly -printtoconsole=0 (COMMIT = 71eb6eaa740ad0b28737e90e59b89a8e951d90d9)
          1.00          COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=4500 -blocksonly -printtoconsole=0 (COMMIT = 46854038e7984b599d25640de26d4680e62caba7)
  ```

  </details>

ACKs for top commit:
  achow101:
    ACK 248b6a27c3
  maflcko:
    review ACK 248b6a27c3 🎻
  ryanofsky:
    Code review ACK 248b6a27c3. Looks good! Thanks for adapting this and considering all the suggestions. I did leave more comments below but non are important and this looks good as-is

Tree-SHA512: ef541cd8a1f1dc504613c4eaa708202e32ae5ac86f9c875e03bcdd6357121f6af0860ef83d513c473efa5445b701e59439d416effae1085a559716b0fd45ecd6
2025-07-18 22:17:11 -07:00
furszy
249889bee6 orphanage: avoid vtx iteration when no orphans 2025-07-18 16:54:16 -07:00
Lőrinc
478d40afc6 refactor: encapsulate vector/array keys into Obfuscation 2025-07-16 14:33:07 -07:00
Lőrinc
fa5d296e3b refactor: prepare mempool_persist for obfuscation key change
These changes are meant to simplify the diffs for the riskier optimization commits later.
2025-07-16 14:33:07 -07:00
Lőrinc
0b8bec8aa6 scripted-diff: unify xor-vs-obfuscation nomenclature
Mechanical refactor of the low-level "xor" wording to signal the intent instead of the implementation used.
The renames are ordered by heaviest-hitting substitutions first, and were constructed such that after each replacement the code is still compilable.

-BEGIN VERIFY SCRIPT-
sed -i \
  -e 's/\bGetObfuscateKey\b/GetObfuscation/g' \
  -e 's/\bxor_key\b/obfuscation/g' \
  -e 's/\bxor_pat\b/obfuscation/g' \
  -e 's/\bm_xor_key\b/m_obfuscation/g' \
  -e 's/\bm_xor\b/m_obfuscation/g' \
  -e 's/\bobfuscate_key\b/m_obfuscation/g' \
  -e 's/\bOBFUSCATE_KEY_KEY\b/OBFUSCATION_KEY_KEY/g' \
  -e 's/\bSetXor(/SetObfuscation(/g' \
  -e 's/\bdata_xor\b/obfuscation/g' \
  -e 's/\bCreateObfuscateKey\b/CreateObfuscation/g' \
  -e 's/\bobfuscate key\b/obfuscation key/g' \
  $(git ls-files '*.cpp' '*.h')
-END VERIFY SCRIPT-
2025-07-16 14:32:01 -07:00
Lőrinc
54ab0bd64c refactor: commit to 8 byte obfuscation keys
Since 31 byte xor-keys are not used in the codebase, using the common size (8 bytes) makes the benchmarks more realistic.

Co-authored-by: maflcko <6399679+maflcko@users.noreply.github.com>
2025-07-16 13:19:18 -07:00
Pieter Wuille
03aaaedc6d [prep] Return the made-reconsiderable announcements in AddChildrenToWorkSet
This is preparation for the simulation fuzz test added in a later commit. Since
AddChildrenToWorkSet consumes randomness, there is no way for the simulator to
exactly predict its behavior. By returning the set of made-reconsiderable announcements
instead, the simulator can instead test that it is *a* valid choice, and then
apply it to its own data structures.
2025-07-14 16:13:47 -04:00
glozow
ea29c4371e [p2p] bump DEFAULT_MAX_ORPHANAGE_LATENCY_SCORE to 3,000
For the default number of peers (125), allows each to relay a default
descendant package (up to 25-1=24 can be missing inputs) of small (9
inputs or fewer) transactions out of order.

This limit also gives acceptable bounds for worst case LimitOrphans iterations.

Functional tests aren't changed to check for larger cap because it would
make the runtime too long.

Also deletes the now-unused DEFAULT_MAX_ORPHAN_TRANSACTIONS.
2025-07-14 16:13:47 -04:00
glozow
4d23d1d7e7 [cleanup] remove unused rng param from LimitOrphans 2025-07-14 16:13:47 -04:00
glozow
067365d2a8 [p2p] overhaul TxOrphanage with smarter limits
This is largely a reimplementation using boost::multi_index_container.
All the same public methods are available. It has an index by outpoint,
per-peer tracking, peer worksets, etc.

A few differences:
- Limits have changed: instead of a global limit of 100 unique orphans,
  we have a maximum number of announcements (which can include duplicate
orphans) and a global memory limit which scales with the number of
peers.
- The maximum announcements limit is 100 to match the original limit,
  but this is actually a stricter limit because the announcement count
is not de-duplicated.
- Eviction strategy: when global limits are reached, a per-peer limit
  comes into play. While limits are exceeded, we choose the peer whose
“DoS score” (max usage / limit ratio for announcements and memory
limits) is highest and evict announcements by entry time, sorting
non-reconsiderable ones before reconsiderable ones. Since announcements
are unique by (wtxid, peer), as long as 1 announcement remains for a
transaction, it remains in the orphanage.
- This eviction strategy means no peer can influence the eviction of
  another peer’s orphans.
- Also, since global limits are a multiple of per-peer limits, as long
  as a peer does not exceed its limits, its orphans are protected from
eviction.
- Orphans no longer expire, since older announcements are generally
  removed before newer ones.
- GetChildrenFromSamePeer returns the transactions from newest to
  oldest.

Co-authored-by: Pieter Wuille <pieter@wuille.net>
2025-07-14 16:13:47 -04:00
glozow
1a41e7962d [refactor] create aliases for TxOrphanage Count and Usage 2025-07-14 16:13:47 -04:00
glozow
b50bd72c42 [prep] change return type of EraseTx to bool
This function only ever returns 0 or 1 (number of unique orphans
erased).
2025-07-14 16:13:47 -04:00
glozow
3da6d7f8f6 [prep/refactor] make TxOrphanage a virtual class implemented by TxOrphanageImpl 2025-07-14 16:13:46 -04:00
glozow
77ebe8f280 [prep/test] have TxOrphanage remember its own limits in LimitOrphans
Move towards a model where TxOrphanage is initialized with limits that
it remembers throughout its lifetime.
Remove the param. Limiting by number of unique orphans will be removed
in a later commit.
Now that -maxorphantx is gone, this does not change the node behavior.
The parameter is only used in tests.
2025-07-14 16:13:10 -04:00
glozow
d0af4239b7 [prep/refactor] move DEFAULT_MAX_ORPHAN_TRANSACTIONS to txorphanage.h
This is move only.
2025-07-14 16:13:10 -04:00
glozow
51365225b8 [prep/config] remove -maxorphantx
The orphanage will no longer have a maximum number of unique orphans.
2025-07-14 16:13:10 -04:00
glozow
15a4ec9069 [prep/rpc] remove entry and expiry time from getorphantxs
Expiry is going away in a later commit.
This is only an RPC change. Behavior of the orphanage does not change.
Note that getorphantxs is marked experimental.
2025-07-11 13:52:50 -04:00
glozow
08e58fa911 [prep/refactor] move txorphanage to node namespace and directory
This is move-only.
2025-07-11 13:52:50 -04:00
merge-script
23e15d40b9 Merge bitcoin/bitcoin#32631: refactor: Convert GenTxid to std::variant
a60f863d3e scripted-diff: Replace GenTxidVariant with GenTxid (marcofleon)
c8ba199598 Remove old GenTxid class (marcofleon)
072a198ea4 Convert remaining instances of GenTxid to GenTxidVariant (marcofleon)
1b528391c7 Convert `txrequest` to GenTxidVariant (marcofleon)
bde4579b07 Convert `txdownloadman_impl` to GenTxidVariant (marcofleon)
c876a892ec Replace GenTxid with Txid/Wtxid overloads in `txmempool` (marcofleon)
de858ce2be move-only: make GetInfo a private CTxMemPool member (stickies-v)
eee473d9f3 Convert `CompareInvMempoolOrder` to GenTxidVariant (marcofleon)
243553d590 refactor: replace get_iter_from_wtxid with GetIter(const Wtxid&) (stickies-v)
fcf92fd640 refactor: make CTxMemPool::GetIter strongly typed (marcofleon)
11d28f21bb Implement GenTxid as a variant (marcofleon)

Pull request description:

  Part of the [type safety refactor](https://github.com/bitcoin/bitcoin/pull/32189).

  This PR changes the GenTxid class to a variant, which holds both Txids and Wtxids. This provides compile-time type safety and eliminates the manual type check (bool m_is_wtxid). Variables that can be either a Txid or a Wtxid are now using the new GenTxid variant, instead of uint256.

ACKs for top commit:
  w0xlt:
    ACK a60f863d3e
  dergoegge:
    Code review ACK a60f863d3e
  maflcko:
    review ACK a60f863d3e 🎽
  theStack:
    Code-review ACK a60f863d3e

Tree-SHA512: da9b73b7bdffee2eb9281a409205519ac330d3336094d17681896703fbca8099608782c9c85801e388e4d90af5af8abf1f34931f57bbbe6e9674d802d6066047
2025-07-11 13:47:19 -04:00
marcofleon
a60f863d3e scripted-diff: Replace GenTxidVariant with GenTxid
-BEGIN VERIFY SCRIPT-
sed -i 's/GenTxidVariant/GenTxid/g' $(git grep -l 'GenTxidVariant')
-END VERIFY SCRIPT-
2025-07-08 20:00:51 +01:00
marcofleon
1b528391c7 Convert txrequest to GenTxidVariant
Switch all instances of GenTxid to the new variant
in `txrequest` and complete `txdownloadman_impl` by
converting `GetRequestsToSend`.
2025-07-08 20:00:51 +01:00
marcofleon
bde4579b07 Convert txdownloadman_impl to GenTxidVariant
Convert all of `txdownloadman_impl` to the new variant except for
`GetRequestsToSend`, which will be easier to switch at the same
time as `txrequest`.
2025-07-08 20:00:43 +01:00
marcofleon
c876a892ec Replace GenTxid with Txid/Wtxid overloads in txmempool
Co-authored-by: stickies-v <stickies-v@protonmail.com>
2025-07-08 19:31:02 +01:00
marcofleon
eee473d9f3 Convert CompareInvMempoolOrder to GenTxidVariant
Now that we are storing `CTxMemPool::CompareDepthAndScore` parameters using
`std::variant` we have no portable zero-overhead way of accessing them,
so use `std::visit` and drop `bool wtxid` in-parameter.

Co-authored-by: stickies-v <stickies-v@protonmail.com>
2025-07-08 16:03:05 +01:00
Ava Chow
a8bff38236 Merge bitcoin/bitcoin#32862: rpc: use CScheduler for relocking wallet and remove RPCTimer
fcfd3db563 remove RPCTimerInterface and RPCRunLater (Matthew Zipkin)
8a1765795f use WalletContext scheduler for walletpassphrase callback (Matthew Zipkin)

Pull request description:

  This removes the dependency on libevent for events scheduled by RPC commands, like re-locking a wallet some time after decryption with walletpassphrase. Since walletpassphrase is currently the only RPC that does this, `RPCRunLater`, `RPCTimerInterface` and all related methods are left unused, and deleted in the second commit. Any future RPC that needs to execute a callback in the future can follow the pattern in this PR and just use a scheduler from node or wallet context.

  This is an alternative approach to #32796, described in https://github.com/bitcoin/bitcoin/pull/32796#issuecomment-3014309449

ACKs for top commit:
  fjahr:
    Code Review ACK fcfd3db563
  achow101:
    ACK fcfd3db563
  furszy:
    ACK fcfd3db563

Tree-SHA512: 04f5e9c3f73f598c3d41d6e35bb59c64c7b93b03ad9fce3c40901733147ce7764f41f475fef1527d44af18f722759996a31ca83b48cb52153795d5022fecfd14
2025-07-07 17:59:21 -07:00
Ava Chow
ea4285775e Merge bitcoin/bitcoin#29307: util: explicitly close all AutoFiles that have been written
c10e382d2a flatfile: check whether the file has been closed successfully (Vasil Dimov)
4bb5dd78ea util: check that a file has been closed before ~AutoFile() is called (Vasil Dimov)
8bb34f07df Explicitly close all AutoFiles that have been written (Vasil Dimov)
a69c4098b2 rpc: take ownership of the file by WriteUTXOSnapshot() (Hodlinator)

Pull request description:

  `fclose(3)` may fail to flush the previously written data to disk, thus a failing `fclose(3)` is as serious as a failing `fwrite(3)`.

  Previously the code ignored `fclose(3)` failures. This PR improves that by changing all users of `AutoFile` that use it to write data to explicitly close the file and handle a possible error.

  ---

  Other alternatives are:

  1. `fflush(3)` after each write to the file (and throw if it fails from the `AutoFile::write()` method) and hope that `fclose(3)` will then always succeed. Assert that it succeeds from the destructor 🙄. Will hurt performance.
  2. Throw nevertheless from the destructor. Exception within the exception in C++ I think results in terminating the program without a useful message.
  3. (this is implemented in the latest incarnation of this PR) Redesign `AutoFile` so that its destructor cannot fail. Adjust _all_ its users 😭. For example, if the file has been written to, then require the callers to explicitly call the `AutoFile::fclose()` method before the object goes out of scope. In the destructor, as a sanity check, assume/assert that this is indeed the case. Defeats the purpose of a RAII wrapper for `FILE*` which automatically closes the file when it goes out of scope and there are a lot of users of `AutoFile`.
  4. Pass a new callback function to the `AutoFile` constructor which will be called from the destructor to handle `fclose()` errors, as described in https://github.com/bitcoin/bitcoin/pull/29307#issuecomment-2243842400. My thinking is that if that callback is going to only log a message, then we can log the message directly from the destructor without needing a callback. If the callback is going to do more complicated error handling then it is easier to do that at the call site by directly calling `AutoFile::fclose()` instead of getting the `AutoFile` object out of scope (so that its destructor is called) and inspecting for side effects done by the callback (e.g. set a variable to indicate a failed `fclose()`).

ACKs for top commit:
  l0rinc:
    ACK c10e382d2a
  achow101:
    ACK c10e382d2a
  hodlinator:
    re-ACK c10e382d2a

Tree-SHA512: 3994ca57e5b2b649fc84f24dad144173b7500fc0e914e06291d5c32fbbf8d2b1f8eae0040abd7a5f16095ddf4e11fe1636c6092f49058cda34f3eb2ee536d7ba
2025-07-03 15:37:44 -07:00
Matthew Zipkin
fcfd3db563 remove RPCTimerInterface and RPCRunLater 2025-07-03 06:26:23 -04:00
Ava Chow
319ff58bbd Merge bitcoin/bitcoin#32638: blocks: force hash validations on disk read
9341b5333a blockstorage: make block read hash checks explicit (Lőrinc)
2371b9f4ee test/bench: verify hash in `ComputeFilter` reads (Lőrinc)
5d235d50d6 net: assert block hash in `ProcessGetBlockData` and `ProcessMessage` (Lőrinc)

Pull request description:

  A follow-up to https://github.com/bitcoin/bitcoin/pull/32487#discussion_r2094072165, after which validating the hash of a read block from disk doesn't incur the cost of calculating its hash anymore.

  ### Summary

  This PR adds explicit checks that the read block header's hash matches the one we were expecting.

  ### Context

  After the previous PR, validating a block's hash during read operations became essentially free. This PR leverages that by requiring callers to provide a block's expected hash (or `std::nullopt`), preventing silent failures caused by corrupted or mismatched data. Most `ReadBlock` usages were updated with expected hashes and now fail on mismatch.

  ### Changes

  * added hash assertions in `ProcessGetBlockData` and `ProcessMessage` to validate that the block read from disk matches the expected hash;
  * updated tests and benchmark to pass the correct block hash to `ReadBlock()`, ensuring the hash validation is tested - or none if we already expect PoW failure;
  * removed the default value for `expected_hash`, requiring an explicit hash for all block reads.

  ### Why is the hash still optional (but no longer has a default value)

  * for header-error tests, where the goal is to trigger failures early in the parsing process;
  * for out-of-order orphan blocks, where the child hash isn't available before the initial disk read.

ACKs for top commit:
  maflcko:
    review ACK 9341b5333a 🕙
  achow101:
    ACK 9341b5333a
  hodlinator:
    ACK 9341b5333a
  janb84:
    re ACK 9341b5333a

Tree-SHA512: cf1d4fff4c15e3f8898ec284929cb83d7e747125d4ee759e77d369f1716728e843ef98030be32c8d608956a96ae2fbefa0e801200c333b9eefd6c086ec032e1f
2025-06-27 13:28:26 -07:00
merge-script
c43cc48aaa Merge bitcoin/bitcoin#32530: node: cap -maxmempool and -dbcache values for 32-bit
9f8e7b0b3b node: cap -dbcache to 1GiB on 32-bit architectures (Antoine Poinsot)
2c43b6adeb init: cap -maxmempool to 500 MB on 32-bit systems (Antoine Poinsot)

Pull request description:

  32-bit architecture is limited to 4GiB of RAM, so it doesn't make sense to set a too high value. A too high value could cause an OOM unbeknownst to the user a while after startup as mempool / dbcache fills.

ACKs for top commit:
  achow101:
    ACK 9f8e7b0b3b
  instagibbs:
    utACK 9f8e7b0b3b
  dergoegge:
    Code review ACK 9f8e7b0b3b
  glozow:
    utACK 9f8e7b0b3b

Tree-SHA512: cc7541b2c0040fc21a43916caec464dfb443af808f4e85deffa1187448ffff6edb0d69f9ebdb43915d145b8b4694d8465afe548f88da53ccebc9ce4b7c34b735
2025-06-26 17:33:23 +01:00
Ava Chow
9a7eece5a4 Merge bitcoin/bitcoin#31981: Add checkBlock() to Mining interface
a18e572328 test: more template verification tests (Sjors Provoost)
10c908808f test: move gbt proposal mode tests to new file (Sjors Provoost)
94959b8dee Add checkBlock to Mining interface (Sjors Provoost)
6077157531 ipc: drop BlockValidationState special handling (Sjors Provoost)
74690f4ed8 validation: refactor TestBlockValidity (Sjors Provoost)

Pull request description:

  This PR adds the IPC equivalent of the `getblocktemplate` RPC in `proposal` mode.

  In order to do so it has `TestBlockValidity` return error reasons as a string instead of `BlockValidationState`. This avoids complexity in IPC code for handling the latter struct.

  The new Mining interface method is used in `miner_tests`.

  It's not used by the `getblocktemplate` and `generateblock` RPC calls, see https://github.com/bitcoin/bitcoin/pull/31981#discussion_r2096473337

  The `inconclusive-not-best-prevblk` check is moved from RPC
  code to `TestBlockValidity`.

  Test coverage is increased by `mining_template_verification.py`.

  Superseedes #31564

  ## Background

  ### Verifying block templates (no PoW)

  Stratum v2 allows miners to generate their own block template. Pools may wish (or need) to verify these templates. This typically involves comparing mempools, asking miners to providing missing transactions and then reconstructing the proposed block.[^0] This is not sufficient to ensure a proposed block is actually valid. In some schemes miners could take advantage of incomplete validation[^1].

  The Stratum Reference Implementation (SRI), currently the only Stratum v2 implementation, collects all missing mempool transactions, but does not yet fully verify the block.[^2]. It could use the `getblocktemplate` RPC in `proposal` mode, but using IPC is more performant, as it avoids serialising up to 4 MB of transaction data as JSON.

  (although SRI could use this PR, the Template Provider role doesn't need it, so this is _not_ part of #31098)

  [^0]: https://github.com/stratum-mining/sv2-spec/blob/main/06-Job-Declaration-Protocol.md
  [^1]: https://delvingbitcoin.org/t/pplns-with-job-declaration/1099/45?u=sjors
  [^2]: https://github.com/stratum-mining/stratum/blob/v1.1.0/roles/jd-server/src/lib/job_declarator/message_handler.rs#L196

ACKs for top commit:
  davidgumberg:
    reACK a18e572328
  achow101:
    ACK a18e572328
  TheCharlatan:
    ACK a18e572328
  ryanofsky:
    Code review ACK a18e572328 just adding another NONFATAL_UNREACHABLE since last review

Tree-SHA512: 1a6c29f45a1666114f10f55aed155980b90104db27761c78aada4727ce3129e6ae7a522d90a56314bd767bd7944dfa46e85fb9f714370fc83e6a585be7b044f1
2025-06-18 17:07:21 -07:00
Ryan Ofsky
5e6dbfd14e Merge bitcoin/bitcoin#32465: thread-safety: fix annotations with REVERSE_LOCK
a201a99f8c thread-safety: fix annotations with REVERSE_LOCK (Cory Fields)
aeea5f0ec1 thread-safety: add missing lock annotation (Cory Fields)
832c57a534 thread-safety: modernize thread safety macros (Cory Fields)

Pull request description:

  This is one of several PRs to cleanup/modernize our threading primitives.

  While replacing the old critical section locks in the mining code with a `REVERSE_LOCK`, I noticed that our thread-safety annotations weren't hooked up to it. This PR gets `REVERSE_LOCK` working properly.

  Firstly it modernizes the attributes as-recommended by the [clang docs](https://clang.llvm.org/docs/ThreadSafetyAnalysis.html) (ctrl+f for `USE_LOCK_STYLE_THREAD_SAFETY_ATTRIBUTES`). There's a subtle difference between the old `unlock_function` and new `release_capability`, where our `reverse_lock` only works with the latter. I believe this is an upstream bug. I've [reported and attempted a fix here](https://github.com/llvm/llvm-project/pull/139343), but either way it makes sense to me to modernize.

  The second adds a missing annotation pointed out by a fixed `REVERSE_LOCK`. Because clang's thread-safety annotations aren't passed through a reference to `UniqueLock` as one may assume (see [here](https://clang.llvm.org/docs/ThreadSafetyAnalysis.html#no-alias-analysis) for more details), `cs_main` has to be listed explicitly as a requirement.

  The last commit actually fixes the `reverse_lock` by making it a `SCOPED_LOCK` and using the pattern [found in a clang test](https://github.com/llvm/llvm-project/blob/main/clang/test/SemaCXX/warn-thread-safety-analysis.cpp#L3126). Though the docs don't describe how to accomplish it, the functionality was added [in this commit](6a68efc959). Due to aliasing issues (see link above), in order to work correctly, the original mutex has to be passed along with the lock, so all existing `REVERSE_LOCK`s have been updated. To ensure that the mutexes actually match, a runtime assertion is added.

ACKs for top commit:
  fjahr:
    re-ACK a201a99f8c
  davidgumberg:
    reACK a201a99f8c
  theuni:
    Ok, done. Those last pushes can be ignored. ACKs on a201a99 are still fresh.
  ryanofsky:
    Code review ACK a201a99f8c. Just dropping 0065b9673db5da2994b0b07c1d50ebfb19af39d0 and fixing incorrect `reverse_lock::lockname` initialization since last review.
  TheCharlatan:
    Re-ACK a201a99f8c

Tree-SHA512: 2755fae0c41021976a1a633014a86d927f104ccbc8014c01c06dae89af363f92e5bc5d4276ad6d759302ac4679fe02a543758124d48318074db1c370989af7a7
2025-06-17 14:12:43 -04:00
Cory Fields
a201a99f8c thread-safety: fix annotations with REVERSE_LOCK
Without proper annotations, clang thinks that mutexes are still held for the
duration of a reverse_lock. This could lead to subtle bugs as
EXCLUSIVE_LOCKS_REQUIRED(foo) passes when it shouldn't.

As mentioned in the docs [0], clang's thread-safety analyzer is unable to deal
with aliases of mutexes, so it is not possible to use the lock's copy of the
mutex for that purpose. Instead, the original mutex needs to be passed back to
the reverse_lock for the sake of thread-safety analysis, but it is not actually
used otherwise.

[0]: https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
2025-06-16 18:09:14 +00:00
Vasil Dimov
8bb34f07df Explicitly close all AutoFiles that have been written
There is no way to report a close error from `AutoFile` destructor.
Such an error could be serious if the file has been written to because
it may mean the file is now corrupted (same as if write fails).

So, change all users of `AutoFile` that use it to write data to
explicitly close the file and handle a possible error.
2025-06-16 15:33:15 +02:00
Sjors Provoost
94959b8dee Add checkBlock to Mining interface
Use it in miner_tests.

The getblocktemplate and generateblock RPC calls don't use this,
because it would make the code more verbose.
2025-06-14 14:32:45 +02:00
Sjors Provoost
74690f4ed8 validation: refactor TestBlockValidity
Comments are expanded.

Return BlockValidationState instead of passing a reference.
Lock Chainman mutex instead of cs_main.
Remove redundant chainparams and pindexPrev arguments.
Drop defaults for checking proof-of-work and merkle root.

The ContextualCheckBlockHeader check is moved to after CheckBlock,
which is more similar to normal validation where context-free checks
are done first.

Validation failure reasons are no longer printed through LogError(),
since it depends on the caller whether this implies an actual bug
in the node, or an externally sourced block that happens to be invalid.
When called from getblocktemplate, via BlockAssembler::CreateNewBlock(),
this method already throws an std::runtime_error if validation fails.

Additionally it moves the inconclusive-not-best-prevblk check from RPC
code to TestBlockValidity.

There is no behavior change when callling getblocktemplate with proposal.
Previously this would return a BIP22ValidationResult which can throw for
state.IsError(). But CheckBlock() and the functions it calls only use
state.IsValid().

The final assert is changed into Assume, with a LogError.

Co-authored-by: <Ryan Ofsky <ryan@ofsky.org>
2025-06-14 14:32:45 +02:00
Roman Zeyde
6ecb9fc65f chore: use std::vector<std::byte> for BlockManager::ReadRawBlock() 2025-06-13 19:19:44 +03:00
Lőrinc
9341b5333a blockstorage: make block read hash checks explicit
Dropped the default expected_hash parameter from `ReadBlock()`.

In `blockmanager_flush_block_file` tests, we pass {} since the tests would already fail at PoW validation for corrupted blocks.

In `ChainstateManager::LoadExternalBlockFile`, we pass {} when processing child blocks because their hashes aren't known beforehand.
2025-06-13 12:32:56 +02:00
MarcoFalke
fa9ca13f35 refactor: Sort includes of touched source files 2025-06-03 19:56:55 +02:00
MarcoFalke
facb152697 scripted-diff: Bump copyright headers after include changes
Historically, the headers have been bumped some time after a file has
been touched. Do it now to avoid having to touch them again in the
future for that reason.

-BEGIN VERIFY SCRIPT-
 sed -i --regexp-extended 's;( 20[0-2][0-9])(-20[0-2][0-9])? The Bitcoin Core developers;\1-present The Bitcoin Core developers;g' $( git show --pretty="" --name-only HEAD~0 )
-END VERIFY SCRIPT-
2025-06-03 15:13:57 +02:00
MarcoFalke
fae71d30f7 clang-tidy: Apply modernize-deprecated-headers
This can be reproduced according to the developer notes with something
like

( cd ./src/ && ../contrib/devtools/run-clang-tidy.py -p ../bld-cmake -fix -j $(nproc) )

Also, the header related changes were done manually.
2025-06-03 15:13:54 +02:00
Ava Chow
88b22acc3d Merge bitcoin/bitcoin#32528: rpc: Round verificationprogress to 1 for a recent tip
fab1e02086 refactor: Pass verification_progress into block tip notifications (MarcoFalke)
fa76b378e4 rpc: Round verificationprogress to exactly 1 for a recent tip (MarcoFalke)
faf6304bdf test: Use mockable time in GuessVerificationProgress (MarcoFalke)

Pull request description:

  Some users really seem to care about this. While it shouldn't matter much, the diff is so trivial that it is probably worth doing.

  Fixes #31127

  One could also consider to split the field into two dedicated ones (https://github.com/bitcoin/bitcoin/issues/28847#issuecomment-1807115357), but this is left for a more involved follow-up and may also be controversial.

ACKs for top commit:
  achow101:
    ACK fab1e02086
  pinheadmz:
    ACK fab1e02086
  sipa:
    utACK fab1e02086

Tree-SHA512: a3c24e3c446d38fbad9399c1e7f1ffa7904490a3a7d12623b44e583b435cc8b5f1ba83b84d29c7ffaf22028bc909c7cec07202b825480449c6419d2a190938f5
2025-05-27 16:45:23 -07:00
Ava Chow
9bd9aee5a6 Merge bitcoin/bitcoin#32487: blocks: avoid recomputing block header hash in ReadBlock
09ee8b7f27 node: avoid recomputing block hash in `ReadBlock` (Lőrinc)
2bf173210f test: exercise `ReadBlock` hash‑mismatch path (Lőrinc)

Pull request description:

  Eliminate one block header hash calculation per block-read by reusing the hash for:
  * proof‑of‑work verification;
  * (optional) integrity check against the supplied hash.

  This part of the code wasn't covered by tests either, so the first commit exercises this part first, before pushing the validation to the delegate method.

ACKs for top commit:
  maflcko:
    lgtm ACK 09ee8b7f27
  achow101:
    ACK 09ee8b7f27
  jonatack:
    ACK 09ee8b7f27
  pinheadmz:
    ACK 09ee8b7f27

Tree-SHA512: 43fe51b478ea574b6d4c952684b13ca54fb8cbd67c3b6c136f460122d9ee953cc70b88778537117eecea71ccb8d88311faeac21b866e11d117f1145973204ed4
2025-05-27 13:02:27 -07:00
Lőrinc
09ee8b7f27 node: avoid recomputing block hash in ReadBlock
Eliminate one SHA‑256 double‑hash computation of the header per block read by reusing the hash for:
* proof‑of‑work verification;
* (optional) integrity check against the supplied hash.
2025-05-26 23:23:44 +02:00
MarcoFalke
fab1e02086 refactor: Pass verification_progress into block tip notifications
It is cheap to calculate and the caller does not have to take a lock to
calculate it.

Also turn pointers that can never be null into references.
2025-05-24 13:49:32 +02:00
merge-script
ec81204694 Merge bitcoin/bitcoin#31622: psbt: add non-default sighash types to PSBTs and unify sighash type match checking
ee045b61ef rpc, psbt: Require sighashes match for descriptorprocesspsbt (Ava Chow)
2b7682c372 psbt: use sighash type field to determine whether to remove non-witness utxos (Ava Chow)
28781b5f06 psbt: Add sighash types to PSBT when not DEFAULT or ALL (Ava Chow)
15ce1bd73f psbt: Enforce sighash type of signatures matches psbt (Ava Chow)
1f71cd337a wallet: Remove sighash type enforcement from FillPSBT (Ava Chow)
4c7d767e49 psbt: Check sighash types in SignPSBTInput and take sighash as optional (Ava Chow)
a118256948 script: Add IsPayToTaproot() (Ava Chow)
d6001dcd4a wallet: change FillPSBT to take sighash as optional (Ava Chow)
e58b680923 psbt: Return PSBTError from SignPSBTInput (Ava Chow)
2adfd81532 tests: Test PSBT sighash type mismatch (Ava Chow)
5a5d26d612 psbt: Require ECDSA signatures to be validly encoded (Ava Chow)

Pull request description:

  Currently, we do not add the sighash field to PSBTs at all, even when we have signed with a non-default sighash. This PR changes the behavior such that when we (attempt to) sign with a sighash other than DEFAULT or ALL, the sighash type field will be added to the PSBT to inform the later signers that a different sighash type was used by a signer. Notably, this is necessary for MuSig2 support as all signers must sign using the same sighash type, but the sighash is not provided in partial signatures.

  Furthermore, because the sighash type can also be provided on the command line, we require that if both a command line sighash type and the sighash field is present, they must specify the same sighash type. However, this was being checked by the wallet, rather than the signing code, so the `descriptorprocesspsbt` RPC was not enforcing this restriction at all, and in fact ignored the sighash field entirely. This PR refactors the checking code so that the underlying PSBT signing function `SignPSBTInput` does the check.

ACKs for top commit:
  theStack:
    re-ACK ee045b61ef
  rkrux:
    re-ACK ee045b61ef
  fjahr:
    Code review ACK ee045b61ef

Tree-SHA512: 4ead5be1ef6756251b827f594beba868a145d75bf7f4ef6f15ad21f0ae4b8d71b38c83494e5a6b75f37fadd097178cddd93d614b962a2c72fc134f00ba2f74ae
2025-05-21 10:02:49 +01:00
MarcoFalke
fa76b378e4 rpc: Round verificationprogress to exactly 1 for a recent tip
This requires a new lock annotation, but all relevant callers already
held the lock.
2025-05-20 11:17:29 +02:00
fanquake
c7c3bfadfc doc: add & amend copyright headers 2025-05-20 09:43:21 +01:00