01dde6b205 fuzz: Fix assertion in txorphan (marcofleon)
Pull request description:
`EraseTx()` calls `LimitOrphans()`, which may evict announcements from a peer that didn't announce the erased transaction, causing that peer's usage to decrease. Relax the assertion in the `EraseTx()` branch that claimed usage of a non-announcer peer should be unchanged. Also, add assertions for the other cases.
ACKs for top commit:
dergoegge:
utACK 01dde6b205
instagibbs:
ACK 01dde6b205
Tree-SHA512: 2e597b85fd41058c2fa79fa55f0d37e12505065b5e27aba7b9680e0c249a5450e6fa97b45394d6ffe1318f42538134ffa9c423b126c455f6f8e6d8ca59eed4b6
9954aa7728 http: don't parse any new requests from a client if m_req_busy = true (Matthew Zipkin)
c7db3ae1f9 test: cover HTTPRequest state machine (Matthew Zipkin)
90676e24ad Add state to HTTPRequest to avoid duplicate work over I/O cycles (Matthew Zipkin)
507e528e84 http: reuse HTTPHeaders to parse chunked trailer (Matthew Zipkin)
902d8908c9 http: only read one HTTPRequest at a time per client (Matthew Zipkin)
Pull request description:
This PR reduces the memory consumption of the HTTP Server when reading data from connected clients, and improves performance especially when requests are large (i.e. requiring multiple TCP packets).
In https://github.com/bitcoin/bitcoin/pull/35182 the server copies as much data as it can from the socket into application memory, and then tries to parse as many complete HTTP requests as possible from that data. If a request is discovered to be incomplete, the in-progress request is abandoned. The server tries again on the next I/O cycle to read the same data from the buffer, duplicating work as many times as it takes before the client finishes sending the request (or times out).
This PR implements two improvements to this:
1. Only parse one request at a time from the receive buffer. The server processes requests from each client in series anyway.
2. Add state to `HTTPRequest` so it can be filled with data from the receive buffer over multiple I/O loop iterations without losing progress.
If a client sends large or multiple requests, that data will sit in the kernel's socket buffer instead of the application memory. Eventually the socket buffer will fill up and TCP backpressure will kick in, dropping the TCP window to 0 and blocking the client from sending any more.
A state machine for `HTTPRemoteClient` was [discussed previously](https://github.com/bitcoin/bitcoin/pull/35182#pullrequestreview-4322490068) to control resource consumption. Another nice benefit of this model (for a follow-up PR) will be to insert the RPC authentication check after reading 8kB-limited headers but before the 32MB-limited request body.
ACKs for top commit:
winterrdog:
re-ACK 9954aa7728
janb84:
re ACK 9954aa7728
frankomosh:
ACK 9954aa7728.
fjahr:
ACK 9954aa7728
Tree-SHA512: b7c913114283fbf1f360b40f6c65a01390a26731bf3b166f460ec260f9206f25d738b3a06887bfa839911c1c6aaf634448181da47a752a9a881aebd907e44868
fabe100c2b test: Use throwing config parser getters without fallback (MarcoFalke)
fa8acd57cd test: Write true/false values in config.ini (MarcoFalke)
Pull request description:
Currently, the called `getboolean` member function is *not* the throwing https://docs.python.org/3/library/configparser.html#configparser.ConfigParser.getboolean, but a non-throwing member function on a dict-like proxy object.
This is confusing and brittle, because tests shouldn't silently skip when a config key is missing. Instead, tests should loudly fail, e.g. when the config key is renamed in one place, but not the other.
ACKs for top commit:
jeanpablojp:
tACK fabe100c2b
willcl-ark:
ACK fabe100c2b
Tree-SHA512: a970d74ad285372b8adcce8e2a52b01f5a3b563899dfc5262e6ffbf3d8aba43e72f7b03111e8d5188924c7d3d789992407cddb5d182d9be5e42f07896d8ad4a3
b3d77ea027 test: Speedup fee estimation functional test with batching (sedited)
Pull request description:
The fee estimation functional test is currently the slowest one by a good margin. It is a bit annoying, because it also increases the total runtime of the functional tests.
It seems like most of the slowness comes from the transactions propagating between the nodes. This patch helps them do that by submitting them directly to all the nodes. Also take this opportunity to batch the transaction submissions.
On my machine this speeds up the fee estimation functional test from around 71 seconds to 25 seconds.
ACKs for top commit:
151henry151:
tACK b3d77ea027
maflcko:
review ACK b3d77ea027🐇
ismaelsadeeq:
ACK b3d77ea027
Tree-SHA512: f76415dca7997577ca39ac6b95dfdf32b930dd64b4311e3da34e34adb16107ff4ea2d9fa679f3ca50540e80c38af7f9390b44f21ad1b8107fbbc3586edb2ef19
25bed560be test: add forward-compat functional test for txindex (sedited)
703304ed8c doc: add release notes for txindex disk usage and downgrading (Andrew Toth)
8e5320a2d2 tests: cover txindex hash prefix collisions and legacy fallback (Andrew Toth)
b75efa19ba txindex: skip bloom filters and legacy lookups for new databases (Andrew Toth)
004d7c098c txindex: hash key prefixes and pack block positions (Andrew Toth)
5a255970fd refactor: move txindex db constants and legacy key to txindex_key.h (Andrew Toth)
327660134c txindex: pass the full block to DB::WriteTxs (Andrew Toth)
42771e7998 txindex: use a new block locator for downgrade safety (Andrew Toth)
4b08baed72 txindex: return optional tx and block hash from FindTx (Andrew Toth)
Pull request description:
The current txindex uses the full 32-byte txid as keys, which takes up about 66 GB of disk space today on mainnet. Using a 5-byte key prefix instead drops the disk usage to 26 GB - cutting the size to less than half.
Using the full 32-bytes is unnecessary since a 5-byte salted siphash will produce collisions in about 1 in 1.1 trillion. Some collisions will occur, but the penalty is just an extra disk read, deserialization and hash.
The tx position can be appended to the key instead of used as a value, and a LevelDB iterator can seek to the prefix and then scan for the correct tx. This is an almost identical approach to `txospenderindex`.
Also instead of storing the file position of the block, we can store only the sequence of the connected block and offset of the transaction in the block. This can be packed into a 6-byte key suffix using 3-byte representations of the sequence and offset in the block. The block file can be recovered by the CBlockIndex that is already in memory. The sequence is mapped to the block hash in the db, so we can lookup the block hash to find the CBlockIndex during reads.
If a tx is not found with this method, we fallback to looking up the legacy entry. With this method a user with an existing db can opt to erase the `indexes/txindex` folder and reindex, or keep the current index and new entries will be appended with the smaller footprint.
The time to index was faster on my machine with this method, 1h19m vs current 1h50m.
Lookups are roughly the same, around 0.2ms per lookup with `getrawtransaction`.
When testing on mainnet, I got 894,549 2-way collisions, 395 3-way collision, and 1 4-way collision that worst case could cause an extra 3 false positives when reading.
ACKs for top commit:
l0rinc:
diff reACK 25bed560be
sedited:
ACK 25bed560be
ajtowns:
ACK 25bed560be
Tree-SHA512: a25c79ca7e722e2f372b65f5fc11c8b194ad49f2240b4881c7e606306aabbd3604aede3f1c33606b467486affac3a3f503638f513c896935cebbc02709cb60d8
75a4e6c678 gui: fix allow restore wallets without .dat file extension (Pol Espinasa)
6ed7e05e20 gui: fix add .dat file extension automatically when exporting watchonly (Pol Espinasa)
Pull request description:
fixes https://github.com/bitcoin-core/gui/issues/956
Unlike `backup wallet`, `export watch-only wallet` was not automatically adding the file extension to the exported file, making restoring difficult if the user doesn't manually add the file extension after exporting.
Allows also to restore a wallet from a non specified `.dat` file extension. This is achieved by removing the filter in the select file screen, matching the RPC behavior.
ACKs for top commit:
hebasto:
ACK 75a4e6c678.
Tree-SHA512: 7c45d51205f9abf2b67233e8abd3297e49a4230eb32aa4118b37ab9da0a8d692aae4b67a8880881e5fab42256d1cf52ccf1b289b8f23f85930354130c192b33d
c3945bfd2b doc: use derivehdkey in multisig tutorial (Sjors Provoost)
3662e33669 test: use derivehdkey in M-of-N multisig demo (Sjors Provoost)
d9570f0838 rpc: add derivehdkey (Sjors Provoost)
62da9f9614 wallet: add GetExtKey helper (Sjors Provoost)
aaf1548475 wallet: generalize GetActiveHDPubKeys helper (Sjors Provoost)
3821452c4a refactor: add hardened derivation helper (Sjors Provoost)
0ab61caafd rpc: ParsePathBIP32 helper (Sjors Provoost)
e36c4b76e1 util: reject out-of-range BIP32 keypath indices (Sjors Provoost)
ba78c31a00 fuzz: check ParseHDKeypath/WriteHDKeypath round-trip (Sjors Provoost)
8cce969085 Have ParseHDKeypath handle h derivation marker (Sjors Provoost)
fc53077762 test: move parse_hd_keypath test to bip32_tests (Sjors Provoost)
dab525eb77 key: add DeriveExtKey() helper (Sjors Provoost)
Pull request description:
Adds a `derivehdkey` RPC that returns an xpub, or optionally the xprv, at an arbitrary BIP32 path (with at least one hardened step), derived from a wallet HD key.
The main use case is coordinating a multisig setup, where each participant shares an xpub derived at a hardened path (e.g. `m/87h/0h/0h`) distinct from their default single-signature descriptors. See the (updated) `doc/multisig-tutorial.md` and (updated) functional test to see how that workflow improves.
The first commits are some helpful helpers:
- _key: add DeriveExtKey() helper_ - performs the actual derivation
- _test: move parse_hd_keypath test to bip32_tests_ - from `psbt_wallet_tests`
- _Have ParseHDKeypath handle h derivation marker_
- _util: reject out-of-range BIP32 keypath indices_ - `ParseHDKeypath` would previously map overflowing values without `h` to hardened.
- _fuzz: check ParseHDKeypath/WriteHDKeypath round-trip_
- _rpc: ParsePathBIP32 helper_
- _refactor: add hardened derivation helper_ - `HasHardenedDerivation()`, to enforce the "at least one hardened step" rule
- _wallet: generalize GetActiveHDPubKeys helper_ - extracts code from `gethdkeys` which `derivehdkey` needs
- _wallet: add GetExtKey helper_ - reconstruct an xprv from a wallet xpub (analog of `GetKey()`); behavior-preserving prep, also simplifies `gethdkeys`.
Meat and potatoes:
- _rpc: add derivehdkey_ - the RPC itself, plus the `UnusedKey` filter on `GetHDPubKeys` that drives key selection.
- _test: use derivehdkey in M-of-N multisig demo_ - rewrites the functional multisig test to use the RPC and `<0;1>` syntax.
- _doc: use derivehdkey in multisig tutorial_ - same for the prose tutorial.
ACKs for top commit:
pseudoramdom:
code review ACK c3945bfd2b
achow101:
ACK c3945bfd2b
w0xlt:
That being the case, ACK c3945bfd2b
Tree-SHA512: 661f17c9bfe26017eb14c27ba7af37093387100d3baa25f5d29bba9c1aedc40d19afe1bdfc126a18d018857bb02f1fc84386f10b8f4f4b8e9d6f4b0691d9e302
`verify-commits.py` must not authorize checkout for a commit whose history diverges from configured trust roots.
Require proof that the commit is an ancestor of a root before skipping checks, and identify the failing root in errors.
Co-authored-by: Rob Hamilton <6456095+Rob1Ham@users.noreply.github.com>
`verify-commits.py` must not authorize checkout when Git cannot inspect the requested commit or its ancestry.
Reject ancestry command errors and validate the exact trusted root through Git before reporting success.
Co-authored-by: Rob Hamilton <6456095+Rob1Ham@users.noreply.github.com>
Rather than rebroadcasting forever, bound attempts at
private broadcast, report remaining attempts over RPC
results, and allow exhausted transactions to be
retried when submitted.
ae36e2ef79 rpc: avoid quadratic prevout resolution (Lőrinc)
da1eaeb350 rpc: preserve `gettxspendingprevout` order (Lőrinc)
f98753e762 refactor: identify prevouts by request index (Lőrinc)
221a3fe5cf test: cover mixed `gettxspendingprevout` order (Lőrinc)
Pull request description:
**Problem:** `gettxspendingprevout` erases each mempool result from a vector while holding `mempool.cs`, shifting the remaining requests every time and making large calls quadratic in the critical section.
For 10,000 mempool matches, an [operation-count model](https://godbolt.org/z/nzch7McPG) reaches nearly 50 million moves.
For mixed requests, the RPC returns mempool results before `txospenderindex` results instead of following request order.
#34749 introduced both regressions.
**Fix:** `gettxspendingprevout` stores each result at its request position and collects unresolved requests in a reserved worklist for the `txospenderindex` lookup.
The mempool pass is linear, the response follows request order, and Clang can verify the lock requirement on `GetConflictTx`.
**Benchmark:** The [functional benchmark](https://gist.github.com/l0rinc/c3231e287cacfdefd100dbf95cd0c3ad) sends mempool-only requests ranging from 8,000 to 128,000 entries ten times per size.
Using the same settings for the unfixed and fixed commits:
```text
AMD Ryzen 7 3700X (8 cores)
unfixed ██████████████████████████████ 90 s
fixed ███▒░░░░░░░░░░░░░░░░░░░░░░░░░░ 10 s (-80 s, 9.0x faster)
Raspberry Pi 5 (4 cores)
unfixed ██████████████████████████████ 685 s
fixed ▓░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 22 s (-663 s, 31.1x faster)
```
The unfixed run timed out after ~9 minutes on a Raspberry Pi 4 with 1 GB RAM.
<details><summary>Benchmark command</summary>
```bash
for commit in 963b061358 46e7173550a93cbe9d4e8ea28cfe7216286d8197; do \
git fetch origin "$commit" && git checkout --detach "$commit" && \
rm -rfd build && cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=OFF -DENABLE_WALLET=OFF >/dev/null 2>&1 && \
ninja -C build -j1 bitcoind >/dev/null 2>&1 && \
build/test/functional/test_runner.py rpc_gettxspendingprevout_quadratic.py --repeats=10 || break; \
done
```
</details>
ACKs for top commit:
andrewtoth:
ACK ae36e2ef79
sedited:
Re-ACK ae36e2ef79
Tree-SHA512: c7734cae481f6638228c8fd3cc6d4c3fbc26cec6981dae7902292af08eae37455d37ff196da2cca5c3694271c99154838431ff21c12855f81f5f10edf85a793a
`gettxspendingprevout` erases each mempool result from its worklist while holding `mempool.cs`, shifting the remaining requests every time and making the pass quadratic when it resolves many requests.
Collect unresolved requests in a reserved worklist so the mempool pass is linear and the compiler can verify the lock requirement on `GetConflictTx`.
Co-authored-by: Andrew Toth <andrewstoth@gmail.com>
`LegacyScriptPubKeyMan` was removed in 83af1a3cca. Remove the orphaned
forward declaration in `wallet/rpc/util.h`, rename
`CWallet::SetupLegacyScriptPubKeyMan()` to `SetupLegacyDataSPKM()`, and
update the comments and inactive-HD-chain log message that still use
the deleted class name.
Co-authored-by: shuv-amp <i@shuvamp.com.np>
The definition of GetAffectedKeys() and its only caller were removed in
83af1a3cca ("wallet: Delete LegacySPKM"), but the declaration in
scriptpubkeyman.h was left behind. The symbol has had no definition and
no callers since then, so drop the leftover declaration.
EraseTx calls LimitOrphans, which may evict announcements from a peer
that didn't announce the erased transaction, causing that peer's usage
to decrease. Relax the assertion in the EraseTx branch that claimed
usage of a non-announcer peer is unchanged. Also, add assertions for
the other cases.
fab74a0e92 refactor: Use C++14 digit separator for large int literals (MarcoFalke)
fae759be79 scripted-diff: Use inline constexpr over plain constexpr (MarcoFalke)
fa74f58a26 scripted-diff: Use inline const over (static) const (MarcoFalke)
fab1a62c87 refactor: Use inline constexpr for string literals in headers (MarcoFalke)
fa08bbed8d contrib: Adjust generate-seeds.py to write inline constexpr (MarcoFalke)
fad753611b scripted-diff: Use inline constexpr over (static) const (MarcoFalke)
faedb52583 refactor: Make CFeeRate(integral) ctor constexpr (MarcoFalke)
5555d5dcb5 scripted-diff: Use inline constexpr over static constexpr (MarcoFalke)
fa6e1a1e85 refactor: Remove static from constexpr functions in headers (MarcoFalke)
Pull request description:
Both are fine and this refactor doesn't change any behavior.
However, `inline constexpr` from C++17 will ensure each symbol has a single address
across all TU, making the release binary minimally smaller. (For me it is smaller by about 1kB)
ACKs for top commit:
l0rinc:
reACK fab74a0e92
rustaceanrob:
ACK fab74a0e92
hebasto:
ACK fab74a0e92, I have reviewed the code and it looks OK.
Tree-SHA512: 6ec94136c12bcbf696812d0661c9857318a69e367c79fc00b9ca0b4068f269d10e5548d95c9ba2070225308c12d7a54fe8cb8447de7e0979cba99f48892b35f9
8b5da677d7 common: remove ::runtime_error from RunCommandParseJSON (fanquake)
Pull request description:
I don't think there's a code path that can reach `RunCommandParseJSON` if we compile with `ENABLE_EXTERNAL_SIGNER=OFF`. If there is a reason for having the code this way, it could be good to document.
This also requires more workarounds in #35911.
ACKs for top commit:
stickies-v:
re-ACK 8b5da677d7
sedited:
ACK 8b5da677d7
willcl-ark:
ACK 8b5da677d7
Tree-SHA512: b0c50372fed35afe47713310851f0b58cd1803fbe87a3a5a75877772172a3881283394a91663251de22a9972f56b46d84ddc868686dec8b970474cfaf5dc0d32
34c03075a5 test: run generic baseindex tests against every index type (Martin Zumsande)
11b3e251c4 test: make baseindex flush test chain-length agnostic (Martin Zumsande)
8b959f4c6a test: move unclean_shutdown test to baseindex_tests (Martin Zumsande)
2232d6afbe test: move index_reorg_crash to baseindex_tests (Martin Zumsande)
a3597e2683 test: move BuildChain helper into test mining util (Martin Zumsande)
954985e6a3 test: simplify blockfilter test's BuildChain helper (Martin Zumsande)
Pull request description:
In #34897, the `baseindex_tests` unit test was introduced, meant for tests that test basic index functionality (e.g. reorg or unclean shutdown behavior) that should work regardless of the particular index type.
This PR moves two more of these tests (`index_reorg_crash`, `coinstatsindex_unclean_shutdown`) from test files of specific indexes into that folder.
In the second part, tests are executed sequentially for all index types instead of just one particular one, where applicable.
Before moving `index_reorg_crash` I extracted the `BuildChain` helper to `util/mining` so that it can be used by multiple tests. While doing that, I simplified the helper a bit.
ACKs for top commit:
jeanpablojp:
tACK 34c03075a5
sedited:
ACK 34c03075a5
Tree-SHA512: 1d7a43160a9b7ec3c75a8c806967f2031da4855fe449c9c8aac8e44b1940e5ee28fde9473406666e74a682cf135b87f8c0eb9ddba50fce156dce9fc54a7763eb
CMasterKey::nDeriveIterations values are deserialized from wallet
files as unsigned 32-bit integers, but key derivation narrowed the
count to a signed int. A count above INT_MAX became negative in the
conversion, and the derivation loop counter then overflowed, which
is undefined behavior.
Keep the count unsigned through the derivation path to match the
serialized type. Add a unit test for zero and normal counts.
test_alternate_witness_tx mines the taproot funding output on node0
with sync_fun=self.no_op and immediately disconnects. node1 later
includes the script-path spend via generateblock. If the funding
block has not reached node1, that call fails with
bad-txns-inputs-missingorspent.
Drop the no_op so generate() uses the default sync_all before the
partition.
Signed-off-by: shaurya2k06 <shaurya2k06@gmail.com>
I don't think there's a code path that can reach RunCommandParseJSON if
we compile with `-DENABLE_EXTERNAL_SIGNER=OFF`. This also requires more
workarounds in #35911.
Co-authored-by: stickies-v <stickies-v@protonmail.com>
New databases never contain legacy ('t' + txid) entries.
Peek at the database before opening it, and if no legacy
entries are found, skip building bloom filters (hashed
entries are only read via iterators, which do not consult
them) and return early from lookups instead of checking
for legacy entries.
Use a 5-byte salted siphash to key txindex entries,
instead of the full 32-byte txid. Store the block sequence and tx position
after the hash in the key, so an iterator can scan
through any collisions and return the correct tx.
Fall back to the legacy key lookup if the tx is not found.
Co-authored-by: Pieter Wuille <pieter@wuille.net>
Co-authored-by: l0rinc <pap.lorinc@gmail.com>
Co-authored-by: Anthony Towns <aj@erisian.com.au>
Move the per-transaction position computation from CustomAppend into
DB::WriteTxs, so the DB layer receives the whole block instead of a
pre-built vector of positions. This is a non-functional refactor.
The hashed txindex entries cannot be found by older nodes. Record sync
progress under a new locator key so a downgraded node will not rely on
entries indexed by upgraded nodes, and instead continue syncing from the
legacy locator.
`Range` iterators point to the view, so iterators saved from temporary views dangle.
Point them to the underlying container, use the range's getter for element access, and remove `operator->`, which returned elements by value and could not support arrow expressions.
Co-authored-by: Daniel Pfeifer <daniel@pfeifer-mail.de>
Document the `mutated` output flag on the `ComputeMerkleRoot` declaration and explain in the inner loop why the mutation check runs at every tree level even after a duplicate is found.
Add a direct regression test for the duplicate-subtree construction described in the code comments for CVE-2012-2459: `[1,2,3,4,5,6]` and `[1,2,3,4,5,6,5,6]` produce the same root.
The test also verifies that mutation detection checks equal pairs before the final pair of a tree level.
The existing `merkle_test` already exercises this behavior indirectly through random duplications and old-vs-new comparisons.
The new test pins it down explicitly through the `ComputeMerkleRoot` API.
Both would fail under a refactor that stops the outer reduction once mutation is detected.
Co-authored-by: Hodlinator <172445034+hodlinator@users.noreply.github.com>
