This RPC has no helpful use while being both dangerous and a maintenance
burden.
Despite what the name says, it allows the deletion of arbitrary
transactions, and `importprunedfunds` does not allow the importing of
transactions not belonging to the user, and `listtransactions` does not
list transactions not belonging to the wallet, so this RPC can only be
used to delete transactions actually belonging to the wallet, and in the
unlikely event that transactions not belonging to the wallet are
present, they cause no harm except for occupying a few bytes on the
users disk.
0cdd817a82 add release note (Pol Espinasa)
517d37ce3e test: tests wallet migration with load_wallet disabled (Pol Espinasa)
b98dd63da7 rpc: Add load_wallet argument to migratewallet RPC (Pol Espinasa)
4acd063ba6 wallet: make loading the wallet after migrating optional (Pol Espinasa)
97d08d62ba refactor: store wallet names to MigrationResult (Pol Espinasa)
Pull request description:
This PR is motivated by this [Stack Exchange question](https://bitcoin.stackexchange.com/questions/130713/bitcoin-core-quickest-method-legacy-descriptor-wallet-migration).
Long story short, someone who has a node pruned before his legacy wallet birthday, is unable to migrate the wallet as it is not possible to load it.
Loading is not necessary for migration, and migrating without wanting to use the wallet in that node is a valid use-case.
This PR adds a new RPC argument to `migratewallet` that allow the user disabling the wallet loading.
Second commits adds tests for it.
Follow-up: Add an option to the GUI to not load the wallet after migrating.
ACKs for top commit:
achow101:
ACK 0cdd817a82
w0xlt:
ACK 0cdd817a82
pablomartin4btc:
ACK 0cdd817a82
Tree-SHA512: 8389599e63603b1a532e1bfba0b6c652653386c001f5a881bd49843302b74ff4dbaa4131b5b377c24f483d42e0e70a92b96f760244e3c2e2b44ce08cd04ca1e0
703a671fbc fuzz: compact coins view db during fuzzing (Lőrinc)
0868c85fd5 refactor: rename async coin compaction (Lőrinc)
Pull request description:
**Problem:** #35465 added async chainstate compaction, but the `coins_view_db` fuzz target did not exercise scheduling compaction alongside ordinary coins view operations.
The async wrapper also shared the `CompactFull()` name with the blocking `CDBWrapper` primitive.
**Fix:** Rename the coins DB wrapper to `CompactFullAsync()` and let `coins_view_db` randomly schedule it under `cs_main` (like in production).
The fuzz operation only starts compaction and any running job is joined by the `CCoinsViewDB` destructor at the end of the fuzz input.
ACKs for top commit:
sedited:
ACK 703a671fbc
andrewtoth:
ACK 703a671fbc
Tree-SHA512: 9854c3acbaace795155e7469cb10938fbd872726cbb8a4b4ef71d6d352d00824498747215aee1b237e64452eae2e690a3f7d7daa6b6b0030e75a6f2ccc0802fb
abc33ff043 test: announce field must be 0 or 1 in sendcmpct (brunoerg)
2d0dce0af5 net_processing: fix BIP152 first integer interpretation (brunoerg)
Pull request description:
Fixes#35542
According to the BIP152, the first integer in `sendcmpct` message shall be interpreted as a boolean (and MUST have a value of either 1 or 0). We currently correctly interpret it as boolean, however, we accept any value >=1 and treat it as `true`, deviating from the specification. This PR fixes it.
ACKs for top commit:
edilmedeiros:
utACK abc33ff043
davidgumberg:
crACK abc33ff043 Seems reasonable to comply with BIP152 strictly, test looks good as well.
Sjors:
ACK abc33ff043
jonatack:
re-ACK abc33ff043
achow101:
ACK abc33ff043
w0xlt:
ACK abc33ff043
Tree-SHA512: 77fed86d4de81f7c35ff002b6e1b2a90882ea55f159075da4d34a619d1075f625fca34f929cdd981f1b2eb06f76b64f42cda46502dcd1b4a634c690b0882ec7c
Exercise `CCoinsViewDB::CompactFullAsync()` from the `coins_view_db` fuzz target so the new chainstate compaction wrapper can run concurrently with ordinary coins view operations.
The fuzz operation only schedules compaction, matching production; outstanding work is waited for by the `CCoinsViewDB` destructor at the end of the fuzz input.
Rename the `CCoinsViewDB` async compaction wrapper to `CompactFullAsync()` so it is distinct from the blocking `CDBWrapper::CompactFull()` primitive it calls.
b847626562 test: refresh MiniWallet after node restart (Sjors Provoost)
f4e643cb15 test: merge mining options in package feerate check (Sjors Provoost)
280ce6a0ae miner: ensure block_max_weight is flattened before limit checks (Sjors Provoost)
65bd3164fb mining: clarify test_block_validity comment (Sjors Provoost)
978e7216e6 test: use shared default_ipc_timeout (Sjors Provoost)
Pull request description:
This implement the suggested followups from #33966. Each commit links to the original comment.
The most important change is the extra asserts added in `miner: ensure block_max_weight is flattened before limit checks`.
ACKs for top commit:
achow101:
ACK b847626562
enirox001:
tACK b847626562
sedited:
ACK b847626562
w0xlt:
ACK b847626562
Tree-SHA512: 47678eaed604228269bd892ccf8ff58804745bbc7675b4a93528da9a9292a2eb1e0562cdb8341edac77178563420885b48282bb9e5c2b997b28f2fc64ceeff3d
2fe34808fa wallet: reject sendtoaddress and sendmany for external signers (Sjors Provoost)
bd5a32f7db doc: add taproot descriptor to getdescriptors example (woltx)
7131c82937 doc: clarify which commands receive --chain, --fingerprint and --stdin (woltx)
4fdd4d8d29 doc: replace stale signtransaction wording with current signtx flow (woltx)
fab92257fe doc, rpc: document enumerate model field and fingerprint deduplication (woltx)
Pull request description:
This PR aligns the external signer documentation with current behavior, and makes one previously implicit behavior explicit.
Per review feedback, each commit fixes a limited set of issues:
* **doc, rpc: document enumerate model field and fingerprint deduplication** — the `enumerate` response uses the optional `model` field, which Bitcoin Core maps to the `name` field of the `enumeratesigners` RPC result. Duplicate fingerprints are skipped, and wallet operations require exactly one connected signer.
* **doc: replace stale signtransaction wording with current signtx flow** — spending from an external signer wallet uses `send`/`sendall` (and `bumpfee` for fee-bumping), which invoke `<cmd> --stdin` and pass the `signtx` subcommand and PSBT over stdin.
* **doc: clarify which commands receive --chain, --fingerprint and --stdin** — mark `--chain` and `--fingerprint` as required except for `enumerate`, keep `--stdin` required for protocol flexibility, and match the order and form of the actual invocations in the usage examples.
* **doc: add taproot descriptor to getdescriptors example** — show the BIP86 `tr()` descriptor alongside the other address types.
* **wallet: reject sendtoaddress and sendmany for external signers** — return a specific error instead of the misleading "Private keys are disabled for this wallet", with functional test coverage. Cherry-picked from #33112 (thanks Sjors).
How the documentation went stale:
* The `enumerate` example has shown a `name` field since external signer support landed in #16546, but the implementation has always read `model`.
* `sendtoaddress`/`sendmany` external signer support was effectively precluded by #21201, which was merged a few days before #16546, so the interaction was missed in review and the documented `signtransaction` flow never existed in this form.
* Fingerprint deduplication was added in #35251.
* The documentation was last updated in #33765.
ACKs for top commit:
Sjors:
ACK 2fe34808fa
optout21:
ACK 2fe34808fa
naiyoma:
ACK 2fe34808fa
Tree-SHA512: 86859d2f81ac337f3b4b6578c6ee0151ffb76b8374dfa58e28e00ce4eb69dc200cd6bd2d0a99f73d0475c3824d6ac1cb9e2542b119ca124dd835132dc95cd023
d6359937bf validation: check invariants when inserting into m_blocks_unlinked (stratospher)
0852925bd8 test/doc: remove misleading comment and improve tests (stratospher)
ca4a380281 test: add coverage for UB caused by FindMostWorkChain (stratospher)
c787b3b99b validation: avoid duplicates in m_blocks_unlinked (stratospher)
Pull request description:
This is joint work with @ mzumsande.
note: this requires a pruned node with deep reorgs to trigger. still it breaks assumptions in the codebase and is good to fix. A similar UB was fixed in https://github.com/bitcoin/bitcoin/pull/34521.
This PR prevents duplicate insertions into `m_blocks_unlinked` in `FindMostWorkChain`. There are 3 ways to insert into `m_blocks_unlinked`:
1. `LoadBlockIndex` - not problematic, as each block index is processed only once.
2. `ReceivedBlockTransactions` - not problematic, as this is usually only called once per block when it is first accepted in `AcceptBlock`. in the rare case it’s triggered again after pruning, the block would have been removed from `m_blocks_unlinked` when it was initially pruned, so duplicates still can’t arise.
3. `FindMostWorkChain` - problematic when multiple candidate tips share common chains of ancestors, traversals from each tip to the fork point may insert duplicate (`pprev`, `pindex`) entries for blocks whose parents have been pruned.
When the missing parent is later received and `ReceivedBlockTransactions` processes `m_blocks_unlinked`, the same entry may be processed multiple times. This can result in the block being re-added to `setBlockIndexCandidates` with a modified `nSequenceId`, violating its ordering invariants and leading to undefined behavior. So avoid duplicate insertions into `m_blocks_unlinked` in `FindMostWorkChain`.
### how to test:
use the updated `feature_pruning.py` which adds coverage for this scenario.
- on master: the test (with the below diff) fails since `nSequenceId` is being modified for an entry in `setBlockIndexCandidates`
- on this branch: the test (with the below diff) passes
```diff --git a/src/validation.cpp b/src/validation.cpp
--- a/src/validation.cpp
+++ b/src/validation.cpp
@@ -3814,6 +3814,12 @@ void ChainstateManager::ReceivedBlockTransactions(const CBlock& block, CBlockInd
pindex->nHeight, pindex->m_chain_tx_count, prev_tx_sum(*pindex), CLIENT_NAME, FormatFullVersion(), CLIENT_BUGREPORT);
}
pindex->m_chain_tx_count = prev_tx_sum(*pindex);
+ for (const auto& c : m_chainstates) {
+ if (c->setBlockIndexCandidates.contains(pindex)) {
+ LogInfo("### pindex UB = %s", pindex);
+ assert(false);
+ }
+ }
pindex->nSequenceId = nBlockSequenceId++;
for (const auto& c : m_chainstates) {
c->TryAddBlockIndexCandidate(pindex);
```
ACKs for top commit:
sedited:
Re-ACK d6359937bf
marcofleon:
crACK d6359937bf
stringintech:
ACK d6359937bf
mzumsande:
Sure - Code Review ACK [d635993](d6359937bf)
Tree-SHA512: bb21adc2d92fe1865bbbcebf775a850ca3eccac6fe83d7bca10b78eee4c0abf782e44fa0ddfec9d9a70f42fa40bdc49b68a1d1b4905cdc371ba29117d3120619
Loading the wallet after migrating is not a necessary step.
By making it optional pruned nodes can also migrate legacy wallets.
Also remove the migrated wallet from the list of wallets to load on startup.
48df0939e7 fuzz: Remove unnecessary thread pool mutexes (marcofleon)
a4c3b003f8 fuzz: Speed up dbwrapper_concurrent_reads harness (marcofleon)
Pull request description:
Limiting how many read queries each worker executes significantly speeds up this test, especially when running with sanitizers. This still builds the full query list, and then takes the first 128/2000 after each worker shuffles it. I think that keeps some more operation diversity vs just lowering the query max directly. It also allowed me to reuse and test against a corpus I already had. Let me know if I'm wrong, but I don't think this test needs every worker to execute an identical query list to be effective.
This PR also reverts the `num_entries` max from 3000 back to 5000, as that didn't have much effect on input speed and restores a bit of lost coverage.
Lastly, as https://github.com/bitcoin/bitcoin/pull/35455#discussion_r3394357385 points out, remove the unnecessary `Mutex` from `StartReadPoolIfNeeded()`. Fuzz targets are entered sequentially within a process and parallel fuzzing uses separate processes/forks, so a mutex to prevent two in-process callers from racing to start the pool isn't needed.
ACKs for top commit:
sedited:
ACK 48df0939e7
brunoerg:
ACK 48df0939e7
Tree-SHA512: 24c35e13fa26790b36e5190283b78dd8511165d576841cb701f541a36c5b7c0a73f9bc265e71598a879fa1df9a9438e7841736f72b967214196fa6fe68569eff
Remove the `Mutex` from the `threadpool` and `dbwrapper_concurrent_reads`
pool startup helpers. Fuzz targets are entered sequentially within a
process and parallel fuzzing uses separate processes/forks, which each
have their own copy of the global thread pool. Therefore, a mutex to
prevent two in-process callers from racing to start the pool isn't needed.
Limit how many read queries each worker executes. This significantly
speeds up the test, as each worker runs >90% fewer (2000 to 128)
expensive LevelDB operations (like `IteratorSeek`) but still
ends up hitting the intended target code.
Revert the `num_entries` max from 3000 back to 5000, as that didn't
have much effect on input speed and restores a bit of lost coverage.
Creates logic to deprecate logging categories but still
"support" them so the software doesn't quit with unknown category
on startup. Deprecated categories are always false and attempts
to switch them are logged as warnings.
The original function was already naturally split into two chunks:
First, we parse and validate the users' RPC configuration for IPs and
ports. Next we bind libevent's http server to the appropriate
endpoints.
This commit splits these chunks into two separate functions, leaving
the argument parsing in the common space of the module and moving the
libevent-specific binding into the http_libevent namespace.
A future commit will implement http_bitcoin::HTTPBindAddresses to
bind the validate list of endpoints by the new HTTP server.
The original function is passed to libevent as a callback when HTTP
requests are received and processed. It wrapped the libevent request
object in a http_libevent::HTTPRequest and then handed that off to
bitcoin for basic checks and finally dispatch to worker threads.
In this commit we split the function after the
http_libevent::HTTPRequest is created, and pass that object to a new
function that maintains the logic of checking and dispatching.
This will be the merge point for http_libevent and http_bitcoin,
where HTTPRequest objects from either namespace have the same
downstream lifecycle.
This is a refactor to prepare for matching the API of HTTPRequest
definitions in both namespaces http_bitcoin and http_libevent. In
particular, to provide a consistent return type for GetRequestMethod()
in both classes.
See https://www.rfc-editor.org/rfc/rfc7230#section-6.3.2
> A server MAY process a sequence of pipelined requests in
parallel if they all have safe methods (Section 4.2.1 of [RFC7231]),
but it MUST send the corresponding responses in the same order that
the requests were received.
We choose NOT to process requests in parallel. They are executed in
the order recevied as well as responded to in the order received.
This prevents race conditions where old state may get sent in response
to requests that are very quick to process but were requested later on
in the queue.
Sockets-touching bits copied and adapted from `CConnman::SocketSendData()`
Testing this requires adding a new feature to the SocketTestingSetup,
returning the DynSock I/O pipes from the mock socket so the received
data can be checked.
Co-authored-by: Vasil Dimov <vd@FreeBSD.org>
`SocketHandlerConnected()` adapted from CConnman
Testing this requires adding a new feature to the SocketTestingSetup,
inserting a "request" payload into the mock client that connects
to us.
This commit also moves IOErrorIsPermanent() from sock.cpp to sock.h
so it can be called from the socket handler in httpserver.cpp
Co-authored-by: Vasil Dimov <vd@FreeBSD.org>
Socket handling methods are copied from CConnMan:
`CConnman::GenerateWaitSockets()`
`CConnman::SocketHandlerListening()`
`CConnman::ThreadSocketHandler()` and `CConnman::SocketHandler()` are combined into ThreadSocketHandler()`.
Co-authored-by: Vasil Dimov <vd@FreeBSD.org>
Introduce a new low-level socket managing class `HTTPServer`.
BindAndStartListening() was copied from CConnMan's BindListenPort()
in net.cpp and modernized.
Unit-test it with a new class `SocketTestingSetup` which mocks
`CreateSock()` and will enable mock client I/O in future commits.
Co-authored-by: Vasil Dimov <vd@FreeBSD.org>
This commit is a no-op to isolate HTTP methods and objects that
depend on libevent. Following commits will add replacement objects
and methods in a new namespace for testing and review before
switching over the server.
The utility can not be opinionated about CR or SP on either end of a
line it reads. That decision is up to the caller and in fact in the
case of HTTP should be allowed in some places and rejected in others.
Replace TrimStringView() with more surgical operation.
394e473d42 coins: compact chainstate in background (Lőrinc)
aa021b26f3 validation: randomly compact chainstate (Lőrinc)
b10889d107 coins: test chainstate flush baseline (Lőrinc)
Pull request description:
**Problem:** https://github.com/bitcoin-core/leveldb-subtree/pull/61 disabled read-triggered seek compactions to avoid large chainstate write amplification from random UTXO lookups.
That avoids repeated read-driven rewrites, but it also removes opportunistic cleanup that previously helped compact old chainstate data.
After IBD, normal chainstate churn can leave obsolete entries behind until ordinary LevelDB compaction naturally reaches the affected levels, keeping the chainstate database larger than necessary.
Also, chainstates created by pre-29 nodes can contain thousands of files from the old 2 MiB LevelDB table target.
After the mmap limit dropped back to 1000 and seek compaction was disabled, continuing from such a chainstate can leave many table reads on the non-mmap path until the database is compacted.
**Fix:** After each completed post-IBD full chainstate flush, give the chainstate a 1/320 chance to compact.
With roughly hourly full flushes, this averages about once every two weeks and makes a six-month stretch without compaction about a one-in-a-million event.
The randomized recurring trigger spreads compactions across nodes and keeps maintenance stateless, without storing last-compaction height or timestamp metadata in the chainstate database.
Compaction runs on a background thread (`utxocompact`) so validation only schedules the work.
Partially fixes#35298 and #35457
ACKs for top commit:
sipa:
ACK 394e473d42
optout21:
ACK 394e473d42
andrewtoth:
ACK 394e473d42
sedited:
ACK 394e473d42
Tree-SHA512: 536a298f86ca7d87a2dd5428354b05b693931c090c7a4afd4d09b95ce53e2bd8319e01c940489a1520d8cb1010a7ea9724d2c18d2e12887bd5ec70cb68c03517
855a3fee88 scripted-diff: Rename SteadyClockContext to FakeSteadyClock (Hao Xu)
Pull request description:
See: https://github.com/bitcoin/bitcoin/pull/35536#issuecomment-4715945494
`SteadyClockContext` and `FakeNodeClock` are both LimitOne RAII helpers that mock a clock in tests -- the steady clock and the node clock, respectively. Rename the former so the two follow a consistent FakeXClock naming scheme.
ACKs for top commit:
maflcko:
lgtm ACK 855a3fee88
w0xlt:
ACK 855a3fee88
sedited:
ACK 855a3fee88
Tree-SHA512: 72fd8dd3217d19f4b9f587e379a0aea2b7760ca6d61be6c363e7c06757cba40487df8b40cf3d62f9f9256fb4d0373610dd6f62c6e9ef3edde02265fc967a6bdf
735b25519a support: clamp RLIMIT_MEMLOCK to size_t (Sjors Provoost)
8ab4b9fc85 init: clamp fd limits to int (Sjors Provoost)
4afbabdcef Fix startup failure with RLIM_INFINITY fd limits (Sjors Provoost)
Pull request description:
When setting the fd limit to unlimited, the node fails to start:
```sh
ulimit -n unlimited
build/bin/bitcoind
Error: Not enough file descriptors available. -1 available, 160 required.
```
This was caused by `RaiseFileDescriptorLimit()` (introduced in #2568) casting `limitFD.rlim_cur` to `int`, which for `RLIM_INFINITY` overflows to `-1`. Fix it by returning `std::numeric_limits<int>::max()` instead.
Some platforms implement `RLIM_INFINITY` as the maximum uint64, others as int64 (-1). So simply changing the return type to `uint64_t` wouldn't work.
Similarly, though unlikely to actually happen:
```sh
ulimit -n 214748364
build/bin/bitcoind
Error: Not enough file descriptors available. -2147483648 available, 160 required.
```
The second commit expands the fix by clamping all values above `std::numeric_limits<int>::max()` instead of letting them overflow.
This PR also expands `test/functional/feature_init.py` to cover these, using `resource.setrlimit`. The check is skipped on environments with a hard limit below infinity (or that don't have the Python [Resource module](https://docs.python.org/3/library/resource.html)).
macOS by default has a hard limit of infinity, but on e.g. Ubuntu the default hard limit is 524288.
The third commit applies a similar fix to `PosixLockedPageAllocator::GetLimit()` for 32-bit systems, but without a test.
ACKs for top commit:
winterrdog:
Re-ACK 735b25519a
achow101:
ACK 735b25519a
sedited:
Re-ACK 735b25519a
pinheadmz:
ACK 735b25519a
Tree-SHA512: 0ce0292ecd61456bdec6943b06cbb9ecfc5180ee6dce850f8496ef54af22c1fae6ea473085202f5ba6f72e4dc51a29247620c9a0eae31e96658adc77b293129f
2447385f47 rpc: remove unused RPCResult::Type::ELISION (satsfy (Renato Britto))
7a85118005 rpc: expand decodepsbt output script with explicit fields (satsfy (Renato Britto))
88e2a6ae89 rpc: expand getaddressinfo embedded with explicit fields (Renato Britto)
a9f9e7d17e rpc: extract fee estimate result helpers (Renato Britto)
8a615a8800 rpc: extract ListSinceBlockTxFields() helper (Renato Britto)
372ac283ac rpc: extend TxDoc() for getblock verbosity 2/3 (Renato Britto)
0380a1c46b rpc: extend TxDoc() for getrawtransaction verbosity 2 (Renato Britto)
44fc3a290d rpc: introduce HelpElision variant and ElideGroup helper (Renato Britto)
Pull request description:
Partially addresses #29912. Motivated by #34683, which exports OpenRPC from existing `RPCHelpMan` metadata. [Sample OpenRPC](https://gist.github.com/natobritto/8c4a1da04968d2325082ac4bca7d2408).
Some RPC help definitions rely on `RPCResult::Type::ELISION` entries whose structure is only described in prose. This keeps human-readable help concise, but leaves parts of the result layout implicit and prevents tools from deriving complete machine-readable schemas from `RPCHelpMan` metadata.
This PR replaces ELISION-based reuse with shared structured definitions, so result layouts are represented directly in metadata rather than only in text. At the same time, human-readable help remains compact via explicit help-rendering elision using `HelpElision`, so previously elided sections stay abbreviated without losing schema completeness.
Affected RPCs: `getrawtransaction`, `getblock`, `listsinceblock`, `estimaterawfee`, `getaddressinfo`.
RPC return values are unchanged. Human-readable help remains compact, while structured result metadata becomes explicit enough to derive complete machine-readable schemas.
A related `RPCResult::Type::ELISION` use in `importdescriptors` was split out into the follow-up PR #34867 because it changes the generated help output, per [this review comment](https://github.com/bitcoin/bitcoin/pull/34764#discussion_r2960236111).
Changes:
- Introduce `HelpElision` (`NONE`, `START`, `SKIP`) and `ElideGroup()`, replacing the tri-state `print_elision`
- Add an `RPCResult` copy-with-replacement-options constructor to support applying elision while keeping `m_opts` const
- Extend `TxDoc()` / `TxDocOptions` to support reusable transaction layouts with optional `prevout`, `fee`, `hex`, and elision behavior
- Replace ELISION-based reuse in `getrawtransaction` and `getblock` with explicit structured definitions
- Factor shared result layouts into `GetBlockFields()`, `ListSinceBlockTxFields()`, `FeeRateBucketDoc()`, `GetAddressInfoEmbeddedFields()` and `FeeEstimateHorizonDoc()`
- Expand `listsinceblock.removed`, `estimaterawfee` horizons/buckets and `getaddressinfo.embedded` into explicit metadata while preserving concise help output
ACKs for top commit:
achow101:
ACK 2447385f47
w0xlt:
reACK 2447385f47
janb84:
re ACK 2447385f47
willcl-ark:
ACK 2447385f47
Tree-SHA512: 8dc03c45c388ebdb4f8f1613af2576fc127a8d5425efe518cd0d0ed2439a38b2ed1236413471672c85f52ee22c4a17677c18fd4689bc6570496fc3af4cd4112f
