3e8bf2e10c26ef9c2f58307b523e4b674ac97a2c test: make assumeUTXO test capture the expected fatal error (furszy)
Pull request description:
The test is exercising the error, so it can capture it before the
test framework displays it on the console as an unforeseen
fatal error.
It is odd to observe a fatal error after executing the complete
test suite and seeing it pass successfully.
Reproduction Steps:
Run the unit test suite. A long AssumeUTXO fatal error will be
printed even when all tests pass successfully.
ACKs for top commit:
MarcoFalke:
lgtm ACK 3e8bf2e10c26ef9c2f58307b523e4b674ac97a2c
theStack:
Tested ACK 3e8bf2e10c26ef9c2f58307b523e4b674ac97a2c
TheCharlatan:
ACK 3e8bf2e10c26ef9c2f58307b523e4b674ac97a2c
Tree-SHA512: 820a5a4db52085ed72cbe7eb433b8c0f2d283ac6f5d456bc2b3e3f0305301022b2729e32e5fd9002859e4491ae7ac6de568a4c20557c7b249b0e7694ab8bd177
At present, during init, we traverse the chain (once per index)
to confirm that all necessary blocks to sync each index up to
the current tip are present.
To make the process more efficient, we can fetch the oldest block
from the indexers and perform the chain data existence check from
that point only once.
This also moves the pruning violation check to the end of the
'loadinit' thread, which is where the reindex, block loading and
chain activation processes happen.
Making the node's startup process faster, allowing us to remove
the global g_indexes_ready_to_sync flag, and enabling the
execution of the pruning violation verification even when the
reindex or reindex-chainstate flags are enabled (which has being
skipped so far).
By moving the 'StartIndexes()' call into the 'initload'
thread, we can remove the threads active wait. Optimizing
the available resources.
The only difference with the current state is that now the
indexes threads will only be started when they can process
work and not before it.
By generalizing 'GetFirstStoredBlock' and implementing
'CheckBlockDataAvailability' we can dedup code and
avoid repeating work when multiple indexes are enabled.
E.g. get the oldest block across all indexes and
perform the pruning violation check from that point
up to the tip only once (this feature is being introduced
in a follow-up commit).
This commit shouldn't change behavior in any way.
Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
And transfer the responsibility of verifying whether 'start_block'
has data or not to the caller.
This is because the 'GetFirstStoredBlock' function responsibility
is to return the first block containing data. And the current
implementation can return 'start_block' when it has no data!. Which
is misleading at least.
Edge case behavior change:
Previously, if the block tip lacked data but all preceding blocks
contained data, there was no prune violation. And now, such
scenario will result in a prune violation.
Verify that our ChaCha20 implementation using the 96/32 split interface
is compatible with >256 GiB outputs by triggering a 32-bit block counter
overflow and checking that the keystream matches one created with an
alternative implementation using a 64/64 split interface with the
corresponding input data. The test case data was generated with the
following Python script using the PyCryptodome library (version 3.15.0):
----------------------------------------------------------------------------------------------
from Crypto.Cipher import ChaCha20
key = bytes(list(range(32))); nonce = 0xdeadbeef12345678; pos = 2**32 - 1
c = ChaCha20.new(key=key, nonce=nonce.to_bytes(8, 'little'))
c.seek(pos * 64); stream = c.encrypt(bytes([0])*128)
print(f"Key: {key.hex()}\nNonce: {hex(nonce)}\nPos: {hex(pos)}\nStream: {stream.hex()}")
----------------------------------------------------------------------------------------------
No behavior change.
The goal here is to group indexes, so we can perform the same
initialization and verification process equally for all of them.
The checks performed inside `StartIndexes` will be expanded
in the subsequent commits.
The thread does not only load blocks, it loads the mempool and,
in a future commit, will start the indexes as well.
Also, renamed the 'ThreadImport' function to 'ImportBlocks'
And the 'm_load_block' class member to 'm_thread_load'.
-BEGIN VERIFY SCRIPT-
sed -i "s/ThreadImport/ImportBlocks/g" $(git grep -l ThreadImport -- ':!/doc/')
sed -i "s/loadblk/initload/g" $(git grep -l loadblk -- ':!/doc/release-notes/')
sed -i "s/m_load_block/m_thread_load/g" $(git grep -l m_load_block)
-END VERIFY SCRIPT-
The mempool load can take a while, and it is not
needed for the indexes' synchronization.
Also, having the mempool load function call
inside 'blockstorage.cpp' wasn't structurally
correct.
There are two variants of ChaCha20 in use. The original one uses a 64-bit
nonce and a 64-bit block counter, while the one used in RFC8439 uses a
96-bit nonce and 32-bit block counter. This commit changes the interface
to use the 96/32 split (but automatically incrementing the first 32-bit
part of the nonce when the 32-bit block counter overflows, so to retain
compatibility with >256 GiB output).
Simultaneously, also merge the SetIV and Seek64 functions, as we almost
always call both anyway.
Co-authored-by: dhruv <856960+dhruv@users.noreply.github.com>
8b5397c00e821d7eaab22f512e9d71a1a0392ebf wallet: bdb: include bdb header from our implementation files only (Cory Fields)
6e010626af7ed51f1748323ece2f46335e145f2f wallet: bdb: don't use bdb define in header (Cory Fields)
004b184b027520a4f9019d1432a816e6ec891fe3 wallet: bdb: move BerkeleyDatabase constructor to cpp file (Cory Fields)
b3582baa3a2f84db7d2fb5a681121a5f2d6de3a1 wallet: bdb: move SafeDbt to cpp file (Cory Fields)
e5e5aa1da261633c8f73b97d5aefe5dc450a7db9 wallet: bdb: move SpanFromDbt to below SafeDbt's implementation (Cory Fields)
4216f69250937b1ca4650dc0c21678a8444c6650 wallet: bdb: move TxnBegin to cpp file since it uses a bdb function (Cory Fields)
43369f37060a1b4c987672707c500d35c9a27c1d wallet: bdb: drop default parameter (Cory Fields)
Pull request description:
Only `#include` upstream bdb headers from our cpp files.
It's generally good practice to avoid including 3rd party deps in headers as otherwise they tend to sneak into new compilation units. IMO this makes for a nice cleanup.
There's a good bit of code movement here, but each commit is small and _should_ be obviously correct.
Note: in the future, the buildsystem can add the bdb include path for `bdb.cpp` and `salvage.cpp` only, rather than all wallet sources.
ACKs for top commit:
achow101:
reACK 8b5397c00e821d7eaab22f512e9d71a1a0392ebf
hebasto:
ACK 8b5397c00e821d7eaab22f512e9d71a1a0392ebf
Tree-SHA512: 0ef6e8a9c4c6e2d1e5d6a3534495f91900e4175143911a5848258c56da54535b85fad67b6d573da5f7b96e7881299b5a8ca2327e708f305b317b9a3e85038d66
7ecc29a0b7a23d8f5d3c1e6a0dad29b3ad839eb9 test: wallet, add coverage for addressbook migration (furszy)
a277f8357ad8b0eb26f33fc36f919d868c06847b wallet: migration bugfix, persist empty labels (furszy)
1b64f6498c394a143df196172a14204fe3b8a744 wallet: migration bugfix, clone 'send' record label to all wallets (furszy)
Pull request description:
Addressing two specific bugs encountered during the wallet migration process, related to the address book, and improves the test coverage for it.
Bug 1: Non-Cloning of External 'Send' Records
The external 'send' records were not being correctly cloned to all wallets.
Bug 2: Persistence of Empty Labels
As address book entries without associated db label records can be treated as change (the `label` field inside the `CAddressBookData` class is optional, `nullopt` labels make `CAddressBookData ::IsChange()` return true), we must persist empty labels during the migration process.
The user might have called `setlabel` with an "" string for an external address and that must be retained during migration.
ACKs for top commit:
achow101:
ACK 7ecc29a0b7a23d8f5d3c1e6a0dad29b3ad839eb9
Tree-SHA512: b8a8483a4178a37c49af11eb7ba8a82ca95e54a6cd799e155e33f9fbe7f37b259e28372c77d6944d46b6765f9eaca6b8ca8d1cdd9d223120a3653e4e41d0b6b7
fa1e27fe8ec42764d0250c82a83d774c15798c4a fuzz: Generate rpc fuzz targets individually (MarcoFalke)
Pull request description:
The `rpc` fuzz target was added more than two years ago in e45863166f5e44cc2c380f4667812fcd3cddc73b. However, the bug https://github.com/bitcoin/bitcoin/issues/27913 was only found recently. Thus, it is pretty clear that fuzz engines can't deal with a search space that is too broad and can be extended in too many directions.
Fix that by limiting the search space to each RPC method name and then iterate over all names, instead of letting the fuzz engine do the iteration.
With this, the bug can be found in seconds, as opposed to years of CPU time (or never).
ACKs for top commit:
brunoerg:
ACK fa1e27fe8ec42764d0250c82a83d774c15798c4a
dergoegge:
ACK fa1e27fe8ec42764d0250c82a83d774c15798c4a
Tree-SHA512: 45ccba842367650d010320603153276b1b303deda9ba8c6bb31a4d2473b00aa5bca866db95f541485d65efd8276e2575026968c037872ef344fa33cf45bcdcd7
fac6af16f4a254458b8cb3522317422b40362f8d Allow std::byte serialization (MarcoFalke)
fade43edc4405e7c51cec9325d8502f3786f7438 Allow FastRandomContext::randbytes for all byte types (MarcoFalke)
Pull request description:
I need this for some stuff, but it should also be useful by itself for other developers that need it.
ACKs for top commit:
sipa:
utACK fac6af16f4a254458b8cb3522317422b40362f8d
dergoegge:
Code review ACK fac6af16f4a254458b8cb3522317422b40362f8d
Tree-SHA512: db4b1bbd6bf6ef6503d59b0b4ed1681db8d935d2d10f8d89f071978ea59b49a1d319bccb4e9717c0c88a4908bbeca4fd0cbff6c655d8a443554fd14146fe16de
fabed7eb796637c02e3677ebbe183d90b258ba69 test: Restore unlimited timeout in IndexWaitSynced (MarcoFalke)
Pull request description:
The timeout was unlimited before, so just restore that value for now: https://github.com/bitcoin/bitcoin/pull/27988#issuecomment-1619218007 .
(Strictly speaking, this is a behavior change for the blockfilterindex and txindex tests, because it only restores the coinstatsindex behavior.)
ACKs for top commit:
ajtowns:
utACK fabed7eb796637c02e3677ebbe183d90b258ba69
mzumsande:
ACK fabed7eb796637c02e3677ebbe183d90b258ba69
furszy:
ACK fabed7eb
Tree-SHA512: 66a878be58bbe53ad8e0c23f05569dd42df688be747551fbd202ada22d20a8285714e58fa2a71664deadb070ddf86cfad88c01042ff95ed26f6b40e4a10cec0a
bea9fc2600635020fd28ec7a6613c92a6f349a86 wallet: sqlite: force sqlite3.h to be included by the cpp files (Cory Fields)
Pull request description:
Only `#include` upstream sqlite headers from our cpp files.
Like #28039 but simpler :)
ACKs for top commit:
achow101:
ACK bea9fc2600635020fd28ec7a6613c92a6f349a86
TheCharlatan:
Nice, ACK bea9fc2600635020fd28ec7a6613c92a6f349a86
kristapsk:
utACK bea9fc2600635020fd28ec7a6613c92a6f349a86
hebasto:
ACK bea9fc2600635020fd28ec7a6613c92a6f349a86, I have reviewed the code and it looks OK.
Tree-SHA512: cb83ac51eed7e0740f1c75ee87c7849fa7e535bc4836c499290041eb995ccfd82533e3babfe83a164257b62b180f206112d6a1bae7ea290ad0ec7f55d62432da
6eb33bd0c21b3e075fbab596351cacafdc947472 kernel: Add fatalError method to notifications (TheCharlatan)
7320db96f8d2aeff0bc5dc67d8b7b37f5f808990 kernel: Add flushError method to notifications (TheCharlatan)
3fa9094b92c5d37f486b0f8265062d3456796a50 scripted-diff: Rename FatalError to FatalErrorf (TheCharlatan)
edb55e2777063dfeba0a52bbd0b92af8b4688501 kernel: Pass interrupt reference to chainman (TheCharlatan)
e2d680a32d757de0ef8eb836047a0daa1d82e3c4 util: Add SignalInterrupt class and use in shutdown.cpp (TheCharlatan)
Pull request description:
Get rid of all `ShutdownRequested` calls in validation code by introducing an interrupt object that applications can use to cancel long-running kernel operations.
Replace all `AbortNode` calls in validation code with new fatal error and flush error notifications so kernel applications can be notified about failures and choose how to handle them.
---
This pull request is part of the `libbitcoinkernel` project https://github.com/bitcoin/bitcoin/issues/27587https://github.com/orgs/bitcoin/projects/3 and more specifically its "Step 2: Decouple most non-consensus code from libbitcoinkernel".
The pull request mostly allows dropping the kernel dependency on shutdown.cpp. The only dependency left after this is a `StartShutdown` call which will be removed in followup PR https://github.com/bitcoin/bitcoin/pull/27711. This PR also drops the last reference to the `uiInterface` global in kernel code. The process of moving the `uiInterface` out of the kernel was started in https://github.com/bitcoin/bitcoin/pull/27636.
This pull request contains a subset of patches originally proposed in #27711. It will be part of a series of changes required to make handling of interrupts (or in other words the current shutdown procedure) in the kernel library more transparent and less reliable on global mutable state. The set of patches contained here was originally proposed by @ryanofsky [here](https://github.com/bitcoin/bitcoin/pull/27711#issuecomment-1580779869).
ACKs for top commit:
achow101:
light ACK 6eb33bd0c21b3e075fbab596351cacafdc947472
hebasto:
ACK 6eb33bd0c21b3e075fbab596351cacafdc947472, I have reviewed the code and it looks OK.
ryanofsky:
Code review ACK 6eb33bd0c21b3e075fbab596351cacafdc947472. No changes since last review other than rebase.
Tree-SHA512: 7d2d05fa4805428a09466d43c11ae32946cbb25aa5e741b1eec9cd142e4de4bb311e13ebf1bb125ae490c9d08274f2d56c93314e10f3d69e7fec7445e504987c
In `add_p2p_connection()` we connect to `bitcoind` from the Python
client and check that it has received our version string.
This check looked up the last/newest entry from `getpeerinfo` RPC,
assuming that it must be the connection we have just opened. But this
will not be the case if a new inbound or outbound connection is made
to/from `bitcoind` in the meantime.
Instead of the last entry in `getpeerinfo`, check all and find the one
which corresponds to our connection using our outgoing address:port
tuple which is unique.
Co-authored-by: MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz>
Co-authored-by: Jon Atack <jon@atack.com>
fa956d20480b21b0f348b83dd451667620010d8e ci: Print full lscpu output (MarcoFalke)
Pull request description:
Seems odd to withhold the other output, given that it may be useful to debug issues?
ACKs for top commit:
fanquake:
ACK fa956d20480b21b0f348b83dd451667620010d8e
Tree-SHA512: d93a79734a594c2ee180107e3ed1d1c07c1b6324b4b1e239d3f263e72490ca641f60a4e80793229523e2d52059958a896cb210014e3aa747a19871be62f5a961
8fbb6e99bfc85a1b9003cae402a7335843a86abd wallet: Give deprecation warning when loading a legacy wallet (Andrew Chow)
Pull request description:
Next step in legacy wallet deprecation.
ACKs for top commit:
S3RK:
reACK 8fbb6e99bfc85a1b9003cae402a7335843a86abd
jonatack:
re-ACK 8fbb6e99bfc85a1b9003cae402a7335843a86abd
Tree-SHA512: 902984b09452926cf199f06e5fb56e4985325cdd5e0dcc829992158488f42d5fbc33e9a30a29303feac24c8315193e8d31712022e2a0503abd6b67169a0027f4
fac14c4e498f2d38b8b337d4c145129d07403a6d ci: Remove deprecated container.greedy (MarcoFalke)
Pull request description:
The option is to be phased out, so remove it to avoid relying on it. Update container.cpu where needed.
ACKs for top commit:
hebasto:
ACK fac14c4e498f2d38b8b337d4c145129d07403a6d.
Tree-SHA512: 0440b710e607aaa2f78f811f9d5ae786a59af4a44861d7905a25c742ff7f5b4558518b939539455309503266757182a1c8fce92c8d3430983b2103a613fe01d7
99c0eb9701e71f16aa360a420b7e4851d5b92510 Fix RPCConsole wallet selection (John Moffett)
Pull request description:
If a user opens multiple wallets in the GUI from the menu bar, the last one opened is the active one in the main window. However, For the RPC Console window, the _first_ one opened is active. This can be confusing, as wallet RPC commands may be sent to a wallet the user didn't intend.
This PR makes the RPC Console switch to the wallet just opened / restored / created from the menu bar, which is how the main GUI now works.
Similar to https://github.com/bitcoin-core/gui/pull/665 and specifically requested [in a comment](https://github.com/bitcoin-core/gui/pull/665#issuecomment-1270003660).
ACKs for top commit:
luke-jr:
utACK 99c0eb9701e71f16aa360a420b7e4851d5b92510
hebasto:
ACK 99c0eb9701e71f16aa360a420b7e4851d5b92510, tested on Ubuntu 23.04.
Tree-SHA512: d5e5acdaa114130ad4d27fd3f25393bc8d02d92b5001cd39352601d04283cdad3bd62c4da6d369c69764e3b188e9cd3e83152c00b09bd42966082ad09037c328
a582b4141f0756faa3793fb1c772898a984c83e4 gui: send, left alignment for "bytes" and "change" label (furszy)
210ef1e980e0887c5675b66bcd5cbccd00aceec6 qt: remove confusing "Dust" label from coincontrol / sendcoins dialog (Sebastian Falbesoner)
Pull request description:
In contrast to to all other labels on the coin selection dialog, the displayed dust information has nothing to do with the selected coins. All that this label shows is whether at least one of the _outputs_ qualify as dust, but the outputs are set in a different dialog. (Even worse, the dust check is currently simply wrong because it only looks at an output's nValue and just assumes a P2PKH script size.)
As the label clearly doesn't help the user and is, quite the contrary, rather increasing confusion/misguidance, it seems sensible to remove it. The label from the sendcoins dialog is also removed with the same rationale. Additionally, the "bytes" and "change" labels are aligned to the left (second commit).
Closes https://github.com/bitcoin-core/gui/issues/699.
ACKs for top commit:
furszy:
ACK a582b41
hebasto:
Looks good. ACK a582b4141f0756faa3793fb1c772898a984c83e4.
Tree-SHA512: ebc00b68bdeab69f6ab643e4b89301a7e3d04a8a4027b50813314ddddb1387bc97a83313851e375dfbce97751c234686c82af7f4e55fa5ef29f4fed4e8fc11d9
5df988b534df842ddb658ad2a7ff0f12996c8478 test: add coverage for descriptor ID (furszy)
6a9510d2dabde1c9ee6c4226e3d16ca32eb48ac5 wallet: bugfix, always use apostrophe for spkm descriptor ID (furszy)
97a965d98f1582ea1b1377bd258c9088380e1b8b refactor: extract descriptor ID calculation from spkm GetID() (furszy)
1d207e3931cf076f69d4a8335cdd6c8ebb2a963f wallet: do not allow loading descriptor with an invalid ID (furszy)
Pull request description:
Aiming to fix#27915.
As we re-write the descriptor's db record every time that
the wallet is loaded (at `TopUp` time), if the spkm ID differs
from the one in db, the wallet will enter in an unrecoverable
corruption state (due to the storage of a descriptor with an ID
that is not linked to any other descriptor record in DB), and
no soft version will be able to open it anymore.
Because we cannot change the past, to stay compatible between
releases, we need to always use the apostrophe version for the
spkm IDs.
ACKs for top commit:
achow101:
ACK 5df988b534df842ddb658ad2a7ff0f12996c8478
Sjors:
tACK 5df988b534df842ddb658ad2a7ff0f12996c8478
Tree-SHA512: f63fc4aac7d21a4e515657471758d28857575e751865bfa359298f8b89b2568970029ca487a873c1786a5716325f453f06cd417ed193f3366417f6e8c2987332
If a user opens multiple wallets in the GUI from the
menu bar, the last one opened is the active one in
the main window. However, For the RPC Console window,
the _first_ one opened is active. This can be
confusing, as wallet RPC commands may be sent to a
wallet the user didn't intend.
This commit makes the RPC Console switch to the wallet
opened from the menu bar.
In contrast to to all other labels on the coin selection dialog, the
displayed dust information has nothing to do with the selected coins.
All that this label shows is whether at least one of the _outputs_
qualify as dust, but the outputs are set in a different dialog.
(Even worse, the dust check is currently simply wrong because it only
looks at an output's nValue and just assumes a P2PKH script size.)
As the label clearly doesn't help the user and is, quite the contrary,
rather increasing confusion/misguidance, it seems sensible to remove it.
Also, remove the label from the sendcoins dialog with the same rationale.
There are several instances in functional tests and the framework
(MiniWallet, feature_block.py, p2p_segwit.py) where we create a legacy
ECDSA signature for a certain transaction's input by doing the following
steps:
1) calculate the `LegacySignatureHash` with the desired sighash type
2) create the actual digital signature by calling `ECKey.sign_ecdsa`
on the signature message hash calculated above
3) put the DER-encoded result as CScript data push into
tx input's scriptSig
Create a new helper `sign_input_legacy` which hides those details and
takes only the necessary parameters (tx, input index, relevant
scriptPubKey, private key, sighash type [SIGHASH_ALL by default]). For
further convenience, the signature is prepended to already existing
data-pushes in scriptSig, in order to avoid rehashing the transaction
after calling the new signing function.
