Test bits are conserved. This only has an effect on the warnings.
Co-Authored-By: Antoine Poinsot <mail@antoinep.com>
Github-Pull: #34779
Rebased-From: f802edf57c
By now, the vast majority of nodes in the network supports BIP324.
Even if the optimistic guess would turn out to be wrong for a given
node, we would just reconnect with v1.
This is better than making v1 connections with peers when both nodes support v2.
Github-Pull: #35766
Rebased-From: cf0f2aeae0
Serialization parameters should be embedded into the object being
serialized rather than passed as a separate argument. This works here
because only serialization is performed and no new object needs to be
constructed.
Github-Pull: #35679
Rebased-From: afab8d4225
The server isn't running out of memory when the private broadcast
transaction queue is full. Add a new RPC_LIMIT_EXCEEDED code that
can be used whenever a resource is bound and currently at capacity.
Github-Pull: #35678
Rebased-From: 82a02a2a22
This queue may be hundreds to thousands of items, each of
which are being revalidated at short interval. Allow the
node to be more responsive in this scenario.
Github-Pull: #35406
Rebased-From: cbf8c107c1
Add a belt-and-suspenders feature, limit the amount of
memory and cpu possible when unlucky or simply misconfigured.
The worst case limit is roughly 400kB * 10,000 = 4GB, regardless
of usage pattern.
Before this change, sheer volume of broadcasts, mismatches in
standardness rules, or simply fee mismatches may result in unbounded
growth of memory usage. As the feature may be expanded in
the future, explicit bounds helps reasoning going forward.
Github-Pull: #35406
Rebased-From: 5aea3d0373
adc5dc5206 doc: update manual pages for v31.1 (fanquake)
919d74e4c7 doc: update release notes for v31.1 (fanquake)
4219fd4693 build: bump version to v31.1 (fanquake)
Pull request description:
The [`v31.1rc1` bins](https://bitcoincore.org/bin/bitcoin-core-31.1/test.rc1/) have been up for nearly 2 weeks. There doesn't seem to be anything else that needs backporting, and no issues have been reported. Finalise `v31.1`.
ACKs for top commit:
hebasto:
ACK adc5dc5206.
sedited:
ACK adc5dc5206
Tree-SHA512: 4bf7e59a010a08952a60bbb318c514e411d85092086006134521d873c498d544e42cf6880d5f26f02ac1a6bcd0fc155528f2858a81ae8d1b11a3c378f3d11cf9
Full chainstate compaction can take minutes on large databases.
Move `CCoinsViewDB::CompactFull()` to a named `utxocompact` one-shot background thread so validation only schedules the work.
When validation selects compaction after a full flush, the chainstate was just written and another write is less likely to be needed immediately.
The coins view destructor waits for completion, and a mutex prevents compaction from using `m_db` while `ResizeCache()` replaces it.
Co-authored-by: Andrew Toth <andrewstoth@gmail.com>
Github-Pull: #35465
Rebased-From: 394e473d42
Full chainstate flushes are convenient maintenance points for long-term LevelDB cleanup because the chainstate was just written.
Randomize the trigger so nodes that flush near the same height do not compact together.
Add blocking chainstate compaction through `CCoinsViewDB::CompactFull()` and give each post-IBD full flush on the normal chainstate a 1/320 chance to start compaction.
With hourly flushes this averages roughly every two weeks and makes a six-month miss about one in a million.
This keeps the schedule stateless and leaves last-compaction height or timestamp bookkeeping out of chainstate metadata.
Co-authored-by: Andrew Toth <andrewstoth@gmail.com>
Github-Pull: #35465
Rebased-From: aa021b26f3
Add `CDBWrapper::GetProperty()` and expose it through `CCoinsViewDB::GetDBProperty()` so coins tests can inspect LevelDB runtime properties through the coins view.
Use it in a coins DB flush baseline that records the LevelDB layout after flushing while keeping readback coverage for the flushed coin and best block.
Co-authored-by: Andrew Toth <andrewstoth@gmail.com>
Github-Pull: #35465
Rebased-From: b10889d107
In WriteSettings(), verify that writing to the stream and closing it
succeeded before returning true. This prevents RenameOver() from replacing
a valid settings.json with a corrupted or zero-byte file when write limits
or a full disk are encountered.
Additionally, update the ReadSettings() parse failure message to mention
power loss, full disk, or storage error as possible causes.
Fixes#35373
Github-Pull: #35384
Rebased-From: 0654511e1b
Extract the part of `p2p_private_broadcast.py` that fills a given node's
addrman and put it into `test_framework/test_framework.py`.
Github-Pull: #35410
Rebased-From: ab35a028ed
Extract the part of `p2p_private_broadcast.py` that starts
listening on a `P2PConnection` object (or its children classes)
and put it into `test_framework/p2p.py`.
Github-Pull: #35410
Rebased-From: 2333be9cbc
Extract the part of `p2p_private_broadcast.py` that configures and
starts the SOCKS5 server into a reusable function and put it into
`test_framework/socks5.py`.
Use bind port 0 to let the OS pick an available port instead of
hackishly assuming that `p2p_port(N)` is available where N is more
than the number of the nodes the test uses.
Github-Pull: #35410
Rebased-From: 2ffa81fac4
Private broadcast connections use either Tor or I2P, which require a
proxy intrinsically or IPv4 or IPv6 which must use a proxy in the
context of private broadcast to avoid leaking the originator's IP
address.
Add a safety check to guard against future mistakes.
Co-authored-by: Andrew Toth <andrewstoth@gmail.com>
Github-Pull: #35410
Rebased-From: d01b461f71
`OpenNetworkConnection()` supports overriding the proxy to use for
connecting. However when v2 connection is attempted and it fails a v1
connection is tried without that proxy.
Store the override proxy in `CNode` and pass it to
`CConnman::m_reconnections` to be used for v1 retries.
Github-Pull: #35410
Rebased-From: fd230f942d
Speeds of 1MB/s and 15 minute cached docker image pulls during builds
are not uncommon.
Warp runners provide a local GitHub Actions cache protocol proxy for
Docker layer cache traffic. Point BuildKit's gha cache backend at that
proxy on Warp runners so cached image layers do not have to be fetched
from GitHub's slower cache service.
Add a default for provider so other users (e.g. qa-assets) don't have to
update this unless they use custome runners.
Github-Pull: #35447
Rebased-From: 82901981bf
The GHA cache is very slow, taking on the order of minutes to save and
restore from.
Use WarpBuild's cache instead as this is in the same region and much
faster.
WarpBuild cache action does not auto-fallback to GHA if not being run on
Warp. To allow fork runs to fallback to GHA caching, whilst minimising
duplication in the action files, create new "interal" actions which
perform the switching logic, and use these in the (renamed) cache|save
actions.
Without this we would need the `if` logic in our prvious actions, 4
times in each of save and restore.
Plumb the provider through into the action, as a composite action can't
read `env` (`GITHUB_OUTPUT`) from previous steps.
Github-Pull: #35430
Rebased-From: 2ce4ae7d8f
The existing Clang-only no_sanitize("address") guard is extended to
also cover GCC. When GCC compiles this file with -fsanitize=address
in debug builds, the instrumented inline assembly causes a SEGV during
SHA256AutoDetect()'s self-test on CPUs that use the SSE4 code path
(i.e. those without SHA-NI support), regardless of optimization level.
The original Clang code placed the attribute between the function
declarator and the opening brace. GCC's Attribute Syntax
documentation notes that this position in a function definition
"may, in future, be permitted," so it is not currently supported.
The attribute is moved to the start of the function definition,
which is valid form for both GCC and Clang.
The preprocessor guards are restructured so each compiler branch is
explicit: __clang__ with __has_feature, and __GNUC__ with
__SANITIZE_ADDRESS__.
Github-Pull: #34953
Rebased-From: fedeff7f20
Seek compaction is causing a cascade effect in the chainstate DB, causing large parts of the database to be rewritten every ~hour.
Every periodic flush writes around 2 MiB. Since this is roughly the `write_buffer_size`, these writes regularly cause the memtable to rotate into a small L0 file. This file has a small seek budget, and with the random UTXO reads done during validation, it can get scheduled for seek compaction quickly.
That seek compaction pushes the small file down to L1. Since most UTXOs are already lower down in L4/L5, many reads that consult this file do not find the key there and continue downward. The bloom filter makes those misses cheap, but LevelDB still decrements the file's seek budget. The file then gets scheduled for another seek compaction, and the same pattern pushes it down through L2 and L3.
The expensive part happens around L3/L4. L4 has many ~32 MiB files holding the bulk of the UTXO set. When LevelDB compacts into L3, it may split the output into many smaller L3 files to limit how much L4 "grandparent" data any one output overlaps. Each of these small L3 files then gets its own small seek budget. Because chainstate keys are hash-random, each small L3 file can still have a broad key range, so many random reads consult it and quickly drain its budget. Once seek-compacted into L4, each tiny L3 file can overlap many L4 files, so compacting a few hundred KiB from L3 can require rewriting hundreds of MiB from L4. Repeating that across many small L3 files can rewrite most of the chainstate.
This is a poor fit for chainstate because UTXO keys are hash-random, the DB is large enough to have many levels, writes are relatively small and periodic, and reads are frequent. The result is that read misses trigger compactions much earlier than size pressure would, and those compactions have very high write amplification.
Disabling seek compaction may leave more files in upper levels for longer, so reads could theoretically consult more files. But Bitcoin Core enables bloom filters for all its LevelDB instances, so these misses are usually cheap in-memory filter checks rather than disk reads.
For the other DBs, the risk is much smaller. They also use bloom filters, and most are smaller and less read-heavy. With fewer levels and less random read pressure, disabling seek compaction should have little effect there.
Co-authored-by: l0rinc <pap.lorinc@gmail.com>
Github-Pull: #35313
Rebased-From: 6bfdb6093bba4710d0f8313ed0113967a8b5176f
Cirrus is winding down and github now offers more than 10GB cache.
Switch to GH cache for all runner-types. Simplify docker build arg
construction, and reduce the number of needed action permissions.
Github-Pull: #35348
Rebased-From: c03107acf5
`CalculateMaximumSignedInputSize()` is passed the outpoint being sized, but that context was not used when estimating the signed input size.
Pass the outpoint through so externally selected inputs are not underestimated.
Co-authored-by: Antoine Poinsot <darosior@protonmail.com>
Github-Pull: #35228
Rebased-From: cd8d3bd937
Because the corresponding Taproot fields were added in PSBT in PR 22558, so
these restrictions are no longer necessary.
Github-Pull: #35279
Rebased-From: 81348576cc
The logging.warning call referenced `bci["bestblockhash"]`, a variable
from the calling scope `do_generate()` that is not available inside the
`Generate.gbt()` method. This would crash with a NameError when
getblocktemplate returned a template based on an unexpected previous
block.
Use the `bestblockhash` parameter that was already being passed in and
used correctly in the comparison on the line above.
The bug was introduced in 7b31332370 when the gbt logic was extracted
into its own method — the if-condition was updated but the logging
call was not.
Github-Pull: #35044
Rebased-From: 701bc2dc02
The FreeBSD, NetBSD, and OpenBSD build guides state that ZMQ support is compiled in when the package is installed. Since WITH_ZMQ defaults to OFF, update the wording to mention the required CMake option.
Github-Pull: #35283
Rebased-From: ca93ab808c
The BDB metadata field `last_page` stores the last valid page number, not the number of pages.
The read-only wallet migration parser currently checks reset LSNs with a half-open loop, so it skips the final page and may accept a database whose last page still depends on BDB log files.
Github-Pull: #35227
Rebased-From: e2b0984f99
