95a0104f2e9869799db84add108ae8c57b56d360 test: Add tests for directories in place of config files (Hodlinator)
e85abe92c7cc5380489c028479f0d42f91827efd args: Catch directories in place of config files (Hodlinator)
e4b6b1822ce004365be11c54c8f5f02f95303fb0 test: Add tests for -noconf (Hodlinator)
483f0dacc413f4b1ba1b74c2429c4367b87e7f11 args: Properly support -noconf (Hodlinator)
312ec64cc0619f58c6e8abc5855fd2fa0e920c3f test refactor: feature_config_args.py - Stop nodes at the end of tests, not at the beginning (Hodlinator)
7402658bc2b9d835b240edb9c1dac308859687c3 test: -norpccookiefile (Hodlinator)
39cbd4f37c3d3a32cd993cbc78052d53f700989b args: Support -norpccookiefile for bitcoind and bitcoin-cli (Hodlinator)
e82ad88452bce4132e4583727e610d52dcf9ad9e logs: Use correct path and more appropriate macros in cookie-related code (Hodlinator)
6e28c76907ca0012b1a4556d9a982dfffb5abaf6 test: Harden testing of cookie file existence (Hodlinator)
75bacabb55f3d54ad9b2c660cafc82c167e4f644 test: combine_logs.py - Output debug.log paths on error (Hodlinator)
bffd92f00f5bfbb5a622d0bd20bfeed9f8b10928 args: Support -nopid (Hodlinator)
12f8d848fd91b11d5cffe21dfc3ba124102ee236 args: Disallow -nodatadir (Hodlinator)
6ff9662760099c405cf13153dd1de22900045f5e scripted-diff: Avoid printing version information for -noversion (Hodlinator)
e8a2054edc814b2c4661b96a3dce91da9be68fa4 doc args: Document narrow scope of -color (Hodlinator)
Pull request description:
- Document `-color` as only applying to `-getinfo`, to be less confusing for bitcoin-cli users.
- No longer print version information when getting passed `-noversion`.
- Disallow `-nodatadir` as we cannot run without one. It was previously interpreted as a mix of unset and as a relative path of "0".
- Support `-norpccookiefile`
- Support `-nopid`
- Properly support `-noconf` (instead of working by accident). Also detect when directories are specified instead of files.
Prompted by investigation in https://github.com/bitcoin/bitcoin/pull/16545#pullrequestreview-2316714013.
ACKs for top commit:
l0rinc:
utACK 95a0104f2e9869799db84add108ae8c57b56d360
achow101:
ACK 95a0104f2e9869799db84add108ae8c57b56d360
ryanofsky:
Code review ACK 95a0104f2e9869799db84add108ae8c57b56d360. Looks good! Thanks for all your work on this breaking the changes down and making them simple.
Tree-SHA512: 5174251e6b9196a9c6d135eddcb94130295c551bcfccc78e633d9e118ff91523b1be0d72828fb49603ceae312e6e1f8ee2651c6a2b9e0f195603a73a9a622785
fa3e074304780549b1e7972217930e34fa55f59a refactor: Tidy fixups (MarcoFalke)
fa72646f2b197810a324cb0544d9a1fac37d3f9c move-only: Detail_CheckNumFormatSpecifiers and G_TRANSLATION_FUN (MarcoFalke)
faff8403f0aac3b5ec26d3c7fc98240f879f9906 refactor: Pick translated string after format (MarcoFalke)
Pull request description:
The changes are required for https://github.com/bitcoin/bitcoin/pull/31061, however they also make sense on their own. For example, they are fixing up an `inline namespace`, which lead to compile errors otherwise (can be tested by observing the compile error after reverting the changes to `src/util/strencodings.h`). Also, a unit test comment is fixed.
ACKs for top commit:
ryanofsky:
Code review ACK fa3e074304780549b1e7972217930e34fa55f59a. Nice changes! These should allow related PRs to be simpler.
l0rinc:
ACK fa3e074304780549b1e7972217930e34fa55f59a
hodlinator:
cr-ACK fa3e074304780549b1e7972217930e34fa55f59a
Tree-SHA512: 37371181a348610442186b5fbb7a6032d0caf70aae566002ad60be329a3131a2b89f28f6c51e10872079f987986925dc8c0611bde639057bee4f572d2b9ba92a
faf70cc9941ce2b0ce4fd48ecfdbe28194adb8ba Remove wallet::ParseISO8601DateTime, use ParseISO8601DateTime instead (MarcoFalke)
2222aecd5f8059785e655da7b7e3fcc59204245c util: Implement ParseISO8601DateTime based on C++20 (MarcoFalke)
Pull request description:
`boost::posix_time` in `ParseISO8601DateTime` has many issues:
* It parses random strings that are clearly invalid and returns a time value for them, see [1] below.
* None of the separators `-`, or `:`, or `T`, or `Z` are validated.
* It may crash when running under a hardened C++ library, see https://github.com/bitcoin/bitcoin/issues/28917.
* It has been unmaintained for years, so reporting or fixing any issues will most likely be useless.
* It pulls in a third-party dependency, when the functionality is already included in vanilla C++20.
Fix all issues by replacing it with a simple helper function written in C++20.
Fixes https://github.com/bitcoin/bitcoin/issues/28917.
[1] The following patch passes on current master:
```diff
diff --git a/src/wallet/test/rpc_util_tests.cpp b/src/wallet/test/rpc_util_tests.cpp
index 32f6f5ab46..c1c94c7116 100644
--- a/src/wallet/test/rpc_util_tests.cpp
+++ b/src/wallet/test/rpc_util_tests.cpp
@@ -12,6 +12,14 @@ BOOST_AUTO_TEST_SUITE(wallet_util_tests)
BOOST_AUTO_TEST_CASE(util_ParseISO8601DateTime)
{
+ BOOST_CHECK_EQUAL(ParseISO8601DateTime("964296"), 242118028800);
+ BOOST_CHECK_EQUAL(ParseISO8601DateTime("244622"), 15023836800);
+ BOOST_CHECK_EQUAL(ParseISO8601DateTime("+INfINITy"), 9223372036854);
+ BOOST_CHECK_EQUAL(ParseISO8601DateTime("7000802 01"), 158734166400);
+ BOOST_CHECK_EQUAL(ParseISO8601DateTime("7469-2 +INfINITy"), 9223372036854);
+ BOOST_CHECK_EQUAL(ParseISO8601DateTime("maXimum-datE-time"), 253402300799);
+ BOOST_CHECK_EQUAL(ParseISO8601DateTime("577737 114maXimum-datE-time"), 253402300799);
+
BOOST_CHECK_EQUAL(ParseISO8601DateTime("1970-01-01T00:00:00Z"), 0);
BOOST_CHECK_EQUAL(ParseISO8601DateTime("1960-01-01T00:00:00Z"), 0);
BOOST_CHECK_EQUAL(ParseISO8601DateTime("2000-01-01T00:00:01Z"), 946684801);
```
ACKs for top commit:
hebasto:
ACK faf70cc9941ce2b0ce4fd48ecfdbe28194adb8ba, I have reviewed the code and it looks OK.
dergoegge:
utACK faf70cc9941ce2b0ce4fd48ecfdbe28194adb8ba
Tree-SHA512: 9dd745a356d04acf6200e13a6af52c51a9e2a0eeccea110093ce5da147b3c669c0eda918e46db0164c081a78c8feae3fe557a4759bea18449a8ff2d090095931
8f85d36d68ab33ba237407a2ed16667eb149d61f refactor: Clamp worker threads in ChainstateManager constructor (TheCharlatan)
Pull request description:
This ensures the options are applied consistently from contexts where they might not pass through the args manager, such as in some tests, or when used through the kernel library.
This is similar to the patch applied in 09ef322acc0a88a9e119f74923399598984c68f6, used to make applying the mempool options consistent.
---
This is part of the libbitcoinkernel project https://github.com/bitcoin/bitcoin/issues/27587
ACKs for top commit:
maflcko:
ACK 8f85d36d68ab33ba237407a2ed16667eb149d61f 🛳
achow101:
ACK 8f85d36d68ab33ba237407a2ed16667eb149d61f
furszy:
Code ACK 8f85d36d68ab33ba237407a2ed16667eb149d61f
stickies-v:
ACK 8f85d36d68ab33ba237407a2ed16667eb149d61f
Tree-SHA512: 32d7cc177d6726ee9df62ac9eb43e49ba676f35bfcff47834bd97a1e33f2a9ea7be65d0a8a37be149de04e58c9c500ecef730e498f4e3909042324d3136160e9
32fc59796f74a2941772b5ec2755b1319132cd9c rpc: Allow single transaction through submitpackage (glozow)
Pull request description:
There's no particular reason to restrict single transaction submissions with submitpackage. This change relaxes the RPC checks as enables the `AcceptPackage` flow to accept packages of a single transaction.
Resolves#31085
ACKs for top commit:
naumenkogs:
ACK 32fc59796f
achow101:
ACK 32fc59796f74a2941772b5ec2755b1319132cd9c
glozow:
ACK 32fc59796f74a2941772b5ec2755b1319132cd9c
Tree-SHA512: ffed353bfdca610ffcfd53b40b76da05ffc26df6bac4b0421492e067bede930380e03399d2e2d1d17f0e88fb91cd8eb376e3aabebbabcc724590bf068d09807c
73db95c65c1d372822166045ca8b9f173d5fd883 kernel: Make bitcoin-chainstate's block validation mirror submitblock's (TheCharlatan)
bb53ce9bdae2f02d7bd95cf5d8ca4ccf5136466a tests: Add functional test for submitting a previously pruned block (Greg Sanders)
1f7fc738255205a64374686aca9a4c53089360f1 rpc: Remove submitblock duplicate pre-check (TheCharlatan)
e62a8abd7df21795dcd173773f689b6d4c8feab6 rpc: Remove submitblock invalid-duplicate precheck (TheCharlatan)
36dbebafb9b54764005e6fffa7ad28d4cadfe5e4 rpc: Remove submitblock coinbase pre-check (TheCharlatan)
Pull request description:
With the introduction of a mining ipc interface and the potential future introduction of a kernel library API it becomes increasingly important to offer common behaviour between them. An example of this is ProcessNewBlock, which is used by ipc, rpc, net_processing and (potentially) the kernel library. Having divergent behaviour on suggested pre-checks and checks for these functions is confusing to both developers and users and is a maintenance burden.
The rpc interface for ProcessNewBlock (submitblock) currently pre-checks if the block has a coinbase transaction and whether it has been processed before. While the current example binary for how to use the kernel library, bitcoin-chainstate, imitates these checks, the other interfaces do not.
The coinbase check is repeated again early during ProcessNewBlock. Pre-checking it may also shadow more fundamental problems with a block. In most cases the block header is checked first, before validating the transactions. Checking the coinbase first therefore masks potential issues with the header. Fix this by removing the pre-check.
Similary the duplicate checks are repeated early in the contextual checks of ProcessNewBlock. If duplicate blocks are detected much of their validation is skipped. Depending on the constitution of the block, validating the merkle root of the block is part of the more intensive workload when validating a block. This could be an argument for moving the pre-checks into block processing. In net_processing this would have a smaller effect however, since the block mutation check, which also validates the merkle root, is done before.
Testing spamming a node with valid, but duplicate unrequested blocks seems to exhaust a CPU thread, but does not seem to significantly impact keeping up with the tip. The benefits of adding these checks to net_processing are questionable, especially since there are other ways to trigger the more CPU-intensive checks without submitting a duplicate block. Since these DOS concerns apply even less to the RPC interface, which does not have banning mechanics built in, remove them too.
Finally, also remove the pre-checks from `bitcoin-chainstate.cpp`.
---
This PR is part of the [libbitcoinkernel project](https://github.com/bitcoin/bitcoin/issues/27587).
ACKs for top commit:
Sjors:
re-utACK 73db95c65c1d372822166045ca8b9f173d5fd883
achow101:
ACK 73db95c65c1d372822166045ca8b9f173d5fd883
instagibbs:
ACK 73db95c65c1d372822166045ca8b9f173d5fd883
mzumsande:
ACK 73db95c65c1d372822166045ca8b9f173d5fd883
Tree-SHA512: 2d02e851cf402ecf6a1968c058df3576aac407e200cbf922a1a6391b7f97b4f42c6d9f6b0a78b9d1af0a6d40bdd529a7b11a1e6d88885bd7b8b090f6d1411861
492e1f09943fcb6145c21d470299305a19e17d8b [validation] merge all ConnectBlock debug logging code paths (Pieter Wuille)
b49df703f031894b41ec1a624137e5d3206940b5 [validation] include all logged information in BlockValidationState (Pieter Wuille)
7b267c034fdc2c8cb964a763f182ff98a75ba2ac [validation] Add detailed txin/txout information for script error messages (Pieter Wuille)
146a3d542683ddb89a31104f5211dfc757d1dafb [validation] Make script error messages uniform for parallel/single validation (Pieter Wuille)
1ac1c33f3f120bbe0bde4fa948299bc07cac47ee [checkqueue] support user-defined return type through std::optional (Pieter Wuille)
Pull request description:
~~Builds on top of #31097~~ (now merged). Fixes#30960.
So far, detailed information about script validation failures is only reported when running with `-par=1`, due to a lack of ability to transfer information from the script validation threads to the validation thread. Fix this by extending the `CCheckQueue` functionality to pass more results through than just success/failure, and use this to report the exact Script error, as well as the transaction input in which it occurred.
ACKs for top commit:
achow101:
ACK 492e1f09943fcb6145c21d470299305a19e17d8b
furszy:
Code review ACK 492e1f0
maflcko:
re-ACK 492e1f09943fcb6145c21d470299305a19e17d8b 🍈
dergoegge:
ACK 492e1f09943fcb6145c21d470299305a19e17d8b
instagibbs:
ACK 492e1f09943fcb6145c21d470299305a19e17d8b
mzumsande:
Code Review ACK 492e1f09943fcb6145c21d470299305a19e17d8b
Tree-SHA512: 234f2e7dfd03bdcd2a56200875fe370962f211ea7ed334038a6a9279a758030bf94bb6246f60d06dd0473dac4b9dbf050d9a32ecaa4176f7727eff63572bf4fd
fa5e7064597fc51366f082e3d07a4591e576db38 ci: Skip broken Wine64 tests by default (MarcoFalke)
Pull request description:
I don't think the unit tests run in Wine after the Windows cross-compilation have ever shown a true positive since the MSVC task was added. However, they are a source of frequent false-positives.
Thus, disable them by default for now. Anyone can still enable them by setting `RUN_UNIT_TESTS=true`.
A follow-up could run them on real Windows, see https://github.com/bitcoin/bitcoin/pull/31176.
Conceptually there are many other nightly tasks, which rarely find issues and are not run by default, like the valgrind or s390x tasks. So putting the Wine unit tests in the same bucket should be fine.
ACKs for top commit:
hebasto:
ACK fa5e7064597fc51366f082e3d07a4591e576db38, to avoid false-positives.
willcl-ark:
ACK fa5e7064597fc51366f082e3d07a4591e576db38
Tree-SHA512: 6bd54470e4d5ce18923c5d724aba0dbf475d053d7097d3f87e822a455cc537b6ce5f0dfcc8ccd0719c12c5d0c8fc7355a6c84185a6c9b5d484d98aee763d0c49
-noconf would previously lead to an ifstream "successfully" being opened to the ".bitcoin"-directory (not a file). (Guards against the general case of directories as configs are added in grandchild commit to this one).
Other users of AbsPathForConfigVal() in combination with negated args have been updated earlier in this PR ("args: Support -nopid" and "args: Support -norpccookiefile...").
This ensures we don't needlessly start the node, and reduces implicit dependencies between test functions.
test_seed_peers() - Move assert calling RPC to verify correct chain after our own function actually started the node.
This makes the debug output mostly the same for -par=1 and parallel validation runs. Of course,
parallel validation is non-deterministic in what error it may encounter first if there are
multiple issues. Also, the way certain script-related and non-script-related checks are
performed differs between the two modes still, which may result in discrepancies.
The check type function now needs to return a std::optional<R> for some type R,
and the check queue overall will return std::nullopt if all individual checks
return that, or one of the non-nullopt values if there is at least one.
For most tests, we use R=int, but for the actual validation code, we make it return
the ScriptError.
409d0d629378c3e23388ed31516376ad1ae536b5 test: enable running individual independent functional test methods (ismaelsadeeq)
Pull request description:
- Some test methods in the functional test framework are independent and do not require any prior context or setup in `run_test`.
- This commit adds a new option for running these specific methods within a test file, allowing them to be executed individually without running the entire test suite.
- Using this option reduces the time you need to wait before the test you are interested in starts executing.
- The functionality added by this PR can be achieved manually by commenting out code, but having a pragmatic option to do this is more convenient.
Note: Running test methods that require arguments or context will fail.
**Example Usage**:
```zsh
build/test/functional/feature_reindex.py --test_methods continue_reindex_after_shutdown
```
```zsh
build/test/functional/feature_config_args.py --test_methods test_log_buffer test_args_log test_connect_with_seednode
```
ACKs for top commit:
maflcko:
review ACK 409d0d629378c3e23388ed31516376ad1ae536b5
rkrux:
reACK 409d0d629378c3e23388ed31516376ad1ae536b5
ryanofsky:
Code review ACK 409d0d629378c3e23388ed31516376ad1ae536b5. This seems like a good step towards making it easy to run independent tests quickly. I think ideally there would be some naming convention or @ annotation added to test methods that can run independently, so the test framework could provide more functionality like being able to list test methods, being able to show command lines to quickly reproduce problems when tests fails, and calling test methods automatically instead of requiring individual tests to call them. But these ideas are all compatible with the new `--test_methods` option
Tree-SHA512: b0daac7c3b322e6fd9b946962335d8279e8cb004ff76f502c8d597b9c4b0073840945be198a79d44c5aaa64bda421429829d5c84ceeb8c6139eb6ed079a35878
19f49c7489d226e1cebc754fbbae3e4bebc360af doc: Use more precise anchor link to codesigning docs (Jeremy Rand)
Pull request description:
The "Codesigning" section is what users presumably are looking for when they follow this link.
ACKs for top commit:
fanquake:
ACK 19f49c7489d226e1cebc754fbbae3e4bebc360af
Tree-SHA512: 0e25cf0d7160db7d564d67d3e3ac614f9bd209b2399414f1278fa01cfc1ff827aa8311f7c1c2666924d5ac2dc23fe9bc258b80ed8025d5b8d5b11bcf1d12b28c
62f6d9e1a48e3b63c504996e914075cacfdcaedc test: simple ordering optimization to reduce runtime (tdb3)
Pull request description:
Noticed in #31371 that the position of `mempool_ephemeral_dust` within `BASE_SCRIPTS` was lengthening total test runtime. Instead of moving only that test, looked for others to move to reduce runtime.
This is a quick optimization that was found to reduce overall functional test runtime of up to around 20% (depending on jobs and machine characteristics). Since it seems like test ordering could be done in many different ways, with many variables, and bike shedding could creep in, a relatively straightforward approach was taken for now that minimized changes to test_runner.
ACKs for top commit:
maflcko:
lgtm ACK 62f6d9e1a48e3b63c504996e914075cacfdcaedc
TheCharlatan:
ACK 62f6d9e1a48e3b63c504996e914075cacfdcaedc
Tree-SHA512: 6f93fbe4de3fce202383d9f84aa0e96961af3de3c02b8cab73589339d701f32c5e1b57a191eeebf4b06b5cd7a82617f63f24110732940be1a5a4d9237813a570
8bf1b3039cb5b396e7e6d3ac075656952edd56d5 doc: Use more precise anchor links to Xcode SDK extraction (Jeremy Rand)
Pull request description:
The "SDK Extraction" section is what users presumably are looking for when they follow these links.
ACKs for top commit:
fanquake:
ACK 8bf1b3039cb5b396e7e6d3ac075656952edd56d5
Tree-SHA512: 38669a6b171aa102bb80f5b3a343bd6a067c6921c454f6d18087c5add8016eea2ba8196036f9968f0a9b7df1f642c96ff6c657338c32e775beb04038497cde1f
faa16ed4b9edd126b5a2b0c13994f18273096efc test: Add missing node.setmocktime(self.mocktime) to p2p_ibd_stalling.py (MarcoFalke)
Pull request description:
This was forgotten by myself in commit fa5b58ea01fac1adb6336b8b6b5217193295c695.
This time, there is a diff to test, which fails on current master and passes with this pull request.
```diff
diff --git a/src/net_processing.cpp b/src/net_processing.cpp
index e503a68382..16438ebd08 100644
--- a/src/net_processing.cpp
+++ b/src/net_processing.cpp
@@ -112,9 +112,9 @@ static_assert(MAX_BLOCKTXN_DEPTH <= MIN_BLOCKS_TO_KEEP, "MAX_BLOCKTXN_DEPTH too
* want to make this a per-peer adaptive value at some point. */
static const unsigned int BLOCK_DOWNLOAD_WINDOW = 1024;
/** Block download timeout base, expressed in multiples of the block interval (i.e. 10 min) */
-static constexpr double BLOCK_DOWNLOAD_TIMEOUT_BASE = 1;
+static constexpr double BLOCK_DOWNLOAD_TIMEOUT_BASE = .05; // 30 sec
/** Additional block download timeout per parallel downloading peer (i.e. 5 min) */
-static constexpr double BLOCK_DOWNLOAD_TIMEOUT_PER_PEER = 0.5;
+static constexpr double BLOCK_DOWNLOAD_TIMEOUT_PER_PEER = 0.;
/** Maximum number of headers to announce when relaying blocks with headers message.*/
static const unsigned int MAX_BLOCKS_TO_ANNOUNCE = 8;
/** Minimum blocks required to signal NODE_NETWORK_LIMITED */
diff --git a/test/functional/p2p_ibd_stalling.py b/test/functional/p2p_ibd_stalling.py
index fa07873929..f8cdd8998c 100755
--- a/test/functional/p2p_ibd_stalling.py
+++ b/test/functional/p2p_ibd_stalling.py
@@ -82,6 +82,7 @@ class P2PIBDStallingTest(BitcoinTestFramework):
# Need to wait until 1023 blocks are received - the magic total bytes number is a workaround in lack of an rpc
# returning the number of downloaded (but not connected) blocks.
bytes_recv = 172761 if not self.options.v2transport else 169692
+ time.sleep(31);
self.wait_until(lambda: self.total_bytes_recv_for_blocks() == bytes_recv)
self.all_sync_send_with_ping(peers)
ACKs for top commit:
brunoerg:
ACK faa16ed4b9edd126b5a2b0c13994f18273096efc
Tree-SHA512: 5a670e2dcf828ac83b721a3e20d897744cca50080b0583a8460a0d0c7bf2c2c988cf7e35f688dde6a3349f1c21cc83a16ea5242ed06a59d59a04130416690737
ab5c63edccea7957e2c7b3b9d8b04655a7f82f22 cmake: Build `secp256k1` only when required (Hennadii Stepanov)
76a3a540a4c3ff42f84c4701cb2ca6e910a24f50 cmake: Ensure script correctness when no targets are specified (Hennadii Stepanov)
Pull request description:
When no build targets are specified, it is reasonable to expect the configuration step to succeed and produce a build system that does not build any targets.
This PR updates the code to ensure this behaviour:
```
$ cmake -B build -G "Ninja" -DBUILD_DAEMON=OFF -DBUILD_CLI=OFF -DBUILD_TX=OFF -DBUILD_UTIL=OFF -DENABLE_WALLET=OFF -DBUILD_TESTS=OFF
$ cmake --build build
ninja: no work to do.
```
ACKs for top commit:
TheCharlatan:
ACK ab5c63edccea7957e2c7b3b9d8b04655a7f82f22
tdb3:
light test ACK ab5c63edccea7957e2c7b3b9d8b04655a7f82f22
Tree-SHA512: 1b13f406c58b02768d9ba831413aeae1d7e03659e7101de8e598f906ba220f479ac06707965c96a14468ce4ba49011a1ab9adee9cee34ab1e8622f690b94dad8
16b140f225a24b9075ef6e4cf6f0788bf96ace5f doc: correct libfuzzer-nosan preset flag (Niklas Gögge)
Pull request description:
`--prefix` is not the correct option for using a preset (it's not an option at all).
ACKs for top commit:
maflcko:
lgtm ACK 16b140f225a24b9075ef6e4cf6f0788bf96ace5f
Tree-SHA512: 8c5fad4f8573bd9ef972447b2847ede61a3b6af9650a599f6ff7e90a2c009e4422715164261b424c08170c9e179cce241a3ca31ddc234f446316f24fc2c353b1
b73d3319377a4c9d7e2dd279c3d106002585bc36 dbwrapper: Bump max file size to 32 MiB (Maciej S. Szmigiero)
Pull request description:
The default max file size for LevelDB is 2 MiB, which results in the LevelDB compaction code generating ~4 disk cache flushes per second when syncing with the Bitcoin network.
These disk cache flushes are triggered by `fdatasync()` syscall issued by the LevelDB compaction code when reaching the max file size.
If the database is on a HDD this flush rate brings the whole system to a crawl.
It also results in very slow throughput since 2 MiB * 4 flushes per second is about 8 MiB / second max throughput, while even an old HDD can pull 100 - 200 MiB / second streaming throughput.
Increase the max file size for LevelDB to 128 MiB instead so the flush rate drops to about 1 flush / 2 seconds and the system no longer gets so sluggish.
The max file size value chosen also matches the `MAX_BLOCKFILE_SIZE` file size setting already used by the block storage.
ACKs for top commit:
l0rinc:
ACK b73d3319377a4c9d7e2dd279c3d106002585bc36
davidgumberg:
ACK b73d331937
andrewtoth:
ACK b73d3319377a4c9d7e2dd279c3d106002585bc36
TheCharlatan:
ACK b73d3319377a4c9d7e2dd279c3d106002585bc36
willcl-ark:
ACK b73d3319377a4c9d7e2dd279c3d106002585bc36
tdb3:
ACK b73d3319377a4c9d7e2dd279c3d106002585bc36
laanwj:
ACK b73d3319377a4c9d7e2dd279c3d106002585bc36
Tree-SHA512: 5d8fb9ad1ea643fb3e42a9c59f6fc90cc5cc3b82c06d9b8d59de3a5a926fabaeb78efb51b608b1e7925f49d82dfcbd5b72c552993879789f33201efe57c278f3
935973b315fa3b99f5e79b360bcb66f473bb7b95 Remove `src/config` directory (Hennadii Stepanov)
Pull request description:
The `src/config` directory has not been used since the migration to CMake, which disables in-source builds.
ACKs for top commit:
TheCharlatan:
ACK 935973b315fa3b99f5e79b360bcb66f473bb7b95
BrandonOdiwuor:
ACK 935973b315fa3b99f5e79b360bcb66f473bb7b95
Tree-SHA512: cc5b405e39387673fa2fd1e96680295b6eb3dc49a5f9a4d288580b8ea83efba04c96132811ca2cec14bcca509dbaf20c390cd47dabeea2a6ebc973e364e7a43b
The default max file size for LevelDB is 2 MiB, which results in the
LevelDB compaction code generating ~4 disk cache flushes per second when
syncing with the Bitcoin network.
These disk cache flushes are triggered by fdatasync() syscall issued by the
LevelDB compaction code when reaching the max file size.
If the database is on a HDD this flush rate brings the whole system to a
crawl.
It also results in very slow throughput since 2 MiB * 4 flushes per second
is about 8 MiB / second max throughput, while even an old HDD can pull
100 - 200 MiB / second streaming throughput.
Increase the max file size for LevelDB to 32 MiB instead so the flush rate
drops significantly and the system no longer gets so sluggish.
The new max file size value chosen is a compromise between the one that
works best for HDD and SSD performance, as determined by benchmarks done by
various people.
160799d9135528dbdea40690f0bb0d56c6c4803a test: refactor: introduce `create_ephemeral_dust_package` helper (Sebastian Falbesoner)
61e18dec306cfb8bc17ad2133ea1867b78000c62 doc: ephemeral policy: add missing closing double quote (Sebastian Falbesoner)
Pull request description:
This small PR contains ephemeral dust follow-ups mentioned in #30329 that were not tackled in the first follow-up PR #31279:
https://github.com/bitcoin/bitcoin/pull/30239#discussion_r1828577696https://github.com/bitcoin/bitcoin/pull/30239#discussion_r1825279952
Happy to add more if I missed some or anyone has concrete commits to add.
ACKs for top commit:
rkrux:
tACK 160799d9135528dbdea40690f0bb0d56c6c4803a
instagibbs:
ACK 160799d9135528dbdea40690f0bb0d56c6c4803a
tdb3:
Code review ACK 160799d9135528dbdea40690f0bb0d56c6c4803a
Tree-SHA512: e9a80c6733f1e7fe9e834d81b404f6e8ef7a61fe986f61b3dcdbda1a0bc547145fc279ec02f54361df56cb4e62a6fedaa0f3991c6e084c3a703ed1b1bfbdbe4e
ee6185372fc317d3948690997117e42f6b79a5ff gen-manpages: Prompt error if no binaries are found (Andre)
299e2220e9546e7dca1ba5e56bdcfbbf97530fba gen-manpages: implement --skip-missing-binaries (Andre Alves)
Pull request description:
Instead of stopping the execution of gen-manpages.py when a binary is not found, continue generating manpages for the available binaries and skip the missing ones.
A new argument, `--skip-missing-binaries`, has been added to enable this behavior.
```sh
➜ bitcoin git:(fix-gen-manpages) ✗ ./contrib/devtools/gen-manpages.py --help
usage: gen-manpages.py [-h] [-s]
options:
-h, --help show this help message and exit
-s, --skip-missing-binaries
skip generation for binaries that are not found
```
closes#30985
This PR also includes an error prompt if no binaries are found in the build path.
ACKs for top commit:
achow101:
ACK ee6185372fc317d3948690997117e42f6b79a5ff
laanwj:
re-ACK ee6185372fc317d3948690997117e42f6b79a5ff
Tree-SHA512: af4a0a5e26e508a51ab63f8aa9f98a6d6af9d7682a16791d8a6a61d49e44cb0147453f628ad5910f65d4efa6e3c7b6605c007259c23230b54888845bfaeb050c
