Regenerate mainnet seeds from new sources without the need for hardcoded
data. Result has 512 nodes from each network type except cjdns, for
which only eight nodes were found that match the seed node criteria.
Pull additional nodes from virtu's crawler. Data includes sufficient
Onion and I2P nodes to align the uptime requirements for these networks
to that of clearnet nodes (i.e., 50%). Data also includes more than
three times the number of CJDNS nodes currently hardcoded into
nodes_main_manual.txt, so hardcoded nodes becomes obsolete.
41ad84a00c20f54b520aab7f6f975231da0ee2d0 seeds: Use fjahr's more up to date asmap (Ava Chow)
d8fd1e0fafa144a9ff96fc646cf9f21e220d5cd6 seeds: Fixed seeds update (Ava Chow)
f1f24d72141dcd2955420195135cabe5092017ff seeds: Add testnet4 fixed seeds file (Ava Chow)
8ace71c73750e3b537784178f3fc299447c461ed seeds: Remove manual onion and i2p seeds (Ava Chow)
ed5b86cbe47676276f8ff1a48001d5ecd560e153 seeds: Add testnet instructions (Ava Chow)
0676515397fcc8fb580973047e60279ce65bec48 seeds: Also pull from achow101 seeder (Ava Chow)
5bab3175a663610070c1000dd4211a58490e5023 makeseeds: Configurable minimum blocks for testnet4's smaller chain (Ava Chow)
d2465dfac68f96ffdaad88a0bd4891ed37cbfdfc makeseeds: Shuffle ips after parsing (Ava Chow)
af550b3a0fd406f175f197ea9867b41ff4e97af4 makeseeds: Support CJDNS (Ava Chow)
d5a8c4c4bd76f296f4d744184dc80a6a6a0731bd makeseeds: Update user agent regex (Ava Chow)
Pull request description:
The [DNS seeder](https://github.com/achow101/dnsseedrs) that I wrote collects statistics on node reliability in the same way that sipa's seeder does, and also outputs this information in the same file format. Thus it can also be used in our fixed seeds update scripts. My seeder additionally crawls onion v3, i2p, and cjdns, so will now be able to set those fixed seeds automatically rather than curating manual lists.
In doing this update, I've found that `makeseeds.py` is missing newer versions from the regex as well as cjdns support; both of these have been updated.
I also noticed that the testnet fixed seeds are all manually curated and sipa's seeder does not appear to publish any testnet data. Since I am also running the seeder for testnet, I've added the commands to generate testnet fixed seeds from my seeder's data too.
Lastly, I've updated all of the fixed seeds. However, since my seeder has not found any cjdns nodes that met the reliability criteria (possibly due to connectivity issues present in those networks), I've left the previous manual seeds for that network.
ACKs for top commit:
fjahr:
re-ACK 41ad84a00c20f54b520aab7f6f975231da0ee2d0
virtu:
ACK [41ad84a](41ad84a00c)
Tree-SHA512: 6ba0141f053d9d6ae7d8c9574f061be38f3e65b28de1d6660c1885ab942623b5a0ec70754b4fcfc5d98fe970f5f179a940d5880b5061ed698f7932500e01d3ee
5b4f34006dbd76223b55b156421f87d2241ac296 devtools, utxo-snapshot: Fix block height out of range (pablomartin4btc)
Pull request description:
<details>
<summary>Fixing a <a href="https://github.com/bitcoin/bitcoin/pull/28553#pullrequestreview-2251032570">bug</a> in <code>utxo_snapshot.sh</code>.</summary>
```
/contrib/devtools/utxo_snapshot.sh 840000 snapshot2.dat ./src/bitcoin-cli -datadir=${AU_DATADIR}
Do you want to disable network activity (setnetworkactive false) before running invalidateblock? (Y/n):
Disabling network activity
false
error code: -8
error message:
Block height out of range
```
And the user will see the following in the node and it would stay there if not reset:
```
2024-08-21T14:44:13Z UpdateTip: new best=00000000000000afa0cd000a16e244f56032735d41acd32ac00337aceb2a5240 height=235382 version=0x00000002 log2_work=69.987697 tx=17492185 date='2013-05-09T23:54:32Z' progress=0.016219 cache=71.0MiB(571085txo)
2024-08-21T14:44:13Z UpdateTip: new best=0000000000000087c5e0b820afff496b95ba44ad64640c73b234d3261d3f99d2 height=235383 version=0x00000002 log2_work=69.987750 tx=17492341 date='2013-05-09T23:54:47Z' progress=0.016219 cache=71.0MiB(571291txo)
2024-08-21T14:44:13Z UpdateTip: new best=000000000000014a4b5fddf3c8abb6209247255ca9e8df786b271dd1b2ac82a6 height=235384 version=0x00000002 log2_work=69.987804 tx=17492344 date='2013-05-10T00:20:18Z' progress=0.016219 cache=71.0MiB(571297txo)
2024-08-21T14:44:13Z SetNetworkActive: false
```
</details>
This is a "temporary" fix until #29553 gets merged, which will remove the script entirely.
Handle the "Block height out of range" error gracefully by checking if the node has synchronized to or beyond the required block height, otherwise without this validation the node would keep the network disabled if the user selected that option.
<details>
<summary>Provide a user-friendly message if the block height is out of range and exit the script cleanly.</summary>
```
/contrib/devtools/utxo_snapshot.sh 840000 snapshot2.dat ./src/bitcoin-cli -datadir=${AU_DATADIR}
Error: The node has not yet synchronized to block height 840001.
Please wait until the node has synchronized past this block height and try again.
```
</details>
ACKs for top commit:
achow101:
ACK 5b4f34006dbd76223b55b156421f87d2241ac296
fjahr:
tACK 5b4f34006dbd76223b55b156421f87d2241ac296
Tree-SHA512: 2b71286b627872d7cfdb367e29361afa3806a7ef9d65075b93892b735ff2ab729069e2f7259d30262909e73cef17fb7dca231615cc1863968cd042f4a2a4f901
Handle the Block height out of range error gracefully by checking if
the node has synchronized to or beyond the required block height,
otherwise without this validation the node would keep the network
disabled if the user selected that option.
Provide a user-friendly message if the block height is out of range
and exit the script cleanly.
The seeders now produce onion and i2p seeds, so there is no need to keep these
in the manual list.
Although should also be produced, there are not enough
good ones detected by the seeder, so we keep the manual seeds for them.
The crawlers are not guaranteed to output nodes in a random order, so
shuffle the ips list after parsing to break any biasing that may be
caused by the output order.
Update the user agent regex to match all 3 digits of the version number,
not just the first 2 digits.
Also updates it to include 24.2, 25.2, 26.1, 27.0, 27.1, 27.99, 28.0 and
28.99.
5215c925d1382e71c9e1d642fced8a152c629c7f Compare ASMaps with respect to specific addresses (virtu)
Pull request description:
Right now, we have no way to quantify the "degradation" of an ASMap over time in the context of Bitcoin's P2P network in a meaningful way. However, such data would be useful for:
1. Making sure the minimum shelf life of ASMaps is compatible with the release cycle (we wouldn't want to start shipping ASMaps with releases before making sure ASMaps typically do not become obsolete before the time of the next release)
2. Node operators eager to keep their ASMaps up-to-date between releases.
While `asmap-tool.py` has a `diff` command to perform a prefix-based comparison of two ASMaps, it is hard to reason about whether an old ASMap still is "good enough" or should be replaced with a newer one based on a prefix-based diff such as the following:
```shell
$ ./asmap-tool.py diff 1704463200_asmap.dat 1710770400_asmap.dat
[...]
# 2c0f:fc98::/32 was AS37282
# 2c0f:fcb8::/32 was AS37323
2c0f:ff18::/32 AS37044 # was unassigned
2c0f:ff98::/32 AS37113 # was unassigned
2c0f:ffa0::/32 AS37273 # was unassigned
# 76082350 (2^26.18) IPv4 addresses changed; 834271985742505274886878979424260 (2^109.36) IPv6 addresses changed
```
One option for a more Bitcoin-centric ASMap comparison comprises comparing ASNs for the addresses of Bitcoin nodes and reporting on the number/share of addresses of nodes with disagreeing ASNs. By applying this approach to a node's set of known peers, a node operator can estimate how many of the node's peers are mapped to out-of-date AS when using the currently deployed and an up-to-date ASMap as input.
This PR adds this functionality to `asmap-tool.py` by introducing a `diff_addrs` subcommand. In addition to two ASMaps, the subcommand reads addresses from a (`getnodeaddresses`-compatible) file, and computes statistics for those addresses:
```bash
$ ./asmap-tool.py diff_addrs 1704463200_asmap.dat 1710770400_asmap.dat <(bitcoin-cli getnodeaddresses 0)
275 address(es) reassigned from unassigned to AS51167
84 address(es) reassigned from AS198949 to AS15557
66 address(es) reassigned from AS45758 to AS45629
33 address(es) reassigned from AS174 to AS212238
[...]
1 address(es) reassigned from unassigned to AS399619
Summary: 919 (1.67%) of 54,901 addresses were reassigned.
```
When the `-s / --show-addresses` flag is used, addresses subject to reassignment are included in the output.
ACKs for top commit:
fjahr:
tACK 5215c925d1382e71c9e1d642fced8a152c629c7f
achow101:
ACK 5215c925d1382e71c9e1d642fced8a152c629c7f
brunoerg:
reACK 5215c925d1382e71c9e1d642fced8a152c629c7f
Tree-SHA512: ebcf47754bce92794fad9f4c3bfc1c5e9daf077db5975f444c5135092eb6a26ecaa1eca6748a03ae0c87d9e45532426966fe8f3c17249b17f9dcad490d6dd3bf
8fee5355ee1d2f2b410c548bac9e85f2a21a696d guix: fix suggested fake date for openssl -1.1.1l (Sjors Provoost)
Pull request description:
Using `2020-10-01` as the fake timestamp will cause many test failures with `/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv`. I didn't investigate why, but I guess because it's _before_ the test certificates were created. They expired in June 2022. I tried a month before that, which worked.
Also fixes layout of instructions.
ACKs for top commit:
achow101:
ACK 8fee5355ee1d2f2b410c548bac9e85f2a21a696d
maflcko:
review ACK 8fee5355ee1d2f2b410c548bac9e85f2a21a696d
Tree-SHA512: df5dd3aa961e25bd57d0b8b73daeb3ec76856b06e35277f24b6b19be81774512228f75e2b779afa8ea92fcc39beb869f43e0c57fba19ad16a82812e7c0bea38b
77ff0ec1f185b818b30877de2bedc1750319e6c4 contrib: support reading XORed blocks in linearize-data.py script (Sebastian Falbesoner)
Pull request description:
This PR is a small follow-up for #28052, adding support for the block linearization script to handle XORed blocksdir *.dat files. Note that if no xor.dat file exists, the XOR pattern is set to all-zeros, in order to still support blockdirs that have been created with versions earlier than 28.x.
Partly fixes issue #30599.
ACKs for top commit:
achow101:
ACK 77ff0ec1f185b818b30877de2bedc1750319e6c4
tdb3:
ACK 77ff0ec1f185b818b30877de2bedc1750319e6c4
hodlinator:
ACK 77ff0ec1f185b818b30877de2bedc1750319e6c4
Tree-SHA512: 011eb02e2411de373cbbf4b26db4640fc693a20be8c2430529fba6e36a3a3abfdfdc3b005d330f9ec2846bfad9bfbf34231c574ba99289ef37dd51a68e6e7f3d
2925bd537cbd8c70594e23f6c4298b7101f7f73d refactor: use c++20 std::views::reverse instead of reverse_iterator.h (stickies-v)
Pull request description:
C++20 introduces [`std::ranges::views::reverse`](https://en.cppreference.com/w/cpp/ranges/reverse_view), which allows us to drop our own `reverse_iterator.h` implementation and also makes it easier to chain views (even though I think we currently don't use this).
ACKs for top commit:
achow101:
ACK 2925bd537cbd8c70594e23f6c4298b7101f7f73d
maflcko:
ACK 2925bd537cbd8c70594e23f6c4298b7101f7f73d 🎷
Tree-SHA512: 567666ec44af5d1beb7a271836bcc89c4c577abc77f522fcc18bc6d4de516ae9b0df766d0bfa6dd217569e6878331c2aee1d9815620860375e3510dad7fed476
chainparams.cpp - workaround for MSVC bug triggering C7595 - Calling consteval constructors in initializer lists fails, but works on GCC (13.2.0) & Clang (17.0.6).
bda537f7c484d61a55a72c5856f2d4420664db8f depends: remove ENV unsetting for darwin (fanquake)
1807760f094b600fdebe4ebd0018e4fd5d506ce9 guix: improve ENV unsetting for macOS (fanquake)
0b2aeee21d24645394d70086e992408948952952 depends: patch explicit -lm usage out of Qt tools (fanquake)
Pull request description:
Now that we use the native compiler, and have fixed Qt, and these vars
are (almost) unset in Guix, we can remove the unsetting from our compiler
command here.
I couldn't manage to make a darwin-clang-cross only exclusion of `-lm` work properly
for Qt, so opted for just removing the explicit link entirely. I do not think this should have
any other unwanted side-effects.
Fixes#21552.
ACKs for top commit:
TheCharlatan:
ACK bda537f7c484d61a55a72c5856f2d4420664db8f
Tree-SHA512: 97a2d85de7d4b1d65717ecb521399ecba5f53863b8aef21af62ede5ceee59ee1a9392663da3a3852cad1b6d8b420dd4b0b5f0eea38d30a81785d8b2718620b5f
d1592d2eee1913e734a4f92907e796eb3350c64a guix: use gcc-12 to compile winpthreads (fanquake)
b23690e8216f2b47e8e2c21a9ff057b25c4083ae guix: use GCC 12.4.0 over 12.3.0 (fanquake)
8b41ede55ebbc6978deb3f4fad5e18b76b372506 guix: consolidate back to GCC 12 toolchain for all HOSTS (fanquake)
Pull request description:
This PR contains 3 changes:
* Bump GCC in Guix from [12.3.0 to 12.4.0](https://gcc.gnu.org/gcc-12/). A patch was sent upstream, https://lists.gnu.org/archive/html/guix-patches/2024-06/msg01025.html, but has not landed.
* Consolidate all build environments back to using a GCC 12 toolchain. After #21778, the macOS environment is no-longer pinned to 11 (12 would otherwise cause issues building cctools). So, instead of requiring all builders to compile an additional GCC toolchain, use 12.
* Use GCC 12 to compile winpthreads. Currently, GCC 11 is used; which became apparent in https://github.com/bitcoin/bitcoin/pull/30452#issuecomment-2244715566.
ACKs for top commit:
TheCharlatan:
ACK d1592d2eee1913e734a4f92907e796eb3350c64a
hebasto:
ACK d1592d2eee1913e734a4f92907e796eb3350c64a.
Tree-SHA512: e3aa1fa3e69500c93180e07cb4684661247ec6bc45245f746538d81406ff1d8777131590307496dda3287a112b6633e4991168586ca4c2036fa3a57b1efa9c87
Using GCC 11 for the macOS build hasn't been required since #21778, and
at this point, given a toolchain is still needed (#30206), it makes more
sense to (re-)use 12, rather than make all builders compile another
GCC toolchain.
If the binaries don't exist, the Guix build has failed for some other
reason.
There's no need to check for unknown architectures, or executable
formats, as the only ones that could be built are those that we've
configured toolchains for in Guix.
We've also been doing this inconsistently across the two scripts.
9010b1343b9f931f771d3d49dd03b57868c24d5d contrib: c++ify test stubs after switching to c++ compilers (Cory Fields)
261f77033349a83f14d39b8bb8a2df46085b47ad contrib: rename cc to cxx in binary checking scripts (Cory Fields)
a38c9600059a1583be244e0df198ffd9da375ba9 contrib: use c++ rather than c for binary tests (Cory Fields)
Pull request description:
From hebasto's CMake repo. See discussion here: https://github.com/hebasto/bitcoin/pull/252#discussion_r1664657488
Use CXX/CXXFLAGS rather than CC/CFLAGS to test our actual compiler for binary checks rather than the one we only forward to secp256k1.
ACKs for top commit:
hebasto:
ACK 9010b1343b9f931f771d3d49dd03b57868c24d5d.
fanquake:
ACK 9010b1343b9f931f771d3d49dd03b57868c24d5d
Tree-SHA512: 7b8788d7d3760103062eff10056c995e1ad14c0c487d9414683ad54d816c255d0ca751f4d0e2d2ad7f9e8a7101d8c7f1e9333fa5b137558ed68fa593c4b4ce6d
34c9cee380e7276cf3f85f2ce56a293e57afd676 clang-tidy: add check for non-trivial thread_local vars (Cory Fields)
Pull request description:
Forbid thread_local vars with non-trivial destructors.
This is a follow-up from: https://github.com/bitcoin/bitcoin/pull/30095#discussion_r1608423170
ACKs for top commit:
maflcko:
ACK 34c9cee380e7276cf3f85f2ce56a293e57afd676
TheCharlatan:
Re-ACK 34c9cee380e7276cf3f85f2ce56a293e57afd676
Tree-SHA512: 3a798607fb189a5bbc714ed6e86dea462fe29d366b790e96d10a7b4ffcf1f194da9b8f4cd0b82154408709b8e3c58d3f613d6311903bd65a76d8b556ab230d21
Introduce diff_addrs subcommand as means for a Bitcoin-centric
comparison of two ASMaps.
In addition to two ASMaps, the subcommand reads addresses from
a (getnodeaddresses-compatible) file, and provides information on
addresses that have mismatching ASN according to the two ASMaps.
b5fc6d46a3854c18f6e8dfc89540d24ef778caa2 guix: use glibc 2.31 (fanquake)
Pull request description:
Set minimum required glibc to 2.31.
The glibc 2.31 branch is still maintained: https://sourceware.org/git/?p=glibc.git;a=shortlog;h=refs/heads/release/2.31/master.
Remove the stack-protector check from test-security-check, as the test
no-longer fails, and given the control we have of the end, the actual
security-check test seems sufficient (this might also be applied to some
of the other checks).
Drops runtime support for Ubuntu Bionic 18.04 and RHEL-8 from the release binaries.
ACKs for top commit:
TheCharlatan:
ACK b5fc6d46a3854c18f6e8dfc89540d24ef778caa2
Tree-SHA512: ba7e727240fa0ebebfb8b749024c71cbfdec37c33b39627866d78f9318ccdc687fd5103a63ee0e98cf809d9954dde56b1b305691c33d1de275ed0519f716c921
3ab25201909bece9066ac6191670bcee09791d54 contrib: Fixup verify-binaries OS platform parsing (Ben Westgate)
Pull request description:
Closes#30145.
This PR solves two major issues in the `parse_version_string` function of verify-binaries:
1. `-aarch64` binaries cannot be specifically downloaded. The -platform string gets interpreted as a release candidate that doesn't exist due to containing sub-string "rc".
2. Specifying a platform with a "-" in the name causes the parser to ignore both "-platform" AND "-rcN" and download the potentially wrong (non-rc) version for every platform. This also prevented specifying just one platform binary the user wished to download.
It also updates the accompanying `test.py` to cover problem two and adds two examples that were formerly broken to `README.md` to show what is now possible. Including the most useful case of downloading only 1 specific platform's binary.
This improves the Bitcoin verify-binaries tools user experience by not:
1. Failing to download for inexplicable reasons,
2. Downloading more files than what the user told it to, or in the worst case
3. Downloading only the wrong files.
* A test was added to cover the command `verify-binaries/verify.py pub 22.0-x86_64-linux-gnu.tar.gz` which checks that _bitcoin-22.0-x86_64-linux-gnu.tar.gz_ downloads successfully AND ONLY _bitcoin-22.0-x86_64-linux-gnu.tar.gz_ downloads.
* The steps to reproduce each bug are in the referenced issue #30145. Explanation of the potential issue as well as reasoning for the way the bug was fixed are in my commit descriptions.
* This delivers the promised feature of "only download the binaries for a certain platform", by allowing strings with '-' to be accepted, allowing for single file downloads for any specific platform which was not always possible before.
* Removes 6 lines of code from the offending `parse_version_string` function, while fixing the bugs/errors, and extending the functionality to be practical for users with slow connections.
* Makes the error message more helpful when no file matches the provided platform string, now prints "Did you mean: `closest-match`" to help correct typos.
Thanks for reading my PR. I look forward to getting this helpful tool in its best shape yet.
Log of this branch passing the new test.py:
```
python3 test.py
✓ 'Nonexistent version should fail' passed
✓ 'Malformed version should fail' passed
✓ '--min-good-sigs 20 should fail' passed
- testing verification (22.0-x86_64-linux-gnu.tar.gz)
✓ '22.0-x86_64-linux-gnu.tar.gz should succeed' passed
- testing verification (22.0)
✓ '22.0 should succeed' passed
```
Log of master failing the new test.py:
```
python3 test.py
✓ 'Nonexistent version should fail' passed
✓ 'Malformed version should fail' passed
✓ '--min-good-sigs 20 should fail' passed
- testing verification (22.0-x86_64-linux-gnu.tar.gz)
✓ '22.0-x86_64-linux-gnu.tar.gz should succeed' passed
Traceback (most recent call last):
File "/home/ben/Documents/GitHub/bitcoin/contrib/verify-binaries/test.py", line 74, in <module>
main()
File "/home/ben/Documents/GitHub/bitcoin/contrib/verify-binaries/test.py", line 27, in main
assert len(v) == 1
^^^^^^^^^^^
AssertionError
```
ACKs for top commit:
stickies-v:
re-ACK 3ab25201909bece9066ac6191670bcee09791d54
willcl-ark:
re-ACK 3ab25201909bece9066ac6191670bcee09791d54
Tree-SHA512: 6093228bb876cd0ac84d1cd2630074e17a3f09c4b23325b9419d859a5721a802f928844575233b135df52b882287dd18d6fadf4419d88ec0a2cdf82db315329e
