Merge bitcoin/bitcoin#30092: [27.x] Backports and rc1

22701a4346 doc: update manual pages for 27.1rc1 (fanquake)
9e919072bd build: bump version to 27.1rc1 (fanquake)
9b4640c4be doc: update release-notes.md for 27.1 (fanquake)
80032d6917 qt: 27.1rc1 translations update (Hennadii Stepanov)
423bd6dc68 windeploy: Renew certificate (Ava Chow)
77b2321ca0 depends: Fetch miniupnpc sources from an alternative website (Hennadii Stepanov)
31adcfa171 test: add GetAddedNodeInfo() CJDNS regression unit test (Jon Atack)
9cdb9edfb8 p2p, bugfix: detect addnode cjdns peers in GetAddedNodeInfo() (Jon Atack)
3c26058da2 crypto: disable asan for sha256_sse4 with clang and -O0 (Cory Fields)
0ba11cf908 rpc: move UniValue in blockToJSON (willcl-ark)
dedf319b08 gui: don't permit port in proxy IP option (willcl-ark)
d1289a1300 gui: fix create unsigned transaction fee bump (furszy)

Pull request description:

  Backports:
  * https://github.com/bitcoin-core/gui/pull/812
  * https://github.com/bitcoin-core/gui/pull/813
  * https://github.com/bitcoin/bitcoin/pull/30085
  * https://github.com/bitcoin/bitcoin/pull/30094
  * https://github.com/bitcoin/bitcoin/pull/30097
  * https://github.com/bitcoin/bitcoin/pull/30149
  * https://github.com/bitcoin/bitcoin/pull/30151

  Bump to 27.1rc1.

ACKs for top commit:
  stickies-v:
    re-ACK 22701a4346
  willcl-ark:
    reACK 22701a4346
  hebasto:
    re-ACK 22701a4346.

Tree-SHA512: 6eca44ba7e6664eb4677646597dfdaf56a241c8c3e95e0ab8929ee2fc3671303fc6c2634d359b4523dbd452ac5e54fd1f4c7c2bf7e9c5209395f8cb3b4753fb3

.cirrus.yml

@@ -3,19 +3,48 @@ env:  # Global defaults
   PACKAGE_MANAGER_INSTALL: "apt-get update && apt-get install -y"
   MAKEJOBS: "-j10"
   TEST_RUNNER_PORT_MIN: "14000"  # Must be larger than 12321, which is used for the http cache. See https://cirrus-ci.org/guide/writing-tasks/#http-cache
-  CI_FAILFAST_TEST_LEAVE_DANGLING: "1"  # Cirrus CI does not care about dangling process and setting this variable avoids killing the CI script itself on error
-  CCACHE_SIZE: "200M"
+  CI_FAILFAST_TEST_LEAVE_DANGLING: "1"  # Cirrus CI does not care about dangling processes and setting this variable avoids killing the CI script itself on error
+  CCACHE_MAXSIZE: "200M"
   CCACHE_DIR: "/tmp/ccache_dir"
   CCACHE_NOHASHDIR: "1"  # Debug info might contain a stale path if the build dir changes, but this is fine
 
-cirrus_ephemeral_worker_template_env: &CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
-  DANGER_RUN_CI_ON_HOST: "1"  # Containers will be discarded after the run, so there is no risk that the ci scripts modify the system
-
-persistent_worker_template_env: &PERSISTENT_WORKER_TEMPLATE_ENV
-  RESTART_CI_DOCKER_BEFORE_RUN: "1"
-
-persistent_worker_template: &PERSISTENT_WORKER_TEMPLATE
-  persistent_worker: {}  # https://cirrus-ci.org/guide/persistent-workers/
+# https://cirrus-ci.org/guide/persistent-workers/
+#
+# It is possible to select a specific persistent worker by label. Refer to the
+# Cirrus CI docs for more details.
+#
+# Generally, a persistent worker must run Ubuntu 23.04+ or Debian 12+.
+# Specifically,
+# - apt-get is required due to PACKAGE_MANAGER_INSTALL
+# - podman-docker-4.1+ is required due to the use of `podman` when
+#   RESTART_CI_DOCKER_BEFORE_RUN is set and 4.1+ due to the bugfix in 4.1
+#   (https://github.com/bitcoin/bitcoin/pull/21652#issuecomment-1657098200)
+# - The ./ci/ depedencies (with cirrus-cli) should be installed:
+#
+#   ```
+#   apt update && apt install git screen python3 bash podman-docker curl -y && curl -L -o cirrus "https://github.com/cirruslabs/cirrus-cli/releases/latest/download/cirrus-linux-$(dpkg --print-architecture)" && mv cirrus /usr/local/bin/cirrus && chmod +x /usr/local/bin/cirrus
+#   ```
+#
+# - There are no strict requirements on the hardware, because having less CPUs
+#   runs the same CI script (maybe slower). To avoid rare and intermittent OOM
+#   due to short memory usage spikes, it is recommended to add (and persist)
+#   swap:
+#
+#   ```
+#   fallocate -l 16G /swapfile_ci && chmod 600 /swapfile_ci && mkswap /swapfile_ci && swapon /swapfile_ci && ( echo '/swapfile_ci none swap sw 0 0' | tee -a /etc/fstab )
+#   ```
+#
+# - To register the persistent worker, open a `screen` session and run:
+#
+#   ```
+#   RESTART_CI_DOCKER_BEFORE_RUN=1 screen cirrus worker run --labels type=todo_fill_in_type --token todo_fill_in_token
+#   ```
+#
+# The following specific types should exist, with the following requirements:
+# - small: For an x86_64 machine, recommended to have 2 CPUs and 8 GB of memory.
+# - medium: For an x86_64 machine, recommended to have 4 CPUs and 16 GB of memory.
+# - noble: For a machine running the Linux kernel shipped with exaclty Ubuntu Noble 24.04. The machine is recommended to have 4 CPUs and 16 GB of memory.
+# - arm64: For an aarch64 machine, recommended to have 2 CPUs and 8 GB of memory.
 
 # https://cirrus-ci.org/guide/tips-and-tricks/#sharing-configuration-between-tasks
 filter_template: &FILTER_TEMPLATE
@@ -26,7 +55,7 @@ base_template: &BASE_TEMPLATE
   << : *FILTER_TEMPLATE
   merge_base_script:
     # Unconditionally install git (used in fingerprint_script).
-    - bash -c "$PACKAGE_MANAGER_INSTALL git"
+    - git --version || bash -c "$PACKAGE_MANAGER_INSTALL git"
     - if [ "$CIRRUS_PR" = "" ]; then exit 0; fi
     - git fetch --depth=1 $CIRRUS_REPO_CLONE_URL "pull/${CIRRUS_PR}/merge"
     - git checkout FETCH_HEAD  # Use merged changes to detect silent merge conflicts
@@ -34,26 +63,11 @@ base_template: &BASE_TEMPLATE
 
 main_template: &MAIN_TEMPLATE
   timeout_in: 120m  # https://cirrus-ci.org/faq/#instance-timed-out
-  ccache_cache:
-    folder: "/tmp/ccache_dir"
   ci_script:
     - ./ci/test_run_all.sh
 
-container_depends_template: &CONTAINER_DEPENDS_TEMPLATE
-  << : *BASE_TEMPLATE
-  container:
-    # https://cirrus-ci.org/faq/#are-there-any-limits
-    # Each project has 16 CPU in total, assign 2 to each container, so that 8 tasks run in parallel
-    cpu: 2
-    greedy: true
-    memory: 8G  # Set to 8GB to avoid OOM. https://cirrus-ci.org/guide/linux/#linux-containers
-    dockerfile: ci/test_imagefile  # https://cirrus-ci.org/guide/docker-builder-vm/#dockerfile-as-a-ci-environment
-  depends_built_cache:
-    folder: "depends/built"
-    fingerprint_script: echo $CIRRUS_TASK_NAME $(git rev-parse HEAD:depends)
-
 global_task_template: &GLOBAL_TASK_TEMPLATE
-  << : *CONTAINER_DEPENDS_TEMPLATE
+  << : *BASE_TEMPLATE
   << : *MAIN_TEMPLATE
 
 compute_credits_template: &CREDITS_TEMPLATE
@@ -62,7 +76,7 @@ compute_credits_template: &CREDITS_TEMPLATE
   use_compute_credits: $CIRRUS_REPO_FULL_NAME == 'bitcoin/bitcoin' && $CIRRUS_PR != ""
 
 task:
-  name: 'lint [bookworm]'
+  name: 'lint'
   << : *BASE_TEMPLATE
   container:
     image: debian:bookworm
@@ -70,296 +84,126 @@ task:
     memory: 1G
   # For faster CI feedback, immediately schedule the linters
   << : *CREDITS_TEMPLATE
+  test_runner_cache:
+    folder: "/lint_test_runner"
+    fingerprint_script: echo $CIRRUS_TASK_NAME $(git rev-parse HEAD:test/lint/test_runner)
   python_cache:
-    folder: "/tmp/python"
+    folder: "/python_build"
     fingerprint_script: cat .python-version /etc/os-release
   unshallow_script:
     - git fetch --unshallow --no-tags
   lint_script:
     - ./ci/lint_run_all.sh
-  env:
-    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
 
 task:
-  name: 'tidy [lunar]'
+  name: 'tidy'
   << : *GLOBAL_TASK_TEMPLATE
-  container:
-    cpu: 2
-    memory: 5G
-    docker_arguments:
-      CI_IMAGE_NAME_TAG: ubuntu:lunar
-      FILE_ENV: "./ci/test/00_setup_env_native_tidy.sh"
-  # For faster CI feedback, immediately schedule the linters
-  << : *CREDITS_TEMPLATE
+  persistent_worker:
+    labels:
+      type: medium
   env:
-    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
+    FILE_ENV: "./ci/test/00_setup_env_native_tidy.sh"
 
 task:
-  name: "Win64 native [vs2022]"
-  << : *FILTER_TEMPLATE
-  windows_container:
-    cpu: 6
-    memory: 12G
-    image: cirrusci/windowsservercore:visualstudio2022
-  timeout_in: 120m
-  env:
-    PATH: 'C:\jom;C:\Python39;C:\Python39\Scripts;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Current\Bin;%PATH%'
-    PYTHONUTF8: 1
-    CI_VCPKG_TAG: '2023.01.09'
-    VCPKG_DOWNLOADS: 'C:\Users\ContainerAdministrator\AppData\Local\vcpkg\downloads'
-    VCPKG_DEFAULT_BINARY_CACHE: 'C:\Users\ContainerAdministrator\AppData\Local\vcpkg\archives'
-    CCACHE_DIR: 'C:\Users\ContainerAdministrator\AppData\Local\ccache'
-    WRAPPED_CL: 'C:\Users\ContainerAdministrator\AppData\Local\Temp\cirrus-ci-build\ci\test\wrapped-cl.bat'
-    QT_DOWNLOAD_URL: 'https://download.qt.io/official_releases/qt/5.15/5.15.5/single/qt-everywhere-opensource-src-5.15.5.zip'
-    QT_LOCAL_PATH: 'C:\qt-everywhere-opensource-src-5.15.5.zip'
-    QT_SOURCE_DIR: 'C:\qt-everywhere-src-5.15.5'
-    QTBASEDIR: 'C:\Qt_static'
-    x64_NATIVE_TOOLS: '"C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat"'
-    QT_CONFIGURE_COMMAND: '..\configure -release -silent -opensource -confirm-license -opengl desktop -static -static-runtime -mp -qt-zlib -qt-pcre -qt-libpng -nomake examples -nomake tests -nomake tools -no-angle -no-dbus -no-gif -no-gtk -no-ico -no-icu -no-libjpeg -no-libudev -no-sql-sqlite -no-sql-odbc -no-sqlite -no-vulkan -skip qt3d -skip qtactiveqt -skip qtandroidextras -skip qtcharts -skip qtconnectivity -skip qtdatavis3d -skip qtdeclarative -skip doc -skip qtdoc -skip qtgamepad -skip qtgraphicaleffects -skip qtimageformats -skip qtlocation -skip qtlottie -skip qtmacextras -skip qtmultimedia -skip qtnetworkauth -skip qtpurchasing -skip qtquick3d -skip qtquickcontrols -skip qtquickcontrols2 -skip qtquicktimeline -skip qtremoteobjects -skip qtscript -skip qtscxml -skip qtsensors -skip qtserialbus -skip qtserialport -skip qtspeech -skip qtsvg -skip qtvirtualkeyboard -skip qtwayland -skip qtwebchannel -skip qtwebengine -skip qtwebglplugin -skip qtwebsockets -skip qtwebview -skip qtx11extras -skip qtxmlpatterns -no-openssl -no-feature-bearermanagement -no-feature-printdialog -no-feature-printer -no-feature-printpreviewdialog -no-feature-printpreviewwidget -no-feature-sql -no-feature-sqlmodel -no-feature-textbrowser -no-feature-textmarkdownwriter -no-feature-textodfwriter -no-feature-xml'
-    IgnoreWarnIntDirInTempDetected: 'true'
-  merge_script:
-    - PowerShell -NoLogo -Command if ($env:CIRRUS_PR -ne $null) { git fetch $env:CIRRUS_REPO_CLONE_URL pull/$env:CIRRUS_PR/merge; git reset --hard FETCH_HEAD; }
-  msvc_qt_built_cache:
-    folder: "%QTBASEDIR%"
-    reupload_on_changes: false
-    fingerprint_script:
-      - echo %QT_DOWNLOAD_URL% %QT_CONFIGURE_COMMAND%
-      - msbuild -version
-    populate_script:
-      - curl -L -o C:\jom.zip http://download.qt.io/official_releases/jom/jom.zip
-      - mkdir C:\jom
-      - tar -xf C:\jom.zip -C C:\jom
-      - curl -L -o %QT_LOCAL_PATH% %QT_DOWNLOAD_URL%
-      - tar -xf %QT_LOCAL_PATH% -C C:\
-      - '%x64_NATIVE_TOOLS%'
-      - cd %QT_SOURCE_DIR%
-      - mkdir build
-      - cd build
-      - '%QT_CONFIGURE_COMMAND% -prefix %QTBASEDIR%'
-      - jom
-      - jom install
-  vcpkg_tools_cache:
-    folder: '%VCPKG_DOWNLOADS%\tools'
-    reupload_on_changes: false
-    fingerprint_script:
-      - echo %CI_VCPKG_TAG%
-      - msbuild -version
-  vcpkg_binary_cache:
-    folder: '%VCPKG_DEFAULT_BINARY_CACHE%'
-    reupload_on_changes: true
-    fingerprint_script:
-      - echo %CI_VCPKG_TAG%
-      - type build_msvc\vcpkg.json
-      - msbuild -version
-    populate_script:
-      - mkdir %VCPKG_DEFAULT_BINARY_CACHE%
-  ccache_cache:
-    folder: '%CCACHE_DIR%'
-  install_tools_script:
-    - choco install --yes --no-progress ccache --version=4.7.4
-    - choco install --yes --no-progress python3 --version=3.9.6
-    - pip install zmq
-    - ccache --version
-    - python -VV
-  install_vcpkg_script:
-    - cd ..
-    - git clone --quiet https://github.com/microsoft/vcpkg.git
-    - cd vcpkg
-    - git -c advice.detachedHead=false checkout %CI_VCPKG_TAG%
-    - .\bootstrap-vcpkg -disableMetrics
-    - echo set(VCPKG_BUILD_TYPE release) >> triplets\x64-windows-static.cmake
-    - .\vcpkg integrate install
-    - .\vcpkg version
-  build_script:
-    - '%x64_NATIVE_TOOLS%'
-    - cd %CIRRUS_WORKING_DIR%
-    - ccache --zero-stats --max-size=%CCACHE_SIZE%
-    - python build_msvc\msvc-autogen.py
-    - msbuild build_msvc\bitcoin.sln -property:CLToolExe=%WRAPPED_CL%;UseMultiToolTask=true;Configuration=Release -maxCpuCount -verbosity:minimal -noLogo
-    - ccache --show-stats
-  check_script:
-    - src\test_bitcoin.exe -l test_suite
-    - src\bench_bitcoin.exe --sanity-check
-    - python test\util\test_runner.py
-    - python test\util\rpcauth-test.py
-  functional_tests_script:
-    # Increase the dynamic port range to the maximum allowed value to mitigate "OSError: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted".
-    # See: https://learn.microsoft.com/en-us/biztalk/technical-guides/settings-that-can-be-modified-to-improve-network-performance
-    - netsh int ipv4 set dynamicport tcp start=1025 num=64511
-    - netsh int ipv6 set dynamicport tcp start=1025 num=64511
-    # Exclude feature_dbcrash for now due to timeout
-    - python test\functional\test_runner.py --nocleanup --ci --quiet --combinedlogslen=99999999 --jobs=6 --timeout-factor=8 --extended --exclude feature_dbcrash
-
-task:
-  name: 'ARM [unit tests, no functional tests] [bullseye]'
+  name: 'ARM, unit tests, no functional tests'
   << : *GLOBAL_TASK_TEMPLATE
-  arm_container:
-    image: debian:bullseye
-    cpu: 2
-    memory: 8G
-    # docker_arguments:  # Can use dockerfile after https://github.com/cirruslabs/cirrus-ci-docs/issues/1154
+  persistent_worker:
+    labels:
+      type: arm64  # Use arm64 worker to sidestep qemu and avoid a slow CI: https://github.com/bitcoin/bitcoin/pull/28087#issuecomment-1649399453
   env:
-    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
     FILE_ENV: "./ci/test/00_setup_env_arm.sh"
-    QEMU_USER_CMD: ""  # Disable qemu and run the test natively
 
 task:
-  name: 'Win64 [unit tests, no gui tests, no boost::process, no functional tests] [jammy]'
+  name: 'Win64, unit tests, no gui tests, no functional tests'
   << : *GLOBAL_TASK_TEMPLATE
-  container:
-    docker_arguments:
-      CI_IMAGE_NAME_TAG: ubuntu:jammy
-      FILE_ENV: "./ci/test/00_setup_env_win64.sh"
-  << : *CREDITS_TEMPLATE
+  persistent_worker:
+    labels:
+      type: small
   env:
-    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
+    FILE_ENV: "./ci/test/00_setup_env_win64.sh"
 
 task:
-  name: '32-bit + dash [gui] [CentOS 8]'
+  name: '32-bit CentOS, dash, gui'
   << : *GLOBAL_TASK_TEMPLATE
-  container:
-    docker_arguments:
-      CI_IMAGE_NAME_TAG: quay.io/centos/centos:stream8
-      FILE_ENV: "./ci/test/00_setup_env_i686_centos.sh"
-  # For faster CI feedback, immediately schedule one task that runs all tests
-  << : *CREDITS_TEMPLATE
+  persistent_worker:
+    labels:
+      type: small
   env:
-    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
-    PACKAGE_MANAGER_INSTALL: "yum install -y"
+    FILE_ENV: "./ci/test/00_setup_env_i686_centos.sh"
 
 task:
-  name: '[previous releases, uses qt5 dev package and some depends packages, DEBUG] [unsigned char] [buster]'
-  previous_releases_cache:
-    folder: "releases"
+  name: 'previous releases, depends DEBUG'
   << : *GLOBAL_TASK_TEMPLATE
-  << : *PERSISTENT_WORKER_TEMPLATE
+  persistent_worker:
+    labels:
+      type: small
   env:
-    << : *PERSISTENT_WORKER_TEMPLATE_ENV
-    FILE_ENV: "./ci/test/00_setup_env_native_qt5.sh"
+    FILE_ENV: "./ci/test/00_setup_env_native_previous_releases.sh"
 
 task:
-  name: '[TSan, depends, gui] [lunar]'
+  name: 'TSan, depends, gui'
   << : *GLOBAL_TASK_TEMPLATE
-  container:
-    cpu: 6  # Increase CPU and Memory to avoid timeout
-    memory: 24G
-    docker_arguments:
-      CI_IMAGE_NAME_TAG: ubuntu:lunar
-      FILE_ENV: "./ci/test/00_setup_env_native_tsan.sh"
+  persistent_worker:
+    labels:
+      type: medium
   env:
-    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
+    FILE_ENV: "./ci/test/00_setup_env_native_tsan.sh"
 
 task:
-  name: '[MSan, depends] [lunar]'
+  name: 'MSan, depends'
   << : *GLOBAL_TASK_TEMPLATE
-  container:
-    docker_arguments:
-      CI_IMAGE_NAME_TAG: ubuntu:lunar
-      FILE_ENV: "./ci/test/00_setup_env_native_msan.sh"
+  persistent_worker:
+    labels:
+      type: small
+  timeout_in: 300m  # Use longer timeout for the *rare* case where a full build (llvm + msan + depends + ...) needs to be done.
   env:
-    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
-    MAKEJOBS: "-j4"  # Avoid excessive memory use due to MSan
+    FILE_ENV: "./ci/test/00_setup_env_native_msan.sh"
 
 task:
-  name: '[ASan + LSan + UBSan + integer, no depends, USDT] [jammy]'
+  name: 'ASan + LSan + UBSan + integer, no depends, USDT'
+  enable_bpfcc_script:
+    # In the image build step, no external environment variables are available,
+    # so any settings will need to be written to the settings env file:
+    - sed -i "s|\${CIRRUS_CI}|true|g" ./ci/test/00_setup_env_native_asan.sh
   << : *GLOBAL_TASK_TEMPLATE
-  # We can't use a 'container' for the USDT interface tests as the CirrusCI
-  # containers don't have privileges to hook into bitcoind. CirrusCI uses
-  # Google Compute Engine instances: https://cirrus-ci.org/guide/custom-vms/
-  # Images can be found here: https://cloud.google.com/compute/docs/images/os-details
-  compute_engine_instance:
-    image_project: ubuntu-os-cloud
-    image: family/ubuntu-2204-lts # when upgrading, check if we can drop "ADD_UNTRUSTED_BPFCC_PPA"
-    cpu: 4
-    memory: 12G
+  persistent_worker:
+    labels:
+      type: noble  # Must use this specific worker (needed for USDT functional tests)
   env:
-    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
-    HOME: /root/  # Only needed for compute_engine_instance
     FILE_ENV: "./ci/test/00_setup_env_native_asan.sh"
-    MAKEJOBS: "-j4"  # Avoid excessive memory use
 
 task:
-  name: '[fuzzer,address,undefined,integer, no depends] [lunar]'
+  name: 'fuzzer,address,undefined,integer, no depends'
   << : *GLOBAL_TASK_TEMPLATE
-  container:
-    cpu: 4  # Increase CPU and memory to avoid timeout
-    memory: 16G
-    docker_arguments:
-      CI_IMAGE_NAME_TAG: ubuntu:lunar
-      FILE_ENV: "./ci/test/00_setup_env_native_fuzz.sh"
+  persistent_worker:
+    labels:
+      type: medium
   env:
-    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
+    FILE_ENV: "./ci/test/00_setup_env_native_fuzz.sh"
 
 task:
-  name: '[multiprocess, i686, DEBUG] [focal]'
+  name: 'multiprocess, i686, DEBUG'
   << : *GLOBAL_TASK_TEMPLATE
-  container:
-    cpu: 4
-    memory: 16G  # The default memory is sometimes just a bit too small, so double everything
-    docker_arguments:
-      CI_IMAGE_NAME_TAG: ubuntu:focal
-      FILE_ENV: "./ci/test/00_setup_env_i686_multiprocess.sh"
+  persistent_worker:
+    labels:
+      type: medium
   env:
-    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
+    FILE_ENV: "./ci/test/00_setup_env_i686_multiprocess.sh"
 
 task:
-  name: '[no wallet, libbitcoinkernel] [focal]'
+  name: 'no wallet, libbitcoinkernel'
   << : *GLOBAL_TASK_TEMPLATE
-  container:
-    docker_arguments:
-      CI_IMAGE_NAME_TAG: ubuntu:focal
-      FILE_ENV: "./ci/test/00_setup_env_native_nowallet_libbitcoinkernel.sh"
-  << : *CREDITS_TEMPLATE
+  persistent_worker:
+    labels:
+      type: small
   env:
-    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
+    FILE_ENV: "./ci/test/00_setup_env_native_nowallet_libbitcoinkernel.sh"
 
 task:
-  name: 'macOS 10.15 [gui, no tests] [focal]'
-  << : *CONTAINER_DEPENDS_TEMPLATE
-  container:
-    docker_arguments:
-      CI_IMAGE_NAME_TAG: ubuntu:focal
-      FILE_ENV: "./ci/test/00_setup_env_mac.sh"
-  << : *CREDITS_TEMPLATE
-  macos_sdk_cache:
-    folder: "depends/SDKs/$MACOS_SDK"
-    fingerprint_key: "$MACOS_SDK"
-  << : *MAIN_TEMPLATE
+  name: 'macOS-cross, gui, no tests'
+  << : *GLOBAL_TASK_TEMPLATE
+  persistent_worker:
+    labels:
+      type: small
   env:
-    MACOS_SDK: "Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers"
-    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
-
-task:
-  name: 'macOS 13 native arm64 [gui, sqlite only] [no depends]'
-  macos_instance:
-    # Use latest image, but hardcode version to avoid silent upgrades (and breaks)
-    image: ghcr.io/cirruslabs/macos-ventura-xcode:14.1  # https://cirrus-ci.org/guide/macOS
-  << : *BASE_TEMPLATE
-  check_clang_script:
-    - clang --version
-  brew_install_script:
-    - brew install boost libevent qt@5 miniupnpc libnatpmp ccache zeromq qrencode libtool automake gnu-getopt
-  << : *MAIN_TEMPLATE
-  env:
-    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
-    CI_USE_APT_INSTALL: "no"
-    PACKAGE_MANAGER_INSTALL: "echo"  # Nothing to do
-    FILE_ENV: "./ci/test/00_setup_env_mac_native_arm64.sh"
-
-task:
-  name: 'ARM64 Android APK [jammy]'
-  << : *CONTAINER_DEPENDS_TEMPLATE
-  container:
-    docker_arguments:
-      CI_IMAGE_NAME_TAG: ubuntu:jammy
-      FILE_ENV: "./ci/test/00_setup_env_android.sh"
-  << : *CREDITS_TEMPLATE
-  android_sdk_cache:
-    folder: "depends/SDKs/android"
-    fingerprint_key: "ANDROID_API_LEVEL=28 ANDROID_BUILD_TOOLS_VERSION=28.0.3 ANDROID_NDK_VERSION=23.2.8568313"
-  depends_sources_cache:
-    folder: "depends/sources"
-    fingerprint_script: git rev-parse HEAD:depends/packages
-  << : *MAIN_TEMPLATE
-  env:
-    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
+    FILE_ENV: "./ci/test/00_setup_env_mac_cross.sh"

.github/ISSUE_TEMPLATE/good_first_issue.yml

@@ -29,8 +29,10 @@ body:
     attributes:
       label: Useful Skills
       description: For example, “`std::thread`”, “Qt5 GUI and async GUI design” or “basic understanding of Bitcoin mining and the Bitcoin Core RPC interface”.
-    validations:
-      required: false
+      value: |
+        * Compiling Bitcoin Core from source
+        * Running the C++ unit tests and the Python functional tests
+        * ...
   - type: textarea
     attributes:
       label: Guidance for new contributors

.github/workflows/ci.yml

@@ -0,0 +1,296 @@
+# Copyright (c) 2023 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+name: CI
+on:
+  # See: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request.
+  pull_request:
+  # See: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#push.
+  push:
+    branches:
+      - '**'
+    tags-ignore:
+      - '**'
+
+concurrency:
+  group: ${{ github.event_name != 'pull_request' && github.run_id || github.ref }}
+  cancel-in-progress: true
+
+env:
+  DANGER_RUN_CI_ON_HOST: 1
+  CI_FAILFAST_TEST_LEAVE_DANGLING: 1  # GHA does not care about dangling processes and setting this variable avoids killing the CI script itself on error
+  MAKEJOBS: '-j10'
+
+jobs:
+  test-each-commit:
+    name: 'test each commit'
+    runs-on: ubuntu-22.04
+    if: github.event_name == 'pull_request' && github.event.pull_request.commits != 1
+    timeout-minutes: 360  # Use maximum time, see https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes. Assuming a worst case time of 1 hour per commit, this leads to a --max-count=6 below.
+    env:
+      MAX_COUNT: 6
+    steps:
+      - name: Determine fetch depth
+        run: echo "FETCH_DEPTH=$((${{ github.event.pull_request.commits }} + 2))" >> "$GITHUB_ENV"
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+          fetch-depth: ${{ env.FETCH_DEPTH }}
+      - name: Determine commit range
+        run: |
+          # Checkout HEAD~ and find the test base commit
+          # Checkout HEAD~ because it would be wasteful to rerun tests on the PR
+          # head commit that are already run by other jobs.
+          git checkout HEAD~
+          # Figure out test base commit by listing ancestors of HEAD, excluding
+          # ancestors of the most recent merge commit, limiting the list to the
+          # newest MAX_COUNT ancestors, ordering it from oldest to newest, and
+          # taking the first one.
+          #
+          # If the branch contains up to MAX_COUNT ancestor commits after the
+          # most recent merge commit, all of those commits will be tested. If it
+          # contains more, only the most recent MAX_COUNT commits will be
+          # tested.
+          #
+          # In the command below, the ^@ suffix is used to refer to all parents
+          # of the merge commit as described in:
+          # https://git-scm.com/docs/git-rev-parse#_other_rev_parent_shorthand_notations
+          # and the ^ prefix is used to exclude these parents and all their
+          # ancestors from the rev-list output as described in:
+          # https://git-scm.com/docs/git-rev-list
+          echo "TEST_BASE=$(git rev-list -n$((${{ env.MAX_COUNT }} + 1)) --reverse HEAD ^$(git rev-list -n1 --merges HEAD)^@ | head -1)" >> "$GITHUB_ENV"
+      - run: |
+          sudo apt-get update
+          sudo apt-get install clang-15 ccache build-essential libtool autotools-dev automake pkg-config bsdmainutils python3-zmq libevent-dev libboost-dev libsqlite3-dev libdb++-dev systemtap-sdt-dev libminiupnpc-dev libnatpmp-dev qtbase5-dev qttools5-dev qttools5-dev-tools qtwayland5 libqrencode-dev -y
+      - name: Compile and run tests
+        run: |
+          # Run tests on commits after the last merge commit and before the PR head commit
+          # Use clang++, because it is a bit faster and uses less memory than g++
+          git rebase --exec "echo Running test-one-commit on \$( git log -1 ) && ./autogen.sh && CC=clang-15 CXX=clang++-15 ./configure && make clean && make -j $(nproc) check && ./test/functional/test_runner.py -j $(( $(nproc) * 2 ))" ${{ env.TEST_BASE }}
+
+  macos-native-x86_64:
+    name: 'macOS 13 native, x86_64, no depends, sqlite only, gui'
+    # Use latest image, but hardcode version to avoid silent upgrades (and breaks).
+    # See: https://github.com/actions/runner-images#available-images.
+    runs-on: macos-13
+
+    # No need to run on the read-only mirror, unless it is a PR.
+    if: github.repository != 'bitcoin-core/gui' || github.event_name == 'pull_request'
+
+    timeout-minutes: 120
+
+    env:
+      FILE_ENV: './ci/test/00_setup_env_mac_native.sh'
+      BASE_ROOT_DIR: ${{ github.workspace }}
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Clang version
+        run: |
+          sudo xcode-select --switch /Applications/Xcode_15.0.app
+          clang --version
+
+      - name: Install Homebrew packages
+        env:
+          HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK: 1
+        run: |
+          # A workaround for "The `brew link` step did not complete successfully" error.
+          brew install python@3 || brew link --overwrite python@3
+          brew install automake libtool pkg-config gnu-getopt ccache boost libevent miniupnpc libnatpmp zeromq qt@5 qrencode
+
+      - name: Set Ccache directory
+        run: echo "CCACHE_DIR=${RUNNER_TEMP}/ccache_dir" >> "$GITHUB_ENV"
+
+      - name: Restore Ccache cache
+        id: ccache-cache
+        uses: actions/cache/restore@v4
+        with:
+          path: ${{ env.CCACHE_DIR }}
+          key: ${{ github.job }}-ccache-${{ github.run_id }}
+          restore-keys: ${{ github.job }}-ccache-
+
+      - name: CI script
+        run: ./ci/test_run_all.sh
+
+      - name: Save Ccache cache
+        uses: actions/cache/save@v4
+        if: github.event_name != 'pull_request' && steps.ccache-cache.outputs.cache-hit != 'true'
+        with:
+          path: ${{ env.CCACHE_DIR }}
+          # https://github.com/actions/cache/blob/main/tips-and-workarounds.md#update-a-cache
+          key: ${{ github.job }}-ccache-${{ github.run_id }}
+
+  win64-native:
+    name: 'Win64 native, VS 2022'
+    # Use latest image, but hardcode version to avoid silent upgrades (and breaks).
+    # See: https://github.com/actions/runner-images#available-images.
+    runs-on: windows-2022
+
+    # No need to run on the read-only mirror, unless it is a PR.
+    if: github.repository != 'bitcoin-core/gui' || github.event_name == 'pull_request'
+
+    env:
+      CCACHE_MAXSIZE: '200M'
+      CI_CCACHE_VERSION: '4.7.5'
+      CI_QT_CONF: '-release -silent -opensource -confirm-license -opengl desktop -static -static-runtime -mp -qt-zlib -qt-pcre -qt-libpng -nomake examples -nomake tests -nomake tools -no-angle -no-dbus -no-gif -no-gtk -no-ico -no-icu -no-libjpeg -no-libudev -no-sql-sqlite -no-sql-odbc -no-sqlite -no-vulkan -skip qt3d -skip qtactiveqt -skip qtandroidextras -skip qtcharts -skip qtconnectivity -skip qtdatavis3d -skip qtdeclarative -skip doc -skip qtdoc -skip qtgamepad -skip qtgraphicaleffects -skip qtimageformats -skip qtlocation -skip qtlottie -skip qtmacextras -skip qtmultimedia -skip qtnetworkauth -skip qtpurchasing -skip qtquick3d -skip qtquickcontrols -skip qtquickcontrols2 -skip qtquicktimeline -skip qtremoteobjects -skip qtscript -skip qtscxml -skip qtsensors -skip qtserialbus -skip qtserialport -skip qtspeech -skip qtsvg -skip qtvirtualkeyboard -skip qtwayland -skip qtwebchannel -skip qtwebengine -skip qtwebglplugin -skip qtwebsockets -skip qtwebview -skip qtx11extras -skip qtxmlpatterns -no-openssl -no-feature-bearermanagement -no-feature-printdialog -no-feature-printer -no-feature-printpreviewdialog -no-feature-printpreviewwidget -no-feature-sql -no-feature-sqlmodel -no-feature-textbrowser -no-feature-textmarkdownwriter -no-feature-textodfwriter -no-feature-xml'
+      CI_QT_DIR: 'qt-everywhere-src-5.15.11'
+      CI_QT_URL: 'https://download.qt.io/official_releases/qt/5.15/5.15.11/single/qt-everywhere-opensource-src-5.15.11.zip'
+      PYTHONUTF8: 1
+      TEST_RUNNER_TIMEOUT_FACTOR: 40
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Configure Developer Command Prompt for Microsoft Visual C++
+        # Using microsoft/setup-msbuild is not enough.
+        uses: ilammy/msvc-dev-cmd@v1
+        with:
+          arch: x64
+
+      - name: Get tool information
+        run: |
+          msbuild -version | Out-File -FilePath "$env:GITHUB_WORKSPACE\msbuild_version"
+          Get-Content -Path "$env:GITHUB_WORKSPACE\msbuild_version"
+          $env:VCToolsVersion | Out-File -FilePath "$env:GITHUB_WORKSPACE\toolset_version"
+          Write-Host "VCToolsVersion $(Get-Content -Path "$env:GITHUB_WORKSPACE\toolset_version")"
+          $env:CI_QT_URL | Out-File -FilePath "$env:GITHUB_WORKSPACE\qt_url"
+          $env:CI_QT_CONF | Out-File -FilePath "$env:GITHUB_WORKSPACE\qt_conf"
+          py -3 --version
+
+      - name: Restore static Qt cache
+        id: static-qt-cache
+        uses: actions/cache/restore@v4
+        with:
+          path: C:\Qt_static
+          key: ${{ github.job }}-static-qt-${{ hashFiles('msbuild_version', 'qt_url', 'qt_conf') }}
+
+      - name: Build static Qt. Download
+        if: steps.static-qt-cache.outputs.cache-hit != 'true'
+        shell: cmd
+        run: |
+          curl --location --output C:\qt-src.zip %CI_QT_URL%
+          choco install --yes --no-progress jom
+
+      - name: Build static Qt. Expand source archive
+        if: steps.static-qt-cache.outputs.cache-hit != 'true'
+        shell: cmd
+        run: tar -xf C:\qt-src.zip -C C:\
+
+      - name: Build static Qt. Create build directory
+        if: steps.static-qt-cache.outputs.cache-hit != 'true'
+        run: |
+          Rename-Item -Path "C:\$env:CI_QT_DIR" -NewName "C:\qt-src"
+          New-Item -ItemType Directory -Path "C:\qt-src\build"
+
+      - name: Build static Qt. Configure
+        if: steps.static-qt-cache.outputs.cache-hit != 'true'
+        working-directory: C:\qt-src\build
+        shell: cmd
+        run: ..\configure %CI_QT_CONF% -prefix C:\Qt_static
+
+      - name: Build static Qt. Build
+        if: steps.static-qt-cache.outputs.cache-hit != 'true'
+        working-directory: C:\qt-src\build
+        shell: cmd
+        run: jom
+
+      - name: Build static Qt. Install
+        if: steps.static-qt-cache.outputs.cache-hit != 'true'
+        working-directory: C:\qt-src\build
+        shell: cmd
+        run: jom install
+
+      - name: Save static Qt cache
+        if: steps.static-qt-cache.outputs.cache-hit != 'true'
+        uses: actions/cache/save@v4
+        with:
+          path: C:\Qt_static
+          key: ${{ github.job }}-static-qt-${{ hashFiles('msbuild_version', 'qt_url', 'qt_conf') }}
+
+      - name: Ccache installation cache
+        id: ccache-installation-cache
+        uses: actions/cache@v4
+        with:
+          path: |
+            C:\ProgramData\chocolatey\lib\ccache
+            C:\ProgramData\chocolatey\bin\ccache.exe
+            C:\ccache\cl.exe
+          key: ${{ github.job }}-ccache-installation-${{ env.CI_CCACHE_VERSION }}
+
+      - name: Install Ccache
+        if: steps.ccache-installation-cache.outputs.cache-hit != 'true'
+        run: |
+          choco install --yes --no-progress ccache --version=$env:CI_CCACHE_VERSION
+          New-Item -ItemType Directory -Path "C:\ccache"
+          Copy-Item -Path "$env:ChocolateyInstall\lib\ccache\tools\ccache-$env:CI_CCACHE_VERSION-windows-x86_64\ccache.exe" -Destination "C:\ccache\cl.exe"
+
+      - name: Restore Ccache cache
+        id: ccache-cache
+        uses: actions/cache/restore@v4
+        with:
+          path: ~/AppData/Local/ccache
+          key: ${{ github.job }}-ccache-${{ github.run_id }}
+          restore-keys: ${{ github.job }}-ccache-
+
+      - name: Using vcpkg with MSBuild
+        run: |
+          Set-Location "$env:VCPKG_INSTALLATION_ROOT"
+          Add-Content -Path "triplets\x64-windows-static.cmake" -Value "set(VCPKG_BUILD_TYPE release)"
+          Add-Content -Path "triplets\x64-windows-static.cmake" -Value "set(VCPKG_PLATFORM_TOOLSET_VERSION $env:VCToolsVersion)"
+          .\vcpkg.exe --vcpkg-root "$env:VCPKG_INSTALLATION_ROOT" integrate install
+          git rev-parse HEAD | Out-File -FilePath "$env:GITHUB_WORKSPACE\vcpkg_commit"
+          Get-Content -Path "$env:GITHUB_WORKSPACE\vcpkg_commit"
+
+      - name: vcpkg tools cache
+        uses: actions/cache@v4
+        with:
+          path: C:/vcpkg/downloads/tools
+          key: ${{ github.job }}-vcpkg-tools
+
+      - name: vcpkg binary cache
+        uses: actions/cache@v4
+        with:
+          path: ~/AppData/Local/vcpkg/archives
+          key: ${{ github.job }}-vcpkg-binary-${{ hashFiles('vcpkg_commit', 'msbuild_version', 'toolset_version', 'build_msvc/vcpkg.json') }}
+
+      - name: Generate project files
+        run: py -3 build_msvc\msvc-autogen.py
+
+      - name: Build
+        shell: cmd
+        run: |
+          ccache --zero-stats
+          msbuild build_msvc\bitcoin.sln -property:CLToolPath=C:\ccache;CLToolExe=cl.exe;UseMultiToolTask=true;Configuration=Release -maxCpuCount -verbosity:minimal -noLogo
+
+      - name: Ccache stats
+        run: ccache --show-stats
+
+      - name: Save Ccache cache
+        uses: actions/cache/save@v4
+        if: github.event_name != 'pull_request' && steps.ccache-cache.outputs.cache-hit != 'true'
+        with:
+          path: ~/AppData/Local/ccache
+          # https://github.com/actions/cache/blob/main/tips-and-workarounds.md#update-a-cache
+          key: ${{ github.job }}-ccache-${{ github.run_id }}
+
+      - name: Run unit tests
+        run: src\test_bitcoin.exe -l test_suite
+
+      - name: Run benchmarks
+        run: src\bench_bitcoin.exe -sanity-check
+
+      - name: Run util tests
+        run: py -3 test\util\test_runner.py
+
+      - name: Run rpcauth test
+        run: py -3 test\util\rpcauth-test.py
+
+      - name: Run functional tests
+        env:
+          TEST_RUNNER_EXTRA: ${{ github.event_name != 'pull_request' && '--extended' || '' }}
+        shell: cmd
+        run: py -3 test\functional\test_runner.py --jobs %NUMBER_OF_PROCESSORS% --ci --quiet --tmpdirprefix=%RUNNER_TEMP% --combinedlogslen=99999999 --timeout-factor=%TEST_RUNNER_TIMEOUT_FACTOR% %TEST_RUNNER_EXTRA%

.gitignore

@@ -74,7 +74,7 @@ src/qt/bitcoin-qt.includes
 
 *.log
 *.trs
-*.dmg
+*.zip
 
 *.json.h
 *.raw.h

.python-version

@@ -1 +1 @@
-3.7.16
+3.9.18

.style.yapf

@@ -107,7 +107,7 @@ each_dict_entry_on_separate_line=True
 i18n_comment=
 
 # The i18n function call names. The presence of this function stops
-# reformattting on that line, because the string it has cannot be moved
+# reformatting on that line, because the string it has cannot be moved
 # away from the i18n comment.
 i18n_function_call=
 

.tx/config

@@ -1,7 +1,7 @@
 [main]
 host = https://www.transifex.com
 
-[o:bitcoin:p:bitcoin:r:qt-translation-025x]
+[o:bitcoin:p:bitcoin:r:qt-translation-027x]
 file_filter = src/qt/locale/bitcoin_<lang>.xlf
 source_file = src/qt/locale/bitcoin_en.xlf
 source_lang = en

CONTRIBUTING.md

@@ -66,9 +66,10 @@ Discussion about codebase improvements happens in GitHub issues and pull
 requests.
 
 The developer
-[mailing list](https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev)
+[mailing list](https://groups.google.com/g/bitcoindev)
 should be used to discuss complicated or controversial consensus or P2P protocol changes before working on
 a patch set.
+Archives can be found on [https://gnusha.org/pi/bitcoindev/](https://gnusha.org/pi/bitcoindev/).
 
 
 Contributor Workflow

COPYING

@@ -1,7 +1,7 @@
 The MIT License (MIT)
 
-Copyright (c) 2009-2023 The Bitcoin Core developers
-Copyright (c) 2009-2023 Bitcoin Developers
+Copyright (c) 2009-2024 The Bitcoin Core developers
+Copyright (c) 2009-2024 Bitcoin Developers
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

Makefile.am

@@ -14,8 +14,6 @@ endif
 .PHONY: deploy FORCE
 .INTERMEDIATE: $(COVERAGE_INFO)
 
-export PYTHONPATH
-
 if BUILD_BITCOIN_LIBS
 pkgconfigdir = $(libdir)/pkgconfig
 pkgconfig_DATA = libbitcoinconsensus.pc
@@ -37,7 +35,7 @@ space := $(empty) $(empty)
 
 OSX_APP=Bitcoin-Qt.app
 OSX_VOLNAME = $(subst $(space),-,$(PACKAGE_NAME))
-OSX_DMG = $(OSX_VOLNAME).dmg
+OSX_ZIP = $(OSX_VOLNAME).zip
 OSX_DEPLOY_SCRIPT=$(top_srcdir)/contrib/macdeploy/macdeployqtplus
 OSX_INSTALLER_ICONS=$(top_srcdir)/src/qt/res/icons/bitcoin.icns
 OSX_PLIST=$(top_builddir)/share/qt/Info.plist #not installed
@@ -124,15 +122,16 @@ osx_volname:
 	echo $(OSX_VOLNAME) >$@
 
 if BUILD_DARWIN
-$(OSX_DMG): $(OSX_APP_BUILT) $(OSX_PACKAGING)
-	$(PYTHON) $(OSX_DEPLOY_SCRIPT) $(OSX_APP) $(OSX_VOLNAME) -translations-dir=$(QT_TRANSLATION_DIR) -dmg
+$(OSX_ZIP): $(OSX_APP_BUILT) $(OSX_PACKAGING)
+	$(PYTHON) $(OSX_DEPLOY_SCRIPT) $(OSX_APP) $(OSX_VOLNAME) -translations-dir=$(QT_TRANSLATION_DIR) -zip
 
-deploydir: $(OSX_DMG)
+deploydir: $(OSX_ZIP)
 else !BUILD_DARWIN
 APP_DIST_DIR=$(top_builddir)/dist
 
-$(OSX_DMG): deploydir
-	$(XORRISOFS) -D -l -V "$(OSX_VOLNAME)" -no-pad -r -dir-mode 0755 -o $@ $(APP_DIST_DIR) -- $(if $(SOURCE_DATE_EPOCH),-volume_date all_file_dates =$(SOURCE_DATE_EPOCH))
+$(OSX_ZIP): deploydir
+	if [ -n "$(SOURCE_DATE_EPOCH)" ]; then find $(APP_DIST_DIR) -exec touch -d @$(SOURCE_DATE_EPOCH) {} +; fi
+	cd $(APP_DIST_DIR) && find . | sort | $(ZIP) -X@ $@
 
 $(APP_DIST_DIR)/$(OSX_APP)/Contents/MacOS/Bitcoin-Qt: $(OSX_APP_BUILT) $(OSX_PACKAGING)
 	INSTALL_NAME_TOOL=$(INSTALL_NAME_TOOL) OTOOL=$(OTOOL) STRIP=$(STRIP) $(PYTHON) $(OSX_DEPLOY_SCRIPT) $(OSX_APP) $(OSX_VOLNAME) -translations-dir=$(QT_TRANSLATION_DIR)
@@ -140,7 +139,7 @@ $(APP_DIST_DIR)/$(OSX_APP)/Contents/MacOS/Bitcoin-Qt: $(OSX_APP_BUILT) $(OSX_PAC
 deploydir: $(APP_DIST_DIR)/$(OSX_APP)/Contents/MacOS/Bitcoin-Qt
 endif !BUILD_DARWIN
 
-deploy: $(OSX_DMG)
+deploy: $(OSX_ZIP)
 endif
 
 $(BITCOIN_QT_BIN): FORCE
@@ -188,7 +187,7 @@ $(COV_TOOL_WRAPPER):
 	@chmod +x $(COV_TOOL_WRAPPER)
 
 baseline.info: $(COV_TOOL_WRAPPER)
-	$(LCOV) -c -i -d $(abs_builddir)/src -o $@
+	$(LCOV) $(LCOV_OPTS) -c -i -d $(abs_builddir)/src -o $@
 
 baseline_filtered.info: baseline.info
 	$(abs_builddir)/contrib/filter-lcov.py $(LCOV_FILTER_PATTERN) $< $@
@@ -222,13 +221,13 @@ functional_test_filtered.info: functional_test.info
 	$(LCOV) -a $@ $(LCOV_OPTS) -o $@
 
 fuzz_coverage.info: fuzz_filtered.info
-	$(LCOV) -a $(LCOV_OPTS) baseline_filtered.info -a fuzz_filtered.info -o $@ | $(GREP) "\%" | $(AWK) '{ print substr($$3,2,50) "/" $$5 }' > coverage_percent.txt
+	$(LCOV) $(LCOV_OPTS) -a baseline_filtered.info -a fuzz_filtered.info -o $@ | $(GREP) "\%" | $(AWK) '{ print substr($$3,2,50) "/" $$5 }' > coverage_percent.txt
 
 test_bitcoin_coverage.info: baseline_filtered.info test_bitcoin_filtered.info
-	$(LCOV) -a $(LCOV_OPTS) baseline_filtered.info -a test_bitcoin_filtered.info -o $@
+	$(LCOV) $(LCOV_OPTS) -a baseline_filtered.info -a test_bitcoin_filtered.info -o $@
 
 total_coverage.info: test_bitcoin_filtered.info functional_test_filtered.info
-	$(LCOV) -a $(LCOV_OPTS) baseline_filtered.info -a test_bitcoin_filtered.info -a functional_test_filtered.info -o $@ | $(GREP) "\%" | $(AWK) '{ print substr($$3,2,50) "/" $$5 }' > coverage_percent.txt
+	$(LCOV) $(LCOV_OPTS) -a baseline_filtered.info -a test_bitcoin_filtered.info -a functional_test_filtered.info -o $@ | $(GREP) "\%" | $(AWK) '{ print substr($$3,2,50) "/" $$5 }' > coverage_percent.txt
 
 fuzz.coverage/.dirstamp: fuzz_coverage.info
 	$(GENHTML) -s $(LCOV_OPTS) $< -o $(@D)
@@ -311,9 +310,13 @@ EXTRA_DIST += \
     test/util/data/txcreatesignv1.hex \
     test/util/data/txcreatesignv1.json \
     test/util/data/txcreatesignv2.hex \
+    test/util/data/txreplace1.hex \
+    test/util/data/txreplacenoinputs.hex \
+    test/util/data/txreplaceomittedn.hex \
+    test/util/data/txreplacesingleinput.hex \
     test/util/rpcauth-test.py
 
-CLEANFILES = $(OSX_DMG) $(BITCOIN_WIN_INSTALLER)
+CLEANFILES = $(OSX_ZIP) $(BITCOIN_WIN_INSTALLER)
 
 DISTCHECK_CONFIGURE_FLAGS = --enable-man
 

SECURITY.md

@@ -15,6 +15,6 @@ The following keys may be used to communicate sensitive information to developer
 |------|-------------|
 | Pieter Wuille | 133E AC17 9436 F14A 5CF1  B794 860F EB80 4E66 9320 |
 | Michael Ford | E777 299F C265 DD04 7930  70EB 944D 35F9 AC3D B76A |
-| Andrew Chow | 1528 1230 0785 C964 44D3  334D 1756 5732 E08E 5E41 |
+| Ava Chow | 1528 1230 0785 C964 44D3  334D 1756 5732 E08E 5E41 |
 
 You can import a key by running the following command with that individual’s fingerprint: `gpg --keyserver hkps://keys.openpgp.org --recv-keys "<fingerprint>"` Ensure that you put quotes around fingerprints containing spaces.

autogen.sh

@@ -14,3 +14,16 @@ fi
 command -v autoreconf >/dev/null || \
   (echo "configuration failed, please install autoconf first" && exit 1)
 autoreconf --install --force --warnings=all
+
+if expr "'$(build-aux/config.guess --timestamp)" \< "'$(depends/config.guess --timestamp)" > /dev/null; then
+  chmod ug+w build-aux/config.guess
+  chmod ug+w src/secp256k1/build-aux/config.guess
+  cp depends/config.guess build-aux
+  cp depends/config.guess src/secp256k1/build-aux
+fi
+if expr "'$(build-aux/config.sub --timestamp)" \< "'$(depends/config.sub --timestamp)" > /dev/null; then
+  chmod ug+w build-aux/config.sub
+  chmod ug+w src/secp256k1/build-aux/config.sub
+  cp depends/config.sub build-aux
+  cp depends/config.sub src/secp256k1/build-aux
+fi

build-aux/m4/ax_cxx_compile_stdcxx.m4

@@ -983,7 +983,7 @@ m4_define([_AX_CXX_COMPILE_STDCXX_testbody_new_in_20], [[
 
 #error "This is not a C++ compiler"
 
-#elif __cplusplus < 202002L
+#elif __cplusplus < 201709L  // Temporary patch on top of upstream to allow g++-10
 
 #error "This is not a C++20 compiler"
 

build-aux/m4/bitcoin_runtime_lib.m4

@@ -1,42 +0,0 @@
-# On some platforms clang builtin implementations
-# require compiler-rt as a runtime library to use.
-#
-# See:
-# - https://bugs.llvm.org/show_bug.cgi?id=28629
-
-m4_define([_CHECK_RUNTIME_testbody], [[
-  bool f(long long x, long long y, long long* p)
-  {
-    return __builtin_mul_overflow(x, y, p);
-  }
-  int main() { return 0; }
-]])
-
-AC_DEFUN([CHECK_RUNTIME_LIB], [
-
-  AC_LANG_PUSH([C++])
-
-  AC_MSG_CHECKING([for __builtin_mul_overflow])
-  AC_LINK_IFELSE(
-    [AC_LANG_SOURCE([_CHECK_RUNTIME_testbody])],
-    [
-      AC_MSG_RESULT([yes])
-      AC_DEFINE([HAVE_BUILTIN_MUL_OVERFLOW], [1], [Define if you have a working __builtin_mul_overflow])
-    ],
-    [
-      ax_check_save_flags="$LDFLAGS"
-      LDFLAGS="$LDFLAGS --rtlib=compiler-rt -lgcc_s"
-      AC_LINK_IFELSE(
-        [AC_LANG_SOURCE([_CHECK_RUNTIME_testbody])],
-        [
-          AC_MSG_RESULT([yes, with additional linker flags])
-          RUNTIME_LDFLAGS="--rtlib=compiler-rt -lgcc_s"
-          AC_DEFINE([HAVE_BUILTIN_MUL_OVERFLOW], [1], [Define if you have a working __builtin_mul_overflow])
-        ],
-        [AC_MSG_RESULT([no])])
-      LDFLAGS="$ax_check_save_flags"
-    ])
-
-  AC_LANG_POP
-  AC_SUBST([RUNTIME_LDFLAGS])
-])

build-aux/m4/l_atomic.m4

@@ -4,8 +4,10 @@ dnl permitted in any medium without royalty provided the copyright notice
 dnl and this notice are preserved. This file is offered as-is, without any
 dnl warranty.
 
-# Some versions of gcc/libstdc++ require linking with -latomic if
-# using the C++ atomic library.
+# Clang, when building for 32-bit,
+# and linking against libstdc++, requires linking with
+# -latomic if using the C++ atomic library.
+# Can be tested with: clang++ -std=c++20 test.cpp -m32
 #
 # Sourced from http://bugs.debian.org/797228
 
@@ -22,9 +24,14 @@ m4_define([_CHECK_ATOMIC_testbody], [[
 
     std::atomic<std::chrono::seconds> t{0s};
     t.store(2s);
+    auto t1 = t.load();
+    t.compare_exchange_strong(t1, 3s);
+
+    std::atomic<double> d{};
+    d.store(3.14);
+    auto d1 = d.load();
 
     std::atomic<int64_t> a{};
-
     int64_t v = 5;
     int64_t r = a.fetch_add(v);
     return static_cast<int>(r);

build-aux/m4/l_filesystem.m4

@@ -1,47 +0,0 @@
-dnl Copyright (c) 2022 The Bitcoin Core developers
-dnl Distributed under the MIT software license, see the accompanying
-dnl file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-# GCC 8.1 and earlier requires -lstdc++fs
-# Clang 8.0.0 (libc++) and earlier requires -lc++fs
-
-m4_define([_CHECK_FILESYSTEM_testbody], [[
-  #include <filesystem>
-
-  namespace fs = std::filesystem;
-
-  int main() {
-    (void)fs::current_path().root_name();
-    return 0;
-  }
-]])
-
-AC_DEFUN([CHECK_FILESYSTEM], [
-
-  AC_LANG_PUSH(C++)
-
-  AC_MSG_CHECKING([whether std::filesystem can be used without link library])
-
-  AC_LINK_IFELSE([AC_LANG_SOURCE([_CHECK_FILESYSTEM_testbody])],[
-      AC_MSG_RESULT([yes])
-    ],[
-      AC_MSG_RESULT([no])
-      SAVED_LIBS="$LIBS"
-      LIBS="$SAVED_LIBS -lstdc++fs"
-      AC_MSG_CHECKING([whether std::filesystem needs -lstdc++fs])
-      AC_LINK_IFELSE([AC_LANG_SOURCE([_CHECK_FILESYSTEM_testbody])],[
-          AC_MSG_RESULT([yes])
-        ],[
-          AC_MSG_RESULT([no])
-          AC_MSG_CHECKING([whether std::filesystem needs -lc++fs])
-          LIBS="$SAVED_LIBS -lc++fs"
-          AC_LINK_IFELSE([AC_LANG_SOURCE([_CHECK_FILESYSTEM_testbody])],[
-            AC_MSG_RESULT([yes])
-        ],[
-            AC_MSG_FAILURE([cannot figure out how to use std::filesystem])
-          ])
-        ])
-    ])
-
-  AC_LANG_POP
-])

build_msvc/README.md

@@ -32,7 +32,9 @@ Qt
 ---------------------
 To build Bitcoin Core with the GUI, a static build of Qt is required.
 
-1. Download a single ZIP archive of Qt source code from https://download.qt.io/official_releases/qt/ (e.g., [`qt-everywhere-opensource-src-5.15.5.zip`](https://download.qt.io/official_releases/qt/5.15/5.15.5/single/qt-everywhere-opensource-src-5.15.5.zip)), and expand it into a dedicated folder. The following instructions assume that this folder is `C:\dev\qt-source`.
+1. Download a single ZIP archive of Qt source code from https://download.qt.io/official_releases/qt/ (e.g., [`qt-everywhere-opensource-src-5.15.11.zip`](https://download.qt.io/official_releases/qt/5.15/5.15.11/single/qt-everywhere-opensource-src-5.15.11.zip)), and expand it into a dedicated folder. The following instructions assume that this folder is `C:\dev\qt-source`.
+
+> 💡 **Tip:** If you use the default path with "Extract All" for the Qt source code zip file, and end up with something like `C:\dev\qt-everywhere-opensource-src-5.15.11\qt-everywhere-src-5.15.11`, you are likely to encounter a "path too long" error when building. To fix the problem move the source files to a shorter path such as the recommended `C:\dev\qt-source`.
 
 2. Open "x64 Native Tools Command Prompt for VS 2022", and input the following commands:
 ```cmd

build_msvc/bench_bitcoin/bench_bitcoin.vcxproj.in

@@ -12,7 +12,7 @@
 @SOURCE_FILES@
   </ItemGroup>
   <ItemGroup>
-    <ProjectReference Include="..\libbitcoinconsensus\libbitcoinconsensus.vcxproj">
+    <ProjectReference Include="..\libbitcoin_consensus\libbitcoin_consensus.vcxproj">
       <Project>{2b384fa8-9ee1-4544-93cb-0d733c25e8ce}</Project>
     </ProjectReference>
     <ProjectReference Include="..\libbitcoin_common\libbitcoin_common.vcxproj">

build_msvc/bitcoin-qt/bitcoin-qt.vcxproj

@@ -13,7 +13,7 @@
     <ResourceCompile Include="..\..\src\qt\res\bitcoin-qt-res.rc" />
   </ItemGroup>
   <ItemGroup>
-    <ProjectReference Include="..\libbitcoinconsensus\libbitcoinconsensus.vcxproj">
+    <ProjectReference Include="..\libbitcoin_consensus\libbitcoin_consensus.vcxproj">
       <Project>{2b384fa8-9ee1-4544-93cb-0d733c25e8ce}</Project>
     </ProjectReference>
     <ProjectReference Include="..\libbitcoin_cli\libbitcoin_cli.vcxproj">
@@ -58,7 +58,7 @@
     <Link>
       <SubSystem>Windows</SubSystem>
       <AdditionalDependencies>$(QtReleaseLibraries);%(AdditionalDependencies)</AdditionalDependencies>
-      <AdditionalOptions>/ignore:4206 /LTCG:OFF</AdditionalOptions>
+      <AdditionalOptions>/LTCG:OFF</AdditionalOptions>
     </Link>
     <ResourceCompile>
       <AdditionalIncludeDirectories>..\..\src;</AdditionalIncludeDirectories>
@@ -72,7 +72,6 @@
     </ClCompile>
     <Link>
       <AdditionalDependencies>$(QtDebugLibraries);%(AdditionalDependencies)</AdditionalDependencies>
-      <AdditionalOptions>/ignore:4206</AdditionalOptions>
     </Link>
     <ResourceCompile>
       <AdditionalIncludeDirectories>..\..\src;</AdditionalIncludeDirectories>

build_msvc/bitcoin-tx/bitcoin-tx.vcxproj

@@ -12,7 +12,7 @@
     <ClCompile Include="..\..\src\bitcoin-tx.cpp" />
   </ItemGroup>
   <ItemGroup>
-    <ProjectReference Include="..\libbitcoinconsensus\libbitcoinconsensus.vcxproj">
+    <ProjectReference Include="..\libbitcoin_consensus\libbitcoin_consensus.vcxproj">
       <Project>{2b384fa8-9ee1-4544-93cb-0d733c25e8ce}</Project>
     </ProjectReference>
     <ProjectReference Include="..\libbitcoin_common\libbitcoin_common.vcxproj">

build_msvc/bitcoin-util/bitcoin-util.vcxproj

@@ -12,7 +12,7 @@
     <ClCompile Include="..\..\src\bitcoin-util.cpp" />
   </ItemGroup>
   <ItemGroup>
-    <ProjectReference Include="..\libbitcoinconsensus\libbitcoinconsensus.vcxproj">
+    <ProjectReference Include="..\libbitcoin_consensus\libbitcoin_consensus.vcxproj">
       <Project>{2b384fa8-9ee1-4544-93cb-0d733c25e8ce}</Project>
     </ProjectReference>
     <ProjectReference Include="..\libbitcoin_common\libbitcoin_common.vcxproj">

build_msvc/bitcoin-wallet/bitcoin-wallet.vcxproj

@@ -15,7 +15,7 @@
     </ClCompile>
   </ItemGroup>
   <ItemGroup>
-    <ProjectReference Include="..\libbitcoinconsensus\libbitcoinconsensus.vcxproj">
+    <ProjectReference Include="..\libbitcoin_consensus\libbitcoin_consensus.vcxproj">
       <Project>{2b384fa8-9ee1-4544-93cb-0d733c25e8ce}</Project>
     </ProjectReference>
     <ProjectReference Include="..\libbitcoin_common\libbitcoin_common.vcxproj">

build_msvc/bitcoin.sln

@@ -2,7 +2,7 @@ Microsoft Visual Studio Solution File, Format Version 12.00
 # Visual Studio Version 16
 VisualStudioVersion = 16.0.28803.452
 MinimumVisualStudioVersion = 10.0.40219.1
-Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "libbitcoinconsensus", "libbitcoinconsensus\libbitcoinconsensus.vcxproj", "{2B384FA8-9EE1-4544-93CB-0D733C25E8CE}"
+Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "libbitcoin_consensus", "libbitcoin_consensus\libbitcoin_consensus.vcxproj", "{2B384FA8-9EE1-4544-93CB-0D733C25E8CE}"
 EndProject
 Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "bitcoind", "bitcoind\bitcoind.vcxproj", "{D4513DDF-6013-44DC-ADCC-12EAF6D1F038}"
 EndProject

build_msvc/bitcoin_config.h.in

@@ -38,15 +38,9 @@
 /* Define to 1 to enable SQLite wallet */
 #define USE_SQLITE 1
 
-/* Define to 1 to enable ZMQ functions */
+/* Define this symbol to enable ZMQ functions */
 #define ENABLE_ZMQ 1
 
-/* define if external signer support is enabled (requires Boost::Process) */
-#define ENABLE_EXTERNAL_SIGNER /**/
-
-/* Define this symbol if the consensus lib has been built */
-#define HAVE_CONSENSUS_LIB 1
-
 /* Define to 1 if you have the declaration of `be16toh', and to 0 if you
    don't. */
 #define HAVE_DECL_BE16TOH 0

build_msvc/bitcoind/bitcoind.vcxproj

@@ -15,7 +15,7 @@
     </ClCompile>
   </ItemGroup>
   <ItemGroup>
-    <ProjectReference Include="..\libbitcoinconsensus\libbitcoinconsensus.vcxproj">
+    <ProjectReference Include="..\libbitcoin_consensus\libbitcoin_consensus.vcxproj">
       <Project>{2b384fa8-9ee1-4544-93cb-0d733c25e8ce}</Project>
     </ProjectReference>
     <ProjectReference Include="..\libbitcoin_common\libbitcoin_common.vcxproj">
@@ -73,16 +73,20 @@
                   Replace="@USE_SQLITE_TRUE@" By=""></ReplaceInFile>
     <ReplaceInFile FilePath="$(ConfigIniOut)"
                   Replace="@BUILD_BITCOIN_CLI_TRUE@" By=""></ReplaceInFile>
+    <ReplaceInFile FilePath="$(ConfigIniOut)"
+                  Replace="@BUILD_BITCOIN_UTIL_TRUE@" By=""></ReplaceInFile>
     <ReplaceInFile FilePath="$(ConfigIniOut)"
                   Replace="@BUILD_BITCOIN_WALLET_TRUE@" By=""></ReplaceInFile>
     <ReplaceInFile FilePath="$(ConfigIniOut)"
                   Replace="@BUILD_BITCOIND_TRUE@" By=""></ReplaceInFile>
     <ReplaceInFile FilePath="$(ConfigIniOut)"
-                  Replace="@ENABLE_FUZZ_TRUE@" By=""></ReplaceInFile>
+                  Replace="@ENABLE_FUZZ_BINARY_TRUE@" By="#"></ReplaceInFile>
     <ReplaceInFile FilePath="$(ConfigIniOut)"
                   Replace="@ENABLE_ZMQ_TRUE@" By=""></ReplaceInFile>
     <ReplaceInFile FilePath="$(ConfigIniOut)"
-                  Replace="@ENABLE_EXTERNAL_SIGNER_TRUE@" By=""></ReplaceInFile>
+                  Replace="@ENABLE_EXTERNAL_SIGNER_TRUE@" By="#"></ReplaceInFile>
+    <ReplaceInFile FilePath="$(ConfigIniOut)"
+                  Replace="@ENABLE_USDT_TRACEPOINTS_TRUE@" By="#"></ReplaceInFile>
   </Target>
   <Import Project="..\common.vcxproj" />
 </Project>

build_msvc/common.init.vcxproj.in

@@ -57,7 +57,7 @@
 
 <ItemDefinitionGroup Condition="'$(Configuration)'=='Release'">
     <ClCompile>
-      <Optimization>Disabled</Optimization>
+      <Optimization>MaxSpeed</Optimization>
       <WholeProgramOptimization>false</WholeProgramOptimization>
       <FunctionLevelLinking>true</FunctionLevelLinking>
       <IntrinsicFunctions>true</IntrinsicFunctions>
@@ -90,7 +90,7 @@
       <AdditionalOptions>/utf-8 /Zc:__cplusplus /std:c++20 %(AdditionalOptions)</AdditionalOptions>
       <DisableSpecificWarnings>4018;4244;4267;4715;4805</DisableSpecificWarnings>
       <TreatWarningAsError>true</TreatWarningAsError>
-      <PreprocessorDefinitions>_SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING;ZMQ_STATIC;NOMINMAX;WIN32;HAVE_CONFIG_H;_CRT_SECURE_NO_WARNINGS;_SCL_SECURE_NO_WARNINGS;_CONSOLE;_WIN32_WINNT=0x0601;_WIN32_IE=0x0501;WIN32_LEAN_AND_MEAN;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <PreprocessorDefinitions>_SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING;SECP256K1_STATIC;ZMQ_STATIC;NOMINMAX;WIN32;HAVE_CONFIG_H;_CRT_SECURE_NO_WARNINGS;_CONSOLE;_WIN32_WINNT=0x0601;_WIN32_IE=0x0501;WIN32_LEAN_AND_MEAN;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <AdditionalIncludeDirectories>..\..\src;..\..\src\minisketch\include;..\..\src\univalue\include;..\..\src\secp256k1\include;..\..\src\leveldb\include;..\..\src\leveldb\helpers\memenv;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
     </ClCompile>
     <Link>
@@ -98,9 +98,6 @@
       <AdditionalDependencies>Iphlpapi.lib;ws2_32.lib;Shlwapi.lib;kernel32.lib;user32.lib;gdi32.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies)</AdditionalDependencies>
       <RandomizedBaseAddress>true</RandomizedBaseAddress>
     </Link>
-    <Lib>
-      <AdditionalOptions>/ignore:4221</AdditionalOptions>
-    </Lib>
   </ItemDefinitionGroup>
   <Import Project="common.init.vcxproj.user" Condition="Exists('common.init.vcxproj.user')" />
 </Project>

build_msvc/common.qt.init.vcxproj

@@ -13,4 +13,10 @@
     <QtDebugLibraries>$(QtPluginsLibraryDir)\platforms\qwindowsd.lib;$(QtPluginsLibraryDir)\platforms\qminimald.lib;$(QtPluginsLibraryDir)\styles\qwindowsvistastyled.lib;$(QtLibraryDir)\*d.lib;Wtsapi32.lib;crypt32.lib;userenv.lib;netapi32.lib;imm32.lib;Dwmapi.lib;version.lib;winmm.lib;UxTheme.lib</QtDebugLibraries>
   </PropertyGroup>
 
+  <ItemDefinitionGroup>
+    <ClCompile>
+      <PreprocessorDefinitions>QT_NO_KEYWORDS;QT_USE_QSTRINGBUILDER;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+    </ClCompile>
+  </ItemDefinitionGroup>
+
 </Project>

build_msvc/libbitcoin_consensus/libbitcoin_consensus.vcxproj

@@ -11,15 +11,6 @@
     <ClCompile Include="..\..\src\arith_uint256.cpp" />
     <ClCompile Include="..\..\src\consensus\merkle.cpp" />
     <ClCompile Include="..\..\src\consensus\tx_check.cpp" />
-    <ClCompile Include="..\..\src\crypto\aes.cpp" />
-    <ClCompile Include="..\..\src\crypto\chacha20.cpp" />
-    <ClCompile Include="..\..\src\crypto\hmac_sha256.cpp" />
-    <ClCompile Include="..\..\src\crypto\hmac_sha512.cpp" />
-    <ClCompile Include="..\..\src\crypto\ripemd160.cpp" />
-    <ClCompile Include="..\..\src\crypto\sha1.cpp" />
-    <ClCompile Include="..\..\src\crypto\sha256.cpp" />
-    <ClCompile Include="..\..\src\crypto\sha256_sse4.cpp" />
-    <ClCompile Include="..\..\src\crypto\sha512.cpp" />
     <ClCompile Include="..\..\src\hash.cpp" />
     <ClCompile Include="..\..\src\primitives\block.cpp" />
     <ClCompile Include="..\..\src\primitives\transaction.cpp" />

build_msvc/libleveldb/libleveldb.vcxproj

@@ -51,7 +51,7 @@
   <ItemDefinitionGroup>
      <ClCompile>
        <PreprocessorDefinitions>HAVE_CRC32C=0;HAVE_SNAPPY=0;LEVELDB_IS_BIG_ENDIAN=0;_UNICODE;UNICODE;_CRT_NONSTDC_NO_DEPRECATE;LEVELDB_PLATFORM_WINDOWS;%(PreprocessorDefinitions)</PreprocessorDefinitions>
-       <DisableSpecificWarnings>4244;4267</DisableSpecificWarnings>
+       <DisableSpecificWarnings>4244;4267;4722</DisableSpecificWarnings>
        <AdditionalIncludeDirectories>..\..\src\leveldb;..\..\src\leveldb\include;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
      </ClCompile>
   </ItemDefinitionGroup>

build_msvc/libminisketch/libminisketch.vcxproj

@@ -28,7 +28,7 @@
   </ItemGroup>
   <ItemDefinitionGroup>
     <ClCompile>
-      <DisableSpecificWarnings>4060;4065;4146;4244;4267;4554</DisableSpecificWarnings>
+      <DisableSpecificWarnings>4060;4065;4146;4244;4267</DisableSpecificWarnings>
       <PreprocessorDefinitions>HAVE_CLMUL;DISABLE_DEFAULT_FIELDS;ENABLE_FIELD_32;%(PreprocessorDefinitions)</PreprocessorDefinitions>
     </ClCompile>
   </ItemDefinitionGroup>

build_msvc/libsecp256k1/libsecp256k1.vcxproj

@@ -14,9 +14,10 @@
   </ItemGroup>
   <ItemDefinitionGroup>
     <ClCompile>
-      <PreprocessorDefinitions>ENABLE_MODULE_RECOVERY;ENABLE_MODULE_EXTRAKEYS;ENABLE_MODULE_SCHNORRSIG;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <PreprocessorDefinitions>ENABLE_MODULE_RECOVERY;ENABLE_MODULE_EXTRAKEYS;ENABLE_MODULE_SCHNORRSIG;ENABLE_MODULE_ELLSWIFT;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <UndefinePreprocessorDefinitions>USE_ASM_X86_64;%(UndefinePreprocessorDefinitions)</UndefinePreprocessorDefinitions>
       <AdditionalIncludeDirectories>..\..\src\secp256k1;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
-      <DisableSpecificWarnings>4146;4244;4267;4334</DisableSpecificWarnings>
+      <DisableSpecificWarnings>4146;4244;4267</DisableSpecificWarnings>
     </ClCompile>
   </ItemDefinitionGroup>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />

build_msvc/libsecp256k1_config.h

@@ -1,15 +0,0 @@
-/**********************************************************************
- * Copyright (c) 2013, 2014 Pieter Wuille                             *
- * Distributed under the MIT software license, see the accompanying   *
- * file COPYING or http://www.opensource.org/licenses/mit-license.php.*
- **********************************************************************/
-
-#ifndef BITCOIN_LIBSECP256K1_CONFIG_H
-#define BITCOIN_LIBSECP256K1_CONFIG_H
-
-#undef USE_ASM_X86_64
-
-#define ECMULT_GEN_PREC_BITS 4
-#define ECMULT_WINDOW_SIZE 15
-
-#endif // BITCOIN_LIBSECP256K1_CONFIG_H

build_msvc/libtest_util/libtest_util.vcxproj.in

@@ -8,6 +8,7 @@
     <ConfigurationType>StaticLibrary</ConfigurationType>
   </PropertyGroup>
   <ItemGroup>
+    <ClCompile Include="..\..\src\wallet\test\util.cpp" />
 @SOURCE_FILES@
   </ItemGroup>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />

build_msvc/msvc-autogen.py

@@ -111,7 +111,6 @@ def main():
         set_properties(vcxproj_filename, '@SOURCE_FILES@\n', content)
     parse_config_into_btc_config()
     copyfile(os.path.join(SOURCE_DIR,'../build_msvc/bitcoin_config.h'), os.path.join(SOURCE_DIR, 'config/bitcoin-config.h'))
-    copyfile(os.path.join(SOURCE_DIR,'../build_msvc/libsecp256k1_config.h'), os.path.join(SOURCE_DIR, 'secp256k1/src/libsecp256k1-config.h'))
 
 if __name__ == '__main__':
     main()

build_msvc/test_bitcoin-qt/test_bitcoin-qt.vcxproj

@@ -9,7 +9,6 @@
   </PropertyGroup>
   <ItemGroup>
     <ClCompile Include="..\..\src\init\bitcoin-qt.cpp" />
-    <ClCompile Include="..\..\src\test\util\setup_common.cpp" />
     <ClCompile Include="..\..\src\qt\test\addressbooktests.cpp" />
     <ClCompile Include="..\..\src\qt\test\apptests.cpp" />
     <ClCompile Include="..\..\src\qt\test\optiontests.cpp" />
@@ -27,7 +26,7 @@
     <ClCompile Include="$(GeneratedFilesOutDir)\moc\moc_wallettests.cpp" />
   </ItemGroup>
   <ItemGroup>
-    <ProjectReference Include="..\libbitcoinconsensus\libbitcoinconsensus.vcxproj">
+    <ProjectReference Include="..\libbitcoin_consensus\libbitcoin_consensus.vcxproj">
       <Project>{2b384fa8-9ee1-4544-93cb-0d733c25e8ce}</Project>
     </ProjectReference>
     <ProjectReference Include="..\libbitcoin_cli\libbitcoin_cli.vcxproj">
@@ -77,7 +76,7 @@
     </ClCompile>
     <Link>
       <AdditionalDependencies>$(QtLibraryDir)\Qt5Test.lib;$(QtReleaseLibraries);%(AdditionalDependencies)</AdditionalDependencies>
-      <AdditionalOptions>/ignore:4206 /LTCG:OFF</AdditionalOptions>
+      <AdditionalOptions>/LTCG:OFF</AdditionalOptions>
     </Link>
   </ItemDefinitionGroup>
 
@@ -87,7 +86,6 @@
     </ClCompile>
     <Link>
       <AdditionalDependencies>$(QtDebugLibraries);%(AdditionalDependencies)</AdditionalDependencies>
-      <AdditionalOptions>/ignore:4206</AdditionalOptions>
     </Link>
   </ItemDefinitionGroup>
   <ItemGroup>

build_msvc/test_bitcoin/test_bitcoin.vcxproj

@@ -10,7 +10,7 @@
   </PropertyGroup>
   <ItemGroup>
     <ClCompile Include="..\..\src\test\*_properties.cpp" />
-    <ClCompile Include="..\..\src\test\*_tests.cpp" />
+    <ClCompile Include="..\..\src\test\*_tests.cpp" Exclude="..\..\src\test\ipc_tests.cpp" />
     <ClCompile Include="..\..\src\test\gen\*_gen.cpp" />
     <ClCompile Include="..\..\src\test\main.cpp" />
     <ClCompile Include="..\..\src\test\util\*.cpp" />
@@ -22,7 +22,7 @@
     <ProjectReference Include="..\libminisketch\libminisketch.vcxproj">
       <Project>{542007e3-be0d-4b0d-a6b0-aa8813e2558d}</Project>
     </ProjectReference>
-    <ProjectReference Include="..\libbitcoinconsensus\libbitcoinconsensus.vcxproj">
+    <ProjectReference Include="..\libbitcoin_consensus\libbitcoin_consensus.vcxproj">
       <Project>{2b384fa8-9ee1-4544-93cb-0d733c25e8ce}</Project>
     </ProjectReference>
     <ProjectReference Include="..\libbitcoin_cli\libbitcoin_cli.vcxproj">
@@ -59,6 +59,11 @@
       <Project>{18430fef-6b61-4c53-b396-718e02850f1b}</Project>
     </ProjectReference>
   </ItemGroup>
+  <ItemDefinitionGroup>
+     <ClCompile>
+       <DisableSpecificWarnings>4018;4244;4267;4703;4715;4805</DisableSpecificWarnings>
+     </ClCompile>
+  </ItemDefinitionGroup>
   <Target Name="RawBenchHeaderGen" BeforeTargets="PrepareForBuild">
     <PropertyGroup>
       <ErrorText>There was an error executing the JSON test header generation task.</ErrorText>
@@ -68,7 +73,7 @@
       <RawTestFile Include="..\..\src\test\data\*.raw" />
     </ItemGroup>
     <HeaderFromHexdump RawFilePath="%(RawTestFile.FullPath)" HeaderFilePath="%(RawTestFile.FullPath).h" SourceHeader="static unsigned const char %(RawTestFile.Filename)_raw[] = {" SourceFooter="};" />
-    <HeaderFromHexdump RawFilePath="%(JsonTestFile.FullPath)" HeaderFilePath="%(JsonTestFile.FullPath).h" SourceHeader="namespace json_tests{ static unsigned const char %(JsonTestFile.Filename)[] = {" SourceFooter="};}" />
+    <HeaderFromHexdump RawFilePath="%(JsonTestFile.FullPath)" HeaderFilePath="%(JsonTestFile.FullPath).h" SourceHeader="#include &lt;string&gt;&#x0D;&#x0A;namespace json_tests{ static const std::string %(JsonTestFile.Filename){" SourceFooter="};}" />
   </Target>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />

build_msvc/vcpkg.json

@@ -3,18 +3,15 @@
   "version-string": "1",
   "dependencies": [
     "berkeleydb",
+    "boost-date-time",
     "boost-multi-index",
-    "boost-process",
     "boost-signals2",
     "boost-test",
+    "libevent",
     "sqlite3",
-    {
-      "name": "libevent",
-      "features": ["thread"]
-    },
     "zeromq"
   ],
-  "builtin-baseline": "f14984af3738e69f197bf0e647a8dca12de92996",
+  "builtin-baseline": "9edb1b8e590cc086563301d735cae4b6e732d2d2",
   "overrides": [
     {
       "name": "libevent",

ci/README.md

@@ -14,22 +14,17 @@ testing compared to other parts of the codebase. If you want to keep the work tr
 system in a virtual machine with a Linux operating system of your choice.
 
 To allow for a wide range of tested environments, but also ensure reproducibility to some extent, the test stage
-requires `docker` to be installed. To install all requirements on Ubuntu, run
+requires `bash`, `docker`, and `python3` to be installed. To run on different architectures than the host `qemu` is also required. To install all requirements on Ubuntu, run
 
 ```
-sudo apt install docker.io bash
+sudo apt install bash docker.io python3 qemu-user-static
 ```
 
-To run the default test stage,
+It is recommended to run the ci system in a clean env. To run the test stage
+with a specific configuration,
 
 ```
-./ci/test_run_all.sh
-```
-
-To run the test stage with a specific configuration,
-
-```
-FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh
+env -i HOME="$HOME" PATH="$PATH" USER="$USER" bash -c 'FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh'
 ```
 
 ### Configurations
@@ -44,14 +39,11 @@ the system package manager to install build dependencies. This guarantees that
 the tester is using the same versions as the release builds, which also use
 `./depends`.
 
-If no `FILE_ENV` has been specified or values are left out, `00_setup_env.sh`
-is used as the default configuration with fallback values.
-
 It is also possible to force a specific configuration without modifying the
 file. For example,
 
 ```
-MAKEJOBS="-j1" FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh
+env -i HOME="$HOME" PATH="$PATH" USER="$USER" bash -c 'MAKEJOBS="-j1" FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh'
 ```
 
 The files starting with `0n` (`n` greater than 0) are the scripts that are run
@@ -60,5 +52,5 @@ in order.
 ### Cache
 
 In order to avoid rebuilding all dependencies for each build, the binaries are
-cached and re-used when possible. Changes in the dependency-generator will
+cached and reused when possible. Changes in the dependency-generator will
 trigger cache-invalidation and rebuilds as necessary.

ci/lint/04_install.sh

@@ -6,38 +6,52 @@
 
 export LC_ALL=C
 
+export PATH=$PWD/ci/retry:$PATH
+
 ${CI_RETRY_EXE} apt-get update
 # Lint dependencies:
+# - automake pkg-config libtool (for lint_includes_build_config)
 # - curl/xz-utils (to install shellcheck)
 # - git (used in many lint scripts)
 # - gpg (used by verify-commits)
-${CI_RETRY_EXE} apt-get install -y curl xz-utils git gpg
+${CI_RETRY_EXE} apt-get install -y automake pkg-config libtool curl xz-utils git gpg
 
-if [ -z "${SKIP_PYTHON_INSTALL}" ]; then
-    PYTHON_PATH=/tmp/python
-    if [ ! -d "${PYTHON_PATH}/bin" ]; then
-      (
-        git clone https://github.com/pyenv/pyenv.git
-        cd pyenv/plugins/python-build || exit 1
-        ./install.sh
-      )
-      # For dependencies see https://github.com/pyenv/pyenv/wiki#suggested-build-environment
-      ${CI_RETRY_EXE} apt-get install -y build-essential libssl-dev zlib1g-dev \
-        libbz2-dev libreadline-dev libsqlite3-dev curl llvm \
-        libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev \
-        clang
-      env CC=clang python-build "$(cat "${BASE_ROOT_DIR}/.python-version")" "${PYTHON_PATH}"
-    fi
-    export PATH="${PYTHON_PATH}/bin:${PATH}"
-    command -v python3
-    python3 --version
+PYTHON_PATH="/python_build"
+if [ ! -d "${PYTHON_PATH}/bin" ]; then
+  (
+    ${CI_RETRY_EXE} git clone https://github.com/pyenv/pyenv.git
+    cd pyenv/plugins/python-build || exit 1
+    ./install.sh
+  )
+  # For dependencies see https://github.com/pyenv/pyenv/wiki#suggested-build-environment
+  ${CI_RETRY_EXE} apt-get install -y build-essential libssl-dev zlib1g-dev \
+    libbz2-dev libreadline-dev libsqlite3-dev curl llvm \
+    libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev \
+    clang
+  env CC=clang python-build "$(cat "./.python-version")" "${PYTHON_PATH}"
+fi
+export PATH="${PYTHON_PATH}/bin:${PATH}"
+command -v python3
+python3 --version
+
+export LINT_RUNNER_PATH="/lint_test_runner"
+if [ ! -d "${LINT_RUNNER_PATH}" ]; then
+  ${CI_RETRY_EXE} apt-get install -y cargo
+  (
+    cd ./test/lint/test_runner || exit 1
+    cargo build
+    mkdir -p "${LINT_RUNNER_PATH}"
+    mv target/debug/test_runner "${LINT_RUNNER_PATH}"
+  )
 fi
 
-${CI_RETRY_EXE} pip3 install codespell==2.2.1
-${CI_RETRY_EXE} pip3 install flake8==5.0.4
-${CI_RETRY_EXE} pip3 install mypy==0.971
-${CI_RETRY_EXE} pip3 install pyzmq==24.0.1
-${CI_RETRY_EXE} pip3 install vulture==2.6
+${CI_RETRY_EXE} pip3 install \
+  codespell==2.2.5 \
+  flake8==6.1.0 \
+  lief==0.13.2 \
+  mypy==1.4.1 \
+  pyzmq==25.1.0 \
+  vulture==2.6
 
 SHELLCHECK_VERSION=v0.8.0
 curl -sL "https://github.com/koalaman/shellcheck/releases/download/${SHELLCHECK_VERSION}/shellcheck-${SHELLCHECK_VERSION}.linux.x86_64.tar.xz" | \

ci/lint/06_script.sh

@@ -6,6 +6,8 @@
 
 export LC_ALL=C
 
+set -ex
+
 if [ -n "$LOCAL_BRANCH" ]; then
   # To faithfully recreate CI linting locally, specify all commits on the current
   # branch.
@@ -21,15 +23,7 @@ else
 fi
 export COMMIT_RANGE
 
-# This only checks that the trees are pure subtrees, it is not doing a full
-# check with -r to not have to fetch all the remotes.
-test/lint/git-subtree-check.sh src/crypto/ctaes
-test/lint/git-subtree-check.sh src/secp256k1
-test/lint/git-subtree-check.sh src/minisketch
-test/lint/git-subtree-check.sh src/leveldb
-test/lint/git-subtree-check.sh src/crc32c
-test/lint/check-doc.py
-test/lint/all-lint.py
+RUST_BACKTRACE=1 "${LINT_RUNNER_PATH}/test_runner"
 
 if [ "$CIRRUS_REPO_FULL_NAME" = "bitcoin/bitcoin" ] && [ "$CIRRUS_PR" = "" ] ; then
     # Sanity check only the last few commits to get notified of missing sigs,

ci/lint/Dockerfile

@@ -1,29 +0,0 @@
-# See test/lint/README.md for usage.
-#
-# This container basically has to live in this directory in order to pull in the CI
-# install scripts. If it lived in the root directory, it would have to pull in the
-# entire repo as docker context during build; if it lived elsewhere, it wouldn't be
-# able to make back-references to pull in the install scripts. So here it lives.
-
-FROM python:3.7-buster
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV LC_ALL=C.UTF-8
-
-# This is used by the 04_install.sh script; we can't read the Python version from
-# .python-version for the same reasons as above, and it's more efficient to pull a
-# preexisting Python image than it is to build from source.
-ENV SKIP_PYTHON_INSTALL=1
-
-# Must be built from ./ci/lint/ for these paths to work.
-COPY ./docker-entrypoint.sh /entrypoint.sh
-COPY ./04_install.sh /install.sh
-
-RUN /install.sh && \
-  echo 'alias lint="./ci/lint/06_script.sh"' >> ~/.bashrc && \
-  chmod 755 /entrypoint.sh && \
-  rm -rf /var/lib/apt/lists/*
-
-
-WORKDIR /bitcoin
-ENTRYPOINT ["/entrypoint.sh"]

ci/lint/container-entrypoint.sh

@@ -1,10 +1,18 @@
 #!/usr/bin/env bash
+#
+# Copyright (c) The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or https://opensource.org/license/mit/.
+
 export LC_ALL=C
 
 # Fixes permission issues when there is a container UID/GID mismatch with the owner
 # of the mounted bitcoin src dir.
 git config --global --add safe.directory /bitcoin
 
+export PATH="/python_build/bin:${PATH}"
+export LINT_RUNNER_PATH="/lint_test_runner"
+
 if [ -z "$1" ]; then
   LOCAL_BRANCH=1 bash -ic "./ci/lint/06_script.sh"
 else

ci/lint_imagefile

@@ -0,0 +1,24 @@
+# Copyright (c) The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or https://opensource.org/license/mit/.
+
+# See test/lint/README.md for usage.
+
+FROM debian:bookworm
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV LC_ALL=C.UTF-8
+
+COPY ./.python-version /.python-version
+COPY ./ci/lint/container-entrypoint.sh /entrypoint.sh
+COPY ./ci/lint/04_install.sh /install.sh
+COPY ./test/lint/test_runner /test/lint/test_runner
+
+RUN /install.sh && \
+  echo 'alias lint="./ci/lint/06_script.sh"' >> ~/.bashrc && \
+  chmod 755 /entrypoint.sh && \
+  rm -rf /var/lib/apt/lists/*
+
+
+WORKDIR /bitcoin
+ENTRYPOINT ["/entrypoint.sh"]

ci/lint_run_all.sh

@@ -8,4 +8,5 @@ export LC_ALL=C.UTF-8
 
 set -o errexit; source ./ci/test/00_setup_env.sh
 set -o errexit; source ./ci/lint/04_install.sh
-set -o errexit; source ./ci/lint/06_script.sh
+set -o errexit
+./ci/lint/06_script.sh

ci/test/00_setup_env.sh

@@ -6,16 +6,25 @@
 
 export LC_ALL=C.UTF-8
 
-# The root dir.
+set -ex
+
+# The source root dir, usually from git, usually read-only.
 # The ci system copies this folder.
-BASE_ROOT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )"/../../ >/dev/null 2>&1 && pwd )
-export BASE_ROOT_DIR
+BASE_READ_ONLY_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )"/../../ >/dev/null 2>&1 && pwd )
+export BASE_READ_ONLY_DIR
+# The destination root dir inside the container.
+# This folder will also hold any SDKs.
+# This folder only exists on the ci guest and will be a copy of BASE_READ_ONLY_DIR
+export BASE_ROOT_DIR="${BASE_ROOT_DIR:-/ci_container_base}"
 # The depends dir.
 # This folder exists only on the ci guest, and on the ci host as a volume.
 export DEPENDS_DIR=${DEPENDS_DIR:-$BASE_ROOT_DIR/depends}
-# A folder for the ci system to put temporary files (ccache, datadirs for tests, ...)
-# This folder only exists on the ci host.
+# A folder for the ci system to put temporary files (build result, datadirs for tests, ...)
+# This folder only exists on the ci guest.
 export BASE_SCRATCH_DIR=${BASE_SCRATCH_DIR:-$BASE_ROOT_DIR/ci/scratch}
+# A folder for the ci system to put executables.
+# This folder only exists on the ci guest.
+export BINS_SCRATCH_DIR="${BASE_SCRATCH_DIR}/bins/"
 
 echo "Setting specific values in env"
 if [ -n "${FILE_ENV}" ]; then
@@ -27,48 +36,37 @@ fi
 echo "Fallback to default values in env (if not yet set)"
 # The number of parallel jobs to pass down to make and test_runner.py
 export MAKEJOBS=${MAKEJOBS:--j4}
-# What host to compile for. See also ./depends/README.md
-# Tests that need cross-compilation export the appropriate HOST.
-# Tests that run natively guess the host
-export HOST=${HOST:-$("$BASE_ROOT_DIR/depends/config.guess")}
 # Whether to prefer BusyBox over GNU utilities
 export USE_BUSY_BOX=${USE_BUSY_BOX:-false}
 
 export RUN_UNIT_TESTS=${RUN_UNIT_TESTS:-true}
 export RUN_FUNCTIONAL_TESTS=${RUN_FUNCTIONAL_TESTS:-true}
 export RUN_TIDY=${RUN_TIDY:-false}
-export RUN_SECURITY_TESTS=${RUN_SECURITY_TESTS:-false}
 # By how much to scale the test_runner timeouts (option --timeout-factor).
 # This is needed because some ci machines have slow CPU or disk, so sanitizers
 # might be slow or a reindex might be waiting on disk IO.
 export TEST_RUNNER_TIMEOUT_FACTOR=${TEST_RUNNER_TIMEOUT_FACTOR:-40}
-export TEST_RUNNER_ENV=${TEST_RUNNER_ENV:-}
 export RUN_FUZZ_TESTS=${RUN_FUZZ_TESTS:-false}
 
-export CONTAINER_NAME=${CONTAINER_NAME:-ci_unnamed}
-export CI_IMAGE_NAME_TAG=${CI_IMAGE_NAME_TAG:-ubuntu:20.04}
 # Randomize test order.
 # See https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/utf_reference/rt_param_reference/random.html
 export BOOST_TEST_RANDOM=${BOOST_TEST_RANDOM:-1}
 # See man 7 debconf
 export DEBIAN_FRONTEND=noninteractive
-export CCACHE_SIZE=${CCACHE_SIZE:-100M}
+export CCACHE_MAXSIZE=${CCACHE_MAXSIZE:-100M}
 export CCACHE_TEMPDIR=${CCACHE_TEMPDIR:-/tmp/.ccache-temp}
 export CCACHE_COMPRESS=${CCACHE_COMPRESS:-1}
 # The cache dir.
 # This folder exists only on the ci guest, and on the ci host as a volume.
 export CCACHE_DIR=${CCACHE_DIR:-$BASE_SCRATCH_DIR/.ccache}
 # Folder where the build result is put (bin and lib).
-export BASE_OUTDIR=${BASE_OUTDIR:-$BASE_SCRATCH_DIR/out/$HOST}
+export BASE_OUTDIR=${BASE_OUTDIR:-$BASE_SCRATCH_DIR/out}
 # Folder where the build is done (dist and out-of-tree build).
 export BASE_BUILD_DIR=${BASE_BUILD_DIR:-$BASE_SCRATCH_DIR/build}
 # The folder for previous release binaries.
 # This folder exists only on the ci guest, and on the ci host as a volume.
-export PREVIOUS_RELEASES_DIR=${PREVIOUS_RELEASES_DIR:-$BASE_ROOT_DIR/releases/$HOST}
-export DIR_IWYU="${BASE_SCRATCH_DIR}/iwyu"
-export SDK_URL=${SDK_URL:-https://bitcoincore.org/depends-sources/sdks}
-export CI_BASE_PACKAGES=${CI_BASE_PACKAGES:-build-essential libtool autotools-dev automake pkg-config bsdmainutils curl ca-certificates ccache python3 rsync git procps bison}
+export PREVIOUS_RELEASES_DIR=${PREVIOUS_RELEASES_DIR:-$BASE_ROOT_DIR/prev_releases}
+export CI_BASE_PACKAGES=${CI_BASE_PACKAGES:-build-essential libtool autotools-dev automake pkg-config bsdmainutils curl ca-certificates ccache python3 rsync git procps bison e2fsprogs cmake}
 export GOAL=${GOAL:-install}
 export DIR_QA_ASSETS=${DIR_QA_ASSETS:-${BASE_SCRATCH_DIR}/qa-assets}
-export PATH=${BASE_ROOT_DIR}/ci/retry:$PATH
 export CI_RETRY_EXE=${CI_RETRY_EXE:-"retry --"}

ci/test/00_setup_env_android.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-2022 The Bitcoin Core developers
+# Copyright (c) 2019-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
@@ -9,7 +9,7 @@ export LC_ALL=C.UTF-8
 export HOST=aarch64-linux-android
 export PACKAGES="unzip openjdk-8-jdk gradle"
 export CONTAINER_NAME=ci_android
-export CI_IMAGE_NAME_TAG="ubuntu:jammy"
+export CI_IMAGE_NAME_TAG="docker.io/amd64/ubuntu:22.04"
 
 export RUN_UNIT_TESTS=false
 export RUN_FUNCTIONAL_TESTS=false

ci/test/00_setup_env_arm.sh

@@ -7,18 +7,10 @@
 export LC_ALL=C.UTF-8
 
 export HOST=arm-linux-gnueabihf
-# The host arch is unknown, so we run the tests through qemu.
-# If the host is arm and wants to run the tests natively, it can set QEMU_USER_CMD to the empty string.
-if [ -z ${QEMU_USER_CMD+x} ]; then export QEMU_USER_CMD="${QEMU_USER_CMD:-"qemu-arm -L /usr/arm-linux-gnueabihf/"}"; fi
 export DPKG_ADD_ARCH="armhf"
 export PACKAGES="python3-zmq g++-arm-linux-gnueabihf busybox libc6:armhf libstdc++6:armhf libfontconfig1:armhf libxcb1:armhf"
-if [ -n "$QEMU_USER_CMD" ]; then
-  # Likely cross-compiling, so install the needed gcc and qemu-user
-  export PACKAGES="$PACKAGES qemu-user"
-fi
 export CONTAINER_NAME=ci_arm_linux
-# Use debian to avoid 404 apt errors when cross compiling
-export CI_IMAGE_NAME_TAG="debian:bullseye"
+export CI_IMAGE_NAME_TAG="docker.io/arm64v8/debian:bookworm"
 export USE_BUSY_BOX=true
 export RUN_UNIT_TESTS=true
 export RUN_FUNCTIONAL_TESTS=false

ci/test/00_setup_env_i686_centos.sh

@@ -8,11 +8,10 @@ export LC_ALL=C.UTF-8
 
 export HOST=i686-pc-linux-gnu
 export CONTAINER_NAME=ci_i686_centos
-export CI_IMAGE_NAME_TAG=quay.io/centos/centos:stream8
-export CI_BASE_PACKAGES="gcc-c++ glibc-devel.x86_64 libstdc++-devel.x86_64 glibc-devel.i686 libstdc++-devel.i686 ccache libtool make git python38 python38-pip which patch lbzip2 xz procps-ng dash rsync coreutils bison"
+export CI_IMAGE_NAME_TAG="quay.io/centos/amd64:stream9"
+export CI_BASE_PACKAGES="gcc-c++ glibc-devel.x86_64 libstdc++-devel.x86_64 glibc-devel.i686 libstdc++-devel.i686 ccache libtool make git python3 python3-pip which patch lbzip2 xz procps-ng dash rsync coreutils bison util-linux e2fsprogs cmake"
 export PIP_PACKAGES="pyzmq"
 export GOAL="install"
-export NO_WERROR=1 # GCC 8
+export NO_WERROR=1  # Suppress error: #warning _FORTIFY_SOURCE > 2 is treated like 2 on this platform [-Werror=cpp]
 export BITCOIN_CONFIG="--enable-zmq --with-gui=qt5 --enable-reduce-exports"
 export CONFIG_SHELL="/bin/dash"
-export TEST_RUNNER_ENV="LC_ALL=en_US.UTF-8"

ci/test/00_setup_env_i686_multiprocess.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2020-2022 The Bitcoin Core developers
+# Copyright (c) 2020-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
@@ -8,10 +8,10 @@ export LC_ALL=C.UTF-8
 
 export HOST=i686-pc-linux-gnu
 export CONTAINER_NAME=ci_i686_multiprocess
-export CI_IMAGE_NAME_TAG=ubuntu:20.04
-export PACKAGES="cmake python3 llvm clang g++-multilib"
+export CI_IMAGE_NAME_TAG="docker.io/amd64/ubuntu:22.04"
+export PACKAGES="llvm clang g++-multilib"
 export DEP_OPTS="DEBUG=1 MULTIPROCESS=1"
 export GOAL="install"
-export BITCOIN_CONFIG="--enable-debug CC='clang -m32' CXX='clang++ -m32' LDFLAGS='--rtlib=compiler-rt -lgcc_s'"
-export TEST_RUNNER_ENV="BITCOIND=bitcoin-node"
-export TEST_RUNNER_EXTRA="--nosandbox"
+export BITCOIN_CONFIG="--enable-debug CC='clang -m32' CXX='clang++ -m32' \
+CPPFLAGS='-DBOOST_MULTI_INDEX_ENABLE_SAFE_MODE'"
+export BITCOIND=bitcoin-node  # Used in functional tests

ci/test/00_setup_env_mac_cross.sh

@@ -1,17 +1,19 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-2021 The Bitcoin Core developers
+# Copyright (c) 2019-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
+export SDK_URL=${SDK_URL:-https://bitcoincore.org/depends-sources/sdks}
+
 export CONTAINER_NAME=ci_macos_cross
-export CI_IMAGE_NAME_TAG=ubuntu:20.04  # Check that Focal can cross-compile to macos
+export CI_IMAGE_NAME_TAG="docker.io/ubuntu:22.04"
 export HOST=x86_64-apple-darwin
-export PACKAGES="cmake libz-dev libtinfo5 python3-setuptools xorriso"
-export XCODE_VERSION=12.2
-export XCODE_BUILD_ID=12B45b
+export PACKAGES="zip"
+export XCODE_VERSION=15.0
+export XCODE_BUILD_ID=15A240d
 export RUN_UNIT_TESTS=false
 export RUN_FUNCTIONAL_TESTS=false
 export GOAL="deploy"

ci/test/00_setup_env_mac_native.sh

@@ -1,16 +1,19 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-2022 The Bitcoin Core developers
+# Copyright (c) 2019-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-export HOST=arm64-apple-darwin
-export PIP_PACKAGES="zmq"
+export HOST=x86_64-apple-darwin
+# Homebrew's python@3.12 is marked as externally managed (PEP 668).
+# Therefore, `--break-system-packages` is needed.
+export PIP_PACKAGES="--break-system-packages zmq"
 export GOAL="install"
 export BITCOIN_CONFIG="--with-gui --with-miniupnpc --with-natpmp --enable-reduce-exports"
 export CI_OS_NAME="macos"
 export NO_DEPENDS=1
 export OSX_SDK=""
-export CCACHE_SIZE=300M
+export CCACHE_MAXSIZE=400M
+export RUN_FUZZ_TESTS=true

ci/test/00_setup_env_native_asan.sh

@@ -1,26 +1,26 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-2022 The Bitcoin Core developers
+# Copyright (c) 2019-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
+export CI_IMAGE_NAME_TAG="docker.io/ubuntu:24.04"
 # Only install BCC tracing packages in Cirrus CI.
 if [[ "${CIRRUS_CI}" == "true" ]]; then
-  # We install an up-to-date 'bpfcc-tools' package from an untrusted PPA.
-  # This can be dropped with the next Ubuntu or Debian release that includes up-to-date packages.
-  # See the if-then in ci/test/04_install.sh too.
-  export ADD_UNTRUSTED_BPFCC_PPA=true
-  export BPFCC_PACKAGE="bpfcc-tools"
+  BPFCC_PACKAGE="bpfcc-tools linux-headers-$(uname --kernel-release)"
+  export CI_CONTAINER_CAP="--privileged -v /sys/kernel:/sys/kernel:rw"
 else
-  export ADD_UNTRUSTED_BPFCC_PPA=false
-  export BPFCC_PACKAGE=""
+  BPFCC_PACKAGE=""
+  export CI_CONTAINER_CAP="--cap-add SYS_PTRACE"  # If run with (ASan + LSan), the container needs access to ptrace (https://github.com/google/sanitizers/issues/764)
 fi
 
 export CONTAINER_NAME=ci_native_asan
-export PACKAGES="systemtap-sdt-dev clang llvm python3-zmq qtbase5-dev qttools5-dev-tools libevent-dev bsdmainutils libboost-dev libdb5.3++-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev libqrencode-dev libsqlite3-dev ${BPFCC_PACKAGE}"
-export CI_IMAGE_NAME_TAG=ubuntu:22.04
+export PACKAGES="systemtap-sdt-dev clang-17 llvm-17 libclang-rt-17-dev python3-zmq qtbase5-dev qttools5-dev-tools libevent-dev libboost-dev libdb5.3++-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev libqrencode-dev libsqlite3-dev ${BPFCC_PACKAGE}"
 export NO_DEPENDS=1
 export GOAL="install"
-export BITCOIN_CONFIG="--enable-c++20 --enable-usdt --enable-zmq --with-incompatible-bdb --with-gui=qt5 CPPFLAGS='-DARENA_DEBUG -DDEBUG_LOCKORDER' --with-sanitizers=address,integer,undefined CC=clang CXX=clang++"
+export BITCOIN_CONFIG="--enable-c++20 --enable-usdt --enable-zmq --with-incompatible-bdb --with-gui=qt5 \
+CPPFLAGS='-DARENA_DEBUG -DDEBUG_LOCKORDER' \
+--with-sanitizers=address,float-divide-by-zero,integer,undefined \
+CC='clang-17 -ftrivial-auto-var-init=pattern' CXX='clang++-17 -ftrivial-auto-var-init=pattern'"

ci/test/00_setup_env_native_fuzz.sh

@@ -1,18 +1,21 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-2022 The Bitcoin Core developers
+# Copyright (c) 2019-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-export CI_IMAGE_NAME_TAG="ubuntu:23.04" # Version 23.04 will reach EOL in Jan 2024, and can be replaced by "ubuntu:24.04" (or anything else that ships the wanted clang version).
+export CI_IMAGE_NAME_TAG="docker.io/ubuntu:24.04"
 export CONTAINER_NAME=ci_native_fuzz
-export PACKAGES="clang-16 llvm-16 libclang-rt-16-dev python3 libevent-dev bsdmainutils libboost-dev libsqlite3-dev"
+export PACKAGES="clang-17 llvm-17 libclang-rt-17-dev libevent-dev libboost-dev libsqlite3-dev"
 export NO_DEPENDS=1
 export RUN_UNIT_TESTS=false
 export RUN_FUNCTIONAL_TESTS=false
 export RUN_FUZZ_TESTS=true
 export GOAL="install"
-export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer,address,undefined,integer CC='clang-16 -ftrivial-auto-var-init=pattern' CXX='clang++-16 -ftrivial-auto-var-init=pattern'"
-export CCACHE_SIZE=200M
+export CI_CONTAINER_CAP="--cap-add SYS_PTRACE"  # If run with (ASan + LSan), the container needs access to ptrace (https://github.com/google/sanitizers/issues/764)
+export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer,address,undefined,float-divide-by-zero,integer \
+CC='clang-17 -ftrivial-auto-var-init=pattern' CXX='clang++-17 -ftrivial-auto-var-init=pattern'"
+export CCACHE_MAXSIZE=200M
+export LLVM_SYMBOLIZER_PATH="/usr/bin/llvm-symbolizer-17"

ci/test/00_setup_env_native_fuzz_with_msan.sh

@@ -1,25 +1,25 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2020-2022 The Bitcoin Core developers
+# Copyright (c) 2020-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-export CI_IMAGE_NAME_TAG="ubuntu:23.04" # Version 23.04 will reach EOL in Jan 2024, and can be replaced by "ubuntu:24.04" (or anything else that ships the wanted clang version).
-LIBCXX_DIR="${BASE_SCRATCH_DIR}/msan/build/"
+export CI_IMAGE_NAME_TAG="docker.io/ubuntu:24.04"
+LIBCXX_DIR="/msan/cxx_build/"
 export MSAN_FLAGS="-fsanitize=memory -fsanitize-memory-track-origins=2 -fno-omit-frame-pointer -g -O1 -fno-optimize-sibling-calls"
-LIBCXX_FLAGS="-nostdinc++ -stdlib=libc++ -L${LIBCXX_DIR}lib -lc++abi -I${LIBCXX_DIR}include -I${LIBCXX_DIR}include/c++/v1 -lpthread -Wl,-rpath,${LIBCXX_DIR}lib -Wno-unused-command-line-argument"
+LIBCXX_FLAGS="-nostdinc++ -nostdlib++ -isystem ${LIBCXX_DIR}include/c++/v1 -L${LIBCXX_DIR}lib -Wl,-rpath,${LIBCXX_DIR}lib -lc++ -lc++abi -lpthread -Wno-unused-command-line-argument"
 export MSAN_AND_LIBCXX_FLAGS="${MSAN_FLAGS} ${LIBCXX_FLAGS}"
 
 export CONTAINER_NAME="ci_native_fuzz_msan"
-export PACKAGES="clang-16 llvm-16 libclang-rt-16-dev cmake"
+export PACKAGES="ninja-build"
 # BDB generates false-positives and will be removed in future
-export DEP_OPTS="NO_BDB=1 NO_QT=1 CC='clang' CXX='clang++' CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'"
+export DEP_OPTS="DEBUG=1 NO_BDB=1 NO_QT=1 CC=clang CXX=clang++ CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'"
 export GOAL="install"
-export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer,memory --disable-hardening --with-asm=no CC=clang CXX=clang++ CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'"
+export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer,memory --disable-hardening --with-asm=no CFLAGS='${MSAN_FLAGS}' CPPFLAGS='-DBOOST_MULTI_INDEX_ENABLE_SAFE_MODE' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'"
 export USE_MEMORY_SANITIZER="true"
 export RUN_UNIT_TESTS="false"
 export RUN_FUNCTIONAL_TESTS="false"
 export RUN_FUZZ_TESTS=true
-export CCACHE_SIZE=250M
+export CCACHE_MAXSIZE=250M

ci/test/00_setup_env_native_fuzz_with_valgrind.sh

@@ -1,14 +1,14 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-2022 The Bitcoin Core developers
+# Copyright (c) 2019-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-export CI_IMAGE_NAME_TAG="debian:bookworm"
+export CI_IMAGE_NAME_TAG="docker.io/debian:bookworm"
 export CONTAINER_NAME=ci_native_fuzz_valgrind
-export PACKAGES="clang llvm libclang-rt-dev python3 libevent-dev bsdmainutils libboost-dev libsqlite3-dev valgrind"
+export PACKAGES="clang llvm libclang-rt-dev libevent-dev libboost-dev libsqlite3-dev valgrind"
 export NO_DEPENDS=1
 export RUN_UNIT_TESTS=false
 export RUN_FUNCTIONAL_TESTS=false
@@ -16,5 +16,5 @@ export RUN_FUZZ_TESTS=true
 export FUZZ_TESTS_CONFIG="--valgrind"
 export GOAL="install"
 # Temporarily pin dwarf 4, until using Valgrind 3.20 or later
-export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer CC=clang CXX=clang++ CFLAGS='-gdwarf-4' CXXFLAGS='-gdwarf-4'"
-export CCACHE_SIZE=200M
+export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer CC=clang CXX=clang++ CFLAGS=-gdwarf-4 CXXFLAGS=-gdwarf-4"
+export CCACHE_MAXSIZE=200M

ci/test/00_setup_env_native_msan.sh

@@ -1,23 +1,23 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2020-2022 The Bitcoin Core developers
+# Copyright (c) 2020-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-export CI_IMAGE_NAME_TAG="ubuntu:23.04" # Version 23.04 will reach EOL in Jan 2024, and can be replaced by "ubuntu:24.04" (or anything else that ships the wanted clang version).
-LIBCXX_DIR="${BASE_SCRATCH_DIR}/msan/build/"
+export CI_IMAGE_NAME_TAG="docker.io/ubuntu:24.04"
+LIBCXX_DIR="/msan/cxx_build/"
 export MSAN_FLAGS="-fsanitize=memory -fsanitize-memory-track-origins=2 -fno-omit-frame-pointer -g -O1 -fno-optimize-sibling-calls"
-LIBCXX_FLAGS="-nostdinc++ -stdlib=libc++ -L${LIBCXX_DIR}lib -lc++abi -I${LIBCXX_DIR}include -I${LIBCXX_DIR}include/c++/v1 -lpthread -Wl,-rpath,${LIBCXX_DIR}lib -Wno-unused-command-line-argument"
+LIBCXX_FLAGS="-nostdinc++ -nostdlib++ -isystem ${LIBCXX_DIR}include/c++/v1 -L${LIBCXX_DIR}lib -Wl,-rpath,${LIBCXX_DIR}lib -lc++ -lc++abi -lpthread -Wno-unused-command-line-argument"
 export MSAN_AND_LIBCXX_FLAGS="${MSAN_FLAGS} ${LIBCXX_FLAGS}"
 
 export CONTAINER_NAME="ci_native_msan"
-export PACKAGES="clang-16 llvm-16 libclang-rt-16-dev cmake"
+export PACKAGES="ninja-build"
 # BDB generates false-positives and will be removed in future
-export DEP_OPTS="NO_BDB=1 NO_QT=1 CC='clang' CXX='clang++' CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'"
+export DEP_OPTS="DEBUG=1 NO_BDB=1 NO_QT=1 CC=clang CXX=clang++ CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'"
 export GOAL="install"
-export BITCOIN_CONFIG="--with-sanitizers=memory --disable-hardening --with-asm=no CC=clang CXX=clang++ CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'"
+export BITCOIN_CONFIG="--with-sanitizers=memory --disable-hardening --with-asm=no CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'"
 export USE_MEMORY_SANITIZER="true"
 export RUN_FUNCTIONAL_TESTS="false"
-export CCACHE_SIZE=250M
+export CCACHE_MAXSIZE=250M

ci/test/00_setup_env_native_nowallet_libbitcoinkernel.sh

@@ -1,16 +1,15 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-2022 The Bitcoin Core developers
+# Copyright (c) 2019-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
 export CONTAINER_NAME=ci_native_nowallet_libbitcoinkernel
-export CI_IMAGE_NAME_TAG=ubuntu:focal
-# Use minimum supported python3.7 (or python3.8, as best-effort) and clang-8, see doc/dependencies.md
-export PACKAGES="python3-zmq clang-8 llvm-8 libc++abi-8-dev libc++-8-dev"
-export DEP_OPTS="NO_WALLET=1 CC=clang-8 CXX='clang++-8 -stdlib=libc++'"
+export CI_IMAGE_NAME_TAG="docker.io/ubuntu:22.04"
+# Use minimum supported python3.9 (or best-effort 3.10) and clang-14, see doc/dependencies.md
+export PACKAGES="python3-zmq clang-14 llvm-14 libc++abi-14-dev libc++-14-dev"
+export DEP_OPTS="NO_WALLET=1 CC=clang-14 CXX='clang++-14 -stdlib=libc++'"
 export GOAL="install"
-export NO_WERROR=1
-export BITCOIN_CONFIG="--enable-reduce-exports CC=clang-8 CXX='clang++-8 -stdlib=libc++' --enable-experimental-util-chainstate --with-experimental-kernel-lib --enable-shared"
+export BITCOIN_CONFIG="--enable-reduce-exports --enable-experimental-util-chainstate --with-experimental-kernel-lib --enable-shared"

ci/test/00_setup_env_native_previous_releases.sh

@@ -1,21 +1,20 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-2022 The Bitcoin Core developers
+# Copyright (c) 2019-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-export CONTAINER_NAME=ci_native_qt5
-export CI_IMAGE_NAME_TAG=debian:buster
-# Use minimum supported python3.7 and gcc-8, see doc/dependencies.md
-export PACKAGES="gcc-8 g++-8 python3-zmq qtbase5-dev qttools5-dev-tools libdbus-1-dev libharfbuzz-dev"
-export DEP_OPTS="NO_QT=1 NO_UPNP=1 NO_NATPMP=1 DEBUG=1 ALLOW_HOST_PACKAGES=1 CC=gcc-8 CXX=g++-8"
+export CONTAINER_NAME=ci_native_previous_releases
+export CI_IMAGE_NAME_TAG="docker.io/debian:bullseye"
+# Use minimum supported python3.9 and gcc-10, see doc/dependencies.md
+export PACKAGES="gcc-10 g++-10 python3-zmq"
+export DEP_OPTS="NO_UPNP=1 NO_NATPMP=1 DEBUG=1 CC=gcc-10 CXX=g++-10"
 export TEST_RUNNER_EXTRA="--previous-releases --coverage --extended --exclude feature_dbcrash"  # Run extended tests so that coverage does not fail, but exclude the very slow dbcrash
 export RUN_UNIT_TESTS_SEQUENTIAL="true"
 export RUN_UNIT_TESTS="false"
 export GOAL="install"
-export NO_WERROR=1
 export DOWNLOAD_PREVIOUS_RELEASES="true"
-export BITCOIN_CONFIG="--enable-zmq --with-libs=no --with-gui=qt5 --enable-reduce-exports \
---enable-debug CFLAGS=\"-g0 -O2 -funsigned-char\" CXXFLAGS=\"-g0 -O2 -funsigned-char\" CC=gcc-8 CXX=g++-8"
+export BITCOIN_CONFIG="--enable-zmq --with-libs=no --with-gui=qt5 --enable-reduce-exports --enable-debug \
+CFLAGS=\"-g0 -O2 -funsigned-char\" CPPFLAGS='-DBOOST_MULTI_INDEX_ENABLE_SAFE_MODE' CXXFLAGS=\"-g0 -O2 -funsigned-char\""

ci/test/00_setup_env_native_tidy.sh

@@ -1,19 +1,20 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2022 The Bitcoin Core developers
+# Copyright (c) 2023-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-export CI_IMAGE_NAME_TAG="ubuntu:lunar" # Version 23.04 will reach EOL in Jan 2024, and can be replaced by "ubuntu:24.04" (or anything else that ships the wanted clang version).
+export CI_IMAGE_NAME_TAG="docker.io/ubuntu:24.04"
 export CONTAINER_NAME=ci_native_tidy
-export PACKAGES="clang-16 libclang-16-dev llvm-16-dev libomp-16-dev clang-tidy-16 bear cmake libevent-dev libboost-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev systemtap-sdt-dev libqt5gui5 libqt5core5a libqt5dbus5 qttools5-dev qttools5-dev-tools libqrencode-dev libsqlite3-dev libdb++-dev"
+export TIDY_LLVM_V="17"
+export PACKAGES="clang-${TIDY_LLVM_V} libclang-${TIDY_LLVM_V}-dev llvm-${TIDY_LLVM_V}-dev libomp-${TIDY_LLVM_V}-dev clang-tidy-${TIDY_LLVM_V} jq bear libevent-dev libboost-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev systemtap-sdt-dev qtbase5-dev qttools5-dev qttools5-dev-tools libqrencode-dev libsqlite3-dev libdb++-dev"
 export NO_DEPENDS=1
 export RUN_UNIT_TESTS=false
 export RUN_FUNCTIONAL_TESTS=false
 export RUN_FUZZ_TESTS=false
 export RUN_TIDY=true
 export GOAL="install"
-export BITCOIN_CONFIG="CC=clang-16 CXX=clang++-16 --with-incompatible-bdb --disable-hardening CFLAGS='-O0 -g0' CXXFLAGS='-O0 -g0 -I/usr/lib/llvm-16/lib/clang/16/include'"
-export CCACHE_SIZE=200M
+export BITCOIN_CONFIG="CC=clang-${TIDY_LLVM_V} CXX=clang++-${TIDY_LLVM_V} --with-incompatible-bdb --disable-hardening CFLAGS='-O0 -g0' CXXFLAGS='-O0 -g0 -I/usr/lib/llvm-${TIDY_LLVM_V}/lib/clang/${TIDY_LLVM_V}/include'"
+export CCACHE_MAXSIZE=200M

ci/test/00_setup_env_native_tsan.sh

@@ -1,14 +1,14 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-2022 The Bitcoin Core developers
+# Copyright (c) 2019-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
 export CONTAINER_NAME=ci_native_tsan
-export CI_IMAGE_NAME_TAG=ubuntu:23.04  # Version 23.04 will reach EOL in Jan 2024, and can be replaced by "ubuntu:24.04" (or anything else that ships the wanted clang version).
-export PACKAGES="clang-16 llvm-16 libclang-rt-16-dev libc++abi-16-dev libc++-16-dev python3-zmq"
-export DEP_OPTS="CC=clang-16 CXX='clang++-16 -stdlib=libc++'"
+export CI_IMAGE_NAME_TAG="docker.io/ubuntu:24.04"
+export PACKAGES="clang-17 llvm-17 libclang-rt-17-dev libc++abi-17-dev libc++-17-dev python3-zmq"
+export DEP_OPTS="CC=clang-17 CXX='clang++-17 -stdlib=libc++'"
 export GOAL="install"
 export BITCOIN_CONFIG="--enable-zmq CPPFLAGS='-DARENA_DEBUG -DDEBUG_LOCKORDER -DDEBUG_LOCKCONTENTION' CXXFLAGS='-g' --with-sanitizers=thread"

ci/test/00_setup_env_native_valgrind.sh

@@ -1,17 +1,17 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-2022 The Bitcoin Core developers
+# Copyright (c) 2019-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-export CI_IMAGE_NAME_TAG="debian:bookworm"
+export CI_IMAGE_NAME_TAG="docker.io/debian:bookworm"
 export CONTAINER_NAME=ci_native_valgrind
-export PACKAGES="valgrind clang llvm libclang-rt-dev python3-zmq libevent-dev bsdmainutils libboost-dev libdb5.3++-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev libsqlite3-dev"
+export PACKAGES="valgrind clang llvm libclang-rt-dev python3-zmq libevent-dev libboost-dev libdb5.3++-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev libsqlite3-dev"
 export USE_VALGRIND=1
 export NO_DEPENDS=1
-export TEST_RUNNER_EXTRA="--nosandbox --exclude feature_init,rpc_bind,feature_bind_extra"  # Excluded for now, see https://github.com/bitcoin/bitcoin/issues/17765#issuecomment-602068547
+export TEST_RUNNER_EXTRA="--exclude feature_init,rpc_bind,feature_bind_extra"  # Excluded for now, see https://github.com/bitcoin/bitcoin/issues/17765#issuecomment-602068547
 export GOAL="install"
 # Temporarily pin dwarf 4, until using Valgrind 3.20 or later
-export BITCOIN_CONFIG="--enable-zmq --with-incompatible-bdb --with-gui=no CC=clang CXX=clang++ CFLAGS='-gdwarf-4' CXXFLAGS='-gdwarf-4'"  # TODO enable GUI
+export BITCOIN_CONFIG="--enable-zmq --with-incompatible-bdb --with-gui=no CC=clang CXX=clang++ CFLAGS=-gdwarf-4 CXXFLAGS=-gdwarf-4"  # TODO enable GUI

ci/test/00_setup_env_s390x.sh

@@ -1,26 +1,16 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-2022 The Bitcoin Core developers
+# Copyright (c) 2019-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
 export HOST=s390x-linux-gnu
-# The host arch is unknown, so we run the tests through qemu.
-# If the host is s390x and wants to run the tests natively, it can set QEMU_USER_CMD to the empty string.
-if [ -z ${QEMU_USER_CMD+x} ]; then export QEMU_USER_CMD="${QEMU_USER_CMD:-"qemu-s390x"}"; fi
 export PACKAGES="python3-zmq"
-if [ -n "$QEMU_USER_CMD" ]; then
-  # Likely cross-compiling, so install the needed gcc and qemu-user
-  export DPKG_ADD_ARCH="s390x"
-  export PACKAGES="$PACKAGES g++-s390x-linux-gnu qemu-user libc6:s390x libstdc++6:s390x"
-fi
-# Use debian to avoid 404 apt errors
 export CONTAINER_NAME=ci_s390x
-export CI_IMAGE_NAME_TAG="debian:bookworm"
-export TEST_RUNNER_ENV="LC_ALL=C"
-export TEST_RUNNER_EXTRA="--exclude feature_init,rpc_bind,feature_bind_extra"  # Excluded for now, see https://github.com/bitcoin/bitcoin/issues/17765#issuecomment-602068547
+export CI_IMAGE_NAME_TAG="docker.io/s390x/ubuntu:24.04"
+export TEST_RUNNER_EXTRA="--exclude rpc_bind,feature_bind_extra"  # Excluded for now, see https://github.com/bitcoin/bitcoin/issues/17765#issuecomment-602068547
 export RUN_FUNCTIONAL_TESTS=true
 export GOAL="install"
-export BITCOIN_CONFIG="--enable-reduce-exports --disable-gui-tests"  # GUI tests disabled for now, see https://github.com/bitcoin/bitcoin/issues/23730
+export BITCOIN_CONFIG="--enable-reduce-exports"

ci/test/00_setup_env_win64.sh

@@ -1,16 +1,19 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-2022 The Bitcoin Core developers
+# Copyright (c) 2019-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
 export CONTAINER_NAME=ci_win64
-export CI_IMAGE_NAME_TAG=ubuntu:22.04  # Check that Jammy can cross-compile to win64
+export CI_IMAGE_NAME_TAG="docker.io/amd64/ubuntu:22.04"  # Check that Jammy can cross-compile to win64
 export HOST=x86_64-w64-mingw32
 export DPKG_ADD_ARCH="i386"
-export PACKAGES="python3 nsis g++-mingw-w64-x86-64-posix wine-binfmt wine64 wine32 file"
+export PACKAGES="nsis g++-mingw-w64-x86-64-posix wine-binfmt wine64 wine32 file"
 export RUN_FUNCTIONAL_TESTS=false
 export GOAL="deploy"
-export BITCOIN_CONFIG="--enable-reduce-exports --enable-external-signer --disable-gui-tests"
+# Prior to 11.0.0, the mingw-w64 headers were missing noreturn attributes, causing warnings when
+# cross-compiling for Windows. https://sourceforge.net/p/mingw-w64/bugs/306/
+# https://github.com/mingw-w64/mingw-w64/commit/1690994f515910a31b9fb7c7bd3a52d4ba987abe
+export BITCOIN_CONFIG="--enable-reduce-exports --disable-gui-tests CXXFLAGS=-Wno-return-type"

ci/test/01_base_install.sh

@@ -1,11 +1,13 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2018-2022 The Bitcoin Core developers
+# Copyright (c) 2018-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
+set -ex
+
 CFG_DONE="ci.base-install-done"  # Use a global git setting to remember whether this script ran to avoid running it twice
 
 if [ "$(git config --global ${CFG_DONE})" == "true" ]; then
@@ -18,17 +20,9 @@ if [ -n "$DPKG_ADD_ARCH" ]; then
 fi
 
 if [[ $CI_IMAGE_NAME_TAG == *centos* ]]; then
-  ${CI_RETRY_EXE} bash -c "dnf -y install epel-release"
-  ${CI_RETRY_EXE} bash -c "dnf -y --allowerasing install $CI_BASE_PACKAGES $PACKAGES"
-elif [ "$CI_USE_APT_INSTALL" != "no" ]; then
-  if [[ "${ADD_UNTRUSTED_BPFCC_PPA}" == "true" ]]; then
-    # Ubuntu 22.04 LTS and Debian 11 both have an outdated bpfcc-tools packages.
-    # The iovisor PPA is outdated as well. The next Ubuntu and Debian releases will contain updated
-    # packages. Meanwhile, use an untrusted PPA to install an up-to-date version of the bpfcc-tools
-    # package.
-    # TODO: drop this once we can use newer images in GCE
-    add-apt-repository ppa:hadret/bpfcc
-  fi
+  bash -c "dnf -y install epel-release"
+  bash -c "dnf -y --allowerasing install $CI_BASE_PACKAGES $PACKAGES"
+elif [ "$CI_OS_NAME" != "macos" ]; then
   if [[ -n "${APPEND_APT_SOURCES_LIST}" ]]; then
     echo "${APPEND_APT_SOURCES_LIST}" >> /etc/apt/sources.list
   fi
@@ -37,28 +31,46 @@ elif [ "$CI_USE_APT_INSTALL" != "no" ]; then
 fi
 
 if [ -n "$PIP_PACKAGES" ]; then
-  if [ "$CI_OS_NAME" == "macos" ]; then
-    sudo -H pip3 install --upgrade pip
-    # shellcheck disable=SC2086
-    IN_GETOPT_BIN="$(brew --prefix gnu-getopt)/bin/getopt" ${CI_RETRY_EXE} pip3 install --user $PIP_PACKAGES
-  else
-    # shellcheck disable=SC2086
-    ${CI_RETRY_EXE} pip3 install --user $PIP_PACKAGES
-  fi
+  # shellcheck disable=SC2086
+  ${CI_RETRY_EXE} pip3 install --user $PIP_PACKAGES
 fi
 
 if [[ ${USE_MEMORY_SANITIZER} == "true" ]]; then
-  update-alternatives --install /usr/bin/clang++ clang++ "$(which clang++-16)" 100
-  update-alternatives --install /usr/bin/clang clang "$(which clang-16)" 100
-  git clone --depth=1 https://github.com/llvm/llvm-project -b llvmorg-16.0.1 "${BASE_SCRATCH_DIR}"/msan/llvm-project
-  cmake -B "${BASE_SCRATCH_DIR}"/msan/build/ -DLLVM_ENABLE_RUNTIMES='libcxx;libcxxabi' -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=MemoryWithOrigins -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF -DLIBCXX_ENABLE_DEBUG_MODE=ON -DLIBCXX_ENABLE_ASSERTIONS=ON -S "${BASE_SCRATCH_DIR}"/msan/llvm-project/runtimes
-  make -C "${BASE_SCRATCH_DIR}"/msan/build/ "$MAKEJOBS"
+  ${CI_RETRY_EXE} git clone --depth=1 https://github.com/llvm/llvm-project -b "llvmorg-18.1.1" /msan/llvm-project
+
+  cmake -G Ninja -B /msan/clang_build/ \
+    -DLLVM_ENABLE_PROJECTS="clang" \
+    -DCMAKE_BUILD_TYPE=Release \
+    -DLLVM_TARGETS_TO_BUILD=Native \
+    -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi;libunwind" \
+    -S /msan/llvm-project/llvm
+
+  ninja -C /msan/clang_build/ "-j$( nproc )"  # Use nproc, because MAKEJOBS is the default in docker image builds
+  ninja -C /msan/clang_build/ install-runtimes
+
+  update-alternatives --install /usr/bin/clang++ clang++ /msan/clang_build/bin/clang++ 100
+  update-alternatives --install /usr/bin/clang clang /msan/clang_build/bin/clang 100
+  update-alternatives --install /usr/bin/llvm-symbolizer llvm-symbolizer /msan/clang_build/bin/llvm-symbolizer 100
+
+  cmake -G Ninja -B /msan/cxx_build/ \
+    -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind" \
+    -DCMAKE_BUILD_TYPE=Release \
+    -DLLVM_USE_SANITIZER=MemoryWithOrigins \
+    -DCMAKE_C_COMPILER=clang \
+    -DCMAKE_CXX_COMPILER=clang++ \
+    -DLLVM_TARGETS_TO_BUILD=Native \
+    -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF \
+    -DLIBCXXABI_USE_LLVM_UNWINDER=OFF \
+    -DLIBCXX_HARDENING_MODE=debug \
+    -S /msan/llvm-project/runtimes
+
+  ninja -C /msan/cxx_build/ "-j$( nproc )"  # Use nproc, because MAKEJOBS is the default in docker image builds
 fi
 
 if [[ "${RUN_TIDY}" == "true" ]]; then
-  git clone --depth=1 https://github.com/include-what-you-use/include-what-you-use -b clang_16 "${DIR_IWYU}"/include-what-you-use
-  cmake -B "${DIR_IWYU}"/build/ -G 'Unix Makefiles' -DCMAKE_PREFIX_PATH=/usr/lib/llvm-16 -S "${DIR_IWYU}"/include-what-you-use
-  make -C "${DIR_IWYU}"/build/ install "$MAKEJOBS"
+  ${CI_RETRY_EXE} git clone --depth=1 https://github.com/include-what-you-use/include-what-you-use -b clang_"${TIDY_LLVM_V}" /include-what-you-use
+  cmake -B /iwyu-build/ -G 'Unix Makefiles' -DCMAKE_PREFIX_PATH=/usr/lib/llvm-"${TIDY_LLVM_V}" -S /include-what-you-use
+  make -C /iwyu-build/ install "-j$( nproc )"  # Use nproc, because MAKEJOBS is the default in docker image builds
 fi
 
 mkdir -p "${DEPENDS_DIR}/SDKs" "${DEPENDS_DIR}/sdk-sources"
@@ -69,7 +81,7 @@ if [ -n "$XCODE_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${OSX_SDK_BASENAME}" ]
   OSX_SDK_FILENAME="${OSX_SDK_BASENAME}.tar.gz"
   OSX_SDK_PATH="${DEPENDS_DIR}/sdk-sources/${OSX_SDK_FILENAME}"
   if [ ! -f "$OSX_SDK_PATH" ]; then
-    curl --location --fail "${SDK_URL}/${OSX_SDK_FILENAME}" -o "$OSX_SDK_PATH"
+    ${CI_RETRY_EXE} curl --location --fail "${SDK_URL}/${OSX_SDK_FILENAME}" -o "$OSX_SDK_PATH"
   fi
   tar -C "${DEPENDS_DIR}/SDKs" -xf "$OSX_SDK_PATH"
 fi
@@ -77,11 +89,11 @@ fi
 if [ -n "$ANDROID_HOME" ] && [ ! -d "$ANDROID_HOME" ]; then
   ANDROID_TOOLS_PATH=${DEPENDS_DIR}/sdk-sources/android-tools.zip
   if [ ! -f "$ANDROID_TOOLS_PATH" ]; then
-    curl --location --fail "${ANDROID_TOOLS_URL}" -o "$ANDROID_TOOLS_PATH"
+    ${CI_RETRY_EXE} curl --location --fail "${ANDROID_TOOLS_URL}" -o "$ANDROID_TOOLS_PATH"
   fi
   mkdir -p "$ANDROID_HOME"
   unzip -o "$ANDROID_TOOLS_PATH" -d "$ANDROID_HOME"
-  yes | "${ANDROID_HOME}"/cmdline-tools/bin/sdkmanager --sdk_root="${ANDROID_HOME}" --install "build-tools;${ANDROID_BUILD_TOOLS_VERSION}" "platform-tools" "platforms;android-${ANDROID_API_LEVEL}" "ndk;${ANDROID_NDK_VERSION}"
+  yes | "${ANDROID_HOME}"/cmdline-tools/bin/sdkmanager --sdk_root="${ANDROID_HOME}" --install "build-tools;${ANDROID_BUILD_TOOLS_VERSION}" "platform-tools" "platforms;android-31" "platforms;android-${ANDROID_API_LEVEL}" "ndk;${ANDROID_NDK_VERSION}"
 fi
 
 git config --global ${CFG_DONE} "true"

ci/test/02_run_container.sh

@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2018-present The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+export LC_ALL=C.UTF-8
+export CI_IMAGE_LABEL="bitcoin-ci-test"
+
+set -ex
+
+if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
+  # Export all env vars to avoid missing some.
+  # Though, exclude those with newlines to avoid parsing problems.
+  python3 -c 'import os; [print(f"{key}={value}") for key, value in os.environ.items() if "\n" not in value and "HOME" != key and "PATH" != key and "USER" != key]' | tee "/tmp/env-$USER-$CONTAINER_NAME"
+  # System-dependent env vars must be kept as is. So read them from the container.
+  docker run --rm "${CI_IMAGE_NAME_TAG}" bash -c "env | grep --extended-regexp '^(HOME|PATH|USER)='" | tee --append "/tmp/env-$USER-$CONTAINER_NAME"
+  echo "Creating $CI_IMAGE_NAME_TAG container to run in"
+  DOCKER_BUILDKIT=1 docker build \
+      --file "${BASE_READ_ONLY_DIR}/ci/test_imagefile" \
+      --build-arg "CI_IMAGE_NAME_TAG=${CI_IMAGE_NAME_TAG}" \
+      --build-arg "FILE_ENV=${FILE_ENV}" \
+      --label="${CI_IMAGE_LABEL}" \
+      --tag="${CONTAINER_NAME}" \
+      "${BASE_READ_ONLY_DIR}"
+  docker volume create "${CONTAINER_NAME}_ccache" || true
+  docker volume create "${CONTAINER_NAME}_depends" || true
+  docker volume create "${CONTAINER_NAME}_depends_sources" || true
+  docker volume create "${CONTAINER_NAME}_depends_SDKs_android" || true
+  docker volume create "${CONTAINER_NAME}_previous_releases" || true
+
+  if [ -n "${RESTART_CI_DOCKER_BEFORE_RUN}" ] ; then
+    echo "Restart docker before run to stop and clear all containers started with --rm"
+    podman container rm --force --all  # Similar to "systemctl restart docker"
+
+    # Still prune everything in case the filtered pruning doesn't work, or if labels were not set
+    # on a previous run. Belt and suspenders approach, should be fine to remove in the future.
+    # Prune images used by --external containers (e.g. build containers) when
+    # using podman.
+    echo "Prune all dangling images"
+    podman image prune --force --external
+  fi
+  echo "Prune all dangling $CI_IMAGE_LABEL images"
+  # When detecting podman-docker, `--external` should be added.
+  docker image prune --force --filter "label=$CI_IMAGE_LABEL"
+
+  # Append $USER to /tmp/env to support multi-user systems and $CONTAINER_NAME
+  # to allow support starting multiple runs simultaneously by the same user.
+  # shellcheck disable=SC2086
+  CI_CONTAINER_ID=$(docker run --cap-add LINUX_IMMUTABLE $CI_CONTAINER_CAP --rm --interactive --detach --tty \
+                  --mount "type=bind,src=$BASE_READ_ONLY_DIR,dst=$BASE_READ_ONLY_DIR,readonly" \
+                  --mount "type=volume,src=${CONTAINER_NAME}_ccache,dst=$CCACHE_DIR" \
+                  --mount "type=volume,src=${CONTAINER_NAME}_depends,dst=$DEPENDS_DIR/built" \
+                  --mount "type=volume,src=${CONTAINER_NAME}_depends_sources,dst=$DEPENDS_DIR/sources" \
+                  --mount "type=volume,src=${CONTAINER_NAME}_depends_SDKs_android,dst=$DEPENDS_DIR/SDKs/android" \
+                  --mount "type=volume,src=${CONTAINER_NAME}_previous_releases,dst=$PREVIOUS_RELEASES_DIR" \
+                  --env-file /tmp/env-$USER-$CONTAINER_NAME \
+                  --name "$CONTAINER_NAME" \
+                  "$CONTAINER_NAME")
+  export CI_CONTAINER_ID
+  export CI_EXEC_CMD_PREFIX="docker exec ${CI_CONTAINER_ID}"
+else
+  echo "Running on host system without docker wrapper"
+  echo "Create missing folders"
+  mkdir -p "${CCACHE_DIR}"
+  mkdir -p "${PREVIOUS_RELEASES_DIR}"
+fi
+
+if [ "$CI_OS_NAME" == "macos" ]; then
+  IN_GETOPT_BIN="$(brew --prefix gnu-getopt)/bin/getopt"
+  export IN_GETOPT_BIN
+fi
+
+CI_EXEC () {
+  $CI_EXEC_CMD_PREFIX bash -c "export PATH=\"/path_with space:${BINS_SCRATCH_DIR}:${BASE_ROOT_DIR}/ci/retry:\$PATH\" && cd \"${BASE_ROOT_DIR}\" && $*"
+}
+export -f CI_EXEC
+
+# Normalize all folders to BASE_ROOT_DIR
+CI_EXEC rsync --archive --stats --human-readable "${BASE_READ_ONLY_DIR}/" "${BASE_ROOT_DIR}" || echo "Nothing to copy from ${BASE_READ_ONLY_DIR}/"
+CI_EXEC "${BASE_ROOT_DIR}/ci/test/01_base_install.sh"
+
+# Fixes permission issues when there is a container UID/GID mismatch with the owner
+# of the git source code directory.
+CI_EXEC git config --global --add safe.directory \"*\"
+
+CI_EXEC mkdir -p "${BINS_SCRATCH_DIR}"
+
+CI_EXEC "${BASE_ROOT_DIR}/ci/test/03_test_script.sh"
+
+if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
+  echo "Stop and remove CI container by ID"
+  docker container kill "${CI_CONTAINER_ID}"
+fi

ci/test/03_test_script.sh

@@ -0,0 +1,204 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2018-present The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+export LC_ALL=C.UTF-8
+
+set -ex
+
+export ASAN_OPTIONS="detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1"
+export LSAN_OPTIONS="suppressions=${BASE_ROOT_DIR}/test/sanitizer_suppressions/lsan"
+export TSAN_OPTIONS="suppressions=${BASE_ROOT_DIR}/test/sanitizer_suppressions/tsan:halt_on_error=1"
+export UBSAN_OPTIONS="suppressions=${BASE_ROOT_DIR}/test/sanitizer_suppressions/ubsan:print_stacktrace=1:halt_on_error=1:report_error_type=1"
+
+if [ "$CI_OS_NAME" == "macos" ]; then
+  top -l 1 -s 0 | awk ' /PhysMem/ {print}'
+  echo "Number of CPUs: $(sysctl -n hw.logicalcpu)"
+else
+  free -m -h
+  echo "Number of CPUs (nproc): $(nproc)"
+  echo "System info: $(uname --kernel-name --kernel-release)"
+  lscpu
+fi
+echo "Free disk space:"
+df -h
+
+# What host to compile for. See also ./depends/README.md
+# Tests that need cross-compilation export the appropriate HOST.
+# Tests that run natively guess the host
+export HOST=${HOST:-$("$BASE_ROOT_DIR/depends/config.guess")}
+
+(
+  # compact->outputs[i].file_size is uninitialized memory, so reading it is UB.
+  # The statistic bytes_written is only used for logging, which is disabled in
+  # CI, so as a temporary minimal fix to work around UB and CI failures, leave
+  # bytes_written unmodified.
+  # See https://github.com/bitcoin/bitcoin/pull/28359#issuecomment-1698694748
+  # Tee patch to stdout to make it clear CI is testing modified code.
+  tee >(patch -p1) <<'EOF'
+--- a/src/leveldb/db/db_impl.cc
++++ b/src/leveldb/db/db_impl.cc
+@@ -1028,9 +1028,6 @@ Status DBImpl::DoCompactionWork(CompactionState* compact) {
+       stats.bytes_read += compact->compaction->input(which, i)->file_size;
+     }
+   }
+-  for (size_t i = 0; i < compact->outputs.size(); i++) {
+-    stats.bytes_written += compact->outputs[i].file_size;
+-  }
+
+   mutex_.Lock();
+   stats_[compact->compaction->level() + 1].Add(stats);
+EOF
+)
+
+if [ "$RUN_FUZZ_TESTS" = "true" ]; then
+  export DIR_FUZZ_IN=${DIR_QA_ASSETS}/fuzz_seed_corpus/
+  if [ ! -d "$DIR_FUZZ_IN" ]; then
+    ${CI_RETRY_EXE} git clone --depth=1 https://github.com/bitcoin-core/qa-assets "${DIR_QA_ASSETS}"
+  fi
+  (
+    cd "${DIR_QA_ASSETS}"
+    echo "Using qa-assets repo from commit ..."
+    git log -1
+  )
+elif [ "$RUN_UNIT_TESTS" = "true" ] || [ "$RUN_UNIT_TESTS_SEQUENTIAL" = "true" ]; then
+  export DIR_UNIT_TEST_DATA=${DIR_QA_ASSETS}/unit_test_data/
+  if [ ! -d "$DIR_UNIT_TEST_DATA" ]; then
+    mkdir -p "$DIR_UNIT_TEST_DATA"
+    ${CI_RETRY_EXE} curl --location --fail https://github.com/bitcoin-core/qa-assets/raw/main/unit_test_data/script_assets_test.json -o "${DIR_UNIT_TEST_DATA}/script_assets_test.json"
+  fi
+fi
+
+if [ "$USE_BUSY_BOX" = "true" ]; then
+  echo "Setup to use BusyBox utils"
+  # tar excluded for now because it requires passing in the exact archive type in ./depends (fixed in later BusyBox version)
+  # ar excluded for now because it does not recognize the -q option in ./depends (unknown if fixed)
+  for util in $(busybox --list | grep -v "^ar$" | grep -v "^tar$" ); do ln -s "$(command -v busybox)" "${BINS_SCRATCH_DIR}/$util"; done
+  # Print BusyBox version
+  patch --help
+fi
+
+# Make sure default datadir does not exist and is never read by creating a dummy file
+if [ "$CI_OS_NAME" == "macos" ]; then
+  echo > "${HOME}/Library/Application Support/Bitcoin"
+else
+  echo > "${HOME}/.bitcoin"
+fi
+
+if [ -z "$NO_DEPENDS" ]; then
+  if [[ $CI_IMAGE_NAME_TAG == *centos* ]]; then
+    SHELL_OPTS="CONFIG_SHELL=/bin/dash"
+  else
+    SHELL_OPTS="CONFIG_SHELL="
+  fi
+  bash -c "$SHELL_OPTS make $MAKEJOBS -C depends HOST=$HOST $DEP_OPTS LOG=1"
+fi
+if [ "$DOWNLOAD_PREVIOUS_RELEASES" = "true" ]; then
+  test/get_previous_releases.py -b -t "$PREVIOUS_RELEASES_DIR"
+fi
+
+BITCOIN_CONFIG_ALL="--disable-dependency-tracking"
+if [ -z "$NO_DEPENDS" ]; then
+  BITCOIN_CONFIG_ALL="${BITCOIN_CONFIG_ALL} CONFIG_SITE=$DEPENDS_DIR/$HOST/share/config.site"
+fi
+if [ -z "$NO_WERROR" ]; then
+  BITCOIN_CONFIG_ALL="${BITCOIN_CONFIG_ALL} --enable-werror"
+fi
+
+ccache --zero-stats
+PRINT_CCACHE_STATISTICS="ccache --version | head -n 1 && ccache --show-stats"
+
+if [ -n "$ANDROID_TOOLS_URL" ]; then
+  make distclean || true
+  ./autogen.sh
+  bash -c "./configure $BITCOIN_CONFIG_ALL $BITCOIN_CONFIG" || ( (cat config.log) && false)
+  make "${MAKEJOBS}" && cd src/qt && ANDROID_HOME=${ANDROID_HOME} ANDROID_NDK_HOME=${ANDROID_NDK_HOME} make apk
+  bash -c "${PRINT_CCACHE_STATISTICS}"
+  exit 0
+fi
+
+BITCOIN_CONFIG_ALL="${BITCOIN_CONFIG_ALL} --enable-external-signer --prefix=$BASE_OUTDIR"
+
+if [ -n "$CONFIG_SHELL" ]; then
+  "$CONFIG_SHELL" -c "./autogen.sh"
+else
+  ./autogen.sh
+fi
+
+mkdir -p "${BASE_BUILD_DIR}"
+cd "${BASE_BUILD_DIR}"
+
+bash -c "${BASE_ROOT_DIR}/configure --cache-file=config.cache $BITCOIN_CONFIG_ALL $BITCOIN_CONFIG" || ( (cat config.log) && false)
+
+make distdir VERSION="$HOST"
+
+cd "${BASE_BUILD_DIR}/bitcoin-$HOST"
+
+bash -c "./configure --cache-file=../config.cache $BITCOIN_CONFIG_ALL $BITCOIN_CONFIG" || ( (cat config.log) && false)
+
+if [[ "${RUN_TIDY}" == "true" ]]; then
+  MAYBE_BEAR="bear --config src/.bear-tidy-config"
+  MAYBE_TOKEN="--"
+fi
+
+bash -c "${MAYBE_BEAR} ${MAYBE_TOKEN} make $MAKEJOBS $GOAL" || ( echo "Build failure. Verbose build follows." && make "$GOAL" V=1 ; false )
+
+bash -c "${PRINT_CCACHE_STATISTICS}"
+du -sh "${DEPENDS_DIR}"/*/
+du -sh "${PREVIOUS_RELEASES_DIR}"
+
+if [[ $HOST = *-mingw32 ]]; then
+  # Generate all binaries, so that they can be wrapped
+  make "$MAKEJOBS" -C src/secp256k1 VERBOSE=1
+  make "$MAKEJOBS" -C src minisketch/test.exe VERBOSE=1
+  "${BASE_ROOT_DIR}/ci/test/wrap-wine.sh"
+fi
+
+if [ -n "$USE_VALGRIND" ]; then
+  "${BASE_ROOT_DIR}/ci/test/wrap-valgrind.sh"
+fi
+
+if [ "$RUN_UNIT_TESTS" = "true" ]; then
+  DIR_UNIT_TEST_DATA="${DIR_UNIT_TEST_DATA}" LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" make "${MAKEJOBS}" check VERBOSE=1
+fi
+
+if [ "$RUN_UNIT_TESTS_SEQUENTIAL" = "true" ]; then
+  DIR_UNIT_TEST_DATA="${DIR_UNIT_TEST_DATA}" LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" "${BASE_OUTDIR}"/bin/test_bitcoin --catch_system_errors=no -l test_suite
+fi
+
+if [ "$RUN_FUNCTIONAL_TESTS" = "true" ]; then
+  # shellcheck disable=SC2086
+  LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" test/functional/test_runner.py --ci "${MAKEJOBS}" --tmpdirprefix "${BASE_SCRATCH_DIR}"/test_runner/ --ansi --combinedlogslen=99999999 --timeout-factor="${TEST_RUNNER_TIMEOUT_FACTOR}" ${TEST_RUNNER_EXTRA} --quiet --failfast
+fi
+
+if [ "${RUN_TIDY}" = "true" ]; then
+  cmake -B /tidy-build -DLLVM_DIR=/usr/lib/llvm-"${TIDY_LLVM_V}"/cmake -DCMAKE_BUILD_TYPE=Release -S "${BASE_ROOT_DIR}"/contrib/devtools/bitcoin-tidy
+  cmake --build /tidy-build "$MAKEJOBS"
+  cmake --build /tidy-build --target bitcoin-tidy-tests "$MAKEJOBS"
+
+  set -eo pipefail
+  cd "${BASE_BUILD_DIR}/bitcoin-$HOST/src/"
+  ( run-clang-tidy-"${TIDY_LLVM_V}" -quiet -load="/tidy-build/libbitcoin-tidy.so" "${MAKEJOBS}" ) | grep -C5 "error"
+  # Filter out files by regex here, because regex may not be
+  # accepted in src/.bear-tidy-config
+  # Filter out:
+  # * qt qrc and moc generated files
+  jq 'map(select(.file | test("src/qt/qrc_.*\\.cpp$|/moc_.*\\.cpp$") | not))' ../compile_commands.json > tmp.json
+  mv tmp.json ../compile_commands.json
+  cd "${BASE_BUILD_DIR}/bitcoin-$HOST/"
+  python3 "/include-what-you-use/iwyu_tool.py" \
+           -p . "${MAKEJOBS}" \
+           -- -Xiwyu --cxx17ns -Xiwyu --mapping_file="${BASE_BUILD_DIR}/bitcoin-$HOST/contrib/devtools/iwyu/bitcoin.core.imp" \
+           -Xiwyu --max_line_length=160 \
+           2>&1 | tee /tmp/iwyu_ci.out
+  cd "${BASE_ROOT_DIR}/src"
+  python3 "/include-what-you-use/fix_includes.py" --nosafe_headers < /tmp/iwyu_ci.out
+  git --no-pager diff
+fi
+
+if [ "$RUN_FUZZ_TESTS" = "true" ]; then
+  # shellcheck disable=SC2086
+  LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" test/fuzz/test_runner.py ${FUZZ_TESTS_CONFIG} "${MAKEJOBS}" -l DEBUG "${DIR_FUZZ_IN}" --empty_min_time=60
+fi

ci/test/04_install.sh

@@ -1,110 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2018-2022 The Bitcoin Core developers
-# Distributed under the MIT software license, see the accompanying
-# file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-export LC_ALL=C.UTF-8
-
-if [[ $QEMU_USER_CMD == qemu-s390* ]]; then
-  export LC_ALL=C
-fi
-
-# Create folders that are mounted into the docker
-mkdir -p "${CCACHE_DIR}"
-mkdir -p "${PREVIOUS_RELEASES_DIR}"
-
-export ASAN_OPTIONS="detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1"
-export LSAN_OPTIONS="suppressions=${BASE_ROOT_DIR}/test/sanitizer_suppressions/lsan"
-export TSAN_OPTIONS="suppressions=${BASE_ROOT_DIR}/test/sanitizer_suppressions/tsan:halt_on_error=1:log_path=${BASE_SCRATCH_DIR}/sanitizer-output/tsan"
-export UBSAN_OPTIONS="suppressions=${BASE_ROOT_DIR}/test/sanitizer_suppressions/ubsan:print_stacktrace=1:halt_on_error=1:report_error_type=1"
-env | grep -E '^(BITCOIN_CONFIG|BASE_|QEMU_|CCACHE_|LC_ALL|BOOST_TEST_RANDOM|DEBIAN_FRONTEND|CONFIG_SHELL|(ASAN|LSAN|TSAN|UBSAN)_OPTIONS|PREVIOUS_RELEASES_DIR)' | tee /tmp/env
-if [[ $BITCOIN_CONFIG = *--with-sanitizers=*address* ]]; then # If ran with (ASan + LSan), Docker needs access to ptrace (https://github.com/google/sanitizers/issues/764)
-  CI_CONTAINER_CAP="--cap-add SYS_PTRACE"
-fi
-
-export P_CI_DIR="$PWD"
-export BINS_SCRATCH_DIR="${BASE_SCRATCH_DIR}/bins/"
-
-if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
-  echo "Creating $CI_IMAGE_NAME_TAG container to run in"
-  DOCKER_BUILDKIT=1 ${CI_RETRY_EXE} docker build \
-      --file "${BASE_ROOT_DIR}/ci/test_imagefile" \
-      --build-arg "CI_IMAGE_NAME_TAG=${CI_IMAGE_NAME_TAG}" \
-      --build-arg "FILE_ENV=${FILE_ENV}" \
-      --tag="${CONTAINER_NAME}" \
-      "${BASE_ROOT_DIR}"
-  docker volume create "${CONTAINER_NAME}_ccache" || true
-  docker volume create "${CONTAINER_NAME}_depends" || true
-  docker volume create "${CONTAINER_NAME}_previous_releases" || true
-
-  if [ -n "${RESTART_CI_DOCKER_BEFORE_RUN}" ] ; then
-    echo "Restart docker before run to stop and clear all containers started with --rm"
-    systemctl restart docker
-  fi
-
-  # shellcheck disable=SC2086
-  CI_CONTAINER_ID=$(docker run $CI_CONTAINER_CAP --rm --interactive --detach --tty \
-                  --mount type=bind,src=$BASE_ROOT_DIR,dst=/ro_base,readonly \
-                  --mount "type=volume,src=${CONTAINER_NAME}_ccache,dst=$CCACHE_DIR" \
-                  --mount "type=volume,src=${CONTAINER_NAME}_depends,dst=$DEPENDS_DIR" \
-                  --mount "type=volume,src=${CONTAINER_NAME}_previous_releases,dst=$PREVIOUS_RELEASES_DIR" \
-                  -w $BASE_ROOT_DIR \
-                  --env-file /tmp/env \
-                  --name $CONTAINER_NAME \
-                  $CONTAINER_NAME)
-  export CI_CONTAINER_ID
-  export CI_EXEC_CMD_PREFIX="docker exec ${CI_CONTAINER_ID}"
-else
-  echo "Running on host system without docker wrapper"
-fi
-
-CI_EXEC () {
-  $CI_EXEC_CMD_PREFIX bash -c "export PATH=${BINS_SCRATCH_DIR}:\$PATH && cd \"$P_CI_DIR\" && $*"
-}
-export -f CI_EXEC
-
-CI_EXEC rsync --archive --stats --human-readable /ci_base_install/ "${BASE_ROOT_DIR}" || echo "/ci_base_install/ missing"
-CI_EXEC "${BASE_ROOT_DIR}/ci/test/01_base_install.sh"
-CI_EXEC rsync --archive --stats --human-readable /ro_base/ "${BASE_ROOT_DIR}" || echo "Nothing to copy from ro_base"
-# Fixes permission issues when there is a container UID/GID mismatch with the owner
-# of the git source code directory.
-CI_EXEC git config --global --add safe.directory \"*\"
-
-CI_EXEC mkdir -p "${BINS_SCRATCH_DIR}"
-
-if [ "$CI_OS_NAME" == "macos" ]; then
-  top -l 1 -s 0 | awk ' /PhysMem/ {print}'
-  echo "Number of CPUs: $(sysctl -n hw.logicalcpu)"
-else
-  CI_EXEC free -m -h
-  CI_EXEC echo "Number of CPUs \(nproc\):" \$\(nproc\)
-  CI_EXEC echo "$(lscpu | grep Endian)"
-fi
-CI_EXEC echo "Free disk space:"
-CI_EXEC df -h
-
-if [ "$RUN_FUZZ_TESTS" = "true" ]; then
-  export DIR_FUZZ_IN=${DIR_QA_ASSETS}/fuzz_seed_corpus/
-  if [ ! -d "$DIR_FUZZ_IN" ]; then
-    CI_EXEC git clone --depth=1 https://github.com/bitcoin-core/qa-assets "${DIR_QA_ASSETS}"
-  fi
-elif [ "$RUN_UNIT_TESTS" = "true" ] || [ "$RUN_UNIT_TESTS_SEQUENTIAL" = "true" ]; then
-  export DIR_UNIT_TEST_DATA=${DIR_QA_ASSETS}/unit_test_data/
-  if [ ! -d "$DIR_UNIT_TEST_DATA" ]; then
-    CI_EXEC mkdir -p "$DIR_UNIT_TEST_DATA"
-    CI_EXEC curl --location --fail https://github.com/bitcoin-core/qa-assets/raw/main/unit_test_data/script_assets_test.json -o "${DIR_UNIT_TEST_DATA}/script_assets_test.json"
-  fi
-fi
-
-CI_EXEC mkdir -p "${BASE_SCRATCH_DIR}/sanitizer-output/"
-
-if [ "$USE_BUSY_BOX" = "true" ]; then
-  echo "Setup to use BusyBox utils"
-  # tar excluded for now because it requires passing in the exact archive type in ./depends (fixed in later BusyBox version)
-  # ar excluded for now because it does not recognize the -q option in ./depends (unknown if fixed)
-  # shellcheck disable=SC1010
-  CI_EXEC for util in \$\(busybox --list \| grep -v "^ar$" \| grep -v "^tar$" \)\; do ln -s \$\(command -v busybox\) "${BINS_SCRATCH_DIR}/\$util"\; done
-  # Print BusyBox version
-  CI_EXEC patch --help
-fi

ci/test/05_before_script.sh

@@ -1,29 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2018-2022 The Bitcoin Core developers
-# Distributed under the MIT software license, see the accompanying
-# file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-export LC_ALL=C.UTF-8
-
-# Make sure default datadir does not exist and is never read by creating a dummy file
-if [ "$CI_OS_NAME" == "macos" ]; then
-  echo > "${HOME}/Library/Application Support/Bitcoin"
-else
-  CI_EXEC echo \> \$HOME/.bitcoin
-fi
-
-if [ -z "$NO_DEPENDS" ]; then
-  if [[ $CI_IMAGE_NAME_TAG == *centos* ]]; then
-    # CentOS has problems building the depends if the config shell is not explicitly set
-    # (i.e. for libevent a Makefile with an empty SHELL variable is generated, leading to
-    #  an error as the first command is executed)
-    SHELL_OPTS="LC_ALL=en_US.UTF-8 CONFIG_SHELL=/bin/dash"
-  else
-    SHELL_OPTS="CONFIG_SHELL="
-  fi
-  CI_EXEC "$SHELL_OPTS" make "$MAKEJOBS" -C depends HOST="$HOST" "$DEP_OPTS" LOG=1
-fi
-if [ "$DOWNLOAD_PREVIOUS_RELEASES" = "true" ]; then
-  CI_EXEC test/get_previous_releases.py -b -t "$PREVIOUS_RELEASES_DIR"
-fi

ci/test/06_script_a.sh

@@ -1,68 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2018-2022 The Bitcoin Core developers
-# Distributed under the MIT software license, see the accompanying
-# file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-export LC_ALL=C.UTF-8
-
-BITCOIN_CONFIG_ALL="--enable-suppress-external-warnings --disable-dependency-tracking"
-if [ -z "$NO_DEPENDS" ]; then
-  BITCOIN_CONFIG_ALL="${BITCOIN_CONFIG_ALL} CONFIG_SITE=$DEPENDS_DIR/$HOST/share/config.site"
-fi
-if [ -z "$NO_WERROR" ]; then
-  BITCOIN_CONFIG_ALL="${BITCOIN_CONFIG_ALL} --enable-werror"
-fi
-
-CI_EXEC "ccache --zero-stats --max-size=$CCACHE_SIZE"
-PRINT_CCACHE_STATISTICS="ccache --version | head -n 1 && ccache --show-stats"
-
-if [ -n "$ANDROID_TOOLS_URL" ]; then
-  CI_EXEC make distclean || true
-  CI_EXEC ./autogen.sh
-  CI_EXEC ./configure "$BITCOIN_CONFIG_ALL" "$BITCOIN_CONFIG" || ( (CI_EXEC cat config.log) && false)
-  CI_EXEC "make $MAKEJOBS && cd src/qt && ANDROID_HOME=${ANDROID_HOME} ANDROID_NDK_HOME=${ANDROID_NDK_HOME} make apk"
-  CI_EXEC "${PRINT_CCACHE_STATISTICS}"
-  exit 0
-fi
-
-BITCOIN_CONFIG_ALL="${BITCOIN_CONFIG_ALL} --enable-external-signer --prefix=$BASE_OUTDIR"
-
-if [ -n "$CONFIG_SHELL" ]; then
-  CI_EXEC "$CONFIG_SHELL" -c "./autogen.sh"
-else
-  CI_EXEC ./autogen.sh
-fi
-
-CI_EXEC mkdir -p "${BASE_BUILD_DIR}"
-export P_CI_DIR="${BASE_BUILD_DIR}"
-
-CI_EXEC "${BASE_ROOT_DIR}/configure" --cache-file=config.cache "$BITCOIN_CONFIG_ALL" "$BITCOIN_CONFIG" || ( (CI_EXEC cat config.log) && false)
-
-CI_EXEC make distdir VERSION="$HOST"
-
-export P_CI_DIR="${BASE_BUILD_DIR}/bitcoin-$HOST"
-
-CI_EXEC ./configure --cache-file=../config.cache "$BITCOIN_CONFIG_ALL" "$BITCOIN_CONFIG" || ( (CI_EXEC cat config.log) && false)
-
-set -o errtrace
-trap 'CI_EXEC "cat ${BASE_SCRATCH_DIR}/sanitizer-output/* 2> /dev/null"' ERR
-
-if [[ ${USE_MEMORY_SANITIZER} == "true" ]]; then
-  # MemorySanitizer (MSAN) does not support tracking memory initialization done by
-  # using the Linux getrandom syscall. Avoid using getrandom by undefining
-  # HAVE_SYS_GETRANDOM. See https://github.com/google/sanitizers/issues/852 for
-  # details.
-  CI_EXEC 'grep -v HAVE_SYS_GETRANDOM src/config/bitcoin-config.h > src/config/bitcoin-config.h.tmp && mv src/config/bitcoin-config.h.tmp src/config/bitcoin-config.h'
-fi
-
-if [[ "${RUN_TIDY}" == "true" ]]; then
-  MAYBE_BEAR="bear --config src/.bear-tidy-config"
-  MAYBE_TOKEN="--"
-fi
-
-CI_EXEC "${MAYBE_BEAR}" "${MAYBE_TOKEN}" make "$MAKEJOBS" "$GOAL" || ( echo "Build failure. Verbose build follows." && CI_EXEC make "$GOAL" V=1 ; false )
-
-CI_EXEC "${PRINT_CCACHE_STATISTICS}"
-CI_EXEC du -sh "${DEPENDS_DIR}"/*/
-CI_EXEC du -sh "${PREVIOUS_RELEASES_DIR}"

ci/test/06_script_b.sh

@@ -1,104 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2018-2022 The Bitcoin Core developers
-# Distributed under the MIT software license, see the accompanying
-# file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-export LC_ALL=C.UTF-8
-
-if [[ $HOST = *-mingw32 ]]; then
-  # Generate all binaries, so that they can be wrapped
-  CI_EXEC make "$MAKEJOBS" -C src/secp256k1 VERBOSE=1
-  CI_EXEC make "$MAKEJOBS" -C src minisketch/test.exe VERBOSE=1
-  CI_EXEC "${BASE_ROOT_DIR}/ci/test/wrap-wine.sh"
-fi
-
-if [ -n "$QEMU_USER_CMD" ]; then
-  # Generate all binaries, so that they can be wrapped
-  CI_EXEC make "$MAKEJOBS" -C src/secp256k1 VERBOSE=1
-  CI_EXEC make "$MAKEJOBS" -C src minisketch/test VERBOSE=1
-  CI_EXEC "${BASE_ROOT_DIR}/ci/test/wrap-qemu.sh"
-fi
-
-if [ -n "$USE_VALGRIND" ]; then
-  CI_EXEC "${BASE_ROOT_DIR}/ci/test/wrap-valgrind.sh"
-fi
-
-if [ "$RUN_UNIT_TESTS" = "true" ]; then
-  CI_EXEC "${TEST_RUNNER_ENV}" DIR_UNIT_TEST_DATA="${DIR_UNIT_TEST_DATA}" LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" make "$MAKEJOBS" check VERBOSE=1
-fi
-
-if [ "$RUN_UNIT_TESTS_SEQUENTIAL" = "true" ]; then
-  CI_EXEC "${TEST_RUNNER_ENV}" DIR_UNIT_TEST_DATA="${DIR_UNIT_TEST_DATA}" LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" "${BASE_OUTDIR}/bin/test_bitcoin" --catch_system_errors=no -l test_suite
-fi
-
-if [ "$RUN_FUNCTIONAL_TESTS" = "true" ]; then
-  CI_EXEC LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" "${TEST_RUNNER_ENV}" test/functional/test_runner.py --ci "$MAKEJOBS" --tmpdirprefix "${BASE_SCRATCH_DIR}/test_runner/" --ansi --combinedlogslen=99999999 --timeout-factor="${TEST_RUNNER_TIMEOUT_FACTOR}" "${TEST_RUNNER_EXTRA}" --quiet --failfast
-fi
-
-if [ "${RUN_TIDY}" = "true" ]; then
-  set -eo pipefail
-  export P_CI_DIR="${BASE_BUILD_DIR}/bitcoin-$HOST/src/"
-  ( CI_EXEC run-clang-tidy-16 -quiet "${MAKEJOBS}" ) | grep -C5 "error"
-  export P_CI_DIR="${BASE_BUILD_DIR}/bitcoin-$HOST/"
-  CI_EXEC "python3 ${DIR_IWYU}/include-what-you-use/iwyu_tool.py"\
-          " src/common/init.cpp"\
-          " src/common/url.cpp"\
-          " src/compat"\
-          " src/dbwrapper.cpp"\
-          " src/init"\
-          " src/kernel"\
-          " src/node/chainstate.cpp"\
-          " src/node/chainstatemanager_args.cpp"\
-          " src/node/mempool_args.cpp"\
-          " src/node/minisketchwrapper.cpp"\
-          " src/node/utxo_snapshot.cpp"\
-          " src/node/validation_cache_args.cpp"\
-          " src/policy/feerate.cpp"\
-          " src/policy/packages.cpp"\
-          " src/policy/settings.cpp"\
-          " src/primitives/transaction.cpp"\
-          " src/random.cpp"\
-          " src/rpc/fees.cpp"\
-          " src/rpc/signmessage.cpp"\
-          " src/test/fuzz/string.cpp"\
-          " src/test/fuzz/txorphan.cpp"\
-          " src/test/fuzz/util/"\
-          " src/test/util/coins.cpp"\
-          " src/uint256.cpp"\
-          " src/util/bip32.cpp"\
-          " src/util/bytevectorhash.cpp"\
-          " src/util/check.cpp"\
-          " src/util/error.cpp"\
-          " src/util/exception.cpp"\
-          " src/util/getuniquepath.cpp"\
-          " src/util/hasher.cpp"\
-          " src/util/message.cpp"\
-          " src/util/moneystr.cpp"\
-          " src/util/serfloat.cpp"\
-          " src/util/spanparsing.cpp"\
-          " src/util/strencodings.cpp"\
-          " src/util/string.cpp"\
-          " src/util/syserror.cpp"\
-          " src/util/threadinterrupt.cpp"\
-          " src/zmq"\
-          " -p . ${MAKEJOBS}"\
-          " -- -Xiwyu --cxx17ns -Xiwyu --mapping_file=${BASE_BUILD_DIR}/bitcoin-$HOST/contrib/devtools/iwyu/bitcoin.core.imp"\
-          " |& tee /tmp/iwyu_ci.out"
-  export P_CI_DIR="${BASE_ROOT_DIR}/src"
-  CI_EXEC "python3 ${DIR_IWYU}/include-what-you-use/fix_includes.py --nosafe_headers < /tmp/iwyu_ci.out"
-  CI_EXEC "git --no-pager diff"
-fi
-
-if [ "$RUN_SECURITY_TESTS" = "true" ]; then
-  CI_EXEC make test-security-check
-fi
-
-if [ "$RUN_FUZZ_TESTS" = "true" ]; then
-  CI_EXEC LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" test/fuzz/test_runner.py "${FUZZ_TESTS_CONFIG}" "$MAKEJOBS" -l DEBUG "${DIR_FUZZ_IN}"
-fi
-
-if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
-  echo "Stop and remove CI container by ID"
-  docker container kill "${CI_CONTAINER_ID}"
-fi

ci/test/wrap-qemu.sh

@@ -1,18 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2018-2022 The Bitcoin Core developers
-# Distributed under the MIT software license, see the accompanying
-# file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-export LC_ALL=C.UTF-8
-
-for b_name in {"${BASE_OUTDIR}/bin"/*,src/secp256k1/*tests,src/minisketch/test{,-verify},src/univalue/{test_json,unitester,object}}; do
-    # shellcheck disable=SC2044
-    for b in $(find "${BASE_ROOT_DIR}" -executable -type f -name "$(basename "$b_name")"); do
-      echo "Wrap $b ..."
-      mv "$b" "${b}_orig"
-      echo '#!/usr/bin/env bash' > "$b"
-      echo "$QEMU_USER_CMD \"${b}_orig\" \"\$@\"" >> "$b"
-      chmod +x "$b"
-    done
-done

ci/test/wrapped-cl.bat

@@ -1 +0,0 @@
-ccache cl %*

ci/test_imagefile

@@ -1,3 +1,9 @@
+# Copyright (c) The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or https://opensource.org/license/mit/.
+
+# See ci/README.md for usage.
+
 ARG CI_IMAGE_NAME_TAG
 FROM ${CI_IMAGE_NAME_TAG}
 
@@ -5,6 +11,6 @@ ARG FILE_ENV
 ENV FILE_ENV=${FILE_ENV}
 
 COPY ./ci/retry/retry /usr/bin/retry
-COPY ./ci/test/00_setup_env.sh ./${FILE_ENV} ./ci/test/01_base_install.sh /ci_base_install/ci/test/
+COPY ./ci/test/00_setup_env.sh ./${FILE_ENV} ./ci/test/01_base_install.sh /ci_container_base/ci/test/
 
-RUN ["bash", "-c", "cd /ci_base_install/ && set -o errexit && source ./ci/test/00_setup_env.sh && ./ci/test/01_base_install.sh"]
+RUN ["bash", "-c", "cd /ci_container_base/ && set -o errexit && source ./ci/test/00_setup_env.sh && ./ci/test/01_base_install.sh"]

ci/test_run_all.sh

@@ -1,13 +1,11 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-2020 The Bitcoin Core developers
+# Copyright (c) 2019-present The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
 set -o errexit; source ./ci/test/00_setup_env.sh
-set -o errexit; source ./ci/test/04_install.sh
-set -o errexit; source ./ci/test/05_before_script.sh
-set -o errexit; source ./ci/test/06_script_a.sh
-set -o errexit; source ./ci/test/06_script_b.sh
+set -o errexit
+"./ci/test/02_run_container.sh"

configure.ac

@@ -1,10 +1,10 @@
 AC_PREREQ([2.69])
-define(_CLIENT_VERSION_MAJOR, 25)
-define(_CLIENT_VERSION_MINOR, 99)
+define(_CLIENT_VERSION_MAJOR, 27)
+define(_CLIENT_VERSION_MINOR, 1)
 define(_CLIENT_VERSION_BUILD, 0)
-define(_CLIENT_VERSION_RC, 0)
-define(_CLIENT_VERSION_IS_RELEASE, false)
-define(_COPYRIGHT_YEAR, 2023)
+define(_CLIENT_VERSION_RC, 1)
+define(_CLIENT_VERSION_IS_RELEASE, true)
+define(_COPYRIGHT_YEAR, 2024)
 define(_COPYRIGHT_HOLDERS,[The %s developers])
 define(_COPYRIGHT_HOLDERS_SUBSTITUTION,[[Bitcoin Core]])
 AC_INIT([Bitcoin Core],m4_join([.], _CLIENT_VERSION_MAJOR, _CLIENT_VERSION_MINOR, _CLIENT_VERSION_BUILD)m4_if(_CLIENT_VERSION_RC, [0], [], [rc]_CLIENT_VERSION_RC),[https://github.com/bitcoin/bitcoin/issues],[bitcoin],[https://bitcoincore.org/])
@@ -70,11 +70,12 @@ else
 fi
 AC_PROG_CXX
 
-dnl By default, libtool for mingw refuses to link static libs into a dll for
-dnl fear of mixing pic/non-pic objects, and import/export complications. Since
-dnl we have those under control, re-enable that functionality.
+dnl libtool overrides
 case $host in
   *mingw*)
+     dnl By default, libtool for mingw refuses to link static libs into a dll for
+     dnl fear of mixing pic/non-pic objects, and import/export complications. Since
+     dnl we have those under control, re-enable that functionality.
      lt_cv_deplibs_check_method="pass_all"
 
      dnl Remove unwanted -DDLL_EXPORT from these variables.
@@ -83,29 +84,20 @@ case $host in
      lt_cv_prog_compiler_pic="-DPIC"
      lt_cv_prog_compiler_pic_CXX="-DPIC"
   ;;
+  *darwin*)
+     dnl Because it prints a verbose warning, lld fails the following check
+     dnl for "-Wl,-single_module" from libtool.m4:
+     dnl   # If there is a non-empty error log, and "single_module"
+     dnl   # appears in it, assume the flag caused a linker warning
+     dnl "-single_module" works fine on ld64 and lld, so just bypass the test.
+     dnl Failure to set this to "yes" causes libtool to use a very broken
+     dnl link-line for shared libs.
+     lt_cv_apple_cc_single_mod="yes"
+  ;;
 esac
 
-AC_ARG_WITH([seccomp],
-  [AS_HELP_STRING([--with-seccomp],
-  [enable experimental syscall sandbox feature (-sandbox), default is yes if seccomp-bpf is detected under Linux x86_64])],
-  [seccomp_found=$withval],
-  [seccomp_found=auto])
-
-AC_ARG_ENABLE([c++20],
-  [AS_HELP_STRING([--enable-c++20],
-  [enable compilation in c++20 mode (disabled by default)])],
-  [use_cxx20=$enableval],
-  [use_cxx20=no])
-
-dnl Require C++17 compiler (no GNU extensions)
-if test "$use_cxx20" = "no"; then
-AX_CXX_COMPILE_STDCXX([17], [noext], [mandatory])
-else
+dnl Require C++20 compiler (no GNU extensions)
 AX_CXX_COMPILE_STDCXX([20], [noext], [mandatory])
-fi
-
-dnl check if additional link flags are required for std::filesystem
-CHECK_FILESYSTEM
 
 dnl Unless the user specified OBJCXX, force it to be the same as CXX. This ensures
 dnl that we get the same -std flags for both.
@@ -126,8 +118,8 @@ AC_PATH_TOOL([AR], [ar])
 AC_PATH_TOOL([GCOV], [gcov])
 AC_PATH_TOOL([LLVM_COV], [llvm-cov])
 AC_PATH_PROG([LCOV], [lcov])
-dnl Python 3.7 is specified in .python-version and should be used if available, see doc/dependencies.md
-AC_PATH_PROGS([PYTHON], [python3.7 python3.8 python3.9 python3.10 python3.11 python3.12 python3 python])
+dnl The minimum supported version is specified in .python-version and should be used if available, see doc/dependencies.md
+AC_PATH_PROGS([PYTHON], [python3.9 python3.10 python3.11 python3.12 python3 python])
 AC_PATH_PROG([GENHTML], [genhtml])
 AC_PATH_PROG([GIT], [git])
 AC_PATH_PROG([CCACHE], [ccache])
@@ -137,8 +129,6 @@ AC_PATH_TOOL([OBJCOPY], [objcopy])
 AC_PATH_PROG([DOXYGEN], [doxygen])
 AM_CONDITIONAL([HAVE_DOXYGEN], [test -n "$DOXYGEN"])
 
-AC_ARG_VAR([PYTHONPATH], [Augments the default search path for python module files])
-
 AC_ARG_ENABLE([wallet],
   [AS_HELP_STRING([--disable-wallet],
   [disable wallet (enabled by default)])],
@@ -236,10 +226,10 @@ dnl May be useful if warnings from external headers clutter the build output
 dnl too much, so that it becomes difficult to spot Bitcoin Core warnings
 dnl or if they cause a build failure with --enable-werror.
 AC_ARG_ENABLE([suppress-external-warnings],
-  [AS_HELP_STRING([--enable-suppress-external-warnings],
-                  [Suppress warnings from external headers (default is no)])],
+  [AS_HELP_STRING([--disable-suppress-external-warnings],
+                  [Do not suppress warnings from external headers (default is to suppress)])],
   [suppress_external_warnings=$enableval],
-  [suppress_external_warnings=no])
+  [suppress_external_warnings=yes])
 
 AC_ARG_ENABLE([lcov],
   [AS_HELP_STRING([--enable-lcov],
@@ -259,16 +249,6 @@ AC_ARG_ENABLE([threadlocal],
   [use_thread_local=$enableval],
   [use_thread_local=auto])
 
-AC_ARG_ENABLE([asm],
-  [AS_HELP_STRING([--disable-asm],
-  [disable assembly routines (enabled by default)])],
-  [use_asm=$enableval],
-  [use_asm=yes])
-
-if test "$use_asm" = "yes"; then
-  AC_DEFINE([USE_ASM], [1], [Define this symbol to build in assembly routines])
-fi
-
 AC_ARG_ENABLE([zmq],
   [AS_HELP_STRING([--disable-zmq],
   [disable ZMQ notifications])],
@@ -331,13 +311,17 @@ AC_ARG_ENABLE([external-signer],
     [use_external_signer=$enableval],
     [use_external_signer=auto])
 
-AC_ARG_ENABLE([lto],
-    [AS_HELP_STRING([--enable-lto],[build using LTO (default is no)])],
-    [enable_lto=$enableval],
-    [enable_lto=no])
-
 AC_LANG_PUSH([C++])
 
+dnl Always set -g -O2 in our CXXFLAGS. Autoconf will try and set CXXFLAGS to "-g -O2" by default,
+dnl so we suppress that (if CXXFLAGS hasn't been overridden by the user), given we are adding it
+dnl ourselves.
+CORE_CXXFLAGS="$CORE_CXXFLAGS -g -O2"
+
+if test "$CXXFLAGS_overridden" = "no"; then
+  CXXFLAGS=""
+fi
+
 dnl Check for a flag to turn compiler warnings into errors. This is helpful for checks which may
 dnl appear to succeed because by default they merely emit warnings when they fail.
 dnl
@@ -362,12 +346,6 @@ case $host in
 esac
 
 if test "$enable_debug" = "yes"; then
-  dnl If debugging is enabled, and the user hasn't overridden CXXFLAGS, clear
-  dnl them, to prevent autoconfs "-g -O2" being added. Otherwise we'd end up
-  dnl with "-O0 -g3 -g -O2".
-  if test "$CXXFLAGS_overridden" = "no"; then
-  CXXFLAGS=""
-  fi
 
   dnl Disable all optimizations
   AX_CHECK_COMPILE_FLAG([-O0], [DEBUG_CXXFLAGS="$DEBUG_CXXFLAGS -O0"], [], [$CXXFLAG_WERROR])
@@ -387,18 +365,14 @@ if test "$enable_debug" = "yes"; then
   AX_CHECK_COMPILE_FLAG([-ftrapv], [DEBUG_CXXFLAGS="$DEBUG_CXXFLAGS -ftrapv"], [], [$CXXFLAG_WERROR])
 fi
 
-if test "$enable_lto" = "yes"; then
-  AX_CHECK_COMPILE_FLAG([-flto], [LTO_CXXFLAGS="$LTO_CXXFLAGS -flto"], [AC_MSG_ERROR([compile failed with -flto])], [$CXXFLAG_WERROR])
-  AX_CHECK_LINK_FLAG([-flto], [LTO_LDFLAGS="$LTO_LDFLAGS -flto"], [AC_MSG_ERROR([link failed with -flto])], [$CXXFLAG_WERROR])
-fi
-
 if test "$use_sanitizers" != ""; then
   dnl First check if the compiler accepts flags. If an incompatible pair like
   dnl -fsanitize=address,thread is used here, this check will fail. This will also
   dnl fail if a bad argument is passed, e.g. -fsanitize=undfeined
   AX_CHECK_COMPILE_FLAG(
     [-fsanitize=$use_sanitizers],
-    [SANITIZER_CXXFLAGS="-fsanitize=$use_sanitizers"],
+    [SANITIZER_CXXFLAGS="-fsanitize=$use_sanitizers"
+     SANITIZER_CFLAGS="-fsanitize=$use_sanitizers"],
     [AC_MSG_ERROR([compiler did not accept requested flags])])
 
   dnl Some compilers (e.g. GCC) require additional libraries like libasan,
@@ -425,12 +399,6 @@ if test "$enable_werror" = "yes"; then
     AC_MSG_ERROR([enable-werror set but -Werror is not usable])
   fi
   ERROR_CXXFLAGS=$CXXFLAG_WERROR
-
-  dnl -Wreturn-type is broken in GCC for MinGW-w64.
-  dnl https://sourceforge.net/p/mingw-w64/bugs/306/
-  AX_CHECK_COMPILE_FLAG([-Werror=return-type], [], [ERROR_CXXFLAGS="$ERROR_CXXFLAGS -Wno-error=return-type"], [$CXXFLAG_WERROR],
-                        [AC_LANG_SOURCE([[#include <cassert>
-                                          int f(){ assert(false); }]])])
 fi
 
 if test "$CXXFLAGS_overridden" = "no"; then
@@ -451,16 +419,10 @@ if test "$CXXFLAGS_overridden" = "no"; then
   AX_CHECK_COMPILE_FLAG([-Wduplicated-cond], [WARN_CXXFLAGS="$WARN_CXXFLAGS -Wduplicated-cond"], [], [$CXXFLAG_WERROR])
   AX_CHECK_COMPILE_FLAG([-Wlogical-op], [WARN_CXXFLAGS="$WARN_CXXFLAGS -Wlogical-op"], [], [$CXXFLAG_WERROR])
   AX_CHECK_COMPILE_FLAG([-Woverloaded-virtual], [WARN_CXXFLAGS="$WARN_CXXFLAGS -Woverloaded-virtual"], [], [$CXXFLAG_WERROR])
-  dnl -Wsuggest-override is broken with GCC before 9.2
-  dnl https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78010
-  AX_CHECK_COMPILE_FLAG([-Wsuggest-override], [WARN_CXXFLAGS="$WARN_CXXFLAGS -Wsuggest-override"], [], [$CXXFLAG_WERROR],
-                        [AC_LANG_SOURCE([[struct A { virtual void f(); }; struct B : A { void f() final; };]])])
-  AX_CHECK_COMPILE_FLAG([-Wunreachable-code-loop-increment], [WARN_CXXFLAGS="$WARN_CXXFLAGS -Wunreachable-code-loop-increment"], [], [$CXXFLAG_WERROR])
+  AX_CHECK_COMPILE_FLAG([-Wsuggest-override], [WARN_CXXFLAGS="$WARN_CXXFLAGS -Wsuggest-override"], [], [$CXXFLAG_WERROR])
   AX_CHECK_COMPILE_FLAG([-Wimplicit-fallthrough], [WARN_CXXFLAGS="$WARN_CXXFLAGS -Wimplicit-fallthrough"], [], [$CXXFLAG_WERROR])
-
-  if test "$suppress_external_warnings" != "no" ; then
-    AX_CHECK_COMPILE_FLAG([-Wdocumentation], [WARN_CXXFLAGS="$WARN_CXXFLAGS -Wdocumentation"], [], [$CXXFLAG_WERROR])
-  fi
+  AX_CHECK_COMPILE_FLAG([-Wunreachable-code], [WARN_CXXFLAGS="$WARN_CXXFLAGS -Wunreachable-code"], [], [$CXXFLAG_WERROR])
+  AX_CHECK_COMPILE_FLAG([-Wdocumentation], [WARN_CXXFLAGS="$WARN_CXXFLAGS -Wdocumentation"], [], [$CXXFLAG_WERROR])
 
   dnl Some compilers (gcc) ignore unknown -Wno-* options, but warn about all
   dnl unknown options if any other warning is produced. Test the -Wfoo case, and
@@ -475,6 +437,12 @@ fi
 dnl Don't allow extended (non-ASCII) symbols in identifiers. This is easier for code review.
 AX_CHECK_COMPILE_FLAG([-fno-extended-identifiers], [CORE_CXXFLAGS="$CORE_CXXFLAGS -fno-extended-identifiers"], [], [$CXXFLAG_WERROR])
 
+dnl Currently all versions of gcc are subject to a class of bugs, see the
+dnl gccbug_90348 test case (only reproduces on GCC 11 and earlier) and
+dnl https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111843. To work around that, set
+dnl -fstack-reuse=none for all gcc builds. (Only gcc understands this flag)
+AX_CHECK_COMPILE_FLAG([-fstack-reuse=none], [CORE_CXXFLAGS="$CORE_CXXFLAGS -fstack-reuse=none"])
+
 enable_arm_crc=no
 enable_arm_shani=no
 enable_sse42=no
@@ -482,8 +450,6 @@ enable_sse41=no
 enable_avx2=no
 enable_x86_shani=no
 
-if test "$use_asm" = "yes"; then
-
 dnl Check for optional instruction set support. Enabling these does _not_ imply that all code will
 dnl be compiled with them, rather that specific objects/libs may use them after checking for runtime
 dnl compatibility.
@@ -582,7 +548,7 @@ AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
 CXXFLAGS="$TEMP_CXXFLAGS"
 
 # ARM
-AX_CHECK_COMPILE_FLAG([-march=armv8-a+crc], [ARM_CRC_CXXFLAGS="-march=armv8-a+crc"], [], [$CXXFLAG_WERROR])
+AX_CHECK_COMPILE_FLAG([-march=armv8-a+crc+crypto], [ARM_CRC_CXXFLAGS="-march=armv8-a+crc+crypto"], [], [$CXXFLAG_WERROR])
 AX_CHECK_COMPILE_FLAG([-march=armv8-a+crypto], [ARM_SHANI_CXXFLAGS="-march=armv8-a+crypto"], [], [$CXXFLAG_WERROR])
 
 TEMP_CXXFLAGS="$CXXFLAGS"
@@ -622,8 +588,6 @@ AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
 )
 CXXFLAGS="$TEMP_CXXFLAGS"
 
-fi
-
 CORE_CPPFLAGS="$CORE_CPPFLAGS -DHAVE_BUILD_INFO"
 
 AC_ARG_WITH([utils],
@@ -712,6 +676,8 @@ case $host in
        AC_MSG_ERROR([windres not found])
      fi
 
+     CORE_CPPFLAGS="$CORE_CPPFLAGS -DSECP256K1_STATIC"
+
      CORE_CPPFLAGS="$CORE_CPPFLAGS -D_MT -DWIN32 -D_WINDOWS -D_WIN32_WINNT=0x0601 -D_WIN32_IE=0x0501 -DWIN32_LEAN_AND_MEAN"
      dnl Prevent the definition of min/max macros.
      dnl We always want to use the standard library.
@@ -727,6 +693,10 @@ case $host in
 
      dnl We require Windows 7 (NT 6.1) or later
      AX_CHECK_LINK_FLAG([-Wl,--major-subsystem-version -Wl,6 -Wl,--minor-subsystem-version -Wl,1], [CORE_LDFLAGS="$CORE_LDFLAGS -Wl,--major-subsystem-version -Wl,6 -Wl,--minor-subsystem-version -Wl,1"], [], [$LDFLAG_WERROR])
+
+     dnl Avoid the use of aligned vector instructions when building for Windows.
+     dnl See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412.
+     AX_CHECK_COMPILE_FLAG([-Wa,-muse-unaligned-vector-move], [CORE_CXXFLAGS="$CORE_CXXFLAGS -Wa,-muse-unaligned-vector-move"], [], [$CXXFLAG_WERROR])
      ;;
   *darwin*)
      TARGET_OS=darwin
@@ -747,7 +717,7 @@ case $host in
          dnl option to system-ify all /usr/local/include paths without adding it to the list
          dnl of search paths in case it's not already there.
          if test "$suppress_external_warnings" != "no"; then
-           AX_CHECK_PREPROC_FLAG([-Xclang -internal-isystem/usr/local/include], [CORE_CPPFLAGS="$CORE_CPPFLAGS -Xclang -internal-isystem/usr/local/include"], [], [$CXXFLAG_WERROR])
+           AX_CHECK_PREPROC_FLAG([-Xclang -internal-isystem -Xclang /usr/local/include/], [CORE_CPPFLAGS="$CORE_CPPFLAGS -Xclang -internal-isystem -Xclang /usr/local/include/"], [], [$CXXFLAG_WERROR])
          fi
 
          if test "$use_bdb" != "no" && $BREW list --versions berkeley-db@4 >/dev/null && test "$BDB_CFLAGS" = "" && test "$BDB_LIBS" = ""; then
@@ -797,7 +767,7 @@ case $host in
            AC_PATH_TOOL([DSYMUTIL], [dsymutil], [dsymutil])
            AC_PATH_TOOL([INSTALL_NAME_TOOL], [install_name_tool], [install_name_tool])
            AC_PATH_TOOL([OTOOL], [otool], [otool])
-           AC_PATH_PROGS([XORRISOFS], [xorrisofs], [xorrisofs])
+           AC_PATH_PROG([ZIP], [zip], [zip])
 
            dnl libtool will try to strip the static lib, which is a problem for
            dnl cross-builds because strip attempts to call a hard-coded ld,
@@ -876,13 +846,7 @@ if test "$use_lcov" = "yes"; then
     [AC_MSG_ERROR([lcov testing requested but --coverage linker flag does not work])])
   AX_CHECK_COMPILE_FLAG([--coverage],[CORE_CXXFLAGS="$CORE_CXXFLAGS --coverage"],
     [AC_MSG_ERROR([lcov testing requested but --coverage flag does not work])])
-  dnl If coverage is enabled, and the user hasn't overridden CXXFLAGS, clear
-  dnl them, to prevent autoconfs "-g -O2" being added. Otherwise we'd end up
-  dnl with "--coverage -Og -O0 -g -O2".
-  if test "$CXXFLAGS_overridden" = "no"; then
-  CXXFLAGS=""
-  fi
-  CORE_CXXFLAGS="$CORE_CXXFLAGS -Og -O0"
+  CORE_CXXFLAGS="$CORE_CXXFLAGS -Og"
 fi
 
 if test "$use_lcov_branch" != "no"; then
@@ -917,8 +881,6 @@ if test "$ac_cv_sys_large_files" != "" &&
   CORE_CPPFLAGS="$CORE_CPPFLAGS -D_LARGE_FILES=$ac_cv_sys_large_files"
 fi
 
-AC_SEARCH_LIBS([clock_gettime],[rt])
-
 if test "$enable_gprof" = "yes"; then
     dnl -pg is incompatible with -pie. Since hardening and profiling together doesn't make sense,
     dnl we simply make them mutually exclusive here. Additionally, hardened toolchains may force
@@ -941,12 +903,6 @@ if test "$TARGET_OS" != "windows"; then
   AX_CHECK_COMPILE_FLAG([-fPIC], [PIC_FLAGS="-fPIC"])
 fi
 
-dnl Versions of gcc prior to 12.1 (commit
-dnl https://github.com/gcc-mirror/gcc/commit/551aa75778a4c5165d9533cd447c8fc822f583e1)
-dnl are subject to a bug, see the gccbug_90348 test case and
-dnl https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90348. To work around that, set
-dnl -fstack-reuse=none for all gcc builds. (Only gcc understands this flag)
-AX_CHECK_COMPILE_FLAG([-fstack-reuse=none], [HARDENED_CXXFLAGS="$HARDENED_CXXFLAGS -fstack-reuse=none"])
 if test "$use_hardening" != "no"; then
   use_hardening=yes
   AX_CHECK_COMPILE_FLAG([-Wstack-protector], [HARDENED_CXXFLAGS="$HARDENED_CXXFLAGS -Wstack-protector"])
@@ -956,7 +912,8 @@ if test "$use_hardening" != "no"; then
 
   case $host in
     *mingw*)
-      dnl stack-clash-protection doesn't currently work, and likely should just be skipped for Windows.
+      dnl stack-clash-protection doesn't compile with GCC 10 and earlier.
+      dnl In any case, it is a no-op for Windows.
       dnl See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90458 for more details.
       ;;
     *)
@@ -964,6 +921,11 @@ if test "$use_hardening" != "no"; then
       ;;
   esac
 
+  case $host in
+    *aarch64*)
+      AX_CHECK_COMPILE_FLAG([-mbranch-protection=bti], [HARDENED_CXXFLAGS="$HARDENED_CXXFLAGS -mbranch-protection=bti"])
+    ;;
+  esac
 
   dnl When enable_debug is yes, all optimizations are disabled.
   dnl However, FORTIFY_SOURCE requires that there is some level of optimization, otherwise it does nothing and just creates a compiler warning.
@@ -985,12 +947,6 @@ if test "$use_hardening" != "no"; then
   AX_CHECK_LINK_FLAG([-Wl,-z,now], [HARDENED_LDFLAGS="$HARDENED_LDFLAGS -Wl,-z,now"], [], [$LDFLAG_WERROR])
   AX_CHECK_LINK_FLAG([-Wl,-z,separate-code], [HARDENED_LDFLAGS="$HARDENED_LDFLAGS -Wl,-z,separate-code"], [], [$LDFLAG_WERROR])
   AX_CHECK_LINK_FLAG([-fPIE -pie], [PIE_FLAGS="-fPIE"; HARDENED_LDFLAGS="$HARDENED_LDFLAGS -pie"], [], [$CXXFLAG_WERROR])
-
-  case $host in
-    *mingw*)
-       AC_CHECK_LIB([ssp], [main], [], [AC_MSG_ERROR([libssp missing])])
-    ;;
-  esac
 fi
 
 dnl These flags are specific to ld64, and may cause issues with other linkers.
@@ -999,10 +955,10 @@ dnl "ad_strip" as the symbol for the entry point.
 if test "$TARGET_OS" = "darwin"; then
   AX_CHECK_LINK_FLAG([-Wl,-dead_strip], [CORE_LDFLAGS="$CORE_LDFLAGS -Wl,-dead_strip"], [], [$LDFLAG_WERROR])
   AX_CHECK_LINK_FLAG([-Wl,-dead_strip_dylibs], [CORE_LDFLAGS="$CORE_LDFLAGS -Wl,-dead_strip_dylibs"], [], [$LDFLAG_WERROR])
-  AX_CHECK_LINK_FLAG([-Wl,-bind_at_load], [HARDENED_LDFLAGS="$HARDENED_LDFLAGS -Wl,-bind_at_load"], [], [$LDFLAG_WERROR])
+  AX_CHECK_LINK_FLAG([-Wl,-fixup_chains], [HARDENED_LDFLAGS="$HARDENED_LDFLAGS -Wl,-fixup_chains"], [], [$LDFLAG_WERROR])
 fi
 
-AC_CHECK_HEADERS([endian.h sys/endian.h byteswap.h sys/select.h sys/prctl.h sys/sysctl.h vm/vm_param.h sys/vmmeter.h sys/resources.h])
+AC_CHECK_HEADERS([sys/select.h sys/prctl.h sys/sysctl.h vm/vm_param.h sys/vmmeter.h sys/resources.h])
 
 AC_CHECK_DECLS([getifaddrs, freeifaddrs],[CHECK_SOCKET],,
     [#include <sys/types.h>
@@ -1017,18 +973,6 @@ AC_CHECK_DECLS([pipe2])
 
 AC_CHECK_FUNCS([timingsafe_bcmp])
 
-AC_CHECK_DECLS([le16toh, le32toh, le64toh, htole16, htole32, htole64, be16toh, be32toh, be64toh, htobe16, htobe32, htobe64],,,
-    [#if HAVE_ENDIAN_H
-                 #include <endian.h>
-                 #elif HAVE_SYS_ENDIAN_H
-                 #include <sys/endian.h>
-                 #endif])
-
-AC_CHECK_DECLS([bswap_16, bswap_32, bswap_64],,,
-    [#if HAVE_BYTESWAP_H
-                 #include <byteswap.h>
-                 #endif])
-
 AC_MSG_CHECKING([for __builtin_clzl])
 AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[ ]], [[
  (void) __builtin_clzl(0);
@@ -1064,7 +1008,7 @@ AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <malloc.h>]],
 dnl Check for posix_fallocate
 AC_MSG_CHECKING([for posix_fallocate])
 AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
-                   // same as in src/util/system.cpp
+                   // same as in src/util/fs_helpers.cpp
                    #ifdef __linux__
                    #ifdef _POSIX_C_SOURCE
                    #undef _POSIX_C_SOURCE
@@ -1162,17 +1106,16 @@ AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <ctime>]],
 )
 
 dnl Check for different ways of gathering OS randomness
-AC_MSG_CHECKING([for Linux getrandom syscall])
-AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <unistd.h>
-  #include <sys/syscall.h>
-  #include <linux/random.h>]],
- [[ syscall(SYS_getrandom, nullptr, 32, 0); ]])],
- [ AC_MSG_RESULT([yes]); AC_DEFINE([HAVE_SYS_GETRANDOM], [1], [Define this symbol if the Linux getrandom system call is available]) ],
+AC_MSG_CHECKING([for Linux getrandom function])
+AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
+  #include <sys/random.h>]],
+ [[ getrandom(nullptr, 32, 0); ]])],
+ [ AC_MSG_RESULT([yes]); AC_DEFINE([HAVE_GETRANDOM], [1], [Define this symbol if the Linux getrandom function call is available]) ],
  [ AC_MSG_RESULT([no])]
 )
 
-AC_MSG_CHECKING([for getentropy via random.h])
-AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <unistd.h>
+AC_MSG_CHECKING([for getentropy via sys/random.h])
+AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
  #include <sys/random.h>]],
  [[ getentropy(nullptr, 32) ]])],
  [ AC_MSG_RESULT([yes]); AC_DEFINE([HAVE_GETENTROPY_RAND], [1], [Define this symbol if the BSD getentropy system call is available with sys/random.h]) ],
@@ -1202,14 +1145,6 @@ AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <sys/types.h>
  [ AC_MSG_RESULT([no])]
 )
 
-AC_MSG_CHECKING([for if type char equals int8_t])
-AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <stdint.h>
-  #include <type_traits>]],
- [[ static_assert(std::is_same<int8_t, char>::value, ""); ]])],
- [ AC_MSG_RESULT([yes]); AC_DEFINE([CHAR_EQUALS_INT8], [1], [Define this symbol if type char equals int8_t]) ],
- [ AC_MSG_RESULT([no])]
-)
-
 AC_MSG_CHECKING([for fdatasync])
 AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <unistd.h>]],
  [[ fdatasync(0); ]])],
@@ -1354,8 +1289,6 @@ if test "$enable_fuzz_binary" = "yes"; then
      ]],[[
       */ int not_main() {
      ]])])
-
-  CHECK_RUNTIME_LIB
 fi
 
 if test "$enable_wallet" != "no"; then
@@ -1401,7 +1334,9 @@ if test "$use_usdt" != "no"; then
   AC_COMPILE_IFELSE([
     AC_LANG_PROGRAM(
       [#include <sys/sdt.h>],
-      [DTRACE_PROBE("context", "event");]
+      [DTRACE_PROBE(context, event);
+       int a, b, c, d, e, f, g;
+       DTRACE_PROBE7(context, event, a, b, c, d, e, f, g);]
     )],
     [AC_MSG_RESULT([yes]); AC_DEFINE([ENABLE_TRACING], [1], [Define to 1 to enable tracepoints for Userspace, Statically Defined Tracing])],
     [AC_MSG_RESULT([no]); use_usdt=no;]
@@ -1419,15 +1354,13 @@ dnl Check for libminiupnpc (optional)
 if test "$use_upnp" != "no"; then
   TEMP_CPPFLAGS="$CPPFLAGS"
   CPPFLAGS="$CPPFLAGS $MINIUPNPC_CPPFLAGS"
-  AC_CHECK_HEADERS(
-    [miniupnpc/miniupnpc.h miniupnpc/upnpcommands.h miniupnpc/upnperrors.h],
-    [AC_CHECK_LIB([miniupnpc], [upnpDiscover], [MINIUPNPC_LIBS="$MINIUPNPC_LIBS -lminiupnpc"], [have_miniupnpc=no], [$MINIUPNPC_LIBS])],
-    [have_miniupnpc=no]
-  )
+  AC_CHECK_HEADERS([miniupnpc/miniupnpc.h miniupnpc/upnpcommands.h miniupnpc/upnperrors.h], [], [have_miniupnpc=no])
 
-  dnl The minimum supported miniUPnPc API version is set to 17. This excludes
-  dnl versions with known vulnerabilities.
   if test "$have_miniupnpc" != "no"; then
+    AC_CHECK_LIB([miniupnpc], [upnpDiscover], [MINIUPNPC_LIBS="$MINIUPNPC_LIBS -lminiupnpc"], [have_miniupnpc=no], [$MINIUPNPC_LIBS])
+
+    dnl The minimum supported miniUPnPc API version is set to 17. This excludes
+    dnl versions with known vulnerabilities.
     AC_MSG_CHECKING([whether miniUPnPc API version is supported])
     AC_PREPROC_IFELSE([AC_LANG_PROGRAM([[
         @%:@include <miniupnpc/miniupnpc.h>
@@ -1452,9 +1385,12 @@ dnl Check for libnatpmp (optional).
 if test "$use_natpmp" != "no"; then
   TEMP_CPPFLAGS="$CPPFLAGS"
   CPPFLAGS="$CPPFLAGS $NATPMP_CPPFLAGS"
-  AC_CHECK_HEADERS([natpmp.h],
-                   [AC_CHECK_LIB([natpmp], [initnatpmp], [NATPMP_LIBS="$NATPMP_LIBS -lnatpmp"], [have_natpmp=no], [$NATPMP_LIBS])],
-                   [have_natpmp=no])
+  AC_CHECK_HEADERS([natpmp.h], [], [have_natpmp=no])
+
+  if test "$have_natpmp" != "no"; then
+    AC_CHECK_LIB([natpmp], [initnatpmp], [NATPMP_LIBS="$NATPMP_LIBS -lnatpmp"], [have_natpmp=no], [$NATPMP_LIBS])
+  fi
+
   CPPFLAGS="$TEMP_CPPFLAGS"
 fi
 
@@ -1467,7 +1403,7 @@ fi
 if test "$use_boost" = "yes"; then
 
   dnl Check for Boost headers
-  AX_BOOST_BASE([1.64.0],[],[AC_MSG_ERROR([Boost is not available!])])
+  AX_BOOST_BASE([1.73.0],[],[AC_MSG_ERROR([Boost is not available!])])
   if test "$want_boost" = "no"; then
     AC_MSG_ERROR([only libbitcoinconsensus can be built without Boost])
   fi
@@ -1482,10 +1418,6 @@ if test "$use_boost" = "yes"; then
   AX_CHECK_PREPROC_FLAG([-DBOOST_NO_CXX98_FUNCTION_BASE], [BOOST_CPPFLAGS="$BOOST_CPPFLAGS -DBOOST_NO_CXX98_FUNCTION_BASE"], [], [$CXXFLAG_WERROR],
                         [AC_LANG_PROGRAM([[#include <boost/config.hpp>]])])
 
-  if test "$enable_debug" = "yes" || test "$enable_fuzz" = "yes"; then
-    BOOST_CPPFLAGS="$BOOST_CPPFLAGS -DBOOST_MULTI_INDEX_ENABLE_SAFE_MODE"
-  fi
-
   if test "$suppress_external_warnings" != "no"; then
     BOOST_CPPFLAGS=SUPPRESS_WARNINGS($BOOST_CPPFLAGS)
   fi
@@ -1522,9 +1454,19 @@ if test "$use_external_signer" != "no"; then
   CXXFLAGS="$TEMP_CXXFLAGS"
   AC_MSG_RESULT([$have_boost_process])
   if test "$have_boost_process" = "yes"; then
-    use_external_signer="yes"
-    AC_DEFINE([ENABLE_EXTERNAL_SIGNER], [1], [Define if external signer support is enabled])
-    AC_DEFINE([BOOST_PROCESS_USE_STD_FS], [1], [Defined to avoid Boost::Process trying to use Boost Filesystem])
+    case $host in
+      dnl Boost Process for Windows uses Boost ASIO. Boost ASIO performs
+      dnl pre-main init of Windows networking libraries, which we do not
+      dnl want.
+      *mingw*)
+        use_external_signer="no"
+      ;;
+      *)
+        use_external_signer="yes"
+        AC_DEFINE([ENABLE_EXTERNAL_SIGNER], [1], [Define if external signer support is enabled])
+        AC_DEFINE([BOOST_PROCESS_USE_STD_FS], [1], [Defined to avoid Boost::Process trying to use Boost Filesystem])
+      ;;
+    esac
   else
     if test "$use_external_signer" = "yes"; then
       AC_MSG_ERROR([External signing is not supported for this Boost version])
@@ -1534,36 +1476,6 @@ if test "$use_external_signer" != "no"; then
 fi
 AM_CONDITIONAL([ENABLE_EXTERNAL_SIGNER], [test "$use_external_signer" = "yes"])
 
-dnl Do not compile with syscall sandbox support when compiling under the sanitizers.
-dnl The sanitizers introduce use of syscalls that are not typically used in bitcoind
-dnl (such as execve when the sanitizers execute llvm-symbolizer).
-if test "$use_sanitizers" != ""; then
-  AC_MSG_WARN([Specifying --with-sanitizers forces --without-seccomp since the sanitizers introduce use of syscalls not allowed by the bitcoind syscall sandbox (-sandbox=<mode>).])
-  seccomp_found=no
-fi
-if test "$seccomp_found" != "no"; then
-  AC_MSG_CHECKING([for seccomp-bpf (Linux x86-64)])
-  AC_PREPROC_IFELSE([AC_LANG_PROGRAM([[
-      @%:@include <linux/seccomp.h>
-    ]], [[
-      #if !defined(__x86_64__)
-      #  error Syscall sandbox is an experimental feature currently available only under Linux x86-64.
-      #endif
-    ]])],[
-      AC_MSG_RESULT([yes])
-      seccomp_found="yes"
-      AC_DEFINE([USE_SYSCALL_SANDBOX], [1], [Define this symbol to build with syscall sandbox support.])
-    ],[
-      AC_MSG_RESULT([no])
-      seccomp_found="no"
-  ])
-fi
-dnl Currently only enable -sandbox=<mode> feature if seccomp is found.
-dnl In the future, sandboxing could be also be supported with other
-dnl sandboxing mechanisms besides seccomp.
-use_syscall_sandbox=$seccomp_found
-AM_CONDITIONAL([ENABLE_SYSCALL_SANDBOX], [test "$use_syscall_sandbox" != "no"])
-
 dnl Check for reduced exports
 if test "$use_reduce_exports" = "yes"; then
   AX_CHECK_COMPILE_FLAG([-fvisibility=hidden], [CORE_CXXFLAGS="$CORE_CXXFLAGS -fvisibility=hidden"],
@@ -1622,12 +1534,9 @@ dnl ZMQ check
 
 if test "$use_zmq" = "yes"; then
   PKG_CHECK_MODULES([ZMQ], [libzmq >= 4],
-    AC_DEFINE([ENABLE_ZMQ], [1], [Define to 1 to enable ZMQ functions]),
-    [AC_DEFINE([ENABLE_ZMQ], [0], [Define to 1 to enable ZMQ functions])
-    AC_MSG_WARN([libzmq version 4.x or greater not found, disabling])
+    AC_DEFINE([ENABLE_ZMQ], [1], [Define this symbol to enable ZMQ functions]),
+    [AC_MSG_WARN([libzmq version 4.x or greater not found, disabling])
     use_zmq=no])
-else
-  AC_DEFINE_UNQUOTED([ENABLE_ZMQ], [0], [Define to 1 to enable ZMQ functions])
 fi
 
 if test "$use_zmq" = "yes"; then
@@ -1639,6 +1548,8 @@ if test "$use_zmq" = "yes"; then
   esac
 fi
 
+AM_CONDITIONAL([ENABLE_ZMQ], [test "$use_zmq" = "yes"])
+
 dnl libmultiprocess library check
 
 libmultiprocess_found=no
@@ -1833,8 +1744,6 @@ if test "$bitcoin_enable_qt" != "no"; then
   fi
 fi
 
-AM_CONDITIONAL([ENABLE_ZMQ], [test "$use_zmq" = "yes"])
-
 AC_MSG_CHECKING([whether to build test_bitcoin])
 if test "$use_tests" = "yes"; then
   if test "$enable_fuzz" = "yes"; then
@@ -1882,7 +1791,6 @@ AM_CONDITIONAL([ENABLE_AVX2], [test "$enable_avx2" = "yes"])
 AM_CONDITIONAL([ENABLE_X86_SHANI], [test "$enable_x86_shani" = "yes"])
 AM_CONDITIONAL([ENABLE_ARM_CRC], [test "$enable_arm_crc" = "yes"])
 AM_CONDITIONAL([ENABLE_ARM_SHANI], [test "$enable_arm_shani" = "yes"])
-AM_CONDITIONAL([USE_ASM], [test "$use_asm" = "yes"])
 AM_CONDITIONAL([WORDS_BIGENDIAN], [test "$ac_cv_c_bigendian" = "yes"])
 AM_CONDITIONAL([USE_NATPMP], [test "$use_natpmp" = "yes"])
 AM_CONDITIONAL([USE_UPNP], [test "$use_upnp" = "yes"])
@@ -1933,8 +1841,6 @@ AC_SUBST(GPROF_LDFLAGS)
 AC_SUBST(HARDENED_CXXFLAGS)
 AC_SUBST(HARDENED_CPPFLAGS)
 AC_SUBST(HARDENED_LDFLAGS)
-AC_SUBST(LTO_CXXFLAGS)
-AC_SUBST(LTO_LDFLAGS)
 AC_SUBST(PIC_FLAGS)
 AC_SUBST(PIE_FLAGS)
 AC_SUBST(SANITIZER_CXXFLAGS)
@@ -1971,7 +1877,6 @@ AC_CONFIG_FILES([contrib/devtools/split-debug.sh],[chmod +x contrib/devtools/spl
 AM_COND_IF([HAVE_DOXYGEN], [AC_CONFIG_FILES([doc/Doxyfile])])
 AC_CONFIG_LINKS([contrib/devtools/iwyu/bitcoin.core.imp:contrib/devtools/iwyu/bitcoin.core.imp])
 AC_CONFIG_LINKS([contrib/filter-lcov.py:contrib/filter-lcov.py])
-AC_CONFIG_LINKS([contrib/macdeploy/background.tiff:contrib/macdeploy/background.tiff])
 AC_CONFIG_LINKS([src/.bear-tidy-config:src/.bear-tidy-config])
 AC_CONFIG_LINKS([src/.clang-tidy:src/.clang-tidy])
 AC_CONFIG_LINKS([test/functional/test_runner.py:test/functional/test_runner.py])
@@ -1989,6 +1894,9 @@ CPPFLAGS_TEMP="$CPPFLAGS"
 unset CPPFLAGS
 CPPFLAGS="$CPPFLAGS_TEMP"
 
+if test -n "$use_sanitizers"; then
+  export SECP_CFLAGS="$SECP_CFLAGS $SANITIZER_CFLAGS"
+fi
 ac_configure_args="${ac_configure_args} --disable-shared --with-pic --enable-benchmark=no --enable-module-recovery --disable-module-ecdh"
 AC_CONFIG_SUBDIRS([src/secp256k1])
 
@@ -2002,11 +1910,21 @@ case ${OS} in
    ;;
 esac
 
+dnl An old hack similar to a98356fee to remove hard-coded
+dnl bind_at_load flag from libtool
+case $host in
+  *darwin*)
+     AC_MSG_RESULT([Removing -Wl,bind_at_load from libtool.])
+     sed < libtool > libtool-2 '/bind_at_load/d'
+     mv libtool-2 libtool
+     chmod 755 libtool
+   ;;
+esac
+
 echo
 echo "Options used to compile and link:"
 echo "  external signer = $use_external_signer"
 echo "  multiprocess    = $build_multiprocess"
-echo "  with experimental syscall sandbox support = $use_syscall_sandbox"
 echo "  with libs       = $build_bitcoin_libs"
 echo "  with wallet     = $enable_wallet"
 if test "$enable_wallet" != "no"; then
@@ -2027,23 +1945,21 @@ echo "  with fuzz binary = $enable_fuzz_binary"
 echo "  with bench      = $use_bench"
 echo "  with upnp       = $use_upnp"
 echo "  with natpmp     = $use_natpmp"
-echo "  use asm         = $use_asm"
 echo "  USDT tracing    = $use_usdt"
 echo "  sanitizers      = $use_sanitizers"
 echo "  debug enabled   = $enable_debug"
 echo "  gprof enabled   = $enable_gprof"
 echo "  werror          = $enable_werror"
-echo "  LTO             = $enable_lto"
 echo
 echo "  target os       = $host_os"
 echo "  build os        = $build_os"
 echo
 echo "  CC              = $CC"
-echo "  CFLAGS          = $PTHREAD_CFLAGS $CFLAGS"
+echo "  CFLAGS          = $PTHREAD_CFLAGS $SANITIZER_CFLAGS $CFLAGS"
 echo "  CPPFLAGS        = $DEBUG_CPPFLAGS $HARDENED_CPPFLAGS $CORE_CPPFLAGS $CPPFLAGS"
 echo "  CXX             = $CXX"
-echo "  CXXFLAGS        = $LTO_CXXFLAGS $DEBUG_CXXFLAGS $HARDENED_CXXFLAGS $WARN_CXXFLAGS $NOWARN_CXXFLAGS $ERROR_CXXFLAGS $GPROF_CXXFLAGS $CORE_CXXFLAGS $CXXFLAGS"
-echo "  LDFLAGS         = $LTO_LDFLAGS $PTHREAD_LIBS $HARDENED_LDFLAGS $GPROF_LDFLAGS $CORE_LDFLAGS $LDFLAGS"
+echo "  CXXFLAGS        = $CORE_CXXFLAGS $DEBUG_CXXFLAGS $HARDENED_CXXFLAGS $WARN_CXXFLAGS $NOWARN_CXXFLAGS $ERROR_CXXFLAGS $GPROF_CXXFLAGS $SANITIZER_CXXFLAGS $CXXFLAGS"
+echo "  LDFLAGS         = $PTHREAD_LIBS $HARDENED_LDFLAGS $GPROF_LDFLAGS $SANITIZER_LDFLAGS $CORE_LDFLAGS $LDFLAGS"
 echo "  AR              = $AR"
 echo "  ARFLAGS         = $ARFLAGS"
 echo

contrib/README.md

@@ -35,7 +35,7 @@ Test and Verify Tools
 ### [TestGen](/contrib/testgen) ###
 Utilities to generate test vectors for the data-driven Bitcoin tests.
 
-### [Verify Binaries](/contrib/verifybinaries) ###
+### [Verify-Binaries](/contrib/verify-binaries) ###
 This script attempts to download and verify the signature file SHA256SUMS.asc from bitcoin.org.
 
 Command Line Tools

contrib/debian/copyright

@@ -5,7 +5,7 @@ Upstream-Contact: Satoshi Nakamoto <satoshin@gmx.com>
 Source: https://github.com/bitcoin/bitcoin
 
 Files: *
-Copyright: 2009-2023, Bitcoin Core Developers
+Copyright: 2009-2024, Bitcoin Core Developers
 License: Expat
 Comment: The Bitcoin Core Developers encompasses all contributors to the
          project, listed in the release notes or the git log.

contrib/devtools/README.md

@@ -83,13 +83,23 @@ A small script to automatically create manpages in ../../doc/man by running the
 This requires help2man which can be found at: https://www.gnu.org/software/help2man/
 
 With in-tree builds this tool can be run from any directory within the
-repostitory. To use this tool with out-of-tree builds set `BUILDDIR`. For
+repository. To use this tool with out-of-tree builds set `BUILDDIR`. For
 example:
 
 ```bash
 BUILDDIR=$PWD/build contrib/devtools/gen-manpages.py
 ```
 
+headerssync-params.py
+=====================
+
+A script to generate optimal parameters for the headerssync module (src/headerssync.cpp). It takes no command-line
+options, as all its configuration is set at the top of the file. It runs many times faster inside PyPy. Invocation:
+
+```bash
+pypy3 contrib/devtools/headerssync-params.py
+```
+
 gen-bitcoin-conf.sh
 ===================
 

contrib/devtools/bitcoin-tidy/CMakeLists.txt

@@ -0,0 +1,56 @@
+cmake_minimum_required(VERSION 3.9)
+
+project(bitcoin-tidy VERSION 1.0.0 DESCRIPTION "clang-tidy checks for Bitcoin Core")
+
+include(GNUInstallDirs)
+
+set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CXX_STANDARD_REQUIRED True)
+set(CMAKE_CXX_EXTENSIONS False)
+
+# TODO: Figure out how to avoid the terminfo check
+find_package(LLVM REQUIRED CONFIG)
+find_program(CLANG_TIDY_EXE NAMES "clang-tidy-${LLVM_VERSION_MAJOR}" "clang-tidy" HINTS ${LLVM_TOOLS_BINARY_DIR})
+message(STATUS "Found LLVM ${LLVM_PACKAGE_VERSION}")
+message(STATUS "Found clang-tidy: ${CLANG_TIDY_EXE}")
+
+add_library(bitcoin-tidy MODULE bitcoin-tidy.cpp logprintf.cpp)
+target_include_directories(bitcoin-tidy SYSTEM PRIVATE ${LLVM_INCLUDE_DIRS})
+
+# Disable RTTI and exceptions as necessary
+if (MSVC)
+    target_compile_options(bitcoin-tidy PRIVATE /GR-)
+else()
+    target_compile_options(bitcoin-tidy PRIVATE -fno-rtti)
+    target_compile_options(bitcoin-tidy PRIVATE -fno-exceptions)
+endif()
+
+if(CMAKE_HOST_APPLE)
+    # ld64 expects no undefined symbols by default
+    target_link_options(bitcoin-tidy PRIVATE -Wl,-flat_namespace)
+    target_link_options(bitcoin-tidy PRIVATE -Wl,-undefined -Wl,suppress)
+endif()
+
+# Add warnings
+if (MSVC)
+    target_compile_options(bitcoin-tidy PRIVATE /W4)
+else()
+    target_compile_options(bitcoin-tidy PRIVATE -Wall)
+    target_compile_options(bitcoin-tidy PRIVATE -Wextra)
+endif()
+
+if(CMAKE_VERSION VERSION_LESS 3.27)
+    set(CLANG_TIDY_COMMAND "${CLANG_TIDY_EXE}" "--load=${CMAKE_BINARY_DIR}/${CMAKE_SHARED_MODULE_PREFIX}bitcoin-tidy${CMAKE_SHARED_MODULE_SUFFIX}" "-checks=-*,bitcoin-*")
+else()
+    # CLANG_TIDY_COMMAND supports generator expressions as of 3.27
+    set(CLANG_TIDY_COMMAND "${CLANG_TIDY_EXE}" "--load=$<TARGET_FILE:bitcoin-tidy>" "-checks=-*,bitcoin-*")
+endif()
+
+# Create a dummy library that runs clang-tidy tests as a side-effect of building
+add_library(bitcoin-tidy-tests OBJECT EXCLUDE_FROM_ALL example_logprintf.cpp)
+add_dependencies(bitcoin-tidy-tests bitcoin-tidy)
+
+set_target_properties(bitcoin-tidy-tests PROPERTIES CXX_CLANG_TIDY "${CLANG_TIDY_COMMAND}")
+
+
+install(TARGETS bitcoin-tidy LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR})

contrib/devtools/bitcoin-tidy/README

@@ -0,0 +1,11 @@
+# Bitcoin Tidy
+
+Example Usage:
+
+```bash
+cmake -S . -B build -DLLVM_DIR=$(llvm-config --cmakedir) -DCMAKE_BUILD_TYPE=Release
+
+cmake --build build -j$(nproc)
+
+cmake --build build --target bitcoin-tidy-tests -j$(nproc)
+```

contrib/devtools/bitcoin-tidy/bitcoin-tidy.cpp

@@ -0,0 +1,22 @@
+// Copyright (c) 2023 Bitcoin Developers
+// Distributed under the MIT software license, see the accompanying
+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#include "logprintf.h"
+
+#include <clang-tidy/ClangTidyModule.h>
+#include <clang-tidy/ClangTidyModuleRegistry.h>
+
+class BitcoinModule final : public clang::tidy::ClangTidyModule
+{
+public:
+    void addCheckFactories(clang::tidy::ClangTidyCheckFactories& CheckFactories) override
+    {
+        CheckFactories.registerCheck<bitcoin::LogPrintfCheck>("bitcoin-unterminated-logprintf");
+    }
+};
+
+static clang::tidy::ClangTidyModuleRegistry::Add<BitcoinModule>
+    X("bitcoin-module", "Adds bitcoin checks.");
+
+volatile int BitcoinModuleAnchorSource = 0;

contrib/devtools/bitcoin-tidy/example_logprintf.cpp

@@ -0,0 +1,108 @@
+// Copyright (c) 2023 Bitcoin Developers
+// Distributed under the MIT software license, see the accompanying
+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#include <string>
+
+// Test for bitcoin-unterminated-logprintf
+
+enum LogFlags {
+    NONE
+};
+
+enum Level {
+    None
+};
+
+template <typename... Args>
+static inline void LogPrintf_(const std::string& logging_function, const std::string& source_file, const int source_line, const LogFlags flag, const Level level, const char* fmt, const Args&... args)
+{
+}
+
+#define LogPrintLevel_(category, level, ...) LogPrintf_(__func__, __FILE__, __LINE__, category, level, __VA_ARGS__)
+#define LogPrintf(...) LogPrintLevel_(LogFlags::NONE, Level::None, __VA_ARGS__)
+
+#define LogPrint(category, ...) \
+    do {                        \
+        LogPrintf(__VA_ARGS__); \
+    } while (0)
+
+
+class CWallet
+{
+    std::string GetDisplayName() const
+    {
+        return "default wallet";
+    }
+
+public:
+    template <typename... Params>
+    void WalletLogPrintf(const char* fmt, Params... parameters) const
+    {
+        LogPrintf(("%s " + std::string{fmt}).c_str(), GetDisplayName(), parameters...);
+    };
+};
+
+struct ScriptPubKeyMan
+{
+    std::string GetDisplayName() const
+    {
+        return "default wallet";
+    }
+
+    template <typename... Params>
+    void WalletLogPrintf(const char* fmt, Params... parameters) const
+    {
+        LogPrintf(("%s " + std::string{fmt}).c_str(), GetDisplayName(), parameters...);
+    };
+};
+
+void good_func()
+{
+    LogPrintf("hello world!\n");
+}
+void good_func2()
+{
+    CWallet wallet;
+    wallet.WalletLogPrintf("hi\n");
+    ScriptPubKeyMan spkm;
+    spkm.WalletLogPrintf("hi\n");
+
+    const CWallet& walletref = wallet;
+    walletref.WalletLogPrintf("hi\n");
+
+    auto* walletptr = new CWallet();
+    walletptr->WalletLogPrintf("hi\n");
+    delete walletptr;
+}
+void bad_func()
+{
+    LogPrintf("hello world!");
+}
+void bad_func2()
+{
+    LogPrintf("");
+}
+void bad_func3()
+{
+    // Ending in "..." has no special meaning.
+    LogPrintf("hello world!...");
+}
+void bad_func4_ignored()
+{
+    LogPrintf("hello world!"); // NOLINT(bitcoin-unterminated-logprintf)
+}
+void bad_func5()
+{
+    CWallet wallet;
+    wallet.WalletLogPrintf("hi");
+    ScriptPubKeyMan spkm;
+    spkm.WalletLogPrintf("hi");
+
+    const CWallet& walletref = wallet;
+    walletref.WalletLogPrintf("hi");
+
+    auto* walletptr = new CWallet();
+    walletptr->WalletLogPrintf("hi");
+    delete walletptr;
+}

contrib/devtools/bitcoin-tidy/logprintf.cpp

@@ -0,0 +1,60 @@
+// Copyright (c) 2023 Bitcoin Developers
+// Distributed under the MIT software license, see the accompanying
+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#include "logprintf.h"
+
+#include <clang/AST/ASTContext.h>
+#include <clang/ASTMatchers/ASTMatchFinder.h>
+
+
+namespace {
+AST_MATCHER(clang::StringLiteral, unterminated)
+{
+    size_t len = Node.getLength();
+    if (len > 0 && Node.getCodeUnit(len - 1) == '\n') {
+        return false;
+    }
+    return true;
+}
+} // namespace
+
+namespace bitcoin {
+
+void LogPrintfCheck::registerMatchers(clang::ast_matchers::MatchFinder* finder)
+{
+    using namespace clang::ast_matchers;
+
+    /*
+      Logprintf(..., ..., ..., ..., ..., "foo", ...)
+    */
+
+    finder->addMatcher(
+        callExpr(
+            callee(functionDecl(hasName("LogPrintf_"))),
+            hasArgument(5, stringLiteral(unterminated()).bind("logstring"))),
+        this);
+
+    /*
+      auto walletptr = &wallet;
+      wallet.WalletLogPrintf("foo");
+      wallet->WalletLogPrintf("foo");
+    */
+    finder->addMatcher(
+        cxxMemberCallExpr(
+            callee(cxxMethodDecl(hasName("WalletLogPrintf"))),
+            hasArgument(0, stringLiteral(unterminated()).bind("logstring"))),
+        this);
+}
+
+void LogPrintfCheck::check(const clang::ast_matchers::MatchFinder::MatchResult& Result)
+{
+    if (const clang::StringLiteral* lit = Result.Nodes.getNodeAs<clang::StringLiteral>("logstring")) {
+        const clang::ASTContext& ctx = *Result.Context;
+        const auto user_diag = diag(lit->getEndLoc(), "Unterminated format string used with LogPrintf");
+        const auto& loc = lit->getLocationOfByte(lit->getByteLength(), *Result.SourceManager, ctx.getLangOpts(), ctx.getTargetInfo());
+        user_diag << clang::FixItHint::CreateInsertion(loc, "\\n");
+    }
+}
+
+} // namespace bitcoin

contrib/devtools/bitcoin-tidy/logprintf.h

@@ -0,0 +1,29 @@
+// Copyright (c) 2023 Bitcoin Developers
+// Distributed under the MIT software license, see the accompanying
+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#ifndef LOGPRINTF_CHECK_H
+#define LOGPRINTF_CHECK_H
+
+#include <clang-tidy/ClangTidyCheck.h>
+
+namespace bitcoin {
+
+// Warn about any use of LogPrintf that does not end with a newline.
+class LogPrintfCheck final : public clang::tidy::ClangTidyCheck
+{
+public:
+    LogPrintfCheck(clang::StringRef Name, clang::tidy::ClangTidyContext* Context)
+        : clang::tidy::ClangTidyCheck(Name, Context) {}
+
+    bool isLanguageVersionSupported(const clang::LangOptions& LangOpts) const override
+    {
+        return LangOpts.CPlusPlus;
+    }
+    void registerMatchers(clang::ast_matchers::MatchFinder* Finder) override;
+    void check(const clang::ast_matchers::MatchFinder::MatchResult& Result) override;
+};
+
+} // namespace bitcoin
+
+#endif // LOGPRINTF_CHECK_H

contrib/devtools/circular-dependencies.py

@@ -5,7 +5,6 @@
 
 import sys
 import re
-from typing import Dict, List, Set
 
 MAPPING = {
     'core_read.cpp': 'core_io.cpp',
@@ -33,7 +32,7 @@ def module_name(path):
     return None
 
 files = dict()
-deps: Dict[str, Set[str]] = dict()
+deps: dict[str, set[str]] = dict()
 
 RE = re.compile("^#include <(.*)>")
 
@@ -65,7 +64,7 @@ while True:
     shortest_cycle = None
     for module in sorted(deps.keys()):
         # Build the transitive closure of dependencies of module
-        closure: Dict[str, List[str]] = dict()
+        closure: dict[str, list[str]] = dict()
         for dep in deps[module]:
             closure[dep] = []
         while True:

contrib/devtools/clang-format-diff.py

@@ -1,166 +1,190 @@
 #!/usr/bin/env python3
 #
-#===- clang-format-diff.py - ClangFormat Diff Reformatter ----*- python -*--===#
+# ===- clang-format-diff.py - ClangFormat Diff Reformatter ----*- python -*--===#
 #
-#                     The LLVM Compiler Infrastructure
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
-# This file is distributed under the University of Illinois Open Source
-# License.
-#
-#           ============================================================
-#
-# University of Illinois/NCSA
-# Open Source License
-#
-# Copyright (c) 2007-2015 University of Illinois at Urbana-Champaign.
-# All rights reserved.
-#
-# Developed by:
-#
-#     LLVM Team
-#
-#     University of Illinois at Urbana-Champaign
-#
-#     http://llvm.org
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy of
-# this software and associated documentation files (the "Software"), to deal with
-# the Software without restriction, including without limitation the rights to
-# use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
-# of the Software, and to permit persons to whom the Software is furnished to do
-# so, subject to the following conditions:
-#
-#     * Redistributions of source code must retain the above copyright notice,
-#       this list of conditions and the following disclaimers.
-#
-#     * Redistributions in binary form must reproduce the above copyright notice,
-#       this list of conditions and the following disclaimers in the
-#       documentation and/or other materials provided with the distribution.
-#
-#     * Neither the names of the LLVM Team, University of Illinois at
-#       Urbana-Champaign, nor the names of its contributors may be used to
-#       endorse or promote products derived from this Software without specific
-#       prior written permission.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
-# FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
-# CONTRIBUTORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS WITH THE
-# SOFTWARE.
-#
-#           ============================================================
-#
-#===------------------------------------------------------------------------===#
-
-r"""
-ClangFormat Diff Reformatter
-============================
+# ===------------------------------------------------------------------------===#
 
+"""
 This script reads input from a unified diff and reformats all the changed
 lines. This is useful to reformat all the lines touched by a specific patch.
 Example usage for git/svn users:
 
-  git diff -U0 HEAD^ | clang-format-diff.py -p1 -i
-  svn diff --diff-cmd=diff -x-U0 | clang-format-diff.py -i
+  git diff -U0 --no-color --relative HEAD^ | {clang_format_diff} -p1 -i
+  svn diff --diff-cmd=diff -x-U0 | {clang_format_diff} -i
 
+It should be noted that the filename contained in the diff is used unmodified
+to determine the source file to update. Users calling this script directly
+should be careful to ensure that the path in the diff is correct relative to the
+current working directory.
 """
+from __future__ import absolute_import, division, print_function
 
 import argparse
 import difflib
-import io
 import re
 import subprocess
 import sys
 
-
-# Change this to the full path if clang-format is not on the path.
-binary = 'clang-format'
+from io import StringIO
 
 
 def main():
-  parser = argparse.ArgumentParser(description=
-                                   'Reformat changed lines in diff. Without -i '
-                                   'option just output the diff that would be '
-                                   'introduced.')
-  parser.add_argument('-i', action='store_true', default=False,
-                      help='apply edits to files instead of displaying a diff')
-  parser.add_argument('-p', metavar='NUM', default=0,
-                      help='strip the smallest prefix containing P slashes')
-  parser.add_argument('-regex', metavar='PATTERN', default=None,
-                      help='custom pattern selecting file paths to reformat '
-                      '(case sensitive, overrides -iregex)')
-  parser.add_argument('-iregex', metavar='PATTERN', default=
-                      r'.*\.(cpp|cc|c\+\+|cxx|c|cl|h|hpp|m|mm|inc|js|ts|proto'
-                      r'|protodevel|java)',
-                      help='custom pattern selecting file paths to reformat '
-                      '(case insensitive, overridden by -regex)')
-  parser.add_argument('-sort-includes', action='store_true', default=False,
-                      help='let clang-format sort include blocks')
-  parser.add_argument('-v', '--verbose', action='store_true',
-                      help='be more verbose, ineffective without -i')
-  args = parser.parse_args()
+    parser = argparse.ArgumentParser(
+        description=__doc__.format(clang_format_diff="%(prog)s"),
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    parser.add_argument(
+        "-i",
+        action="store_true",
+        default=False,
+        help="apply edits to files instead of displaying a diff",
+    )
+    parser.add_argument(
+        "-p",
+        metavar="NUM",
+        default=0,
+        help="strip the smallest prefix containing P slashes",
+    )
+    parser.add_argument(
+        "-regex",
+        metavar="PATTERN",
+        default=None,
+        help="custom pattern selecting file paths to reformat "
+        "(case sensitive, overrides -iregex)",
+    )
+    parser.add_argument(
+        "-iregex",
+        metavar="PATTERN",
+        default=r".*\.(?:cpp|cc|c\+\+|cxx|cppm|ccm|cxxm|c\+\+m|c|cl|h|hh|hpp"
+        r"|hxx|m|mm|inc|js|ts|proto|protodevel|java|cs|json|s?vh?)",
+        help="custom pattern selecting file paths to reformat "
+        "(case insensitive, overridden by -regex)",
+    )
+    parser.add_argument(
+        "-sort-includes",
+        action="store_true",
+        default=False,
+        help="let clang-format sort include blocks",
+    )
+    parser.add_argument(
+        "-v",
+        "--verbose",
+        action="store_true",
+        help="be more verbose, ineffective without -i",
+    )
+    parser.add_argument(
+        "-style",
+        help="formatting style to apply (LLVM, GNU, Google, Chromium, "
+        "Microsoft, Mozilla, WebKit)",
+    )
+    parser.add_argument(
+        "-fallback-style",
+        help="The name of the predefined style used as a"
+        "fallback in case clang-format is invoked with"
+        "-style=file, but can not find the .clang-format"
+        "file to use.",
+    )
+    parser.add_argument(
+        "-binary",
+        default="clang-format",
+        help="location of binary to use for clang-format",
+    )
+    args = parser.parse_args()
 
-  # Extract changed lines for each file.
-  filename = None
-  lines_by_file = {}
-  for line in sys.stdin:
-    match = re.search(r'^\+\+\+\ (.*?/){%s}(\S*)' % args.p, line)
-    if match:
-      filename = match.group(2)
-    if filename is None:
-      continue
+    # Extract changed lines for each file.
+    filename = None
+    lines_by_file = {}
+    for line in sys.stdin:
+        match = re.search(r"^\+\+\+\ (.*?/){%s}(\S*)" % args.p, line)
+        if match:
+            filename = match.group(2)
+        if filename is None:
+            continue
 
-    if args.regex is not None:
-      if not re.match('^%s$' % args.regex, filename):
-        continue
-    else:
-      if not re.match('^%s$' % args.iregex, filename, re.IGNORECASE):
-        continue
+        if args.regex is not None:
+            if not re.match("^%s$" % args.regex, filename):
+                continue
+        else:
+            if not re.match("^%s$" % args.iregex, filename, re.IGNORECASE):
+                continue
 
-    match = re.search(r'^@@.*\+(\d+)(,(\d+))?', line)
-    if match:
-      start_line = int(match.group(1))
-      line_count = 1
-      if match.group(3):
-        line_count = int(match.group(3))
-      if line_count == 0:
-        continue
-      end_line = start_line + line_count - 1
-      lines_by_file.setdefault(filename, []).extend(
-          ['-lines', str(start_line) + ':' + str(end_line)])
+        match = re.search(r"^@@.*\+(\d+)(?:,(\d+))?", line)
+        if match:
+            start_line = int(match.group(1))
+            line_count = 1
+            if match.group(2):
+                line_count = int(match.group(2))
+                # The input is something like
+                #
+                # @@ -1, +0,0 @@
+                #
+                # which means no lines were added.
+                if line_count == 0:
+                    continue
+            # Also format lines range if line_count is 0 in case of deleting
+            # surrounding statements.
+            end_line = start_line
+            if line_count != 0:
+                end_line += line_count - 1
+            lines_by_file.setdefault(filename, []).extend(
+                ["-lines", str(start_line) + ":" + str(end_line)]
+            )
 
-  # Reformat files containing changes in place.
-  for filename, lines in lines_by_file.items():
-    if args.i and args.verbose:
-      print('Formatting {}'.format(filename))
-    command = [binary, filename]
-    if args.i:
-      command.append('-i')
-    if args.sort_includes:
-      command.append('-sort-includes')
-    command.extend(lines)
-    command.extend(['-style=file', '-fallback-style=none'])
-    p = subprocess.Popen(command,
-                         stdout=subprocess.PIPE,
-                         stderr=None,
-                         stdin=subprocess.PIPE,
-                         text=True)
-    stdout, stderr = p.communicate()
-    if p.returncode != 0:
-      sys.exit(p.returncode)
+    # Reformat files containing changes in place.
+    for filename, lines in lines_by_file.items():
+        if args.i and args.verbose:
+            print("Formatting {}".format(filename))
+        command = [args.binary, filename]
+        if args.i:
+            command.append("-i")
+        if args.sort_includes:
+            command.append("-sort-includes")
+        command.extend(lines)
+        if args.style:
+            command.extend(["-style", args.style])
+        if args.fallback_style:
+            command.extend(["-fallback-style", args.fallback_style])
 
-    if not args.i:
-      with open(filename, encoding="utf8") as f:
-        code = f.readlines()
-      formatted_code = io.StringIO(stdout).readlines()
-      diff = difflib.unified_diff(code, formatted_code,
-                                  filename, filename,
-                                  '(before formatting)', '(after formatting)')
-      diff_string = ''.join(diff)
-      if len(diff_string) > 0:
-        sys.stdout.write(diff_string)
+        try:
+            p = subprocess.Popen(
+                command,
+                stdout=subprocess.PIPE,
+                stderr=None,
+                stdin=subprocess.PIPE,
+                universal_newlines=True,
+            )
+        except OSError as e:
+            # Give the user more context when clang-format isn't
+            # found/isn't executable, etc.
+            raise RuntimeError(
+                'Failed to run "%s" - %s"' % (" ".join(command), e.strerror)
+            )
 
-if __name__ == '__main__':
-  main()
+        stdout, stderr = p.communicate()
+        if p.returncode != 0:
+            sys.exit(p.returncode)
+
+        if not args.i:
+            with open(filename, encoding="utf8") as f:
+                code = f.readlines()
+            formatted_code = StringIO(stdout).readlines()
+            diff = difflib.unified_diff(
+                code,
+                formatted_code,
+                filename,
+                filename,
+                "(before formatting)",
+                "(after formatting)",
+            )
+            diff_string = "".join(diff)
+            if len(diff_string) > 0:
+                sys.stdout.write(diff_string)
+                sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

contrib/devtools/headerssync-params.py

@@ -0,0 +1,357 @@
+#!/usr/bin/env python3
+# Copyright (c) 2022 Pieter Wuille
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+"""Script to find the optimal parameters for the headerssync module through simulation."""
+
+from math import log, exp, sqrt
+from datetime import datetime, timedelta
+import random
+
+# Parameters:
+
+# Aim for still working fine at some point in the future. [datetime]
+TIME = datetime(2026, 10, 5)
+
+# Expected block interval. [timedelta]
+BLOCK_INTERVAL = timedelta(seconds=600)
+
+# The number of headers corresponding to the minchainwork parameter. [headers]
+MINCHAINWORK_HEADERS = 804000
+
+# Combined processing bandwidth from all attackers to one victim. [bit/s]
+# 6 Gbit/s is approximately the speed at which a single thread of a Ryzen 5950X CPU thread can hash
+# headers. In practice, the victim's network bandwidth and network processing overheads probably
+# impose a far lower number, but it's a useful upper bound.
+ATTACK_BANDWIDTH = 6000000000
+
+# How much additional permanent memory usage are attackers (jointly) allowed to cause in the victim,
+# expressed as fraction of the normal memory usage due to mainchain growth, for the duration the
+# attack is sustained. [unitless]
+# 0.2 means that attackers, while they keep up the attack, can cause permanent memory usage due to
+# headers storage to grow at 1.2 header per BLOCK_INTERVAL.
+ATTACK_FRACTION = 0.2
+
+# When this is set, the mapping from period size to memory usage (at optimal buffer size for that
+# period) is assumed to be convex. This greatly speeds up the computation, and does not appear
+# to influence the outcome. Set to False for a stronger guarantee to get the optimal result.
+ASSUME_CONVEX = True
+
+# Explanation:
+#
+#  The headerssync module implements a DoS protection against low-difficulty header spam which does
+#  not rely on checkpoints. In short it works as follows:
+#
+#  - (initial) header synchronization is split into two phases:
+#    - A commitment phase, in which headers are downloaded from the peer, and a very compact
+#      commitment to them is remembered in per-peer memory. The commitment phase ends when the
+#      received chain's combined work reaches a predetermined threshold.
+#    - A redownload phase, during which the headers are downloaded a second time from the same peer,
+#      and compared against the commitment constructed in the first phase. If there is a match, the
+#      redownloaded headers are fed to validation and accepted into permanent storage.
+#
+#    This separation guarantees that no headers are accepted into permanent storage without
+#    requiring the peer to first prove the chain actually has sufficient work.
+#
+#  - To actually implement this commitment mechanism, the following approach is used:
+#    - Keep a *1 bit* commitment (constructed using a salted hash function), for every block whose
+#      height is a multiple of {period} plus an offset value. If RANDOMIZE_OFFSET, the offset,
+#      like the salt, is chosen randomly when the synchronization starts and kept fixed afterwards.
+#    - When redownloading, headers are fed through a per-peer queue that holds {bufsize} headers,
+#      before passing them to validation. All the headers in this queue are verified against the
+#      commitment bits created in the first phase before any header is released from it. This means
+#      {bufsize/period} bits are checked "on top of" each header before actually processing it,
+#      which results in a commitment structure with roughly {bufsize/period} bits of security, as
+#      once a header is modified, due to the prevhash inclusion, all future headers necessarily
+#      change as well.
+#
+#  The question is what these {period} and {bufsize} parameters need to be set to. This program
+#  exhaustively tests a range of values to find the optimal choice, taking into account:
+#
+#  - Minimizing the (maximum of) two scenarios that trigger per-peer memory usage:
+#
+#    - When downloading a (likely honest) chain that reaches the chainwork threshold after {n}
+#      blocks, and then redownloads them, we will consume per-peer memory that is sufficient to
+#      store {n/period} commitment bits and {bufsize} headers. We only consider attackers without
+#      sufficient hashpower (as otherwise they are from a PoW perspective not attackers), which
+#      means {n} is restricted to the honest chain's length before reaching minchainwork.
+#
+#    - When downloading a (likely false) chain of {n} headers that never reaches the chainwork
+#      threshold, we will consume per-peer memory that is sufficient to store {n/period}
+#      commitment bits. Such a chain may be very long, by exploiting the timewarp bug to avoid
+#      ramping up difficulty. There is however an absolute limit on how long such a chain can be: 6
+#      blocks per second since genesis, due to the increasing MTP consensus rule.
+#
+#  - Not gratuitously preventing synchronizing any valid chain, however difficult such a chain may
+#    be to construct. In particular, the above scenario with an enormous timewarp-expoiting chain
+#    cannot simply be ignored, as it is legal that the honest main chain is like that. We however
+#    do not bother minimizing the memory usage in that case (because a billion-header long honest
+#    chain will inevitably use far larger amounts of memory than designed for).
+#
+#  - Keep the rate at which attackers can get low-difficulty headers accepted to the block index
+#    negligible. Specifically, the possibility exists for an attacker to send the honest main
+#    chain's headers during the commitment phase, but then start deviating at an attacker-chosen
+#    point by sending novel low-difficulty headers instead. Depending on how high we set the
+#    {bufsize/period} ratio, we can make the probability that such a header makes it in
+#    arbitrarily small, but at the cost of higher memory during the redownload phase. It turns out,
+#    some rate of memory usage growth is expected anyway due to chain growth, so permitting the
+#    attacker to increase that rate by a small factor isn't concerning. The attacker may start
+#    somewhat later than genesis, as long as the difficulty doesn't get too high. This reduces
+#    the attacker bandwidth required at the cost of higher PoW needed for constructing the
+#    alternate chain. This trade-off is ignored here, as it results in at most a small constant
+#    factor in attack rate.
+
+
+# System properties:
+
+# Headers in the redownload buffer are stored without prevhash. [bits]
+COMPACT_HEADER_SIZE = 48 * 8
+
+# How many bits a header uses in P2P protocol. [bits]
+NET_HEADER_SIZE = 81 * 8
+
+# How many headers are sent at once. [headers]
+HEADER_BATCH_COUNT = 2000
+
+# Whether or not the offset of which blocks heights get checksummed is randomized.
+RANDOMIZE_OFFSET = True
+
+# Timestamp of the genesis block
+GENESIS_TIME = datetime(2009, 1, 3)
+
+# Derived values:
+
+# What rate of headers worth of RAM attackers are allowed to cause in the victim. [headers/s]
+LIMIT_HEADERRATE = ATTACK_FRACTION / BLOCK_INTERVAL.total_seconds()
+
+# How many headers can attackers (jointly) send a victim per second. [headers/s]
+NET_HEADERRATE = ATTACK_BANDWIDTH / NET_HEADER_SIZE
+
+# What fraction of headers sent by attackers can at most be accepted by a victim [unitless]
+LIMIT_FRACTION = LIMIT_HEADERRATE / NET_HEADERRATE
+
+# How many headers we permit attackers to cause being accepted per attack. [headers/attack]
+ATTACK_HEADERS = LIMIT_FRACTION * MINCHAINWORK_HEADERS
+
+
+def find_max_headers(when):
+    """Compute the maximum number of headers a valid Bitcoin chain can have at given time."""
+    # When exploiting the timewarp attack, this can be up to 6 per second since genesis.
+    return 6 * ((when - GENESIS_TIME) // timedelta(seconds=1))
+
+
+def lambert_w(value):
+    """Solve the equation x*exp(x)=value (x > 0, value > 0)."""
+    # Initial approximation.
+    approx = max(log(value), 0.0)
+    for _ in range(10):
+        # Newton-Rhapson iteration steps.
+        approx += (value * exp(-approx) - approx) / (approx + 1.0)
+    return approx
+
+
+def attack_rate(period, bufsize, limit=None):
+    """Compute maximal accepted headers per attack in (period, bufsize) configuration.
+
+    If limit is provided, the computation is stopped early when the result is known to exceed the
+    value in limit.
+    """
+
+    max_rate = None
+    max_honest = None
+    # Let the current batch 0 being received be the first one in which the attacker starts lying.
+    # They will only ever start doing so right after a commitment block, but where that is can be
+    # in a number of places. Let honest be the number of honest headers in this current batch,
+    # preceding the forged ones.
+    for honest in range(HEADER_BATCH_COUNT):
+        # The number of headers the attack under consideration will on average get accepted.
+        # This is the number being computed.
+        rate = 0
+
+        # Iterate over the possible alignments of commitments w.r.t. the first batch. In case
+        # the alignments are randomized, try all values. If not, the attacker can know/choose
+        # the alignment, and will always start forging right after a commitment.
+        if RANDOMIZE_OFFSET:
+            align_choices = list(range(period))
+        else:
+            align_choices = [(honest - 1) % period]
+        # Now loop over those possible alignment values, computing the average attack rate
+        # over them by dividing each contribution by len(align_choices).
+        for align in align_choices:
+            # These state variables capture the situation after receiving the first batch.
+            # - The number of headers received after the last commitment for an honest block:
+            after_good_commit = HEADER_BATCH_COUNT - honest + ((honest - align - 1) % period)
+            # - The number of forged headers in the redownload buffer:
+            forged_in_buf = HEADER_BATCH_COUNT - honest
+
+            # Now iterate over the next batches of headers received, adding contributions to the
+            # rate variable.
+            while True:
+                # Process the first HEADER_BATCH_COUNT headers in the buffer:
+                accept_forged_headers = max(forged_in_buf - bufsize, 0)
+                forged_in_buf -= accept_forged_headers
+                if accept_forged_headers:
+                    # The probability the attack has not been detected yet at this point:
+                    prob = 0.5 ** (after_good_commit // period)
+                    # Update attack rate, divided by align_choices to average over the alignments.
+                    rate += accept_forged_headers * prob / len(align_choices)
+                    # If this means we exceed limit, bail out early (performance optimization).
+                    if limit is not None and rate >= limit:
+                        return rate, None
+                    # If the maximal term being added is negligible compared to rate, stop
+                    # iterating.
+                    if HEADER_BATCH_COUNT * prob < 1.0e-16 * rate * len(align_choices):
+                        break
+                # Update state from a new incoming batch (which is all forged)
+                after_good_commit += HEADER_BATCH_COUNT
+                forged_in_buf += HEADER_BATCH_COUNT
+
+        if max_rate is None or rate > max_rate:
+            max_rate = rate
+            max_honest = honest
+
+    return max_rate, max_honest
+
+
+def memory_usage(period, bufsize, when):
+    """How much memory (max,mainchain,timewarp) does the (period,bufsize) configuration need?"""
+
+    # Per-peer memory usage for a timewarp chain that never meets minchainwork
+    mem_timewarp = find_max_headers(when) // period
+    # Per-peer memory usage for being fed the main chain
+    mem_mainchain = (MINCHAINWORK_HEADERS // period) + bufsize * COMPACT_HEADER_SIZE
+    # Maximum per-peer memory usage
+    max_mem = max(mem_timewarp, mem_mainchain)
+
+    return max_mem, mem_mainchain, mem_timewarp
+
+def find_bufsize(period, attack_headers, when, max_mem=None, min_bufsize=1):
+    """Determine how big bufsize needs to be given a specific period length.
+
+    Given a period, find the smallest value of bufsize such that the attack rate against the
+    (period, bufsize) configuration is below attack_headers. If max_mem is provided, and no
+    such bufsize exists that needs less than max_mem bits of memory, None is returned.
+    min_bufsize is the minimal result to be considered."""
+
+    if max_mem is None:
+        succ_buf = min_bufsize - 1
+        fail_buf = min_bufsize
+        # First double iteratively until an upper bound for failure is found.
+        while True:
+            if attack_rate(period, fail_buf, attack_headers)[0] < attack_headers:
+                break
+            succ_buf, fail_buf = fail_buf, 3 * fail_buf - 2 * succ_buf
+    else:
+        # If a long low-work header chain exists that exceeds max_mem already, give up.
+        if find_max_headers(when) // period > max_mem:
+            return None
+        # Otherwise, verify that the maximal buffer size that permits a mainchain sync with less
+        # than max_mem memory is sufficient to get the attack rate below attack_headers. If not,
+        # also give up.
+        max_buf = (max_mem - (MINCHAINWORK_HEADERS // period)) // COMPACT_HEADER_SIZE
+        if max_buf < min_bufsize:
+            return None
+        if attack_rate(period, max_buf, attack_headers)[0] >= attack_headers:
+            return None
+        # If it is sufficient, that's an upper bound to start our search.
+        succ_buf = min_bufsize - 1
+        fail_buf = max_buf
+
+    # Then perform a bisection search to narrow it down.
+    while fail_buf > succ_buf + 1:
+        try_buf = (succ_buf + fail_buf) // 2
+        if attack_rate(period, try_buf, attack_headers)[0] >= attack_headers:
+            succ_buf = try_buf
+        else:
+            fail_buf = try_buf
+    return fail_buf
+
+
+def optimize(when):
+    """Find the best (period, bufsize) configuration."""
+
+    # When period*bufsize = memory_scale, the per-peer memory for a mainchain sync and a maximally
+    # long low-difficulty header sync are equal.
+    memory_scale = (find_max_headers(when) - MINCHAINWORK_HEADERS) / COMPACT_HEADER_SIZE
+    # Compute approximation for {bufsize/period}, using a formula for a simplified problem.
+    approx_ratio = lambert_w(log(4) * memory_scale / ATTACK_HEADERS**2) / log(4)
+    # Use those for a first attempt.
+    print("Searching configurations:")
+    period = int(sqrt(memory_scale / approx_ratio) + 0.5)
+    bufsize = find_bufsize(period, ATTACK_HEADERS, when)
+    mem = memory_usage(period, bufsize, when)
+    best = (period, bufsize, mem)
+    maps = [(period, bufsize), (MINCHAINWORK_HEADERS + 1, None)]
+    print(f"- Initial: period={period}, buffer={bufsize}, mem={mem[0] / 8192:.3f} KiB")
+
+    # Consider all period values between 1 and MINCHAINWORK_HEADERS, except the one just tried.
+    periods = [iv for iv in range(1, MINCHAINWORK_HEADERS + 1) if iv != period]
+    # Iterate, picking a random element from periods, computing its corresponding bufsize, and
+    # then using the result to shrink the period.
+    while True:
+        # Remove all periods whose memory usage for low-work long chain sync exceed the best
+        # memory usage we've found so far.
+        periods = [p for p in periods if find_max_headers(when) // p < best[2][0]]
+        # Stop if there is nothing left to try.
+        if len(periods) == 0:
+            break
+        # Pick a random remaining option for period size, and compute corresponding bufsize.
+        period = periods.pop(random.randrange(len(periods)))
+        # The buffer size (at a given attack level) cannot shrink as the period grows. Find the
+        # largest period smaller than the selected one we know the buffer size for, and use that
+        # as a lower bound to find_bufsize.
+        min_bufsize = max([(p, b) for p, b in maps if p < period] + [(0,0)])[1]
+        bufsize = find_bufsize(period, ATTACK_HEADERS, when, best[2][0], min_bufsize)
+        if bufsize is not None:
+            # We found a (period, bufsize) configuration with better memory usage than our best
+            # so far. Remember it for future lower bounds.
+            maps.append((period, bufsize))
+            mem = memory_usage(period, bufsize, when)
+            assert mem[0] <= best[2][0]
+            if ASSUME_CONVEX:
+                # Remove all periods that are on the other side of the former best as the new
+                # best.
+                periods = [p for p in periods if (p < best[0]) == (period < best[0])]
+            best = (period, bufsize, mem)
+            print(f"- New best: period={period}, buffer={bufsize}, mem={mem[0] / 8192:.3f} KiB")
+        else:
+            # The (period, bufsize) configuration we found is worse than what we already had.
+            if ASSUME_CONVEX:
+                # Remove all periods that are on the other side of the tried configuration as the
+                # best one.
+                periods = [p for p in periods if (p < period) == (best[0] < period)]
+
+    # Return the result.
+    period, bufsize, _ = best
+    return period, bufsize
+
+
+def analyze(when):
+    """Find the best configuration and print it out."""
+
+    period, bufsize = optimize(when)
+    # Compute accurate statistics for the best found configuration.
+    _, mem_mainchain, mem_timewarp = memory_usage(period, bufsize, when)
+    headers_per_attack, _ = attack_rate(period, bufsize)
+    attack_volume = NET_HEADER_SIZE * MINCHAINWORK_HEADERS
+    # And report them.
+    print()
+    print("Optimal configuration:")
+    print()
+    print("//! Store one header commitment per HEADER_COMMITMENT_PERIOD blocks.")
+    print(f"constexpr size_t HEADER_COMMITMENT_PERIOD{{{period}}};")
+    print()
+    print("//! Only feed headers to validation once this many headers on top have been")
+    print("//! received and validated against commitments.")
+    print(f"constexpr size_t REDOWNLOAD_BUFFER_SIZE{{{bufsize}}};"
+          f" // {bufsize}/{period} = ~{bufsize/period:.1f} commitments")
+    print()
+    print("Properties:")
+    print(f"- Per-peer memory for mainchain sync: {mem_mainchain / 8192:.3f} KiB")
+    print(f"- Per-peer memory for timewarp attack: {mem_timewarp / 8192:.3f} KiB")
+    print(f"- Attack rate: {1/headers_per_attack:.1f} attacks for 1 header of memory growth")
+    print(f"  (where each attack costs {attack_volume / 8388608:.3f} MiB bandwidth)")
+
+
+analyze(TIME)

contrib/devtools/iwyu/bitcoin.core.imp

@@ -1,7 +1,4 @@
 # Fixups / upstreamed changes
 [
-  { include: [ "<bits/termios-c_lflag.h>", private, "<termios.h>", public ] },
-  { include: [ "<bits/termios-struct.h>", private, "<termios.h>", public ] },
-  { include: [ "<bits/termios-tcflow.h>", private, "<termios.h>", public ] },
   { include: [ "<bits/chrono.h>", private, "<chrono>", public ] },
 ]

contrib/devtools/security-check.py

@@ -8,9 +8,8 @@ Exit status will be 0 if successful, and the program will be silent.
 Otherwise the exit status will be 1 and it will log which executables failed which checks.
 '''
 import sys
-from typing import List
 
-import lief #type:ignore
+import lief
 
 def check_ELF_RELRO(binary) -> bool:
     '''
@@ -113,7 +112,7 @@ def check_ELF_control_flow(binary) -> bool:
     main = binary.get_function_address('main')
     content = binary.get_content_from_virtual_address(main, 4, lief.Binary.VA_TYPES.AUTO)
 
-    if content == [243, 15, 30, 250]: # endbr64
+    if content.tolist() == [243, 15, 30, 250]: # endbr64
         return True
     return False
 
@@ -142,7 +141,7 @@ def check_PE_control_flow(binary) -> bool:
 
     content = binary.get_content_from_virtual_address(virtual_address, 4, lief.Binary.VA_TYPES.VA)
 
-    if content == [243, 15, 30, 250]: # endbr64
+    if content.tolist() == [243, 15, 30, 250]: # endbr64
         return True
     return False
 
@@ -158,12 +157,11 @@ def check_MACHO_NOUNDEFS(binary) -> bool:
     '''
     return binary.header.has(lief.MachO.HEADER_FLAGS.NOUNDEFS)
 
-def check_MACHO_LAZY_BINDINGS(binary) -> bool:
+def check_MACHO_FIXUP_CHAINS(binary) -> bool:
     '''
-    Check for no lazy bindings.
-    We don't use or check for MH_BINDATLOAD. See #18295.
+    Check for use of chained fixups.
     '''
-    return binary.dyld_info.lazy_bind == (0,0)
+    return binary.has_dyld_chained_fixups
 
 def check_MACHO_Canary(binary) -> bool:
     '''
@@ -190,7 +188,17 @@ def check_MACHO_control_flow(binary) -> bool:
     '''
     content = binary.get_content_from_virtual_address(binary.entrypoint, 4, lief.Binary.VA_TYPES.AUTO)
 
-    if content == [243, 15, 30, 250]: # endbr64
+    if content.tolist() == [243, 15, 30, 250]: # endbr64
+        return True
+    return False
+
+def check_MACHO_branch_protection(binary) -> bool:
+    '''
+    Check for branch protection instrumentation
+    '''
+    content = binary.get_content_from_virtual_address(binary.entrypoint, 4, lief.Binary.VA_TYPES.AUTO)
+
+    if content.tolist() == [95, 36, 3, 213]: # bti
         return True
     return False
 
@@ -214,8 +222,8 @@ BASE_PE = [
 
 BASE_MACHO = [
     ('NOUNDEFS', check_MACHO_NOUNDEFS),
-    ('LAZY_BINDINGS', check_MACHO_LAZY_BINDINGS),
     ('Canary', check_MACHO_Canary),
+    ('FIXUP_CHAINS', check_MACHO_FIXUP_CHAINS),
 ]
 
 CHECKS = {
@@ -233,7 +241,7 @@ CHECKS = {
         lief.ARCHITECTURES.X86: BASE_MACHO + [('PIE', check_PIE),
                                               ('NX', check_NX),
                                               ('CONTROL_FLOW', check_MACHO_control_flow)],
-        lief.ARCHITECTURES.ARM64: BASE_MACHO,
+        lief.ARCHITECTURES.ARM64: BASE_MACHO + [('BRANCH_PROTECTION', check_MACHO_branch_protection)],
     }
 }
 
@@ -256,7 +264,7 @@ if __name__ == '__main__':
                 retval = 1
                 continue
 
-            failed: List[str] = []
+            failed: list[str] = []
             for (name, func) in CHECKS[etype][arch]:
                 if not func(binary):
                     failed.append(name)

contrib/devtools/symbol-check.py

@@ -11,9 +11,8 @@ Example usage:
     find ../path/to/binaries -type f -executable | xargs python3 contrib/devtools/symbol-check.py
 '''
 import sys
-from typing import List, Dict
 
-import lief #type:ignore
+import lief
 
 # Debian 10 (Buster) EOL: 2024. https://wiki.debian.org/LTS
 #
@@ -33,7 +32,7 @@ import lief #type:ignore
 # See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html for more info.
 
 MAX_VERSIONS = {
-'GCC':       (4,8,0),
+'GCC':       (4,3,0),
 'GLIBC': {
     lief.ELF.ARCH.x86_64: (2,27),
     lief.ELF.ARCH.ARM:    (2,27),
@@ -53,7 +52,7 @@ IGNORE_EXPORTS = {
 
 # Expected linker-loader names can be found here:
 # https://sourceware.org/glibc/wiki/ABIList?action=recall&rev=16
-ELF_INTERPRETER_NAMES: Dict[lief.ELF.ARCH, Dict[lief.ENDIANNESS, str]] = {
+ELF_INTERPRETER_NAMES: dict[lief.ELF.ARCH, dict[lief.ENDIANNESS, str]] = {
     lief.ELF.ARCH.x86_64:  {
         lief.ENDIANNESS.LITTLE: "/lib64/ld-linux-x86-64.so.2",
     },
@@ -72,6 +71,25 @@ ELF_INTERPRETER_NAMES: Dict[lief.ELF.ARCH, Dict[lief.ENDIANNESS, str]] = {
     },
 }
 
+ELF_ABIS: dict[lief.ELF.ARCH, dict[lief.ENDIANNESS, list[int]]] = {
+    lief.ELF.ARCH.x86_64: {
+        lief.ENDIANNESS.LITTLE: [3,2,0],
+    },
+    lief.ELF.ARCH.ARM: {
+        lief.ENDIANNESS.LITTLE: [3,2,0],
+    },
+    lief.ELF.ARCH.AARCH64: {
+        lief.ENDIANNESS.LITTLE: [3,7,0],
+    },
+    lief.ELF.ARCH.PPC64: {
+        lief.ENDIANNESS.LITTLE: [3,10,0],
+        lief.ENDIANNESS.BIG: [3,2,0],
+    },
+    lief.ELF.ARCH.RISCV: {
+        lief.ENDIANNESS.LITTLE: [4,15,0],
+    },
+}
+
 # Allowed NEEDED libraries
 ELF_ALLOWED_LIBRARIES = {
 # bitcoind and bitcoin-qt
@@ -79,7 +97,6 @@ ELF_ALLOWED_LIBRARIES = {
 'libc.so.6', # C library
 'libpthread.so.0', # threading
 'libm.so.6', # math library
-'librt.so.1', # real-time (clock)
 'libatomic.so.1',
 'ld-linux-x86-64.so.2', # 64-bit dynamic linker
 'ld-linux.so.2', # 32-bit dynamic linker
@@ -139,21 +156,21 @@ PE_ALLOWED_LIBRARIES = {
 'KERNEL32.dll', # win32 base APIs
 'msvcrt.dll', # C standard library for MSVC
 'SHELL32.dll', # shell API
-'USER32.dll', # user interface
 'WS2_32.dll', # sockets
 # bitcoin-qt only
 'dwmapi.dll', # desktop window manager
 'GDI32.dll', # graphics device interface
 'IMM32.dll', # input method editor
-'NETAPI32.dll',
+'NETAPI32.dll', # network management
 'ole32.dll', # component object model
 'OLEAUT32.dll', # OLE Automation API
 'SHLWAPI.dll', # light weight shell API
-'USERENV.dll',
-'UxTheme.dll',
+'USER32.dll', # user interface
+'USERENV.dll', # user management
+'UxTheme.dll', # visual style
 'VERSION.dll', # version checking
 'WINMM.dll', # WinMM audio API
-'WTSAPI32.dll',
+'WTSAPI32.dll', # Remote Desktop
 }
 
 def check_version(max_versions, version, arch) -> bool:
@@ -213,12 +230,17 @@ def check_MACHO_libraries(binary) -> bool:
     return ok
 
 def check_MACHO_min_os(binary) -> bool:
-    if binary.build_version.minos == [10,15,0]:
+    if binary.build_version.minos == [11,0,0]:
         return True
     return False
 
 def check_MACHO_sdk(binary) -> bool:
-    if binary.build_version.sdk == [11, 0, 0]:
+    if binary.build_version.sdk == [14, 0, 0]:
+        return True
+    return False
+
+def check_MACHO_ld64(binary) -> bool:
+    if binary.build_version.tools[0].version == [711, 0, 0]:
         return True
     return False
 
@@ -242,17 +264,25 @@ def check_ELF_interpreter(binary) -> bool:
 
     return binary.concrete.interpreter == expected_interpreter
 
+def check_ELF_ABI(binary) -> bool:
+    expected_abi = ELF_ABIS[binary.header.machine_type][binary.abstract.header.endianness]
+    note = binary.concrete.get(lief.ELF.NOTE_TYPES.ABI_TAG)
+    assert note.details.abi == lief.ELF.NOTE_ABIS.LINUX
+    return note.details.version == expected_abi
+
 CHECKS = {
 lief.EXE_FORMATS.ELF: [
     ('IMPORTED_SYMBOLS', check_imported_symbols),
     ('EXPORTED_SYMBOLS', check_exported_symbols),
     ('LIBRARY_DEPENDENCIES', check_ELF_libraries),
     ('INTERPRETER_NAME', check_ELF_interpreter),
+    ('ABI', check_ELF_ABI),
 ],
 lief.EXE_FORMATS.MACHO: [
     ('DYNAMIC_LIBRARIES', check_MACHO_libraries),
     ('MIN_OS', check_MACHO_min_os),
     ('SDK', check_MACHO_sdk),
+    ('LD64', check_MACHO_ld64),
 ],
 lief.EXE_FORMATS.PE: [
     ('DYNAMIC_LIBRARIES', check_PE_libraries),
@@ -271,7 +301,7 @@ if __name__ == '__main__':
                 retval = 1
                 continue
 
-            failed: List[str] = []
+            failed: list[str] = []
             for (name, func) in CHECKS[etype]:
                 if not func(binary):
                     failed.append(name)

contrib/devtools/test-security-check.py

@@ -5,10 +5,9 @@
 '''
 Test script for security-check.py
 '''
-import lief #type:ignore
+import lief
 import os
 import subprocess
-from typing import List
 import unittest
 
 from utils import determine_wellknown_cmd
@@ -28,13 +27,13 @@ def clean_files(source, executable):
     os.remove(source)
     os.remove(executable)
 
-def call_security_check(cc, source, executable, options):
+def call_security_check(cc: str, source: str, executable: str, options) -> tuple:
     # This should behave the same as AC_TRY_LINK, so arrange well-known flags
     # in the same order as autoconf would.
     #
     # See the definitions for ac_link in autoconf's lib/autoconf/c.m4 file for
     # reference.
-    env_flags: List[str] = []
+    env_flags: list[str] = []
     for var in ['CFLAGS', 'CPPFLAGS', 'LDFLAGS']:
         env_flags += filter(None, os.environ.get(var, '').split(' '))
 
@@ -119,29 +118,31 @@ class TestSecurityChecks(unittest.TestCase):
         arch = get_arch(cc, source, executable)
 
         if arch == lief.ARCHITECTURES.X86:
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fno-stack-protector']),
-                (1, executable+': failed NOUNDEFS LAZY_BINDINGS Canary PIE NX CONTROL_FLOW'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fstack-protector-all']),
-                (1, executable+': failed NOUNDEFS LAZY_BINDINGS PIE NX CONTROL_FLOW'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-fstack-protector-all']),
-                (1, executable+': failed NOUNDEFS LAZY_BINDINGS PIE CONTROL_FLOW'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-fstack-protector-all']),
-                (1, executable+': failed LAZY_BINDINGS PIE CONTROL_FLOW'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-bind_at_load','-fstack-protector-all']),
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fno-stack-protector', '-Wl,-no_fixup_chains']),
+                (1, executable+': failed NOUNDEFS Canary FIXUP_CHAINS PIE NX CONTROL_FLOW'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fno-stack-protector', '-Wl,-fixup_chains']),
+                (1, executable+': failed NOUNDEFS Canary PIE NX CONTROL_FLOW'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fstack-protector-all', '-Wl,-fixup_chains']),
+                (1, executable+': failed NOUNDEFS PIE NX CONTROL_FLOW'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-fstack-protector-all', '-Wl,-fixup_chains']),
+                (1, executable+': failed NOUNDEFS PIE CONTROL_FLOW'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-fstack-protector-all', '-Wl,-fixup_chains']),
                 (1, executable+': failed PIE CONTROL_FLOW'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-bind_at_load','-fstack-protector-all', '-fcf-protection=full']),
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-fstack-protector-all', '-Wl,-fixup_chains']),
+                (1, executable+': failed PIE CONTROL_FLOW'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-fstack-protector-all', '-fcf-protection=full', '-Wl,-fixup_chains']),
                 (1, executable+': failed PIE'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-pie','-Wl,-bind_at_load','-fstack-protector-all', '-fcf-protection=full']),
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-pie','-fstack-protector-all', '-fcf-protection=full', '-Wl,-fixup_chains']),
                 (0, ''))
         else:
             # arm64 darwin doesn't support non-PIE binaries, control flow or executable stacks
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-flat_namespace','-fno-stack-protector']),
-                (1, executable+': failed NOUNDEFS LAZY_BINDINGS Canary'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-flat_namespace','-fstack-protector-all']),
-                (1, executable+': failed NOUNDEFS LAZY_BINDINGS'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-fstack-protector-all']),
-                (1, executable+': failed LAZY_BINDINGS'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-bind_at_load','-fstack-protector-all']),
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-flat_namespace','-fno-stack-protector', '-Wl,-no_fixup_chains']),
+                (1, executable+': failed NOUNDEFS Canary FIXUP_CHAINS BRANCH_PROTECTION'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-flat_namespace','-fno-stack-protector', '-Wl,-fixup_chains', '-mbranch-protection=bti']),
+                (1, executable+': failed NOUNDEFS Canary'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-flat_namespace','-fstack-protector-all', '-Wl,-fixup_chains', '-mbranch-protection=bti']),
+                (1, executable+': failed NOUNDEFS'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-fstack-protector-all', '-Wl,-fixup_chains', '-mbranch-protection=bti']),
                 (0, ''))
 
 

contrib/devtools/test-symbol-check.py

@@ -7,18 +7,17 @@ Test script for symbol-check.py
 '''
 import os
 import subprocess
-from typing import List
 import unittest
 
 from utils import determine_wellknown_cmd
 
-def call_symbol_check(cc: List[str], source, executable, options):
+def call_symbol_check(cc: list[str], source, executable, options):
     # This should behave the same as AC_TRY_LINK, so arrange well-known flags
     # in the same order as autoconf would.
     #
     # See the definitions for ac_link in autoconf's lib/autoconf/c.m4 file for
     # reference.
-    env_flags: List[str] = []
+    env_flags: list[str] = []
     for var in ['CFLAGS', 'CPPFLAGS', 'LDFLAGS']:
         env_flags += filter(None, os.environ.get(var, '').split(' '))
 
@@ -28,7 +27,7 @@ def call_symbol_check(cc: List[str], source, executable, options):
     os.remove(executable)
     return (p.returncode, p.stdout.rstrip())
 
-def get_machine(cc: List[str]):
+def get_machine(cc: list[str]):
     p = subprocess.run([*cc,'-dumpmachine'], stdout=subprocess.PIPE, text=True)
     return p.stdout.rstrip()
 
@@ -121,7 +120,7 @@ class TestSymbolChecks(unittest.TestCase):
                 }
         ''')
 
-        self.assertEqual(call_symbol_check(cc, source, executable, ['-Wl,-platform_version','-Wl,macos', '-Wl,10.15', '-Wl,11.4']),
+        self.assertEqual(call_symbol_check(cc, source, executable, ['-Wl,-platform_version','-Wl,macos', '-Wl,11.0', '-Wl,11.4']),
                 (1, f'{executable}: failed SDK'))
 
     def test_PE(self):

contrib/devtools/test_utxo_snapshots.sh

@@ -0,0 +1,209 @@
+#!/usr/bin/env bash
+# Demonstrate the creation and usage of UTXO snapshots.
+#
+# A server node starts up, IBDs up to a certain height, then generates a UTXO
+# snapshot at that point.
+#
+# The server then downloads more blocks (to create a diff from the snapshot).
+#
+# We bring a client up, load the UTXO snapshot, and we show the client sync to
+# the "network tip" and then start a background validation of the snapshot it
+# loaded. We see the background validation chainstate removed after validation
+# completes.
+#
+# The shellcheck rule SC2086 (quoted variables) disablements are necessary
+# since this rule needs to be violated in order to get bitcoind to pick up on
+# $EARLY_IBD_FLAGS for the script to work.
+
+export LC_ALL=C
+set -e
+
+BASE_HEIGHT=${1:-30000}
+INCREMENTAL_HEIGHT=20000
+FINAL_HEIGHT=$((BASE_HEIGHT + INCREMENTAL_HEIGHT))
+
+SERVER_DATADIR="$(pwd)/utxodemo-data-server-$BASE_HEIGHT"
+CLIENT_DATADIR="$(pwd)/utxodemo-data-client-$BASE_HEIGHT"
+UTXO_DAT_FILE="$(pwd)/utxo.$BASE_HEIGHT.dat"
+
+# Chosen to try to not interfere with any running bitcoind processes.
+SERVER_PORT=8633
+SERVER_RPC_PORT=8632
+
+CLIENT_PORT=8733
+CLIENT_RPC_PORT=8732
+
+SERVER_PORTS="-port=${SERVER_PORT} -rpcport=${SERVER_RPC_PORT}"
+CLIENT_PORTS="-port=${CLIENT_PORT} -rpcport=${CLIENT_RPC_PORT}"
+
+# Ensure the client exercises all indexes to test that snapshot use works
+# properly with indexes.
+ALL_INDEXES="-txindex -coinstatsindex -blockfilterindex=1"
+
+if ! command -v jq >/dev/null ; then
+  echo "This script requires jq to parse JSON RPC output. Please install it."
+  echo "(e.g. sudo apt install jq)"
+  exit 1
+fi
+
+DUMP_OUTPUT="dumptxoutset-output-$BASE_HEIGHT.json"
+
+finish() {
+  echo
+  echo "Killing server and client PIDs ($SERVER_PID, $CLIENT_PID) and cleaning up datadirs"
+  echo
+  rm -f "$UTXO_DAT_FILE" "$DUMP_OUTPUT"
+  rm -rf "$SERVER_DATADIR" "$CLIENT_DATADIR"
+  kill -9 "$SERVER_PID" "$CLIENT_PID"
+}
+
+trap finish EXIT
+
+# Need to specify these to trick client into accepting server as a peer
+# it can IBD from, otherwise the default values prevent IBD from the server node.
+EARLY_IBD_FLAGS="-maxtipage=9223372036854775207 -minimumchainwork=0x00"
+
+server_rpc() {
+  ./src/bitcoin-cli -rpcport=$SERVER_RPC_PORT -datadir="$SERVER_DATADIR" "$@"
+}
+client_rpc() {
+  ./src/bitcoin-cli -rpcport=$CLIENT_RPC_PORT -datadir="$CLIENT_DATADIR" "$@"
+}
+server_sleep_til_boot() {
+  while ! server_rpc ping >/dev/null 2>&1; do sleep 0.1; done
+}
+client_sleep_til_boot() {
+  while ! client_rpc ping >/dev/null 2>&1; do sleep 0.1; done
+}
+server_sleep_til_shutdown() {
+  while server_rpc ping >/dev/null 2>&1; do sleep 0.1; done
+}
+
+mkdir -p "$SERVER_DATADIR" "$CLIENT_DATADIR"
+
+echo "Hi, welcome to the assumeutxo demo/test"
+echo
+echo "We're going to"
+echo
+echo "  - start up a 'server' node, sync it via mainnet IBD to height ${BASE_HEIGHT}"
+echo "  - create a UTXO snapshot at that height"
+echo "  - IBD ${INCREMENTAL_HEIGHT} more blocks on top of that"
+echo
+echo "then we'll demonstrate assumeutxo by "
+echo
+echo "  - starting another node (the 'client') and loading the snapshot in"
+echo "    * first you'll have to modify the code slightly (chainparams) and recompile"
+echo "    * don't worry, we'll make it easy"
+echo "  - observing the client sync ${INCREMENTAL_HEIGHT} blocks on top of the snapshot from the server"
+echo "  - observing the client validate the snapshot chain via background IBD"
+echo
+read -p "Press [enter] to continue" _
+
+echo
+echo "-- Starting the demo. You might want to run the two following commands in"
+echo "   separate terminal windows:"
+echo
+echo "   watch -n0.1 tail -n 30 $SERVER_DATADIR/debug.log"
+echo "   watch -n0.1 tail -n 30 $CLIENT_DATADIR/debug.log"
+echo
+read -p "Press [enter] to continue" _
+
+echo
+echo "-- IBDing the blocks (height=$BASE_HEIGHT) required to the server node..."
+# shellcheck disable=SC2086
+./src/bitcoind -logthreadnames=1 $SERVER_PORTS \
+    -datadir="$SERVER_DATADIR" $EARLY_IBD_FLAGS -stopatheight="$BASE_HEIGHT" >/dev/null
+
+echo
+echo "-- Creating snapshot at ~ height $BASE_HEIGHT ($UTXO_DAT_FILE)..."
+server_sleep_til_shutdown # wait for stopatheight to be hit
+# shellcheck disable=SC2086
+./src/bitcoind -logthreadnames=1 $SERVER_PORTS \
+    -datadir="$SERVER_DATADIR" $EARLY_IBD_FLAGS -connect=0 -listen=0 >/dev/null &
+SERVER_PID="$!"
+
+server_sleep_til_boot
+server_rpc dumptxoutset "$UTXO_DAT_FILE" > "$DUMP_OUTPUT"
+cat "$DUMP_OUTPUT"
+kill -9 "$SERVER_PID"
+
+RPC_BASE_HEIGHT=$(jq -r .base_height < "$DUMP_OUTPUT")
+RPC_AU=$(jq -r .txoutset_hash < "$DUMP_OUTPUT")
+RPC_NCHAINTX=$(jq -r .nchaintx < "$DUMP_OUTPUT")
+RPC_BLOCKHASH=$(jq -r .base_hash < "$DUMP_OUTPUT")
+
+server_sleep_til_shutdown
+
+echo
+echo "-- Now: add the following to CMainParams::m_assumeutxo_data"
+echo "   in src/kernel/chainparams.cpp, and recompile:"
+echo
+echo "   {${RPC_BASE_HEIGHT}, AssumeutxoHash{uint256S(\"0x${RPC_AU}\")}, ${RPC_NCHAINTX}, uint256S(\"0x${RPC_BLOCKHASH}\")},"
+echo
+echo
+echo "-- IBDing more blocks to the server node (height=$FINAL_HEIGHT) so there is a diff between snapshot and tip..."
+# shellcheck disable=SC2086
+./src/bitcoind $SERVER_PORTS -logthreadnames=1 -datadir="$SERVER_DATADIR" \
+    $EARLY_IBD_FLAGS -stopatheight="$FINAL_HEIGHT" >/dev/null
+
+echo
+echo "-- Starting the server node to provide blocks to the client node..."
+# shellcheck disable=SC2086
+./src/bitcoind $SERVER_PORTS -logthreadnames=1 -debug=net -datadir="$SERVER_DATADIR" \
+    $EARLY_IBD_FLAGS -connect=0 -listen=1 >/dev/null &
+SERVER_PID="$!"
+server_sleep_til_boot
+
+echo
+echo "-- Okay, what you're about to see is the client starting up and activating the snapshot."
+echo "   I'm going to display the top 14 log lines from the client on top of an RPC called"
+echo "   getchainstates, which is like getblockchaininfo but for both the snapshot and "
+echo "   background validation chainstates."
+echo
+echo "   You're going to first see the snapshot chainstate sync to the server's tip, then"
+echo "   the background IBD chain kicks in to validate up to the base of the snapshot."
+echo
+echo "   Once validation of the snapshot is done, you should see log lines indicating"
+echo "   that we've deleted the background validation chainstate."
+echo
+echo "   Once everything completes, exit the watch command with CTRL+C."
+echo
+read -p "When you're ready for all this, hit [enter]" _
+
+echo
+echo "-- Starting the client node to get headers from the server, then load the snapshot..."
+# shellcheck disable=SC2086
+./src/bitcoind $CLIENT_PORTS $ALL_INDEXES -logthreadnames=1 -datadir="$CLIENT_DATADIR" \
+    -connect=0 -addnode=127.0.0.1:$SERVER_PORT -debug=net $EARLY_IBD_FLAGS >/dev/null &
+CLIENT_PID="$!"
+client_sleep_til_boot
+
+echo
+echo "-- Initial state of the client:"
+client_rpc getchainstates
+
+echo
+echo "-- Loading UTXO snapshot into client. Calling RPC in a loop..."
+while ! client_rpc loadtxoutset "$UTXO_DAT_FILE" ; do sleep 10; done
+
+watch -n 0.3 "( tail -n 14 $CLIENT_DATADIR/debug.log ; echo ; ./src/bitcoin-cli -rpcport=$CLIENT_RPC_PORT -datadir=$CLIENT_DATADIR getchainstates) | cat"
+
+echo
+echo "-- Okay, now I'm going to restart the client to make sure that the snapshot chain reloads "
+echo "   as the main chain properly..."
+echo
+echo "   Press CTRL+C after you're satisfied to exit the demo"
+echo
+read -p "Press [enter] to continue"
+
+client_sleep_til_boot
+# shellcheck disable=SC2086
+./src/bitcoind $CLIENT_PORTS $ALL_INDEXES -logthreadnames=1 -datadir="$CLIENT_DATADIR" -connect=0 \
+    -addnode=127.0.0.1:$SERVER_PORT "$EARLY_IBD_FLAGS" >/dev/null &
+CLIENT_PID="$!"
+client_sleep_til_boot
+
+watch -n 0.3 "( tail -n 14 $CLIENT_DATADIR/debug.log ; echo ; ./src/bitcoin-cli -rpcport=$CLIENT_RPC_PORT -datadir=$CLIENT_DATADIR getchainstates) | cat"
+
+echo
+echo "-- Done!"

contrib/devtools/utils.py

@@ -8,10 +8,9 @@ Common utility functions
 import shutil
 import sys
 import os
-from typing import List
 
 
-def determine_wellknown_cmd(envvar, progname) -> List[str]:
+def determine_wellknown_cmd(envvar, progname) -> list[str]:
     maybe_env = os.getenv(envvar)
     maybe_which = shutil.which(progname)
     if maybe_env:

contrib/devtools/utxo_snapshot.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019 The Bitcoin Core developers
+# Copyright (c) 2019-2023 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 #
@@ -8,6 +8,8 @@ export LC_ALL=C
 
 set -ueo pipefail
 
+NETWORK_DISABLED=false
+
 if (( $# < 3 )); then
   echo 'Usage: utxo_snapshot.sh <generate-at-height> <snapshot-out-path> <bitcoin-cli-call ...>'
   echo
@@ -26,19 +28,67 @@ OUTPUT_PATH="${1}"; shift;
 # Most of the calls we make take a while to run, so pad with a lengthy timeout.
 BITCOIN_CLI_CALL="${*} -rpcclienttimeout=9999999"
 
+# Check if the node is pruned and get the pruned block height
+PRUNED=$( ${BITCOIN_CLI_CALL} getblockchaininfo | awk '/pruneheight/ {print $2}' | tr -d ',' )
+
+if (( GENERATE_AT_HEIGHT < PRUNED )); then
+  echo "Error: The requested snapshot height (${GENERATE_AT_HEIGHT}) should be greater than the pruned block height (${PRUNED})."
+  exit 1
+fi
+
+# Early exit if file at OUTPUT_PATH already exists
+if [[ -e "$OUTPUT_PATH" ]]; then
+  (>&2 echo "Error: $OUTPUT_PATH already exists or is not a valid path.")
+  exit 1
+fi
+
+# Validate that the path is correct
+if [[ "${OUTPUT_PATH}" != "-" && ! -d "$(dirname "${OUTPUT_PATH}")" ]]; then
+  (>&2 echo "Error: The directory $(dirname "${OUTPUT_PATH}") does not exist.")
+  exit 1
+fi
+
+function cleanup {
+  (>&2 echo "Restoring chain to original height; this may take a while")
+  ${BITCOIN_CLI_CALL} reconsiderblock "${PIVOT_BLOCKHASH}"
+
+  if $NETWORK_DISABLED; then
+    (>&2 echo "Restoring network activity")
+    ${BITCOIN_CLI_CALL} setnetworkactive true
+  fi
+}
+
+function early_exit {
+  (>&2 echo "Exiting due to Ctrl-C")
+  cleanup
+  exit 1
+}
+
+# Prompt the user to disable network activity
+read -p "Do you want to disable network activity (setnetworkactive false) before running invalidateblock? (Y/n): " -r
+if [[ "$REPLY" =~ ^[Yy]*$ || -z "$REPLY" ]]; then
+  # User input is "Y", "y", or Enter key, proceed with the action
+  NETWORK_DISABLED=true
+  (>&2 echo "Disabling network activity")
+  ${BITCOIN_CLI_CALL} setnetworkactive false
+else
+  (>&2 echo "Network activity remains enabled")
+fi
+
 # Block we'll invalidate/reconsider to rewind/fast-forward the chain.
 PIVOT_BLOCKHASH=$($BITCOIN_CLI_CALL getblockhash $(( GENERATE_AT_HEIGHT + 1 )) )
 
+# Trap for normal exit and Ctrl-C
+trap cleanup EXIT
+trap early_exit INT
+
 (>&2 echo "Rewinding chain back to height ${GENERATE_AT_HEIGHT} (by invalidating ${PIVOT_BLOCKHASH}); this may take a while")
 ${BITCOIN_CLI_CALL} invalidateblock "${PIVOT_BLOCKHASH}"
 
 if [[ "${OUTPUT_PATH}" = "-" ]]; then
   (>&2 echo "Generating txoutset info...")
-  ${BITCOIN_CLI_CALL} gettxoutsetinfo | grep hash_serialized_2 | sed 's/^.*: "\(.\+\)\+",/\1/g'
+  ${BITCOIN_CLI_CALL} gettxoutsetinfo | grep hash_serialized_3 | sed 's/^.*: "\(.\+\)\+",/\1/g'
 else
   (>&2 echo "Generating UTXO snapshot...")
   ${BITCOIN_CLI_CALL} dumptxoutset "${OUTPUT_PATH}"
 fi
-
-(>&2 echo "Restoring chain to original height; this may take a while")
-${BITCOIN_CLI_CALL} reconsiderblock "${PIVOT_BLOCKHASH}"

contrib/guix/INSTALL.md

@@ -62,9 +62,6 @@ so you should log out and log back in.
 Please refer to fanquake's instructions
 [here](https://github.com/fanquake/core-review/tree/master/guix).
 
-Note that the `Dockerfile` is largely equivalent to running through the binary
-tarball installation steps.
-
 ## Option 4: Using a distribution-maintained package
 
 Note that this section is based on the distro packaging situation at the time of
@@ -74,25 +71,15 @@ https://repology.org/project/guix/versions
 
 ### Debian / Ubuntu
 
-Guix v1.2.0 is available as a distribution package starting in [Debian
-11](https://packages.debian.org/bullseye/guix) and [Ubuntu
-21.04](https://packages.ubuntu.com/search?keywords=guix).
-
-Note that if you intend on using Guix without using any substitutes (more
-details [here][security-model]), v1.2.0 has a known problem when building GnuTLS
-from source. Solutions and workarounds are documented
-[here](#gnutls-test-suite-fail-status-request-revoked).
-
+Guix is available as a distribution package in [Debian
+](https://packages.debian.org/search?keywords=guix) and [Ubuntu
+](https://packages.ubuntu.com/search?keywords=guix).
 
 To install:
 ```sh
 sudo apt install guix
 ```
 
-For up-to-date information on Debian and Ubuntu's release history:
-- [Debian release history](https://www.debian.org/releases/)
-- [Ubuntu release history](https://ubuntu.com/about/release-cycle)
-
 ### Arch Linux
 
 Guix is available in the AUR as
@@ -167,80 +154,41 @@ For reference, the graphic below outlines Guix v1.3.0's dependency graph:
 
 ![bootstrap map](https://user-images.githubusercontent.com/6399679/125064185-a9a59880-e0b0-11eb-82c1-9b8e5dc9950d.png)
 
-#### Consider /tmp on tmpfs
+If you do not care about building each dependency from source, and Guix is
+already packaged for your distribution, you can easily install only the build
+dependencies of Guix. For example, to enable deb-src and install the Guix build
+dependencies on Ubuntu/Debian:
 
-If you use an NVME (SSD) drive, you may encounter [cryptic build errors](#coreutils-fail-teststail-2inotify-dir-recreate). Mounting a [tmpfs at /tmp](https://ubuntu.com/blog/data-driven-analysis-tmp-on-tmpfs) should prevent this and may improve performance as a bonus.
+```sh
+sed -i 's|# deb-src|deb-src|g' /etc/apt/sources.list
+apt update
+apt-get build-dep -y guix
+```
+
+If this succeeded, you can likely skip to section
+["Building and Installing Guix itself"](#building-and-installing-guix-itself).
 
 #### Guile
 
-##### Choosing a Guile version and sticking to it
-
-One of the first things you need to decide is which Guile version you want to
-use: Guile v2.2 or Guile v3.0. Unlike the python2 to python3 transition, Guile
-v2.2 and Guile v3.0 are largely compatible, as evidenced by the fact that most
-Guile packages and even [Guix
-itself](https://guix.gnu.org/en/blog/2020/guile-3-and-guix/) support running on
-both.
-
-What is important here is that you **choose one**, and you **remain consistent**
-with your choice throughout **all Guile-related packages**, no matter if they
-are installed via the distribution's package manager or installed from source.
-This is because the files for Guile packages are installed to directories which
-are separated based on the Guile version.
-
-###### Example: Checking that Ubuntu's `guile-git` is compatible with your chosen Guile version
-
-On Ubuntu Focal:
-
-```sh
-$ apt show guile-git
-Package: guile-git
-...
-Depends: guile-2.2, guile-bytestructures, libgit2-dev
-...
-```
-
-As you can see, the package `guile-git` depends on `guile-2.2`, meaning that it
-was likely built for Guile v2.2. This means that if you decided to use Guile
-v3.0 on Ubuntu Focal, you would need to build guile-git from source instead of
-using the distribution package.
-
-On Ubuntu Hirsute:
-
-```sh
-$ apt show guile-git
-Package: guile-git
-...
-Depends: guile-3.0 | guile-2.2, guile-bytestructures (>= 1.0.7-3~), libgit2-dev (>= 1.0)
-...
-```
-
-In this case, `guile-git` depends on either `guile-3.0` or `guile-2.2`, meaning
-that it would work no matter what Guile version you decided to use.
-
 ###### Corner case: Multiple versions of Guile on one system
 
-It is recommended to only install one version of Guile, so that build systems do
+It is recommended to only install the required version of Guile, so that build systems do
 not get confused about which Guile to use.
 
-However, if you insist on having both Guile v2.2 and Guile v3.0 installed on
-your system, then you need to **consistently** specify one of
-`GUILE_EFFECTIVE_VERSION=3.0` or `GUILE_EFFECTIVE_VERSION=2.2` to all
+However, if you insist on having more versions of Guile installed on
+your system, then you need to **consistently** specify
+`GUILE_EFFECTIVE_VERSION=3.0` to all
 `./configure` invocations for Guix and its dependencies.
 
 ##### Installing Guile
 
-Guile is most likely already packaged for your distribution, so after you have
-[chosen a Guile version](#choosing-a-guile-version-and-sticking-to-it), install
-it via your distribution's package manager.
-
 If your distribution splits packages into `-dev`-suffixed and
 non-`-dev`-suffixed sub-packages (as is the case for Debian-derived
 distributions), please make sure to install both. For example, to install Guile
-v2.2 on Debian/Ubuntu:
+v3.0 on Debian/Ubuntu:
 
 ```sh
-apt install guile-2.2 guile-2.2-dev
+apt install guile-3.0 guile-3.0-dev
 ```
 
 #### Mixing distribution packages and source-built packages
@@ -258,16 +206,16 @@ source-built packages, you will need to augment the `GUILE_LOAD_PATH` and
 `GUILE_LOAD_COMPILED_PATH` environment variables so that Guile will look
 under the right prefix and find your source-built packages.
 
-For example, if you are using Guile v2.2, and have Guile packages in the
+For example, if you are using Guile v3.0, and have Guile packages in the
 `/usr/local` prefix, either add the following lines to your `.profile` or
 `.bash_profile` so that the environment variable is properly set for all future
 shell logins, or paste the lines into a POSIX-style shell to temporarily modify
 the environment variables of your current shell session.
 
 ```sh
-# Help Guile v2.2.x find packages in /usr/local
-export GUILE_LOAD_PATH="/usr/local/share/guile/site/2.2${GUILE_LOAD_PATH:+:}$GUILE_LOAD_PATH"
-export GUILE_LOAD_COMPILED_PATH="/usr/local/lib/guile/2.2/site-ccache${GUILE_LOAD_COMPILED_PATH:+:}$GUILE_COMPILED_LOAD_PATH"
+# Help Guile v3.0.x find packages in /usr/local
+export GUILE_LOAD_PATH="/usr/local/share/guile/site/3.0${GUILE_LOAD_PATH:+:}$GUILE_LOAD_PATH"
+export GUILE_LOAD_COMPILED_PATH="/usr/local/lib/guile/3.0/site-ccache${GUILE_LOAD_COMPILED_PATH:+:}$GUILE_COMPILED_LOAD_PATH"
 ```
 
 Note that these environment variables are used to check for packages during
@@ -352,7 +300,7 @@ Relevant for:
 - Those installing `guile-git` from their distribution where `guile-git` is
   built against `libgit2 < 1.1`
 
-As of v0.4.0, `guile-git` claims to only require `libgit2 >= 0.28.0`, however,
+As of v0.5.2, `guile-git` claims to only require `libgit2 >= 0.28.0`, however,
 it actually requires `libgit2 >= 1.1`, otherwise, it will be confused by a
 reference of `origin/keyring`: instead of interpreting the reference as "the
 'keyring' branch of the 'origin' remote", the reference is interpreted as "the
@@ -366,20 +314,6 @@ Should you be in this situation, you need to build both `libgit2 v1.1.x` and
 
 Source: https://logs.guix.gnu.org/guix/2020-11-12.log#232527
 
-##### `{scheme,guile}-bytestructures` v1.0.8 and v1.0.9 are broken for Guile v2.2
-
-Relevant for:
-- Those building `{scheme,guile}-bytestructures` from source against Guile v2.2
-
-Commit
-[707eea3](https://github.com/TaylanUB/scheme-bytestructures/commit/707eea3a85e1e375e86702229ebf73d496377669)
-introduced a regression for Guile v2.2 and was first included in v1.0.8, this
-was later corrected in commit
-[ec9a721](https://github.com/TaylanUB/scheme-bytestructures/commit/ec9a721957c17bcda13148f8faa5f06934431ff7)
-and included in v1.1.0.
-
-TL;DR If you decided to use Guile v2.2, do not use `{scheme,guile}-bytestructures` v1.0.8 or v1.0.9.
-
 ### Building and Installing Guix itself
 
 Start by cloning Guix:
@@ -389,10 +323,8 @@ git clone https://git.savannah.gnu.org/git/guix.git
 cd guix
 ```
 
-You will likely want to build the latest release, however, if the latest release
-when you're reading this is still 1.3.0 then you may want to use 998eda30 instead
-to avoid the issues described in [#25099](
-https://github.com/bitcoin/bitcoin/pull/25099).
+You will likely want to build the latest release.
+At the time of writing (November 2023), the latest release was `v1.4.0`.
 
 ```
 git branch -a -l 'origin/version-*'  # check for the latest release
@@ -578,7 +510,7 @@ sudo --login guix pull --commit=<particular-commit>
 ```
 
 `guix pull` is quite a long process (especially if you're using
-`--no-substitute`). If you encounter build problems, please refer to the
+`--no-substitutes`). If you encounter build problems, please refer to the
 [troubleshooting section](#troubleshooting).
 
 Note that running a bare `guix pull` with no commit or branch specified will
@@ -616,7 +548,7 @@ systemctl enable guix-daemon
 systemctl start guix-daemon
 ```
 
-Remember to set `--no-substitute` in `$libdir/systemd/system/guix-daemon.service` and other customizations if you used them for `guix-daemon-original.service`.
+Remember to set `--no-substitutes` in `$libdir/systemd/system/guix-daemon.service` and other customizations if you used them for `guix-daemon-original.service`.
 
 ##### If you installed Guix via the Debian/Ubuntu distribution packages
 
@@ -726,26 +658,18 @@ $ bzcat /var/log/guix/drvs/../...-foo-3.6.12.drv.bz2 | less
 times, it may be `/tmp/...drv-1` or `/tmp/...drv-2`. Always consult the build
 failure output for the most accurate, up-to-date information.
 
-### openssl-1.1.1l and openssl-1.1.1n
-
-OpenSSL includes tests that will fail once some certificate has expired. A workaround
-is to change your system clock:
-
-```sh
-sudo timedatectl set-ntp no
-sudo date --set "28 may 2022 15:00:00"
-sudo --login guix build --cores=1 /gnu/store/g9alz81w4q03ncm542487xd001s6akd4-openssl-1.1.1l.drv
-sudo --login guix build --cores=1 /gnu/store/mw6ax0gk33gh082anrdrxp2flrbskxv6-openssl-1.1.1n.drv
-sudo timedatectl set-ntp yes
-```
-
 ### python(-minimal): [Errno 84] Invalid or incomplete multibyte or wide character
 
 This error occurs when your `$TMPDIR` (default: /tmp) exists on a filesystem
 which rejects characters not present in the UTF-8 character code set. An example
 is ZFS with the utf8only=on option set.
 
-More information: https://bugs.python.org/issue37584
+More information: https://github.com/python/cpython/issues/81765
+
+### openssl-1.1.1l and openssl-1.1.1n
+
+OpenSSL includes tests that will fail once some certificate has expired.
+The workarounds from the GnuTLS section immediately below can be used.
 
 ### GnuTLS: test-suite FAIL: status-request-revoked
 
@@ -781,13 +705,41 @@ authorized.
 This workaround was described [here](https://issues.guix.gnu.org/44559#5).
 
 Basically:
-1. Turn off networking
 2. Turn off NTP
 3. Set system time to 2020-10-01
 4. guix build --no-substitutes /gnu/store/vhphki5sg9xkdhh2pbc8gi6vhpfzryf0-gnutls-3.6.12.drv
 5. Set system time back to accurate current time
 6. Turn NTP back on
-7. Turn networking back on
+
+For example,
+
+```sh
+sudo timedatectl set-ntp no
+sudo date --set "01 oct 2020 15:00:00"
+guix build /gnu/store/vhphki5sg9xkdhh2pbc8gi6vhpfzryf0-gnutls-3.6.12.drv
+sudo timedatectl set-ntp yes
+```
+
+#### Workaround 3: Disable the tests in the Guix source code for this single derivation
+
+If all of the above workarounds fail, you can also disable the `tests` phase of
+the derivation via the `arguments` option, as described in the official
+[`package`
+reference](https://guix.gnu.org/manual/en/html_node/package-Reference.html).
+
+For example, to disable the openssl-1.1 check phase:
+
+```diff
+diff --git a/gnu/packages/tls.scm b/gnu/packages/tls.scm
+index f1e844b..1077c4b 100644
+--- a/gnu/packages/tls.scm
++++ b/gnu/packages/tls.scm
+@@ -494,4 +494,5 @@ (define-public openssl-1.1
+     (arguments
+      `(#:parallel-tests? #f
++       #:tests? #f
+        #:test-target "test"
+```
 
 ### coreutils: FAIL: tests/tail-2/inotify-dir-recreate
 
@@ -796,7 +748,7 @@ The inotify-dir-create test fails on "remote" filesystems such as overlayfs
 as non-remote.
 
 A relatively easy workaround to this is to make sure that a somewhat traditional
-filesystem is mounted at `/tmp` (where `guix-daemon` performs its builds), see [/tmp on tmpfs](#consider-tmp-on-tmpfs). For
+filesystem is mounted at `/tmp` (where `guix-daemon` performs its builds). For
 Docker users, this might mean [using a volume][docker/volumes], [binding
 mounting][docker/bind-mnt] from host, or (for those with enough RAM and swap)
 [mounting a tmpfs][docker/tmpfs] using the `--tmpfs` flag.

contrib/guix/README.md

@@ -11,7 +11,7 @@ We achieve bootstrappability by using Guix as a functional package manager.
 
 # Requirements
 
-Conservatively, you will need an x86_64 machine with:
+Conservatively, you will need:
 
 - 16GB of free disk space on the partition that /gnu/store will reside in
 - 8GB of free disk space **per platform triple** you're planning on building
@@ -259,7 +259,7 @@ details.
   Override the number of jobs to run simultaneously, you might want to do so on
   a memory-limited machine. This may be passed to:
 
-  - `guix` build commands as in `guix environment --cores="$JOBS"`
+  - `guix` build commands as in `guix shell --cores="$JOBS"`
   - `make` as in `make --jobs="$JOBS"`
   - `xargs` as in `xargs -P"$JOBS"`
 
@@ -301,7 +301,7 @@ details.
 
 * _**ADDITIONAL_GUIX_ENVIRONMENT_FLAGS**_
 
-  Additional flags to be passed to the invocation of `guix environment` inside
+  Additional flags to be passed to the invocation of `guix shell` inside
   `guix time-machine`.
 
 # Choosing your security model