Merge bitcoin/bitcoin#29464: [25.2] Final backports and changes for 25.2rc1

9f13dc1ed3 doc: Update release notes for 25.2rc1 (Ava Chow)
a27662b16a doc: update manpages for 25.2rc1 (Ava Chow)
65c6171784 build: Bump to 25.2rc1 (Ava Chow)
cf0f43ee42 wallet: Fix use-after-free in WalletBatch::EraseRecords (MarcoFalke)
6acfc4324c Use only Span{} constructor for byte-like types where possible (MarcoFalke)
b40d10787b util: Allow std::byte and char Span serialization (MarcoFalke)

Pull request description:

  Backport:

  * #29176
  * #27927

  #29176 does not cleanly backport, and it also requires 27927 to work. Both are still fairly simple backports.

  Also does the rest of the version bump tasks for 25.2rc1.

ACKs for top commit:
  fanquake:
    ACK 9f13dc1ed3

Tree-SHA512: 9d9dbf415f8559410eba9a431b61a8fc94216898d2d1fd8398e1f7a22a04790faade810e65324c7a797456b33396c3a58f991e81319aaaa63d3ab441e5e20dbc

.cirrus.yml

@@ -3,48 +3,19 @@ env:  # Global defaults
   PACKAGE_MANAGER_INSTALL: "apt-get update && apt-get install -y"
   MAKEJOBS: "-j10"
   TEST_RUNNER_PORT_MIN: "14000"  # Must be larger than 12321, which is used for the http cache. See https://cirrus-ci.org/guide/writing-tasks/#http-cache
-  CI_FAILFAST_TEST_LEAVE_DANGLING: "1"  # Cirrus CI does not care about dangling processes and setting this variable avoids killing the CI script itself on error
-  CCACHE_MAXSIZE: "200M"
+  CI_FAILFAST_TEST_LEAVE_DANGLING: "1"  # Cirrus CI does not care about dangling process and setting this variable avoids killing the CI script itself on error
+  CCACHE_SIZE: "200M"
   CCACHE_DIR: "/tmp/ccache_dir"
   CCACHE_NOHASHDIR: "1"  # Debug info might contain a stale path if the build dir changes, but this is fine
 
-# https://cirrus-ci.org/guide/persistent-workers/
-#
-# It is possible to select a specific persistent worker by label. Refer to the
-# Cirrus CI docs for more details.
-#
-# Generally, a persistent worker must run Ubuntu 23.04+ or Debian 12+.
-# Specifically,
-# - apt-get is required due to PACKAGE_MANAGER_INSTALL
-# - podman-docker-4.1+ is required due to the use of `podman` when
-#   RESTART_CI_DOCKER_BEFORE_RUN is set and 4.1+ due to the bugfix in 4.1
-#   (https://github.com/bitcoin/bitcoin/pull/21652#issuecomment-1657098200)
-# - The ./ci/ depedencies (with cirrus-cli) should be installed:
-#
-#   ```
-#   apt update && apt install screen python3 bash podman-docker curl -y && curl -L -o cirrus "https://github.com/cirruslabs/cirrus-cli/releases/latest/download/cirrus-linux-$(dpkg --print-architecture)" && mv cirrus /usr/local/bin/cirrus && chmod +x /usr/local/bin/cirrus
-#   ```
-#
-# - There are no strict requirements on the hardware, because having less CPUs
-#   runs the same CI script (maybe slower). To avoid rare and intermittent OOM
-#   due to short memory usage spikes, it is recommended to add (and persist)
-#   swap:
-#
-#   ```
-#   fallocate -l 16G /swapfile_ci && chmod 600 /swapfile_ci && mkswap /swapfile_ci && swapon /swapfile_ci && ( echo '/swapfile_ci none swap sw 0 0' | tee -a /etc/fstab )
-#   ```
-#
-# - To register the persistent worker, open a `screen` session and run:
-#
-#   ```
-#   RESTART_CI_DOCKER_BEFORE_RUN=1 screen cirrus worker run --labels type=todo_fill_in_type --token todo_fill_in_token
-#   ```
-#
-# The following specific types should exist, with the following requirements:
-# - small: For an x86_64 machine, recommended to have 2 CPUs and 8 GB of memory.
-# - medium: For an x86_64 machine, recommended to have 4 CPUs and 16 GB of memory.
-# - noble: For a machine running the Linux kernel shipped with exaclty Ubuntu Noble 24.04. The machine is recommended to have 4 CPUs and 16 GB of memory.
-# - arm64: For an aarch64 machine, recommended to have 2 CPUs and 8 GB of memory.
+cirrus_ephemeral_worker_template_env: &CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
+  DANGER_RUN_CI_ON_HOST: "1"  # Containers will be discarded after the run, so there is no risk that the ci scripts modify the system
+
+persistent_worker_template_env: &PERSISTENT_WORKER_TEMPLATE_ENV
+  RESTART_CI_DOCKER_BEFORE_RUN: "1"
+
+persistent_worker_template: &PERSISTENT_WORKER_TEMPLATE
+  persistent_worker: {}  # https://cirrus-ci.org/guide/persistent-workers/
 
 # https://cirrus-ci.org/guide/tips-and-tricks/#sharing-configuration-between-tasks
 filter_template: &FILTER_TEMPLATE
@@ -63,11 +34,26 @@ base_template: &BASE_TEMPLATE
 
 main_template: &MAIN_TEMPLATE
   timeout_in: 120m  # https://cirrus-ci.org/faq/#instance-timed-out
+  ccache_cache:
+    folder: "/tmp/ccache_dir"
   ci_script:
     - ./ci/test_run_all.sh
 
-global_task_template: &GLOBAL_TASK_TEMPLATE
+container_depends_template: &CONTAINER_DEPENDS_TEMPLATE
   << : *BASE_TEMPLATE
+  container:
+    # https://cirrus-ci.org/faq/#are-there-any-limits
+    # Each project has 16 CPU in total, assign 2 to each container, so that 8 tasks run in parallel
+    cpu: 2
+    greedy: true
+    memory: 8G  # Set to 8GB to avoid OOM. https://cirrus-ci.org/guide/linux/#linux-containers
+    dockerfile: ci/test_imagefile  # https://cirrus-ci.org/guide/docker-builder-vm/#dockerfile-as-a-ci-environment
+  depends_built_cache:
+    folder: "depends/built"
+    fingerprint_script: echo $CIRRUS_TASK_NAME $(git rev-parse HEAD:depends)
+
+global_task_template: &GLOBAL_TASK_TEMPLATE
+  << : *CONTAINER_DEPENDS_TEMPLATE
   << : *MAIN_TEMPLATE
 
 compute_credits_template: &CREDITS_TEMPLATE
@@ -76,7 +62,7 @@ compute_credits_template: &CREDITS_TEMPLATE
   use_compute_credits: $CIRRUS_REPO_FULL_NAME == 'bitcoin/bitcoin' && $CIRRUS_PR != ""
 
 task:
-  name: 'lint'
+  name: 'lint [bookworm]'
   << : *BASE_TEMPLATE
   container:
     image: debian:bookworm
@@ -85,122 +71,276 @@ task:
   # For faster CI feedback, immediately schedule the linters
   << : *CREDITS_TEMPLATE
   python_cache:
-    folder: "/python_build"
+    folder: "/tmp/python"
     fingerprint_script: cat .python-version /etc/os-release
   unshallow_script:
     - git fetch --unshallow --no-tags
   lint_script:
     - ./ci/lint_run_all.sh
+  env:
+    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
 
 task:
-  name: 'tidy'
+  name: 'tidy [lunar]'
   << : *GLOBAL_TASK_TEMPLATE
-  persistent_worker:
-    labels:
-      type: medium
+  container:
+    cpu: 2
+    memory: 5G
+    docker_arguments:
+      CI_IMAGE_NAME_TAG: ubuntu:lunar
+      FILE_ENV: "./ci/test/00_setup_env_native_tidy.sh"
+  # For faster CI feedback, immediately schedule the linters
+  << : *CREDITS_TEMPLATE
   env:
-    FILE_ENV: "./ci/test/00_setup_env_native_tidy.sh"
+    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
 
 task:
-  name: 'ARM, unit tests, no functional tests'
-  << : *GLOBAL_TASK_TEMPLATE
-  persistent_worker:
-    labels:
-      type: arm64  # Use arm64 worker to sidestep qemu and avoid a slow CI: https://github.com/bitcoin/bitcoin/pull/28087#issuecomment-1649399453
+  name: "Win64 native [vs2022]"
+  << : *FILTER_TEMPLATE
+  windows_container:
+    cpu: 6
+    memory: 12G
+    image: cirrusci/windowsservercore:visualstudio2022
+  timeout_in: 120m
   env:
-    FILE_ENV: "./ci/test/00_setup_env_arm.sh"
+    PATH: 'C:\jom;C:\Python39;C:\Python39\Scripts;C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Current\Bin;%PATH%'
+    PYTHONUTF8: 1
+    CI_VCPKG_TAG: '2023.01.09'
+    VCPKG_DOWNLOADS: 'C:\Users\ContainerAdministrator\AppData\Local\vcpkg\downloads'
+    VCPKG_DEFAULT_BINARY_CACHE: 'C:\Users\ContainerAdministrator\AppData\Local\vcpkg\archives'
+    CCACHE_DIR: 'C:\Users\ContainerAdministrator\AppData\Local\ccache'
+    WRAPPED_CL: 'C:\Users\ContainerAdministrator\AppData\Local\Temp\cirrus-ci-build\ci\test\wrapped-cl.bat'
+    QT_DOWNLOAD_URL: 'https://download.qt.io/official_releases/qt/5.15/5.15.5/single/qt-everywhere-opensource-src-5.15.5.zip'
+    QT_LOCAL_PATH: 'C:\qt-everywhere-opensource-src-5.15.5.zip'
+    QT_SOURCE_DIR: 'C:\qt-everywhere-src-5.15.5'
+    QTBASEDIR: 'C:\Qt_static'
+    x64_NATIVE_TOOLS: '"C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat"'
+    QT_CONFIGURE_COMMAND: '..\configure -release -silent -opensource -confirm-license -opengl desktop -static -static-runtime -mp -qt-zlib -qt-pcre -qt-libpng -nomake examples -nomake tests -nomake tools -no-angle -no-dbus -no-gif -no-gtk -no-ico -no-icu -no-libjpeg -no-libudev -no-sql-sqlite -no-sql-odbc -no-sqlite -no-vulkan -skip qt3d -skip qtactiveqt -skip qtandroidextras -skip qtcharts -skip qtconnectivity -skip qtdatavis3d -skip qtdeclarative -skip doc -skip qtdoc -skip qtgamepad -skip qtgraphicaleffects -skip qtimageformats -skip qtlocation -skip qtlottie -skip qtmacextras -skip qtmultimedia -skip qtnetworkauth -skip qtpurchasing -skip qtquick3d -skip qtquickcontrols -skip qtquickcontrols2 -skip qtquicktimeline -skip qtremoteobjects -skip qtscript -skip qtscxml -skip qtsensors -skip qtserialbus -skip qtserialport -skip qtspeech -skip qtsvg -skip qtvirtualkeyboard -skip qtwayland -skip qtwebchannel -skip qtwebengine -skip qtwebglplugin -skip qtwebsockets -skip qtwebview -skip qtx11extras -skip qtxmlpatterns -no-openssl -no-feature-bearermanagement -no-feature-printdialog -no-feature-printer -no-feature-printpreviewdialog -no-feature-printpreviewwidget -no-feature-sql -no-feature-sqlmodel -no-feature-textbrowser -no-feature-textmarkdownwriter -no-feature-textodfwriter -no-feature-xml'
+    IgnoreWarnIntDirInTempDetected: 'true'
+  merge_script:
+    - PowerShell -NoLogo -Command if ($env:CIRRUS_PR -ne $null) { git fetch $env:CIRRUS_REPO_CLONE_URL pull/$env:CIRRUS_PR/merge; git reset --hard FETCH_HEAD; }
+  msvc_qt_built_cache:
+    folder: "%QTBASEDIR%"
+    reupload_on_changes: false
+    fingerprint_script:
+      - echo %QT_DOWNLOAD_URL% %QT_CONFIGURE_COMMAND%
+      - msbuild -version
+    populate_script:
+      - curl -L -o C:\jom.zip http://download.qt.io/official_releases/jom/jom.zip
+      - mkdir C:\jom
+      - tar -xf C:\jom.zip -C C:\jom
+      - curl -L -o %QT_LOCAL_PATH% %QT_DOWNLOAD_URL%
+      - tar -xf %QT_LOCAL_PATH% -C C:\
+      - '%x64_NATIVE_TOOLS%'
+      - cd %QT_SOURCE_DIR%
+      - mkdir build
+      - cd build
+      - '%QT_CONFIGURE_COMMAND% -prefix %QTBASEDIR%'
+      - jom
+      - jom install
+  vcpkg_tools_cache:
+    folder: '%VCPKG_DOWNLOADS%\tools'
+    reupload_on_changes: false
+    fingerprint_script:
+      - echo %CI_VCPKG_TAG%
+      - msbuild -version
+  vcpkg_binary_cache:
+    folder: '%VCPKG_DEFAULT_BINARY_CACHE%'
+    reupload_on_changes: true
+    fingerprint_script:
+      - echo %CI_VCPKG_TAG%
+      - type build_msvc\vcpkg.json
+      - msbuild -version
+    populate_script:
+      - mkdir %VCPKG_DEFAULT_BINARY_CACHE%
+  ccache_cache:
+    folder: '%CCACHE_DIR%'
+  install_tools_script:
+    - choco install --yes --no-progress ccache --version=4.7.4
+    - choco install --yes --no-progress python3 --version=3.9.6
+    - pip install zmq
+    - ccache --version
+    - python -VV
+  install_vcpkg_script:
+    - cd ..
+    - git clone --quiet https://github.com/microsoft/vcpkg.git
+    - cd vcpkg
+    - git -c advice.detachedHead=false checkout %CI_VCPKG_TAG%
+    - .\bootstrap-vcpkg -disableMetrics
+    - echo set(VCPKG_BUILD_TYPE release) >> triplets\x64-windows-static.cmake
+    - .\vcpkg integrate install
+    - .\vcpkg version
+  build_script:
+    - '%x64_NATIVE_TOOLS%'
+    - cd %CIRRUS_WORKING_DIR%
+    - ccache --zero-stats --max-size=%CCACHE_SIZE%
+    - python build_msvc\msvc-autogen.py
+    - msbuild build_msvc\bitcoin.sln -property:CLToolExe=%WRAPPED_CL%;UseMultiToolTask=true;Configuration=Release -maxCpuCount -verbosity:minimal -noLogo
+    - ccache --show-stats
+  check_script:
+    - src\test_bitcoin.exe -l test_suite
+    - src\bench_bitcoin.exe --sanity-check
+    - python test\util\test_runner.py
+    - python test\util\rpcauth-test.py
+  functional_tests_script:
+    # Increase the dynamic port range to the maximum allowed value to mitigate "OSError: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted".
+    # See: https://learn.microsoft.com/en-us/biztalk/technical-guides/settings-that-can-be-modified-to-improve-network-performance
+    - netsh int ipv4 set dynamicport tcp start=1025 num=64511
+    - netsh int ipv6 set dynamicport tcp start=1025 num=64511
+    # Exclude feature_dbcrash for now due to timeout
+    - python test\functional\test_runner.py --nocleanup --ci --quiet --combinedlogslen=99999999 --jobs=6 --timeout-factor=8 --extended --exclude feature_dbcrash
 
 task:
-  name: 'Win64, unit tests, no gui tests, no boost::process, no functional tests'
+  name: 'ARM [unit tests, no functional tests] [bullseye]'
   << : *GLOBAL_TASK_TEMPLATE
-  persistent_worker:
-    labels:
-      type: small
+  container:
+    docker_arguments:
+      CI_IMAGE_NAME_TAG: debian:bullseye
+      FILE_ENV: "./ci/test/00_setup_env_arm.sh"
+  << : *CREDITS_TEMPLATE
   env:
-    FILE_ENV: "./ci/test/00_setup_env_win64.sh"
+    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
 
 task:
-  name: '32-bit CentOS, dash, gui'
+  name: 'Win64 [unit tests, no gui tests, no boost::process, no functional tests] [jammy]'
   << : *GLOBAL_TASK_TEMPLATE
-  persistent_worker:
-    labels:
-      type: small
+  container:
+    docker_arguments:
+      CI_IMAGE_NAME_TAG: ubuntu:jammy
+      FILE_ENV: "./ci/test/00_setup_env_win64.sh"
+  << : *CREDITS_TEMPLATE
   env:
-    FILE_ENV: "./ci/test/00_setup_env_i686_centos.sh"
+    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
 
 task:
-  name: 'previous releases, qt5 dev package and depends packages, DEBUG'
+  name: '32-bit + dash [gui] [CentOS 8]'
   << : *GLOBAL_TASK_TEMPLATE
-  persistent_worker:
-    labels:
-      type: small
+  container:
+    docker_arguments:
+      CI_IMAGE_NAME_TAG: quay.io/centos/centos:stream8
+      FILE_ENV: "./ci/test/00_setup_env_i686_centos.sh"
+  # For faster CI feedback, immediately schedule one task that runs all tests
+  << : *CREDITS_TEMPLATE
   env:
+    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
+    PACKAGE_MANAGER_INSTALL: "yum install -y"
+
+task:
+  name: '[previous releases, uses qt5 dev package and some depends packages, DEBUG] [unsigned char] [buster]'
+  previous_releases_cache:
+    folder: "releases"
+  << : *GLOBAL_TASK_TEMPLATE
+  << : *PERSISTENT_WORKER_TEMPLATE
+  env:
+    << : *PERSISTENT_WORKER_TEMPLATE_ENV
     FILE_ENV: "./ci/test/00_setup_env_native_qt5.sh"
 
 task:
-  name: 'TSan, depends, gui'
+  name: '[TSan, depends, gui] [lunar]'
   << : *GLOBAL_TASK_TEMPLATE
-  persistent_worker:
-    labels:
-      type: medium
+  container:
+    cpu: 6  # Increase CPU and Memory to avoid timeout
+    memory: 24G
+    docker_arguments:
+      CI_IMAGE_NAME_TAG: ubuntu:lunar
+      FILE_ENV: "./ci/test/00_setup_env_native_tsan.sh"
+  << : *CREDITS_TEMPLATE
   env:
-    FILE_ENV: "./ci/test/00_setup_env_native_tsan.sh"
+    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
 
 task:
-  name: 'MSan, depends'
+  name: '[MSan, depends] [lunar]'
   << : *GLOBAL_TASK_TEMPLATE
-  persistent_worker:
-    labels:
-      type: small
-  timeout_in: 300m  # Use longer timeout for the *rare* case where a full build (llvm + msan + depends + ...) needs to be done.
+  container:
+    docker_arguments:
+      CI_IMAGE_NAME_TAG: ubuntu:lunar
+      FILE_ENV: "./ci/test/00_setup_env_native_msan.sh"
   env:
-    FILE_ENV: "./ci/test/00_setup_env_native_msan.sh"
+    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
+    MAKEJOBS: "-j4"  # Avoid excessive memory use due to MSan
 
 task:
-  name: 'ASan + LSan + UBSan + integer, no depends, USDT'
-  enable_bpfcc_script:
-    # In the image build step, no external environment variables are available,
-    # so any settings will need to be written to the settings env file:
-    - sed -i "s|\${CIRRUS_CI}|true|g" ./ci/test/00_setup_env_native_asan.sh
+  name: '[ASan + LSan + UBSan + integer, no depends, USDT] [jammy]'
   << : *GLOBAL_TASK_TEMPLATE
-  persistent_worker:
-    labels:
-      type: noble  # Must use this specific worker (needed for USDT functional tests)
+  # We can't use a 'container' for the USDT interface tests as the CirrusCI
+  # containers don't have privileges to hook into bitcoind. CirrusCI uses
+  # Google Compute Engine instances: https://cirrus-ci.org/guide/custom-vms/
+  # Images can be found here: https://cloud.google.com/compute/docs/images/os-details
+  compute_engine_instance:
+    image_project: ubuntu-os-cloud
+    image: family/ubuntu-2204-lts # when upgrading, check if we can drop "ADD_UNTRUSTED_BPFCC_PPA"
+    cpu: 4
+    memory: 12G
   env:
+    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
+    HOME: /root/  # Only needed for compute_engine_instance
     FILE_ENV: "./ci/test/00_setup_env_native_asan.sh"
+    MAKEJOBS: "-j4"  # Avoid excessive memory use
 
 task:
-  name: 'fuzzer,address,undefined,integer, no depends'
+  name: '[fuzzer,address,undefined,integer, no depends] [lunar]'
   << : *GLOBAL_TASK_TEMPLATE
-  persistent_worker:
-    labels:
-      type: medium
+  container:
+    cpu: 4  # Increase CPU and memory to avoid timeout
+    memory: 16G
+    docker_arguments:
+      CI_IMAGE_NAME_TAG: ubuntu:lunar
+      FILE_ENV: "./ci/test/00_setup_env_native_fuzz.sh"
   env:
-    FILE_ENV: "./ci/test/00_setup_env_native_fuzz.sh"
+    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
 
 task:
-  name: 'multiprocess, i686, DEBUG'
+  name: '[multiprocess, i686, DEBUG] [focal]'
   << : *GLOBAL_TASK_TEMPLATE
-  persistent_worker:
-    labels:
-      type: medium
+  container:
+    cpu: 4
+    memory: 16G  # The default memory is sometimes just a bit too small, so double everything
+    docker_arguments:
+      CI_IMAGE_NAME_TAG: ubuntu:focal
+      FILE_ENV: "./ci/test/00_setup_env_i686_multiprocess.sh"
   env:
-    FILE_ENV: "./ci/test/00_setup_env_i686_multiprocess.sh"
+    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
 
 task:
-  name: 'no wallet, libbitcoinkernel'
+  name: '[no wallet, libbitcoinkernel] [focal]'
   << : *GLOBAL_TASK_TEMPLATE
-  persistent_worker:
-    labels:
-      type: small
+  container:
+    docker_arguments:
+      CI_IMAGE_NAME_TAG: ubuntu:focal
+      FILE_ENV: "./ci/test/00_setup_env_native_nowallet_libbitcoinkernel.sh"
+  << : *CREDITS_TEMPLATE
   env:
-    FILE_ENV: "./ci/test/00_setup_env_native_nowallet_libbitcoinkernel.sh"
+    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
 
 task:
-  name: 'macOS-cross 11.0, gui, no tests'
-  << : *GLOBAL_TASK_TEMPLATE
-  persistent_worker:
-    labels:
-      type: small
+  name: 'macOS 10.15 [gui, no tests] [focal]'
+  << : *CONTAINER_DEPENDS_TEMPLATE
+  container:
+    docker_arguments:
+      CI_IMAGE_NAME_TAG: ubuntu:focal
+      FILE_ENV: "./ci/test/00_setup_env_mac.sh"
+  << : *CREDITS_TEMPLATE
+  macos_sdk_cache:
+    folder: "depends/SDKs/$MACOS_SDK"
+    fingerprint_key: "$MACOS_SDK"
+  << : *MAIN_TEMPLATE
   env:
-    FILE_ENV: "./ci/test/00_setup_env_mac.sh"
+    MACOS_SDK: "Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers"
+    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
+
+task:
+  name: 'macOS 13 native arm64 [gui, sqlite only] [no depends]'
+  macos_instance:
+    # Use latest image, but hardcode version to avoid silent upgrades (and breaks)
+    image: ghcr.io/cirruslabs/macos-ventura-xcode:14.1  # https://cirrus-ci.org/guide/macOS
+  << : *BASE_TEMPLATE
+  check_clang_script:
+    - clang --version
+  brew_install_script:
+    - brew install boost libevent qt@5 miniupnpc libnatpmp ccache zeromq qrencode libtool automake gnu-getopt
+  << : *MAIN_TEMPLATE
+  env:
+    << : *CIRRUS_EPHEMERAL_WORKER_TEMPLATE_ENV
+    CI_USE_APT_INSTALL: "no"
+    PACKAGE_MANAGER_INSTALL: "echo"  # Nothing to do
+    FILE_ENV: "./ci/test/00_setup_env_mac_native_arm64.sh"

.github/ISSUE_TEMPLATE/good_first_issue.yml

@@ -29,10 +29,8 @@ body:
     attributes:
       label: Useful Skills
       description: For example, “`std::thread`”, “Qt5 GUI and async GUI design” or “basic understanding of Bitcoin mining and the Bitcoin Core RPC interface”.
-      value: |
-        * Compiling Bitcoin Core from source
-        * Running the C++ unit tests and the Python functional tests
-        * ...
+    validations:
+      required: false
   - type: textarea
     attributes:
       label: Guidance for new contributors

.github/workflows/ci.yml

@@ -1,344 +0,0 @@
-# Copyright (c) 2023 The Bitcoin Core developers
-# Distributed under the MIT software license, see the accompanying
-# file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-name: CI
-on:
-  # See: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request.
-  pull_request:
-  # See: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#push.
-  push:
-    branches:
-      - '**'
-    tags-ignore:
-      - '**'
-
-concurrency:
-  group: ${{ github.event_name != 'pull_request' && github.run_id || github.ref }}
-  cancel-in-progress: true
-
-env:
-  DANGER_RUN_CI_ON_HOST: 1
-  CI_FAILFAST_TEST_LEAVE_DANGLING: 1  # GHA does not care about dangling processes and setting this variable avoids killing the CI script itself on error
-  MAKEJOBS: '-j10'
-
-jobs:
-  test-each-commit:
-    name: 'test each commit'
-    runs-on: ubuntu-22.04
-    if: github.event_name == 'pull_request' && github.event.pull_request.commits != 1
-    timeout-minutes: 360  # Use maximum time, see https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes. Assuming a worst case time of 1 hour per commit, this leads to a --max-count=6 below.
-    env:
-      MAX_COUNT: 6
-    steps:
-      - name: Determine fetch depth
-        run: echo "FETCH_DEPTH=$((${{ github.event.pull_request.commits }} + 2))" >> "$GITHUB_ENV"
-      - uses: actions/checkout@v4
-        with:
-          ref: ${{ github.event.pull_request.head.sha }}
-          fetch-depth: ${{ env.FETCH_DEPTH }}
-      - name: Determine commit range
-        run: |
-          # Checkout HEAD~ and find the test base commit
-          # Checkout HEAD~ because it would be wasteful to rerun tests on the PR
-          # head commit that are already run by other jobs.
-          git checkout HEAD~
-          # Figure out test base commit by listing ancestors of HEAD, excluding
-          # ancestors of the most recent merge commit, limiting the list to the
-          # newest MAX_COUNT ancestors, ordering it from oldest to newest, and
-          # taking the first one.
-          #
-          # If the branch contains up to MAX_COUNT ancestor commits after the
-          # most recent merge commit, all of those commits will be tested. If it
-          # contains more, only the most recent MAX_COUNT commits will be
-          # tested.
-          #
-          # In the command below, the ^@ suffix is used to refer to all parents
-          # of the merge commit as described in:
-          # https://git-scm.com/docs/git-rev-parse#_other_rev_parent_shorthand_notations
-          # and the ^ prefix is used to exclude these parents and all their
-          # ancestors from the rev-list output as described in:
-          # https://git-scm.com/docs/git-rev-list
-          echo "TEST_BASE=$(git rev-list -n$((${{ env.MAX_COUNT }} + 1)) --reverse HEAD ^$(git rev-list -n1 --merges HEAD)^@ | head -1)" >> "$GITHUB_ENV"
-      - run: |
-          sudo apt-get update
-          sudo apt-get install clang ccache build-essential libtool autotools-dev automake pkg-config bsdmainutils python3-zmq libevent-dev libboost-dev libsqlite3-dev libdb++-dev systemtap-sdt-dev libminiupnpc-dev libnatpmp-dev qtbase5-dev qttools5-dev qttools5-dev-tools qtwayland5 libqrencode-dev -y
-      - name: Compile and run tests
-        run: |
-          # Run tests on commits after the last merge commit and before the PR head commit
-          # Use clang++, because it is a bit faster and uses less memory than g++
-          git rebase --exec "echo Running test-one-commit on \$( git log -1 ) && ./autogen.sh && CC=clang CXX=clang++ ./configure && make clean && make -j $(nproc) check && ./test/functional/test_runner.py -j $(( $(nproc) * 2 ))" ${{ env.TEST_BASE }}
-
-  macos-native-x86_64:
-    name: 'macOS 13 native, x86_64, no depends, sqlite only, gui'
-    # Use latest image, but hardcode version to avoid silent upgrades (and breaks).
-    # See: https://github.com/actions/runner-images#available-images.
-    runs-on: macos-13  # Use M1 once available https://github.com/github/roadmap/issues/528
-
-    # No need to run on the read-only mirror, unless it is a PR.
-    if: github.repository != 'bitcoin-core/gui' || github.event_name == 'pull_request'
-
-    timeout-minutes: 120
-
-    env:
-      FILE_ENV: './ci/test/00_setup_env_mac_native.sh'
-      BASE_ROOT_DIR: ${{ github.workspace }}
-
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-
-      - name: Clang version
-        run: |
-          sudo xcode-select --switch /Applications/Xcode_15.0.app
-          clang --version
-
-      - name: Install Homebrew packages
-        env:
-          HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK: 1
-        run: |
-          # A workaround for "The `brew link` step did not complete successfully" error.
-          brew install python@3 || brew link --overwrite python@3
-          brew install automake libtool pkg-config gnu-getopt ccache boost libevent miniupnpc libnatpmp zeromq qt@5 qrencode
-
-      - name: Set Ccache directory
-        run: echo "CCACHE_DIR=${RUNNER_TEMP}/ccache_dir" >> "$GITHUB_ENV"
-
-      - name: Restore Ccache cache
-        id: ccache-cache
-        uses: actions/cache/restore@v3
-        with:
-          path: ${{ env.CCACHE_DIR }}
-          key: ${{ github.job }}-ccache-${{ github.run_id }}
-          restore-keys: ${{ github.job }}-ccache-
-
-      - name: CI script
-        run: ./ci/test_run_all.sh
-
-      - name: Save Ccache cache
-        uses: actions/cache/save@v3
-        if: github.event_name != 'pull_request' && steps.ccache-cache.outputs.cache-hit != 'true'
-        with:
-          path: ${{ env.CCACHE_DIR }}
-          # https://github.com/actions/cache/blob/main/tips-and-workarounds.md#update-a-cache
-          key: ${{ github.job }}-ccache-${{ github.run_id }}
-
-  win64-native:
-    name: 'Win64 native, VS 2022'
-    # Use latest image, but hardcode version to avoid silent upgrades (and breaks).
-    # See: https://github.com/actions/runner-images#available-images.
-    runs-on: windows-2022
-
-    # No need to run on the read-only mirror, unless it is a PR.
-    if: github.repository != 'bitcoin-core/gui' || github.event_name == 'pull_request'
-
-    env:
-      CCACHE_MAXSIZE: '200M'
-      CI_CCACHE_VERSION: '4.7.5'
-      CI_QT_CONF: '-release -silent -opensource -confirm-license -opengl desktop -static -static-runtime -mp -qt-zlib -qt-pcre -qt-libpng -nomake examples -nomake tests -nomake tools -no-angle -no-dbus -no-gif -no-gtk -no-ico -no-icu -no-libjpeg -no-libudev -no-sql-sqlite -no-sql-odbc -no-sqlite -no-vulkan -skip qt3d -skip qtactiveqt -skip qtandroidextras -skip qtcharts -skip qtconnectivity -skip qtdatavis3d -skip qtdeclarative -skip doc -skip qtdoc -skip qtgamepad -skip qtgraphicaleffects -skip qtimageformats -skip qtlocation -skip qtlottie -skip qtmacextras -skip qtmultimedia -skip qtnetworkauth -skip qtpurchasing -skip qtquick3d -skip qtquickcontrols -skip qtquickcontrols2 -skip qtquicktimeline -skip qtremoteobjects -skip qtscript -skip qtscxml -skip qtsensors -skip qtserialbus -skip qtserialport -skip qtspeech -skip qtsvg -skip qtvirtualkeyboard -skip qtwayland -skip qtwebchannel -skip qtwebengine -skip qtwebglplugin -skip qtwebsockets -skip qtwebview -skip qtx11extras -skip qtxmlpatterns -no-openssl -no-feature-bearermanagement -no-feature-printdialog -no-feature-printer -no-feature-printpreviewdialog -no-feature-printpreviewwidget -no-feature-sql -no-feature-sqlmodel -no-feature-textbrowser -no-feature-textmarkdownwriter -no-feature-textodfwriter -no-feature-xml'
-      CI_QT_DIR: 'qt-everywhere-src-5.15.10'
-      CI_QT_URL: 'https://download.qt.io/official_releases/qt/5.15/5.15.10/single/qt-everywhere-opensource-src-5.15.10.zip'
-      PYTHONUTF8: 1
-      TEST_RUNNER_TIMEOUT_FACTOR: 40
-
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-
-      - name: Fix Visual Studio installation
-        # Avoid toolset ambiguity that MSVC can't handle.
-        run: |
-          Set-Location "C:\Program Files (x86)\Microsoft Visual Studio\Installer\"
-          $InstallPath = "C:\Program Files\Microsoft Visual Studio\2022\Enterprise"
-          $componentsToRemove= @(
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.ARM.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.ARM"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.ARM64.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.ARM64"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.ATL.ARM.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.ATL.ARM"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.ATL.ARM64.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.ATL.ARM64"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.ATL.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.ATL"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.MFC.ARM.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.MFC.ARM"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.MFC.ARM64.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.MFC.ARM64"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.MFC.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.MFC"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.x86.x64.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.37.17.7.x86.x64"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.ARM"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.ARM.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.ARM64"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.ARM64.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.ATL"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.ATL.ARM"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.ATL.ARM.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.ATL.ARM64"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.ATL.ARM64.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.ATL.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.MFC"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.MFC.ARM"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.MFC.ARM.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.MFC.ARM64"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.MFC.ARM64.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.MFC.Spectre"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.x86.x64"
-            "Microsoft.VisualStudio.Component.VC.14.38.17.8.x86.x64.Spectre"
-          )
-          [string]$workloadArgs = $componentsToRemove | ForEach-Object {" --remove " +  $_}
-          $Arguments = ('/c', "vs_installer.exe", 'modify', '--installPath', "`"$InstallPath`"",$workloadArgs, '--quiet', '--norestart', '--nocache')
-          # should be run twice
-          $process = Start-Process -FilePath cmd.exe -ArgumentList $Arguments -Wait -PassThru -WindowStyle Hidden
-          $process = Start-Process -FilePath cmd.exe -ArgumentList $Arguments -Wait -PassThru -WindowStyle Hidden
-
-      - name: Configure Developer Command Prompt for Microsoft Visual C++
-        # Using microsoft/setup-msbuild is not enough.
-        uses: ilammy/msvc-dev-cmd@v1
-        with:
-          arch: x64
-
-      - name: Check MSBuild and Qt
-        run: |
-          msbuild -version | Out-File -FilePath "$env:GITHUB_WORKSPACE\msbuild_version"
-          Get-Content -Path "$env:GITHUB_WORKSPACE\msbuild_version"
-          $env:CI_QT_URL | Out-File -FilePath "$env:GITHUB_WORKSPACE\qt_url"
-          $env:CI_QT_CONF | Out-File -FilePath "$env:GITHUB_WORKSPACE\qt_conf"
-
-      - name: Restore static Qt cache
-        id: static-qt-cache
-        uses: actions/cache/restore@v3
-        with:
-          path: C:\Qt_static
-          key: ${{ github.job }}-static-qt-${{ hashFiles('msbuild_version', 'qt_url', 'qt_conf') }}
-
-      - name: Build static Qt. Download
-        if: steps.static-qt-cache.outputs.cache-hit != 'true'
-        shell: cmd
-        run: |
-          curl --location --output C:\qt-src.zip %CI_QT_URL%
-          choco install --yes --no-progress jom
-
-      - name: Build static Qt. Expand source archive
-        if: steps.static-qt-cache.outputs.cache-hit != 'true'
-        shell: cmd
-        run: tar -xf C:\qt-src.zip -C C:\
-
-      - name: Build static Qt. Create build directory
-        if: steps.static-qt-cache.outputs.cache-hit != 'true'
-        run: |
-          Rename-Item -Path "C:\$env:CI_QT_DIR" -NewName "C:\qt-src"
-          New-Item -ItemType Directory -Path "C:\qt-src\build"
-
-      - name: Build static Qt. Configure
-        if: steps.static-qt-cache.outputs.cache-hit != 'true'
-        working-directory: C:\qt-src\build
-        shell: cmd
-        run: ..\configure %CI_QT_CONF% -prefix C:\Qt_static
-
-      - name: Build static Qt. Build
-        if: steps.static-qt-cache.outputs.cache-hit != 'true'
-        working-directory: C:\qt-src\build
-        shell: cmd
-        run: jom
-
-      - name: Build static Qt. Install
-        if: steps.static-qt-cache.outputs.cache-hit != 'true'
-        working-directory: C:\qt-src\build
-        shell: cmd
-        run: jom install
-
-      - name: Save static Qt cache
-        if: steps.static-qt-cache.outputs.cache-hit != 'true'
-        uses: actions/cache/save@v3
-        with:
-          path: C:\Qt_static
-          key: ${{ github.job }}-static-qt-${{ hashFiles('msbuild_version', 'qt_url', 'qt_conf') }}
-
-      - name: Ccache installation cache
-        id: ccache-installation-cache
-        uses: actions/cache@v3
-        with:
-          path: |
-            C:\ProgramData\chocolatey\lib\ccache
-            C:\ProgramData\chocolatey\bin\ccache.exe
-            C:\ccache\cl.exe
-          key: ${{ github.job }}-ccache-installation-${{ env.CI_CCACHE_VERSION }}
-
-      - name: Install Ccache
-        if: steps.ccache-installation-cache.outputs.cache-hit != 'true'
-        run: |
-          choco install --yes --no-progress ccache --version=$env:CI_CCACHE_VERSION
-          New-Item -ItemType Directory -Path "C:\ccache"
-          Copy-Item -Path "$env:ChocolateyInstall\lib\ccache\tools\ccache-$env:CI_CCACHE_VERSION-windows-x86_64\ccache.exe" -Destination "C:\ccache\cl.exe"
-
-      - name: Restore Ccache cache
-        id: ccache-cache
-        uses: actions/cache/restore@v3
-        with:
-          path: ~/AppData/Local/ccache
-          key: ${{ github.job }}-ccache-${{ github.run_id }}
-          restore-keys: ${{ github.job }}-ccache-
-
-      - name: Using vcpkg with MSBuild
-        run: |
-          Set-Location "$env:VCPKG_INSTALLATION_ROOT"
-          Add-Content -Path "triplets\x64-windows-static.cmake" -Value "set(VCPKG_BUILD_TYPE release)"
-          vcpkg --vcpkg-root "$env:VCPKG_INSTALLATION_ROOT" integrate install
-          git rev-parse HEAD | Out-File -FilePath "$env:GITHUB_WORKSPACE\vcpkg_commit"
-          Get-Content -Path "$env:GITHUB_WORKSPACE\vcpkg_commit"
-
-      - name: vcpkg tools cache
-        uses: actions/cache@v3
-        with:
-          path: C:/vcpkg/downloads/tools
-          key: ${{ github.job }}-vcpkg-tools
-
-      - name: vcpkg binary cache
-        uses: actions/cache@v3
-        with:
-          path: ~/AppData/Local/vcpkg/archives
-          key: ${{ github.job }}-vcpkg-binary-${{ hashFiles('vcpkg_commit', 'msbuild_version', 'build_msvc/vcpkg.json') }}
-
-      - name: Generate project files
-        run: py -3 build_msvc\msvc-autogen.py
-
-      - name: Build
-        shell: cmd
-        run: |
-          ccache --zero-stats
-          msbuild build_msvc\bitcoin.sln -property:CLToolPath=C:\ccache;CLToolExe=cl.exe;UseMultiToolTask=true;Configuration=Release -maxCpuCount -verbosity:minimal -noLogo
-
-      - name: Ccache stats
-        run: ccache --show-stats
-
-      - name: Save Ccache cache
-        uses: actions/cache/save@v3
-        if: github.event_name != 'pull_request' && steps.ccache-cache.outputs.cache-hit != 'true'
-        with:
-          path: ~/AppData/Local/ccache
-          # https://github.com/actions/cache/blob/main/tips-and-workarounds.md#update-a-cache
-          key: ${{ github.job }}-ccache-${{ github.run_id }}
-
-      - name: Run unit tests
-        run: src\test_bitcoin.exe -l test_suite
-
-      - name: Run benchmarks
-        run: src\bench_bitcoin.exe -sanity-check
-
-      - name: Run util tests
-        run: py -3 test\util\test_runner.py
-
-      - name: Run rpcauth test
-        run: py -3 test\util\rpcauth-test.py
-
-      - name: Run functional tests
-        # Don't run functional tests for pull requests.
-        # The test suit regularly fails to complete in windows native github
-        # actions as a child process stops making progress. The root cause has
-        # not yet been determined.
-        # Discussed in https://github.com/bitcoin/bitcoin/pull/28509
-        if: github.event_name != 'pull_request'
-        run: py -3 test\functional\test_runner.py --jobs $env:NUMBER_OF_PROCESSORS --ci --quiet --tmpdirprefix=$env:RUNNER_TEMP --combinedlogslen=99999999 --timeout-factor=$env:TEST_RUNNER_TIMEOUT_FACTOR --extended

.gitignore

@@ -74,7 +74,7 @@ src/qt/bitcoin-qt.includes
 
 *.log
 *.trs
-*.zip
+*.dmg
 
 *.json.h
 *.raw.h

.python-version

@@ -1 +1 @@
-3.8.17
+3.7.16

.style.yapf

@@ -107,7 +107,7 @@ each_dict_entry_on_separate_line=True
 i18n_comment=
 
 # The i18n function call names. The presence of this function stops
-# reformatting on that line, because the string it has cannot be moved
+# reformattting on that line, because the string it has cannot be moved
 # away from the i18n comment.
 i18n_function_call=
 

.tx/config

@@ -1,7 +1,7 @@
 [main]
 host = https://www.transifex.com
 
-[o:bitcoin:p:bitcoin:r:qt-translation-026x]
+[o:bitcoin:p:bitcoin:r:qt-translation-025x]
 file_filter = src/qt/locale/bitcoin_<lang>.xlf
 source_file = src/qt/locale/bitcoin_en.xlf
 source_lang = en

Makefile.am

@@ -37,7 +37,7 @@ space := $(empty) $(empty)
 
 OSX_APP=Bitcoin-Qt.app
 OSX_VOLNAME = $(subst $(space),-,$(PACKAGE_NAME))
-OSX_ZIP = $(OSX_VOLNAME).zip
+OSX_DMG = $(OSX_VOLNAME).dmg
 OSX_DEPLOY_SCRIPT=$(top_srcdir)/contrib/macdeploy/macdeployqtplus
 OSX_INSTALLER_ICONS=$(top_srcdir)/src/qt/res/icons/bitcoin.icns
 OSX_PLIST=$(top_builddir)/share/qt/Info.plist #not installed
@@ -124,16 +124,15 @@ osx_volname:
 	echo $(OSX_VOLNAME) >$@
 
 if BUILD_DARWIN
-$(OSX_ZIP): $(OSX_APP_BUILT) $(OSX_PACKAGING)
-	$(PYTHON) $(OSX_DEPLOY_SCRIPT) $(OSX_APP) $(OSX_VOLNAME) -translations-dir=$(QT_TRANSLATION_DIR) -zip
+$(OSX_DMG): $(OSX_APP_BUILT) $(OSX_PACKAGING)
+	$(PYTHON) $(OSX_DEPLOY_SCRIPT) $(OSX_APP) $(OSX_VOLNAME) -translations-dir=$(QT_TRANSLATION_DIR) -dmg
 
-deploydir: $(OSX_ZIP)
+deploydir: $(OSX_DMG)
 else !BUILD_DARWIN
 APP_DIST_DIR=$(top_builddir)/dist
 
-$(OSX_ZIP): deploydir
-	if [ -n "$(SOURCE_DATE_EPOCH)" ]; then find $(APP_DIST_DIR) -exec touch -d @$(SOURCE_DATE_EPOCH) {} +; fi
-	cd $(APP_DIST_DIR) && find . | sort | $(ZIP) -X@ $@
+$(OSX_DMG): deploydir
+	$(XORRISOFS) -D -l -V "$(OSX_VOLNAME)" -no-pad -r -dir-mode 0755 -o $@ $(APP_DIST_DIR) -- $(if $(SOURCE_DATE_EPOCH),-volume_date all_file_dates =$(SOURCE_DATE_EPOCH))
 
 $(APP_DIST_DIR)/$(OSX_APP)/Contents/MacOS/Bitcoin-Qt: $(OSX_APP_BUILT) $(OSX_PACKAGING)
 	INSTALL_NAME_TOOL=$(INSTALL_NAME_TOOL) OTOOL=$(OTOOL) STRIP=$(STRIP) $(PYTHON) $(OSX_DEPLOY_SCRIPT) $(OSX_APP) $(OSX_VOLNAME) -translations-dir=$(QT_TRANSLATION_DIR)
@@ -141,7 +140,7 @@ $(APP_DIST_DIR)/$(OSX_APP)/Contents/MacOS/Bitcoin-Qt: $(OSX_APP_BUILT) $(OSX_PAC
 deploydir: $(APP_DIST_DIR)/$(OSX_APP)/Contents/MacOS/Bitcoin-Qt
 endif !BUILD_DARWIN
 
-deploy: $(OSX_ZIP)
+deploy: $(OSX_DMG)
 endif
 
 $(BITCOIN_QT_BIN): FORCE
@@ -314,7 +313,7 @@ EXTRA_DIST += \
     test/util/data/txcreatesignv2.hex \
     test/util/rpcauth-test.py
 
-CLEANFILES = $(OSX_ZIP) $(BITCOIN_WIN_INSTALLER)
+CLEANFILES = $(OSX_DMG) $(BITCOIN_WIN_INSTALLER)
 
 DISTCHECK_CONFIGURE_FLAGS = --enable-man
 

autogen.sh

@@ -14,16 +14,3 @@ fi
 command -v autoreconf >/dev/null || \
   (echo "configuration failed, please install autoconf first" && exit 1)
 autoreconf --install --force --warnings=all
-
-if expr "'$(build-aux/config.guess --timestamp)" \< "'$(depends/config.guess --timestamp)" > /dev/null; then
-  chmod ug+w build-aux/config.guess
-  chmod ug+w src/secp256k1/build-aux/config.guess
-  cp depends/config.guess build-aux
-  cp depends/config.guess src/secp256k1/build-aux
-fi
-if expr "'$(build-aux/config.sub --timestamp)" \< "'$(depends/config.sub --timestamp)" > /dev/null; then
-  chmod ug+w build-aux/config.sub
-  chmod ug+w src/secp256k1/build-aux/config.sub
-  cp depends/config.sub build-aux
-  cp depends/config.sub src/secp256k1/build-aux
-fi

build-aux/m4/l_filesystem.m4

@@ -0,0 +1,47 @@
+dnl Copyright (c) 2022 The Bitcoin Core developers
+dnl Distributed under the MIT software license, see the accompanying
+dnl file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+# GCC 8.1 and earlier requires -lstdc++fs
+# Clang 8.0.0 (libc++) and earlier requires -lc++fs
+
+m4_define([_CHECK_FILESYSTEM_testbody], [[
+  #include <filesystem>
+
+  namespace fs = std::filesystem;
+
+  int main() {
+    (void)fs::current_path().root_name();
+    return 0;
+  }
+]])
+
+AC_DEFUN([CHECK_FILESYSTEM], [
+
+  AC_LANG_PUSH(C++)
+
+  AC_MSG_CHECKING([whether std::filesystem can be used without link library])
+
+  AC_LINK_IFELSE([AC_LANG_SOURCE([_CHECK_FILESYSTEM_testbody])],[
+      AC_MSG_RESULT([yes])
+    ],[
+      AC_MSG_RESULT([no])
+      SAVED_LIBS="$LIBS"
+      LIBS="$SAVED_LIBS -lstdc++fs"
+      AC_MSG_CHECKING([whether std::filesystem needs -lstdc++fs])
+      AC_LINK_IFELSE([AC_LANG_SOURCE([_CHECK_FILESYSTEM_testbody])],[
+          AC_MSG_RESULT([yes])
+        ],[
+          AC_MSG_RESULT([no])
+          AC_MSG_CHECKING([whether std::filesystem needs -lc++fs])
+          LIBS="$SAVED_LIBS -lc++fs"
+          AC_LINK_IFELSE([AC_LANG_SOURCE([_CHECK_FILESYSTEM_testbody])],[
+            AC_MSG_RESULT([yes])
+        ],[
+            AC_MSG_FAILURE([cannot figure out how to use std::filesystem])
+          ])
+        ])
+    ])
+
+  AC_LANG_POP
+])

build_msvc/README.md

@@ -32,7 +32,7 @@ Qt
 ---------------------
 To build Bitcoin Core with the GUI, a static build of Qt is required.
 
-1. Download a single ZIP archive of Qt source code from https://download.qt.io/official_releases/qt/ (e.g., [`qt-everywhere-opensource-src-5.15.10.zip`](https://download.qt.io/official_releases/qt/5.15/5.15.10/single/qt-everywhere-opensource-src-5.15.10.zip)), and expand it into a dedicated folder. The following instructions assume that this folder is `C:\dev\qt-source`.
+1. Download a single ZIP archive of Qt source code from https://download.qt.io/official_releases/qt/ (e.g., [`qt-everywhere-opensource-src-5.15.5.zip`](https://download.qt.io/official_releases/qt/5.15/5.15.5/single/qt-everywhere-opensource-src-5.15.5.zip)), and expand it into a dedicated folder. The following instructions assume that this folder is `C:\dev\qt-source`.
 
 2. Open "x64 Native Tools Command Prompt for VS 2022", and input the following commands:
 ```cmd

build_msvc/bench_bitcoin/bench_bitcoin.vcxproj.in

@@ -12,7 +12,7 @@
 @SOURCE_FILES@
   </ItemGroup>
   <ItemGroup>
-    <ProjectReference Include="..\libbitcoin_consensus\libbitcoin_consensus.vcxproj">
+    <ProjectReference Include="..\libbitcoinconsensus\libbitcoinconsensus.vcxproj">
       <Project>{2b384fa8-9ee1-4544-93cb-0d733c25e8ce}</Project>
     </ProjectReference>
     <ProjectReference Include="..\libbitcoin_common\libbitcoin_common.vcxproj">

build_msvc/bitcoin-qt/bitcoin-qt.vcxproj

@@ -13,7 +13,7 @@
     <ResourceCompile Include="..\..\src\qt\res\bitcoin-qt-res.rc" />
   </ItemGroup>
   <ItemGroup>
-    <ProjectReference Include="..\libbitcoin_consensus\libbitcoin_consensus.vcxproj">
+    <ProjectReference Include="..\libbitcoinconsensus\libbitcoinconsensus.vcxproj">
       <Project>{2b384fa8-9ee1-4544-93cb-0d733c25e8ce}</Project>
     </ProjectReference>
     <ProjectReference Include="..\libbitcoin_cli\libbitcoin_cli.vcxproj">

build_msvc/bitcoin-tx/bitcoin-tx.vcxproj

@@ -12,7 +12,7 @@
     <ClCompile Include="..\..\src\bitcoin-tx.cpp" />
   </ItemGroup>
   <ItemGroup>
-    <ProjectReference Include="..\libbitcoin_consensus\libbitcoin_consensus.vcxproj">
+    <ProjectReference Include="..\libbitcoinconsensus\libbitcoinconsensus.vcxproj">
       <Project>{2b384fa8-9ee1-4544-93cb-0d733c25e8ce}</Project>
     </ProjectReference>
     <ProjectReference Include="..\libbitcoin_common\libbitcoin_common.vcxproj">

build_msvc/bitcoin-util/bitcoin-util.vcxproj

@@ -12,7 +12,7 @@
     <ClCompile Include="..\..\src\bitcoin-util.cpp" />
   </ItemGroup>
   <ItemGroup>
-    <ProjectReference Include="..\libbitcoin_consensus\libbitcoin_consensus.vcxproj">
+    <ProjectReference Include="..\libbitcoinconsensus\libbitcoinconsensus.vcxproj">
       <Project>{2b384fa8-9ee1-4544-93cb-0d733c25e8ce}</Project>
     </ProjectReference>
     <ProjectReference Include="..\libbitcoin_common\libbitcoin_common.vcxproj">

build_msvc/bitcoin-wallet/bitcoin-wallet.vcxproj

@@ -15,7 +15,7 @@
     </ClCompile>
   </ItemGroup>
   <ItemGroup>
-    <ProjectReference Include="..\libbitcoin_consensus\libbitcoin_consensus.vcxproj">
+    <ProjectReference Include="..\libbitcoinconsensus\libbitcoinconsensus.vcxproj">
       <Project>{2b384fa8-9ee1-4544-93cb-0d733c25e8ce}</Project>
     </ProjectReference>
     <ProjectReference Include="..\libbitcoin_common\libbitcoin_common.vcxproj">

build_msvc/bitcoin.sln

@@ -2,7 +2,7 @@ Microsoft Visual Studio Solution File, Format Version 12.00
 # Visual Studio Version 16
 VisualStudioVersion = 16.0.28803.452
 MinimumVisualStudioVersion = 10.0.40219.1
-Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "libbitcoin_consensus", "libbitcoin_consensus\libbitcoin_consensus.vcxproj", "{2B384FA8-9EE1-4544-93CB-0D733C25E8CE}"
+Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "libbitcoinconsensus", "libbitcoinconsensus\libbitcoinconsensus.vcxproj", "{2B384FA8-9EE1-4544-93CB-0D733C25E8CE}"
 EndProject
 Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "bitcoind", "bitcoind\bitcoind.vcxproj", "{D4513DDF-6013-44DC-ADCC-12EAF6D1F038}"
 EndProject

build_msvc/bitcoin_config.h.in

@@ -38,12 +38,15 @@
 /* Define to 1 to enable SQLite wallet */
 #define USE_SQLITE 1
 
-/* Define this symbol to enable ZMQ functions */
+/* Define to 1 to enable ZMQ functions */
 #define ENABLE_ZMQ 1
 
 /* define if external signer support is enabled (requires Boost::Process) */
 #define ENABLE_EXTERNAL_SIGNER /**/
 
+/* Define this symbol if the consensus lib has been built */
+#define HAVE_CONSENSUS_LIB 1
+
 /* Define to 1 if you have the declaration of `be16toh', and to 0 if you
    don't. */
 #define HAVE_DECL_BE16TOH 0

build_msvc/bitcoind/bitcoind.vcxproj

@@ -15,7 +15,7 @@
     </ClCompile>
   </ItemGroup>
   <ItemGroup>
-    <ProjectReference Include="..\libbitcoin_consensus\libbitcoin_consensus.vcxproj">
+    <ProjectReference Include="..\libbitcoinconsensus\libbitcoinconsensus.vcxproj">
       <Project>{2b384fa8-9ee1-4544-93cb-0d733c25e8ce}</Project>
     </ProjectReference>
     <ProjectReference Include="..\libbitcoin_common\libbitcoin_common.vcxproj">

build_msvc/common.init.vcxproj.in

@@ -90,7 +90,7 @@
       <AdditionalOptions>/utf-8 /Zc:__cplusplus /std:c++20 %(AdditionalOptions)</AdditionalOptions>
       <DisableSpecificWarnings>4018;4244;4267;4715;4805</DisableSpecificWarnings>
       <TreatWarningAsError>true</TreatWarningAsError>
-      <PreprocessorDefinitions>_SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING;SECP256K1_STATIC;ZMQ_STATIC;NOMINMAX;WIN32;HAVE_CONFIG_H;_CRT_SECURE_NO_WARNINGS;_SCL_SECURE_NO_WARNINGS;_CONSOLE;_WIN32_WINNT=0x0601;_WIN32_IE=0x0501;WIN32_LEAN_AND_MEAN;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <PreprocessorDefinitions>_SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING;ZMQ_STATIC;NOMINMAX;WIN32;HAVE_CONFIG_H;_CRT_SECURE_NO_WARNINGS;_SCL_SECURE_NO_WARNINGS;_CONSOLE;_WIN32_WINNT=0x0601;_WIN32_IE=0x0501;WIN32_LEAN_AND_MEAN;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <AdditionalIncludeDirectories>..\..\src;..\..\src\minisketch\include;..\..\src\univalue\include;..\..\src\secp256k1\include;..\..\src\leveldb\include;..\..\src\leveldb\helpers\memenv;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
     </ClCompile>
     <Link>

build_msvc/libbitcoinconsensus/libbitcoinconsensus.vcxproj

@@ -11,6 +11,15 @@
     <ClCompile Include="..\..\src\arith_uint256.cpp" />
     <ClCompile Include="..\..\src\consensus\merkle.cpp" />
     <ClCompile Include="..\..\src\consensus\tx_check.cpp" />
+    <ClCompile Include="..\..\src\crypto\aes.cpp" />
+    <ClCompile Include="..\..\src\crypto\chacha20.cpp" />
+    <ClCompile Include="..\..\src\crypto\hmac_sha256.cpp" />
+    <ClCompile Include="..\..\src\crypto\hmac_sha512.cpp" />
+    <ClCompile Include="..\..\src\crypto\ripemd160.cpp" />
+    <ClCompile Include="..\..\src\crypto\sha1.cpp" />
+    <ClCompile Include="..\..\src\crypto\sha256.cpp" />
+    <ClCompile Include="..\..\src\crypto\sha256_sse4.cpp" />
+    <ClCompile Include="..\..\src\crypto\sha512.cpp" />
     <ClCompile Include="..\..\src\hash.cpp" />
     <ClCompile Include="..\..\src\primitives\block.cpp" />
     <ClCompile Include="..\..\src\primitives\transaction.cpp" />

build_msvc/libsecp256k1/libsecp256k1.vcxproj

@@ -14,8 +14,7 @@
   </ItemGroup>
   <ItemDefinitionGroup>
     <ClCompile>
-      <PreprocessorDefinitions>ENABLE_MODULE_RECOVERY;ENABLE_MODULE_EXTRAKEYS;ENABLE_MODULE_SCHNORRSIG;ENABLE_MODULE_ELLSWIFT;%(PreprocessorDefinitions)</PreprocessorDefinitions>
-      <UndefinePreprocessorDefinitions>USE_ASM_X86_64;%(UndefinePreprocessorDefinitions)</UndefinePreprocessorDefinitions>
+      <PreprocessorDefinitions>ENABLE_MODULE_RECOVERY;ENABLE_MODULE_EXTRAKEYS;ENABLE_MODULE_SCHNORRSIG;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <AdditionalIncludeDirectories>..\..\src\secp256k1;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
       <DisableSpecificWarnings>4146;4244;4267;4334</DisableSpecificWarnings>
     </ClCompile>

build_msvc/libsecp256k1_config.h

@@ -0,0 +1,15 @@
+/**********************************************************************
+ * Copyright (c) 2013, 2014 Pieter Wuille                             *
+ * Distributed under the MIT software license, see the accompanying   *
+ * file COPYING or http://www.opensource.org/licenses/mit-license.php.*
+ **********************************************************************/
+
+#ifndef BITCOIN_LIBSECP256K1_CONFIG_H
+#define BITCOIN_LIBSECP256K1_CONFIG_H
+
+#undef USE_ASM_X86_64
+
+#define ECMULT_GEN_PREC_BITS 4
+#define ECMULT_WINDOW_SIZE 15
+
+#endif // BITCOIN_LIBSECP256K1_CONFIG_H

build_msvc/libtest_util/libtest_util.vcxproj.in

@@ -8,7 +8,6 @@
     <ConfigurationType>StaticLibrary</ConfigurationType>
   </PropertyGroup>
   <ItemGroup>
-    <ClCompile Include="..\..\src\wallet\test\util.cpp" />
 @SOURCE_FILES@
   </ItemGroup>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />

build_msvc/msvc-autogen.py

@@ -111,6 +111,7 @@ def main():
         set_properties(vcxproj_filename, '@SOURCE_FILES@\n', content)
     parse_config_into_btc_config()
     copyfile(os.path.join(SOURCE_DIR,'../build_msvc/bitcoin_config.h'), os.path.join(SOURCE_DIR, 'config/bitcoin-config.h'))
+    copyfile(os.path.join(SOURCE_DIR,'../build_msvc/libsecp256k1_config.h'), os.path.join(SOURCE_DIR, 'secp256k1/src/libsecp256k1-config.h'))
 
 if __name__ == '__main__':
     main()

build_msvc/test_bitcoin-qt/test_bitcoin-qt.vcxproj

@@ -10,18 +10,13 @@
   <ItemGroup>
     <ClCompile Include="..\..\src\init\bitcoin-qt.cpp" />
     <ClCompile Include="..\..\src\test\util\setup_common.cpp" />
-    <ClCompile Include="..\..\src\wallet\test\util.cpp">
-      <ObjectFileName>$(IntDir)wallet_test_util.obj</ObjectFileName>
-    </ClCompile>
     <ClCompile Include="..\..\src\qt\test\addressbooktests.cpp" />
     <ClCompile Include="..\..\src\qt\test\apptests.cpp" />
     <ClCompile Include="..\..\src\qt\test\optiontests.cpp" />
     <ClCompile Include="..\..\src\qt\test\rpcnestedtests.cpp" />
     <ClCompile Include="..\..\src\qt\test\test_main.cpp" />
     <ClCompile Include="..\..\src\qt\test\uritests.cpp" />
-    <ClCompile Include="..\..\src\qt\test\util.cpp">
-      <ObjectFileName>$(IntDir)qt_test_util.obj</ObjectFileName>
-    </ClCompile>
+    <ClCompile Include="..\..\src\qt\test\util.cpp" />
     <ClCompile Include="..\..\src\qt\test\wallettests.cpp" />
     <ClCompile Include="..\..\src\wallet\test\wallet_test_fixture.cpp" />
     <ClCompile Include="$(GeneratedFilesOutDir)\moc\moc_addressbooktests.cpp" />
@@ -32,7 +27,7 @@
     <ClCompile Include="$(GeneratedFilesOutDir)\moc\moc_wallettests.cpp" />
   </ItemGroup>
   <ItemGroup>
-    <ProjectReference Include="..\libbitcoin_consensus\libbitcoin_consensus.vcxproj">
+    <ProjectReference Include="..\libbitcoinconsensus\libbitcoinconsensus.vcxproj">
       <Project>{2b384fa8-9ee1-4544-93cb-0d733c25e8ce}</Project>
     </ProjectReference>
     <ProjectReference Include="..\libbitcoin_cli\libbitcoin_cli.vcxproj">

build_msvc/test_bitcoin/test_bitcoin.vcxproj

@@ -22,7 +22,7 @@
     <ProjectReference Include="..\libminisketch\libminisketch.vcxproj">
       <Project>{542007e3-be0d-4b0d-a6b0-aa8813e2558d}</Project>
     </ProjectReference>
-    <ProjectReference Include="..\libbitcoin_consensus\libbitcoin_consensus.vcxproj">
+    <ProjectReference Include="..\libbitcoinconsensus\libbitcoinconsensus.vcxproj">
       <Project>{2b384fa8-9ee1-4544-93cb-0d733c25e8ce}</Project>
     </ProjectReference>
     <ProjectReference Include="..\libbitcoin_cli\libbitcoin_cli.vcxproj">
@@ -68,7 +68,7 @@
       <RawTestFile Include="..\..\src\test\data\*.raw" />
     </ItemGroup>
     <HeaderFromHexdump RawFilePath="%(RawTestFile.FullPath)" HeaderFilePath="%(RawTestFile.FullPath).h" SourceHeader="static unsigned const char %(RawTestFile.Filename)_raw[] = {" SourceFooter="};" />
-    <HeaderFromHexdump RawFilePath="%(JsonTestFile.FullPath)" HeaderFilePath="%(JsonTestFile.FullPath).h" SourceHeader="#include &lt;string&gt;&#x0D;&#x0A;namespace json_tests{ static const std::string %(JsonTestFile.Filename){" SourceFooter="};}" />
+    <HeaderFromHexdump RawFilePath="%(JsonTestFile.FullPath)" HeaderFilePath="%(JsonTestFile.FullPath).h" SourceHeader="namespace json_tests{ static unsigned const char %(JsonTestFile.Filename)[] = {" SourceFooter="};}" />
   </Target>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />

ci/README.md

@@ -14,17 +14,22 @@ testing compared to other parts of the codebase. If you want to keep the work tr
 system in a virtual machine with a Linux operating system of your choice.
 
 To allow for a wide range of tested environments, but also ensure reproducibility to some extent, the test stage
-requires `bash`, `docker`, and `python3` to be installed. To install all requirements on Ubuntu, run
+requires `docker` to be installed. To install all requirements on Ubuntu, run
 
 ```
-sudo apt install bash docker.io python3
+sudo apt install docker.io bash
 ```
 
-It is recommended to run the ci system in a clean env. To run the test stage
-with a specific configuration,
+To run the default test stage,
 
 ```
-env -i HOME="$HOME" PATH="$PATH" USER="$USER" bash -c 'FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh'
+./ci/test_run_all.sh
+```
+
+To run the test stage with a specific configuration,
+
+```
+FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh
 ```
 
 ### Configurations
@@ -39,11 +44,14 @@ the system package manager to install build dependencies. This guarantees that
 the tester is using the same versions as the release builds, which also use
 `./depends`.
 
+If no `FILE_ENV` has been specified or values are left out, `00_setup_env.sh`
+is used as the default configuration with fallback values.
+
 It is also possible to force a specific configuration without modifying the
 file. For example,
 
 ```
-env -i HOME="$HOME" PATH="$PATH" USER="$USER" bash -c 'MAKEJOBS="-j1" FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh'
+MAKEJOBS="-j1" FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh
 ```
 
 The files starting with `0n` (`n` greater than 0) are the scripts that are run

ci/lint/04_install.sh

@@ -6,8 +6,6 @@
 
 export LC_ALL=C
 
-export PATH=$PWD/ci/retry:$PATH
-
 ${CI_RETRY_EXE} apt-get update
 # Lint dependencies:
 # - curl/xz-utils (to install shellcheck)
@@ -15,31 +13,31 @@ ${CI_RETRY_EXE} apt-get update
 # - gpg (used by verify-commits)
 ${CI_RETRY_EXE} apt-get install -y curl xz-utils git gpg
 
-PYTHON_PATH="/python_build"
-if [ ! -d "${PYTHON_PATH}/bin" ]; then
-  (
-    ${CI_RETRY_EXE} git clone https://github.com/pyenv/pyenv.git
-    cd pyenv/plugins/python-build || exit 1
-    ./install.sh
-  )
-  # For dependencies see https://github.com/pyenv/pyenv/wiki#suggested-build-environment
-  ${CI_RETRY_EXE} apt-get install -y build-essential libssl-dev zlib1g-dev \
-    libbz2-dev libreadline-dev libsqlite3-dev curl llvm \
-    libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev \
-    clang
-  env CC=clang python-build "$(cat "./.python-version")" "${PYTHON_PATH}"
+if [ -z "${SKIP_PYTHON_INSTALL}" ]; then
+    PYTHON_PATH=/tmp/python
+    if [ ! -d "${PYTHON_PATH}/bin" ]; then
+      (
+        git clone https://github.com/pyenv/pyenv.git
+        cd pyenv/plugins/python-build || exit 1
+        ./install.sh
+      )
+      # For dependencies see https://github.com/pyenv/pyenv/wiki#suggested-build-environment
+      ${CI_RETRY_EXE} apt-get install -y build-essential libssl-dev zlib1g-dev \
+        libbz2-dev libreadline-dev libsqlite3-dev curl llvm \
+        libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev \
+        clang
+      env CC=clang python-build "$(cat "${BASE_ROOT_DIR}/.python-version")" "${PYTHON_PATH}"
+    fi
+    export PATH="${PYTHON_PATH}/bin:${PATH}"
+    command -v python3
+    python3 --version
 fi
-export PATH="${PYTHON_PATH}/bin:${PATH}"
-command -v python3
-python3 --version
 
-${CI_RETRY_EXE} pip3 install \
-  codespell==2.2.5 \
-  flake8==6.1.0 \
-  lief==0.13.2 \
-  mypy==1.4.1 \
-  pyzmq==25.1.0 \
-  vulture==2.6
+${CI_RETRY_EXE} pip3 install codespell==2.2.1
+${CI_RETRY_EXE} pip3 install flake8==5.0.4
+${CI_RETRY_EXE} pip3 install mypy==0.971
+${CI_RETRY_EXE} pip3 install pyzmq==24.0.1
+${CI_RETRY_EXE} pip3 install vulture==2.6
 
 SHELLCHECK_VERSION=v0.8.0
 curl -sL "https://github.com/koalaman/shellcheck/releases/download/${SHELLCHECK_VERSION}/shellcheck-${SHELLCHECK_VERSION}.linux.x86_64.tar.xz" | \

ci/lint/06_script.sh

@@ -6,8 +6,6 @@
 
 export LC_ALL=C
 
-set -ex
-
 if [ -n "$LOCAL_BRANCH" ]; then
   # To faithfully recreate CI linting locally, specify all commits on the current
   # branch.

ci/lint/Dockerfile

@@ -0,0 +1,29 @@
+# See test/lint/README.md for usage.
+#
+# This container basically has to live in this directory in order to pull in the CI
+# install scripts. If it lived in the root directory, it would have to pull in the
+# entire repo as docker context during build; if it lived elsewhere, it wouldn't be
+# able to make back-references to pull in the install scripts. So here it lives.
+
+FROM python:3.7-buster
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV LC_ALL=C.UTF-8
+
+# This is used by the 04_install.sh script; we can't read the Python version from
+# .python-version for the same reasons as above, and it's more efficient to pull a
+# preexisting Python image than it is to build from source.
+ENV SKIP_PYTHON_INSTALL=1
+
+# Must be built from ./ci/lint/ for these paths to work.
+COPY ./docker-entrypoint.sh /entrypoint.sh
+COPY ./04_install.sh /install.sh
+
+RUN /install.sh && \
+  echo 'alias lint="./ci/lint/06_script.sh"' >> ~/.bashrc && \
+  chmod 755 /entrypoint.sh && \
+  rm -rf /var/lib/apt/lists/*
+
+
+WORKDIR /bitcoin
+ENTRYPOINT ["/entrypoint.sh"]

ci/lint/docker-entrypoint.sh

@@ -1,17 +1,10 @@
 #!/usr/bin/env bash
-#
-# Copyright (c) The Bitcoin Core developers
-# Distributed under the MIT software license, see the accompanying
-# file COPYING or https://opensource.org/license/mit/.
-
 export LC_ALL=C
 
 # Fixes permission issues when there is a container UID/GID mismatch with the owner
 # of the mounted bitcoin src dir.
 git config --global --add safe.directory /bitcoin
 
-export PATH="/python_build/bin:${PATH}"
-
 if [ -z "$1" ]; then
   LOCAL_BRANCH=1 bash -ic "./ci/lint/06_script.sh"
 else

ci/lint_imagefile

@@ -1,23 +0,0 @@
-# Copyright (c) The Bitcoin Core developers
-# Distributed under the MIT software license, see the accompanying
-# file COPYING or https://opensource.org/license/mit/.
-
-# See test/lint/README.md for usage.
-
-FROM debian:bookworm
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV LC_ALL=C.UTF-8
-
-COPY ./.python-version /.python-version
-COPY ./ci/lint/container-entrypoint.sh /entrypoint.sh
-COPY ./ci/lint/04_install.sh /install.sh
-
-RUN /install.sh && \
-  echo 'alias lint="./ci/lint/06_script.sh"' >> ~/.bashrc && \
-  chmod 755 /entrypoint.sh && \
-  rm -rf /var/lib/apt/lists/*
-
-
-WORKDIR /bitcoin
-ENTRYPOINT ["/entrypoint.sh"]

ci/lint_run_all.sh

@@ -8,5 +8,4 @@ export LC_ALL=C.UTF-8
 
 set -o errexit; source ./ci/test/00_setup_env.sh
 set -o errexit; source ./ci/lint/04_install.sh
-set -o errexit
-./ci/lint/06_script.sh
+set -o errexit; source ./ci/lint/06_script.sh

ci/test/00_setup_env.sh

@@ -6,25 +6,16 @@
 
 export LC_ALL=C.UTF-8
 
-set -ex
-
-# The source root dir, usually from git, usually read-only.
+# The root dir.
 # The ci system copies this folder.
-BASE_READ_ONLY_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )"/../../ >/dev/null 2>&1 && pwd )
-export BASE_READ_ONLY_DIR
-# The destination root dir inside the container.
-# This folder will also hold any SDKs.
-# This folder only exists on the ci guest and will be a copy of BASE_READ_ONLY_DIR
-export BASE_ROOT_DIR="${BASE_ROOT_DIR:-/ci_container_base}"
+BASE_ROOT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )"/../../ >/dev/null 2>&1 && pwd )
+export BASE_ROOT_DIR
 # The depends dir.
 # This folder exists only on the ci guest, and on the ci host as a volume.
 export DEPENDS_DIR=${DEPENDS_DIR:-$BASE_ROOT_DIR/depends}
-# A folder for the ci system to put temporary files (build result, datadirs for tests, ...)
-# This folder only exists on the ci guest.
+# A folder for the ci system to put temporary files (ccache, datadirs for tests, ...)
+# This folder only exists on the ci host.
 export BASE_SCRATCH_DIR=${BASE_SCRATCH_DIR:-$BASE_ROOT_DIR/ci/scratch}
-# A folder for the ci system to put executables.
-# This folder only exists on the ci guest.
-export BINS_SCRATCH_DIR="${BASE_SCRATCH_DIR}/bins/"
 
 echo "Setting specific values in env"
 if [ -n "${FILE_ENV}" ]; then
@@ -36,12 +27,17 @@ fi
 echo "Fallback to default values in env (if not yet set)"
 # The number of parallel jobs to pass down to make and test_runner.py
 export MAKEJOBS=${MAKEJOBS:--j4}
+# What host to compile for. See also ./depends/README.md
+# Tests that need cross-compilation export the appropriate HOST.
+# Tests that run natively guess the host
+export HOST=${HOST:-$("$BASE_ROOT_DIR/depends/config.guess")}
 # Whether to prefer BusyBox over GNU utilities
 export USE_BUSY_BOX=${USE_BUSY_BOX:-false}
 
 export RUN_UNIT_TESTS=${RUN_UNIT_TESTS:-true}
 export RUN_FUNCTIONAL_TESTS=${RUN_FUNCTIONAL_TESTS:-true}
 export RUN_TIDY=${RUN_TIDY:-false}
+export RUN_SECURITY_TESTS=${RUN_SECURITY_TESTS:-false}
 # By how much to scale the test_runner timeouts (option --timeout-factor).
 # This is needed because some ci machines have slow CPU or disk, so sanitizers
 # might be slow or a reindex might be waiting on disk IO.
@@ -49,25 +45,30 @@ export TEST_RUNNER_TIMEOUT_FACTOR=${TEST_RUNNER_TIMEOUT_FACTOR:-40}
 export TEST_RUNNER_ENV=${TEST_RUNNER_ENV:-}
 export RUN_FUZZ_TESTS=${RUN_FUZZ_TESTS:-false}
 
+export CONTAINER_NAME=${CONTAINER_NAME:-ci_unnamed}
+export CI_IMAGE_NAME_TAG=${CI_IMAGE_NAME_TAG:-ubuntu:20.04}
 # Randomize test order.
 # See https://www.boost.org/doc/libs/1_71_0/libs/test/doc/html/boost_test/utf_reference/rt_param_reference/random.html
 export BOOST_TEST_RANDOM=${BOOST_TEST_RANDOM:-1}
 # See man 7 debconf
 export DEBIAN_FRONTEND=noninteractive
-export CCACHE_MAXSIZE=${CCACHE_MAXSIZE:-100M}
+export CCACHE_SIZE=${CCACHE_SIZE:-100M}
 export CCACHE_TEMPDIR=${CCACHE_TEMPDIR:-/tmp/.ccache-temp}
 export CCACHE_COMPRESS=${CCACHE_COMPRESS:-1}
 # The cache dir.
 # This folder exists only on the ci guest, and on the ci host as a volume.
 export CCACHE_DIR=${CCACHE_DIR:-$BASE_SCRATCH_DIR/.ccache}
 # Folder where the build result is put (bin and lib).
-export BASE_OUTDIR=${BASE_OUTDIR:-$BASE_SCRATCH_DIR/out}
+export BASE_OUTDIR=${BASE_OUTDIR:-$BASE_SCRATCH_DIR/out/$HOST}
 # Folder where the build is done (dist and out-of-tree build).
 export BASE_BUILD_DIR=${BASE_BUILD_DIR:-$BASE_SCRATCH_DIR/build}
 # The folder for previous release binaries.
 # This folder exists only on the ci guest, and on the ci host as a volume.
-export PREVIOUS_RELEASES_DIR=${PREVIOUS_RELEASES_DIR:-$BASE_ROOT_DIR/prev_releases}
-export CI_BASE_PACKAGES=${CI_BASE_PACKAGES:-build-essential libtool autotools-dev automake pkg-config bsdmainutils curl ca-certificates ccache python3 rsync git procps bison e2fsprogs}
+export PREVIOUS_RELEASES_DIR=${PREVIOUS_RELEASES_DIR:-$BASE_ROOT_DIR/releases/$HOST}
+export DIR_IWYU="${BASE_SCRATCH_DIR}/iwyu"
+export SDK_URL=${SDK_URL:-https://bitcoincore.org/depends-sources/sdks}
+export CI_BASE_PACKAGES=${CI_BASE_PACKAGES:-build-essential libtool autotools-dev automake pkg-config bsdmainutils curl ca-certificates ccache python3 rsync git procps bison}
 export GOAL=${GOAL:-install}
 export DIR_QA_ASSETS=${DIR_QA_ASSETS:-${BASE_SCRATCH_DIR}/qa-assets}
+export PATH=${BASE_ROOT_DIR}/ci/retry:$PATH
 export CI_RETRY_EXE=${CI_RETRY_EXE:-"retry --"}

ci/test/00_setup_env_android.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-present The Bitcoin Core developers
+# Copyright (c) 2019-2022 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
@@ -9,7 +9,7 @@ export LC_ALL=C.UTF-8
 export HOST=aarch64-linux-android
 export PACKAGES="unzip openjdk-8-jdk gradle"
 export CONTAINER_NAME=ci_android
-export CI_IMAGE_NAME_TAG="docker.io/amd64/ubuntu:22.04"
+export CI_IMAGE_NAME_TAG="ubuntu:jammy"
 
 export RUN_UNIT_TESTS=false
 export RUN_FUNCTIONAL_TESTS=false

ci/test/00_setup_env_arm.sh

@@ -7,10 +7,18 @@
 export LC_ALL=C.UTF-8
 
 export HOST=arm-linux-gnueabihf
+# The host arch is unknown, so we run the tests through qemu.
+# If the host is arm and wants to run the tests natively, it can set QEMU_USER_CMD to the empty string.
+if [ -z ${QEMU_USER_CMD+x} ]; then export QEMU_USER_CMD="${QEMU_USER_CMD:-"qemu-arm -L /usr/arm-linux-gnueabihf/"}"; fi
 export DPKG_ADD_ARCH="armhf"
 export PACKAGES="python3-zmq g++-arm-linux-gnueabihf busybox libc6:armhf libstdc++6:armhf libfontconfig1:armhf libxcb1:armhf"
+if [ -n "$QEMU_USER_CMD" ]; then
+  # Likely cross-compiling, so install the needed gcc and qemu-user
+  export PACKAGES="$PACKAGES qemu-user"
+fi
 export CONTAINER_NAME=ci_arm_linux
-export CI_IMAGE_NAME_TAG="docker.io/arm64v8/debian:bookworm"
+# Use debian to avoid 404 apt errors when cross compiling
+export CI_IMAGE_NAME_TAG="debian:bullseye"
 export USE_BUSY_BOX=true
 export RUN_UNIT_TESTS=true
 export RUN_FUNCTIONAL_TESTS=false

ci/test/00_setup_env_i686_centos.sh

@@ -8,10 +8,11 @@ export LC_ALL=C.UTF-8
 
 export HOST=i686-pc-linux-gnu
 export CONTAINER_NAME=ci_i686_centos
-export CI_IMAGE_NAME_TAG="quay.io/centos/amd64:stream9"
-export CI_BASE_PACKAGES="gcc-c++ glibc-devel.x86_64 libstdc++-devel.x86_64 glibc-devel.i686 libstdc++-devel.i686 ccache libtool make git python3 python3-pip which patch lbzip2 xz procps-ng dash rsync coreutils bison util-linux e2fsprogs"
+export CI_IMAGE_NAME_TAG=quay.io/centos/centos:stream8
+export CI_BASE_PACKAGES="gcc-c++ glibc-devel.x86_64 libstdc++-devel.x86_64 glibc-devel.i686 libstdc++-devel.i686 ccache libtool make git python38 python38-pip which patch lbzip2 xz procps-ng dash rsync coreutils bison"
 export PIP_PACKAGES="pyzmq"
 export GOAL="install"
-export NO_WERROR=1  # Suppress error: #warning _FORTIFY_SOURCE > 2 is treated like 2 on this platform [-Werror=cpp]
+export NO_WERROR=1 # GCC 8
 export BITCOIN_CONFIG="--enable-zmq --with-gui=qt5 --enable-reduce-exports"
 export CONFIG_SHELL="/bin/dash"
+export TEST_RUNNER_ENV="LC_ALL=en_US.UTF-8"

ci/test/00_setup_env_i686_multiprocess.sh

@@ -8,10 +8,11 @@ export LC_ALL=C.UTF-8
 
 export HOST=i686-pc-linux-gnu
 export CONTAINER_NAME=ci_i686_multiprocess
-export CI_IMAGE_NAME_TAG="docker.io/amd64/ubuntu:20.04"
-export PACKAGES="cmake llvm clang g++-multilib"
+export CI_IMAGE_NAME_TAG=ubuntu:20.04
+export PACKAGES="cmake python3 llvm clang g++-multilib"
 export DEP_OPTS="DEBUG=1 MULTIPROCESS=1"
 export GOAL="install"
 export BITCOIN_CONFIG="--enable-debug CC='clang -m32' CXX='clang++ -m32' \
 LDFLAGS='--rtlib=compiler-rt -lgcc_s' CPPFLAGS='-DBOOST_MULTI_INDEX_ENABLE_SAFE_MODE'"
 export TEST_RUNNER_ENV="BITCOIND=bitcoin-node"
+export TEST_RUNNER_EXTRA="--nosandbox"

ci/test/00_setup_env_mac.sh

@@ -1,23 +1,18 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-present The Bitcoin Core developers
+# Copyright (c) 2019-2021 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-export SDK_URL=${SDK_URL:-https://bitcoincore.org/depends-sources/sdks}
-
 export CONTAINER_NAME=ci_macos_cross
-export CI_IMAGE_NAME_TAG="docker.io/ubuntu:22.04"
+export CI_IMAGE_NAME_TAG=ubuntu:20.04  # Check that Focal can cross-compile to macos
 export HOST=x86_64-apple-darwin
-export PACKAGES="cmake libz-dev python3-setuptools zip"
+export PACKAGES="cmake libz-dev libtinfo5 python3-setuptools xorriso"
 export XCODE_VERSION=12.2
 export XCODE_BUILD_ID=12B45b
 export RUN_UNIT_TESTS=false
 export RUN_FUNCTIONAL_TESTS=false
 export GOAL="deploy"
-
-# False-positive warning is fixed with clang 17, remove this when that version
-# can be used.
-export BITCOIN_CONFIG="--with-gui --enable-reduce-exports LDFLAGS=-Wno-error=unused-command-line-argument"
+export BITCOIN_CONFIG="--with-gui --enable-reduce-exports"

ci/test/00_setup_env_mac_native.sh

@@ -1,20 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2019-present The Bitcoin Core developers
-# Distributed under the MIT software license, see the accompanying
-# file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-export LC_ALL=C.UTF-8
-
-export HOST=x86_64-apple-darwin
-# Homebrew's python@3.12 is marked as externally managed (PEP 668).
-# Therefore, `--break-system-packages` is needed.
-export PIP_PACKAGES="--break-system-packages zmq"
-export GOAL="install"
-export BITCOIN_CONFIG="--with-gui --with-miniupnpc --with-natpmp --enable-reduce-exports"
-export CI_OS_NAME="macos"
-export NO_DEPENDS=1
-export OSX_SDK=""
-export CCACHE_MAXSIZE=400M
-export RUN_FUZZ_TESTS=true
-export FUZZ_TESTS_CONFIG="--exclude banman"  # https://github.com/bitcoin/bitcoin/issues/27924

ci/test/00_setup_env_mac_native_arm64.sh

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2019-2022 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+export LC_ALL=C.UTF-8
+
+export HOST=arm64-apple-darwin
+export PIP_PACKAGES="zmq"
+export GOAL="install"
+export BITCOIN_CONFIG="--with-gui --with-miniupnpc --with-natpmp --enable-reduce-exports"
+export CI_OS_NAME="macos"
+export NO_DEPENDS=1
+export OSX_SDK=""
+export CCACHE_SIZE=300M

ci/test/00_setup_env_native_asan.sh

@@ -1,26 +1,26 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-present The Bitcoin Core developers
+# Copyright (c) 2019-2022 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-export CI_IMAGE_NAME_TAG="docker.io/ubuntu:24.04"
 # Only install BCC tracing packages in Cirrus CI.
 if [[ "${CIRRUS_CI}" == "true" ]]; then
-  BPFCC_PACKAGE="bpfcc-tools linux-headers-$(uname --kernel-release)"
-  export CI_CONTAINER_CAP="--privileged -v /sys/kernel:/sys/kernel:rw"
+  # We install an up-to-date 'bpfcc-tools' package from an untrusted PPA.
+  # This can be dropped with the next Ubuntu or Debian release that includes up-to-date packages.
+  # See the if-then in ci/test/04_install.sh too.
+  export ADD_UNTRUSTED_BPFCC_PPA=true
+  export BPFCC_PACKAGE="bpfcc-tools"
 else
-  BPFCC_PACKAGE=""
-  export CI_CONTAINER_CAP="--cap-add SYS_PTRACE"  # If run with (ASan + LSan), the container needs access to ptrace (https://github.com/google/sanitizers/issues/764)
+  export ADD_UNTRUSTED_BPFCC_PPA=false
+  export BPFCC_PACKAGE=""
 fi
 
 export CONTAINER_NAME=ci_native_asan
-export PACKAGES="systemtap-sdt-dev clang-17 llvm-17 libclang-rt-17-dev python3-zmq qtbase5-dev qttools5-dev-tools libevent-dev libboost-dev libdb5.3++-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev libqrencode-dev libsqlite3-dev ${BPFCC_PACKAGE}"
+export PACKAGES="systemtap-sdt-dev clang llvm python3-zmq qtbase5-dev qttools5-dev-tools libevent-dev bsdmainutils libboost-dev libdb5.3++-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev libqrencode-dev libsqlite3-dev ${BPFCC_PACKAGE}"
+export CI_IMAGE_NAME_TAG=ubuntu:22.04
 export NO_DEPENDS=1
 export GOAL="install"
-export BITCOIN_CONFIG="--enable-c++20 --enable-usdt --enable-zmq --with-incompatible-bdb --with-gui=qt5 \
-CPPFLAGS='-DARENA_DEBUG -DDEBUG_LOCKORDER' \
---with-sanitizers=address,float-divide-by-zero,integer,undefined \
-CC='clang-17 -ftrivial-auto-var-init=pattern' CXX='clang++-17 -ftrivial-auto-var-init=pattern'"
+export BITCOIN_CONFIG="--enable-c++20 --enable-usdt --enable-zmq --with-incompatible-bdb --with-gui=qt5 CPPFLAGS='-DARENA_DEBUG -DDEBUG_LOCKORDER' --with-sanitizers=address,integer,undefined CC=clang CXX=clang++"

ci/test/00_setup_env_native_fuzz.sh

@@ -1,20 +1,18 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-present The Bitcoin Core developers
+# Copyright (c) 2019-2022 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-export CI_IMAGE_NAME_TAG="docker.io/ubuntu:24.04"
+export CI_IMAGE_NAME_TAG="ubuntu:23.04" # Version 23.04 will reach EOL in Jan 2024, and can be replaced by "ubuntu:24.04" (or anything else that ships the wanted clang version).
 export CONTAINER_NAME=ci_native_fuzz
-export PACKAGES="clang-17 llvm-17 libclang-rt-17-dev libevent-dev libboost-dev libsqlite3-dev"
+export PACKAGES="clang-16 llvm-16 libclang-rt-16-dev python3 libevent-dev bsdmainutils libboost-dev libsqlite3-dev"
 export NO_DEPENDS=1
 export RUN_UNIT_TESTS=false
 export RUN_FUNCTIONAL_TESTS=false
 export RUN_FUZZ_TESTS=true
 export GOAL="install"
-export CI_CONTAINER_CAP="--cap-add SYS_PTRACE"  # If run with (ASan + LSan), the container needs access to ptrace (https://github.com/google/sanitizers/issues/764)
-export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer,address,undefined,float-divide-by-zero,integer \
-CC='clang-17 -ftrivial-auto-var-init=pattern' CXX='clang++-17 -ftrivial-auto-var-init=pattern'"
-export CCACHE_MAXSIZE=200M
+export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer,address,undefined,integer CC='clang-16 -ftrivial-auto-var-init=pattern' CXX='clang++-16 -ftrivial-auto-var-init=pattern'"
+export CCACHE_SIZE=200M

ci/test/00_setup_env_native_fuzz_with_msan.sh

@@ -1,25 +1,25 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2020-present The Bitcoin Core developers
+# Copyright (c) 2020-2022 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-export CI_IMAGE_NAME_TAG="docker.io/ubuntu:22.04"
-LIBCXX_DIR="/msan/cxx_build/"
+export CI_IMAGE_NAME_TAG="ubuntu:23.04" # Version 23.04 will reach EOL in Jan 2024, and can be replaced by "ubuntu:24.04" (or anything else that ships the wanted clang version).
+LIBCXX_DIR="${BASE_SCRATCH_DIR}/msan/build/"
 export MSAN_FLAGS="-fsanitize=memory -fsanitize-memory-track-origins=2 -fno-omit-frame-pointer -g -O1 -fno-optimize-sibling-calls"
-LIBCXX_FLAGS="-nostdinc++ -nostdlib++ -isystem ${LIBCXX_DIR}include/c++/v1 -L${LIBCXX_DIR}lib -Wl,-rpath,${LIBCXX_DIR}lib -lc++ -lc++abi -lpthread -Wno-unused-command-line-argument"
+LIBCXX_FLAGS="-nostdinc++ -stdlib=libc++ -L${LIBCXX_DIR}lib -lc++abi -I${LIBCXX_DIR}include -I${LIBCXX_DIR}include/c++/v1 -lpthread -Wl,-rpath,${LIBCXX_DIR}lib -Wno-unused-command-line-argument"
 export MSAN_AND_LIBCXX_FLAGS="${MSAN_FLAGS} ${LIBCXX_FLAGS}"
 
 export CONTAINER_NAME="ci_native_fuzz_msan"
-export PACKAGES="cmake ninja-build"
+export PACKAGES="clang-16 llvm-16 libclang-rt-16-dev cmake"
 # BDB generates false-positives and will be removed in future
-export DEP_OPTS="NO_BDB=1 NO_QT=1 CC=clang CXX=clang++ CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'"
+export DEP_OPTS="NO_BDB=1 NO_QT=1 CC='clang' CXX='clang++' CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'"
 export GOAL="install"
-export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer,memory --disable-hardening --with-asm=no CFLAGS='${MSAN_FLAGS}' CPPFLAGS='-DBOOST_MULTI_INDEX_ENABLE_SAFE_MODE' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'"
+export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer,memory --disable-hardening --with-asm=no CC=clang CXX=clang++ CFLAGS='${MSAN_FLAGS}' CPPFLAGS='-DBOOST_MULTI_INDEX_ENABLE_SAFE_MODE' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'"
 export USE_MEMORY_SANITIZER="true"
 export RUN_UNIT_TESTS="false"
 export RUN_FUNCTIONAL_TESTS="false"
 export RUN_FUZZ_TESTS=true
-export CCACHE_MAXSIZE=250M
+export CCACHE_SIZE=250M

ci/test/00_setup_env_native_fuzz_with_valgrind.sh

@@ -1,14 +1,14 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-present The Bitcoin Core developers
+# Copyright (c) 2019-2022 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-export CI_IMAGE_NAME_TAG="docker.io/debian:bookworm"
+export CI_IMAGE_NAME_TAG="debian:bookworm"
 export CONTAINER_NAME=ci_native_fuzz_valgrind
-export PACKAGES="clang llvm libclang-rt-dev libevent-dev libboost-dev libsqlite3-dev valgrind"
+export PACKAGES="clang llvm libclang-rt-dev python3 libevent-dev bsdmainutils libboost-dev libsqlite3-dev valgrind"
 export NO_DEPENDS=1
 export RUN_UNIT_TESTS=false
 export RUN_FUNCTIONAL_TESTS=false
@@ -16,5 +16,5 @@ export RUN_FUZZ_TESTS=true
 export FUZZ_TESTS_CONFIG="--valgrind"
 export GOAL="install"
 # Temporarily pin dwarf 4, until using Valgrind 3.20 or later
-export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer CC='clang -gdwarf-4' CXX='clang++ -gdwarf-4'"
-export CCACHE_MAXSIZE=200M
+export BITCOIN_CONFIG="--enable-fuzz --with-sanitizers=fuzzer CC=clang CXX=clang++ CFLAGS='-gdwarf-4' CXXFLAGS='-gdwarf-4'"
+export CCACHE_SIZE=200M

ci/test/00_setup_env_native_msan.sh

@@ -1,23 +1,23 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2020-present The Bitcoin Core developers
+# Copyright (c) 2020-2022 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-export CI_IMAGE_NAME_TAG="docker.io/ubuntu:22.04"
-LIBCXX_DIR="/msan/cxx_build/"
+export CI_IMAGE_NAME_TAG="ubuntu:23.04" # Version 23.04 will reach EOL in Jan 2024, and can be replaced by "ubuntu:24.04" (or anything else that ships the wanted clang version).
+LIBCXX_DIR="${BASE_SCRATCH_DIR}/msan/build/"
 export MSAN_FLAGS="-fsanitize=memory -fsanitize-memory-track-origins=2 -fno-omit-frame-pointer -g -O1 -fno-optimize-sibling-calls"
-LIBCXX_FLAGS="-nostdinc++ -nostdlib++ -isystem ${LIBCXX_DIR}include/c++/v1 -L${LIBCXX_DIR}lib -Wl,-rpath,${LIBCXX_DIR}lib -lc++ -lc++abi -lpthread -Wno-unused-command-line-argument"
+LIBCXX_FLAGS="-nostdinc++ -stdlib=libc++ -L${LIBCXX_DIR}lib -lc++abi -I${LIBCXX_DIR}include -I${LIBCXX_DIR}include/c++/v1 -lpthread -Wl,-rpath,${LIBCXX_DIR}lib -Wno-unused-command-line-argument"
 export MSAN_AND_LIBCXX_FLAGS="${MSAN_FLAGS} ${LIBCXX_FLAGS}"
 
 export CONTAINER_NAME="ci_native_msan"
-export PACKAGES="cmake ninja-build"
+export PACKAGES="clang-16 llvm-16 libclang-rt-16-dev cmake"
 # BDB generates false-positives and will be removed in future
-export DEP_OPTS="NO_BDB=1 NO_QT=1 CC=clang CXX=clang++ CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'"
+export DEP_OPTS="NO_BDB=1 NO_QT=1 CC='clang' CXX='clang++' CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'"
 export GOAL="install"
-export BITCOIN_CONFIG="--with-sanitizers=memory --disable-hardening --with-asm=no CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'"
+export BITCOIN_CONFIG="--with-sanitizers=memory --disable-hardening --with-asm=no CC=clang CXX=clang++ CFLAGS='${MSAN_FLAGS}' CXXFLAGS='${MSAN_AND_LIBCXX_FLAGS}'"
 export USE_MEMORY_SANITIZER="true"
 export RUN_FUNCTIONAL_TESTS="false"
-export CCACHE_MAXSIZE=250M
+export CCACHE_SIZE=250M

ci/test/00_setup_env_native_nowallet_libbitcoinkernel.sh

@@ -1,15 +1,16 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-present The Bitcoin Core developers
+# Copyright (c) 2019-2022 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
 export CONTAINER_NAME=ci_native_nowallet_libbitcoinkernel
-export CI_IMAGE_NAME_TAG="docker.io/ubuntu:20.04"
-# Use minimum supported python3.8 and clang-10, see doc/dependencies.md
-export PACKAGES="python3-zmq clang-10 llvm-10 libc++abi-10-dev libc++-10-dev"
-export DEP_OPTS="NO_WALLET=1 CC=clang-10 CXX='clang++-10 -stdlib=libc++'"
+export CI_IMAGE_NAME_TAG=ubuntu:focal
+# Use minimum supported python3.7 (or python3.8, as best-effort) and clang-8, see doc/dependencies.md
+export PACKAGES="python3-zmq clang-8 llvm-8 libc++abi-8-dev libc++-8-dev"
+export DEP_OPTS="NO_WALLET=1 CC=clang-8 CXX='clang++-8 -stdlib=libc++'"
 export GOAL="install"
-export BITCOIN_CONFIG="--enable-reduce-exports --enable-experimental-util-chainstate --with-experimental-kernel-lib --enable-shared"
+export NO_WERROR=1
+export BITCOIN_CONFIG="--enable-reduce-exports CC=clang-8 CXX='clang++-8 -stdlib=libc++' --enable-experimental-util-chainstate --with-experimental-kernel-lib --enable-shared"

ci/test/00_setup_env_native_qt5.sh

@@ -1,21 +1,21 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-present The Bitcoin Core developers
+# Copyright (c) 2019-2022 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
 export CONTAINER_NAME=ci_native_qt5
-export CI_IMAGE_NAME_TAG="docker.io/ubuntu:20.04"
-# Use minimum supported python3.8 and gcc-9, see doc/dependencies.md
-export PACKAGES="gcc-9 g++-9 python3-zmq qtbase5-dev qttools5-dev-tools libdbus-1-dev libharfbuzz-dev"
-export DEP_OPTS="NO_QT=1 NO_UPNP=1 NO_NATPMP=1 DEBUG=1 ALLOW_HOST_PACKAGES=1 CC=gcc-9 CXX=g++-9"
+export CI_IMAGE_NAME_TAG=debian:buster
+# Use minimum supported python3.7 and gcc-8, see doc/dependencies.md
+export PACKAGES="gcc-8 g++-8 python3-zmq qtbase5-dev qttools5-dev-tools libdbus-1-dev libharfbuzz-dev"
+export DEP_OPTS="NO_QT=1 NO_UPNP=1 NO_NATPMP=1 DEBUG=1 ALLOW_HOST_PACKAGES=1 CC=gcc-8 CXX=g++-8"
 export TEST_RUNNER_EXTRA="--previous-releases --coverage --extended --exclude feature_dbcrash"  # Run extended tests so that coverage does not fail, but exclude the very slow dbcrash
 export RUN_UNIT_TESTS_SEQUENTIAL="true"
 export RUN_UNIT_TESTS="false"
 export GOAL="install"
-export NO_WERROR=1  # -Werror=maybe-uninitialized
+export NO_WERROR=1
 export DOWNLOAD_PREVIOUS_RELEASES="true"
 export BITCOIN_CONFIG="--enable-zmq --with-libs=no --with-gui=qt5 --enable-reduce-exports --enable-debug \
-CFLAGS=\"-g0 -O2 -funsigned-char\" CPPFLAGS='-DBOOST_MULTI_INDEX_ENABLE_SAFE_MODE' CXXFLAGS=\"-g0 -O2 -funsigned-char\""
+CFLAGS=\"-g0 -O2 -funsigned-char\" CPPFLAGS='-DBOOST_MULTI_INDEX_ENABLE_SAFE_MODE' CXXFLAGS=\"-g0 -O2 -funsigned-char\" CC=gcc-8 CXX=g++-8"

ci/test/00_setup_env_native_tidy.sh

@@ -1,20 +1,19 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2023-present The Bitcoin Core developers
+# Copyright (c) 2022 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-export CI_IMAGE_NAME_TAG="docker.io/ubuntu:24.04"
+export CI_IMAGE_NAME_TAG="ubuntu:lunar" # Version 23.04 will reach EOL in Jan 2024, and can be replaced by "ubuntu:24.04" (or anything else that ships the wanted clang version).
 export CONTAINER_NAME=ci_native_tidy
-export TIDY_LLVM_V="17"
-export PACKAGES="clang-${TIDY_LLVM_V} libclang-${TIDY_LLVM_V}-dev llvm-${TIDY_LLVM_V}-dev libomp-${TIDY_LLVM_V}-dev clang-tidy-${TIDY_LLVM_V} jq bear cmake libevent-dev libboost-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev systemtap-sdt-dev qtbase5-dev qttools5-dev qttools5-dev-tools libqrencode-dev libsqlite3-dev libdb++-dev"
+export PACKAGES="clang-16 libclang-16-dev llvm-16-dev libomp-16-dev clang-tidy-16 bear cmake libevent-dev libboost-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev systemtap-sdt-dev libqt5gui5 libqt5core5a libqt5dbus5 qttools5-dev qttools5-dev-tools libqrencode-dev libsqlite3-dev libdb++-dev"
 export NO_DEPENDS=1
 export RUN_UNIT_TESTS=false
 export RUN_FUNCTIONAL_TESTS=false
 export RUN_FUZZ_TESTS=false
 export RUN_TIDY=true
 export GOAL="install"
-export BITCOIN_CONFIG="CC=clang-${TIDY_LLVM_V} CXX=clang++-${TIDY_LLVM_V} --with-incompatible-bdb --disable-hardening CFLAGS='-O0 -g0' CXXFLAGS='-O0 -g0 -I/usr/lib/llvm-${TIDY_LLVM_V}/lib/clang/${TIDY_LLVM_V}/include'"
-export CCACHE_MAXSIZE=200M
+export BITCOIN_CONFIG="CC=clang-16 CXX=clang++-16 --with-incompatible-bdb --disable-hardening CFLAGS='-O0 -g0' CXXFLAGS='-O0 -g0 -I/usr/lib/llvm-16/lib/clang/16/include'"
+export CCACHE_SIZE=200M

ci/test/00_setup_env_native_tsan.sh

@@ -1,14 +1,14 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-present The Bitcoin Core developers
+# Copyright (c) 2019-2022 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
 export CONTAINER_NAME=ci_native_tsan
-export CI_IMAGE_NAME_TAG="docker.io/ubuntu:24.04"
-export PACKAGES="clang-17 llvm-17 libclang-rt-17-dev libc++abi-17-dev libc++-17-dev python3-zmq"
-export DEP_OPTS="CC=clang-17 CXX='clang++-17 -stdlib=libc++'"
+export CI_IMAGE_NAME_TAG=ubuntu:23.04  # Version 23.04 will reach EOL in Jan 2024, and can be replaced by "ubuntu:24.04" (or anything else that ships the wanted clang version).
+export PACKAGES="clang-16 llvm-16 libclang-rt-16-dev libc++abi-16-dev libc++-16-dev python3-zmq"
+export DEP_OPTS="CC=clang-16 CXX='clang++-16 -stdlib=libc++'"
 export GOAL="install"
 export BITCOIN_CONFIG="--enable-zmq CPPFLAGS='-DARENA_DEBUG -DDEBUG_LOCKORDER -DDEBUG_LOCKCONTENTION' CXXFLAGS='-g' --with-sanitizers=thread"

ci/test/00_setup_env_native_valgrind.sh

@@ -1,17 +1,17 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-present The Bitcoin Core developers
+# Copyright (c) 2019-2022 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-export CI_IMAGE_NAME_TAG="docker.io/debian:bookworm"
+export CI_IMAGE_NAME_TAG="debian:bookworm"
 export CONTAINER_NAME=ci_native_valgrind
-export PACKAGES="valgrind clang llvm libclang-rt-dev python3-zmq libevent-dev libboost-dev libdb5.3++-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev libsqlite3-dev"
+export PACKAGES="valgrind clang llvm libclang-rt-dev python3-zmq libevent-dev bsdmainutils libboost-dev libdb5.3++-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev libsqlite3-dev"
 export USE_VALGRIND=1
 export NO_DEPENDS=1
-export TEST_RUNNER_EXTRA="--exclude feature_init,rpc_bind,feature_bind_extra"  # Excluded for now, see https://github.com/bitcoin/bitcoin/issues/17765#issuecomment-602068547
+export TEST_RUNNER_EXTRA="--nosandbox --exclude feature_init,rpc_bind,feature_bind_extra"  # Excluded for now, see https://github.com/bitcoin/bitcoin/issues/17765#issuecomment-602068547
 export GOAL="install"
 # Temporarily pin dwarf 4, until using Valgrind 3.20 or later
-export BITCOIN_CONFIG="--enable-zmq --with-incompatible-bdb --with-gui=no CC='clang -gdwarf-4' CXX='clang++ -gdwarf-4'"  # TODO enable GUI
+export BITCOIN_CONFIG="--enable-zmq --with-incompatible-bdb --with-gui=no CC=clang CXX=clang++ CFLAGS='-gdwarf-4' CXXFLAGS='-gdwarf-4'"  # TODO enable GUI

ci/test/00_setup_env_s390x.sh

@@ -1,16 +1,26 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-present The Bitcoin Core developers
+# Copyright (c) 2019-2022 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
 export HOST=s390x-linux-gnu
+# The host arch is unknown, so we run the tests through qemu.
+# If the host is s390x and wants to run the tests natively, it can set QEMU_USER_CMD to the empty string.
+if [ -z ${QEMU_USER_CMD+x} ]; then export QEMU_USER_CMD="${QEMU_USER_CMD:-"qemu-s390x"}"; fi
 export PACKAGES="python3-zmq"
+if [ -n "$QEMU_USER_CMD" ]; then
+  # Likely cross-compiling, so install the needed gcc and qemu-user
+  export DPKG_ADD_ARCH="s390x"
+  export PACKAGES="$PACKAGES g++-s390x-linux-gnu qemu-user libc6:s390x libstdc++6:s390x"
+fi
+# Use debian to avoid 404 apt errors
 export CONTAINER_NAME=ci_s390x
-export CI_IMAGE_NAME_TAG="docker.io/s390x/ubuntu:24.04"
-export TEST_RUNNER_EXTRA="--exclude rpc_bind,feature_bind_extra"  # Excluded for now, see https://github.com/bitcoin/bitcoin/issues/17765#issuecomment-602068547
+export CI_IMAGE_NAME_TAG="debian:bookworm"
+export TEST_RUNNER_ENV="LC_ALL=C"
+export TEST_RUNNER_EXTRA="--exclude feature_init,rpc_bind,feature_bind_extra"  # Excluded for now, see https://github.com/bitcoin/bitcoin/issues/17765#issuecomment-602068547
 export RUN_FUNCTIONAL_TESTS=true
 export GOAL="install"
-export BITCOIN_CONFIG="--enable-reduce-exports"
+export BITCOIN_CONFIG="--enable-reduce-exports --disable-gui-tests"  # GUI tests disabled for now, see https://github.com/bitcoin/bitcoin/issues/23730

ci/test/00_setup_env_win64.sh

@@ -1,19 +1,16 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-present The Bitcoin Core developers
+# Copyright (c) 2019-2022 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
 export CONTAINER_NAME=ci_win64
-export CI_IMAGE_NAME_TAG="docker.io/amd64/ubuntu:22.04"  # Check that Jammy can cross-compile to win64
+export CI_IMAGE_NAME_TAG=ubuntu:22.04  # Check that Jammy can cross-compile to win64
 export HOST=x86_64-w64-mingw32
 export DPKG_ADD_ARCH="i386"
-export PACKAGES="nsis g++-mingw-w64-x86-64-posix wine-binfmt wine64 wine32 file"
+export PACKAGES="python3 nsis g++-mingw-w64-x86-64-posix wine-binfmt wine64 wine32 file"
 export RUN_FUNCTIONAL_TESTS=false
 export GOAL="deploy"
-# Prior to 11.0.0, the mingw-w64 headers were missing noreturn attributes, causing warnings when
-# cross-compiling for Windows. https://sourceforge.net/p/mingw-w64/bugs/306/
-# https://github.com/mingw-w64/mingw-w64/commit/1690994f515910a31b9fb7c7bd3a52d4ba987abe
-export BITCOIN_CONFIG="--enable-reduce-exports --enable-external-signer --disable-gui-tests CXXFLAGS=-Wno-return-type"
+export BITCOIN_CONFIG="--enable-reduce-exports --enable-external-signer --disable-gui-tests"

ci/test/01_base_install.sh

@@ -6,8 +6,6 @@
 
 export LC_ALL=C.UTF-8
 
-set -ex
-
 CFG_DONE="ci.base-install-done"  # Use a global git setting to remember whether this script ran to avoid running it twice
 
 if [ "$(git config --global ${CFG_DONE})" == "true" ]; then
@@ -20,9 +18,17 @@ if [ -n "$DPKG_ADD_ARCH" ]; then
 fi
 
 if [[ $CI_IMAGE_NAME_TAG == *centos* ]]; then
-  bash -c "dnf -y install epel-release"
-  bash -c "dnf -y --allowerasing install $CI_BASE_PACKAGES $PACKAGES"
-elif [ "$CI_OS_NAME" != "macos" ]; then
+  ${CI_RETRY_EXE} bash -c "dnf -y install epel-release"
+  ${CI_RETRY_EXE} bash -c "dnf -y --allowerasing install $CI_BASE_PACKAGES $PACKAGES"
+elif [ "$CI_USE_APT_INSTALL" != "no" ]; then
+  if [[ "${ADD_UNTRUSTED_BPFCC_PPA}" == "true" ]]; then
+    # Ubuntu 22.04 LTS and Debian 11 both have an outdated bpfcc-tools packages.
+    # The iovisor PPA is outdated as well. The next Ubuntu and Debian releases will contain updated
+    # packages. Meanwhile, use an untrusted PPA to install an up-to-date version of the bpfcc-tools
+    # package.
+    # TODO: drop this once we can use newer images in GCE
+    add-apt-repository ppa:hadret/bpfcc
+  fi
   if [[ -n "${APPEND_APT_SOURCES_LIST}" ]]; then
     echo "${APPEND_APT_SOURCES_LIST}" >> /etc/apt/sources.list
   fi
@@ -31,46 +37,28 @@ elif [ "$CI_OS_NAME" != "macos" ]; then
 fi
 
 if [ -n "$PIP_PACKAGES" ]; then
-  # shellcheck disable=SC2086
-  ${CI_RETRY_EXE} pip3 install --user $PIP_PACKAGES
+  if [ "$CI_OS_NAME" == "macos" ]; then
+    sudo -H pip3 install --upgrade pip
+    # shellcheck disable=SC2086
+    IN_GETOPT_BIN="$(brew --prefix gnu-getopt)/bin/getopt" ${CI_RETRY_EXE} pip3 install --user $PIP_PACKAGES
+  else
+    # shellcheck disable=SC2086
+    ${CI_RETRY_EXE} pip3 install --user $PIP_PACKAGES
+  fi
 fi
 
 if [[ ${USE_MEMORY_SANITIZER} == "true" ]]; then
-  ${CI_RETRY_EXE} git clone --depth=1 https://github.com/llvm/llvm-project -b llvmorg-17.0.2 /msan/llvm-project
-
-  cmake -G Ninja -B /msan/clang_build/ \
-    -DLLVM_ENABLE_PROJECTS="clang" \
-    -DCMAKE_BUILD_TYPE=Release \
-    -DLLVM_TARGETS_TO_BUILD=Native \
-    -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi;libunwind" \
-    -S /msan/llvm-project/llvm
-
-  ninja -C /msan/clang_build/ "-j$( nproc )"  # Use nproc, because MAKEJOBS is the default in docker image builds
-  ninja -C /msan/clang_build/ install-runtimes
-
-  update-alternatives --install /usr/bin/clang++ clang++ /msan/clang_build/bin/clang++ 100
-  update-alternatives --install /usr/bin/clang clang /msan/clang_build/bin/clang 100
-  update-alternatives --install /usr/bin/llvm-symbolizer llvm-symbolizer /msan/clang_build/bin/llvm-symbolizer 100
-
-  cmake -G Ninja -B /msan/cxx_build/ \
-    -DLLVM_ENABLE_RUNTIMES='libcxx;libcxxabi' \
-    -DCMAKE_BUILD_TYPE=Release \
-    -DLLVM_USE_SANITIZER=MemoryWithOrigins \
-    -DCMAKE_C_COMPILER=clang \
-    -DCMAKE_CXX_COMPILER=clang++ \
-    -DLLVM_TARGETS_TO_BUILD=Native \
-    -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF \
-    -DLIBCXX_HARDENING_MODE=debug \
-    -S /msan/llvm-project/runtimes
-
-  ninja -C /msan/cxx_build/ "-j$( nproc )"  # Use nproc, because MAKEJOBS is the default in docker image builds
+  update-alternatives --install /usr/bin/clang++ clang++ "$(which clang++-16)" 100
+  update-alternatives --install /usr/bin/clang clang "$(which clang-16)" 100
+  git clone --depth=1 https://github.com/llvm/llvm-project -b llvmorg-16.0.1 "${BASE_SCRATCH_DIR}"/msan/llvm-project
+  cmake -B "${BASE_SCRATCH_DIR}"/msan/build/ -DLLVM_ENABLE_RUNTIMES='libcxx;libcxxabi' -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=MemoryWithOrigins -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF -DLIBCXX_ENABLE_DEBUG_MODE=ON -DLIBCXX_ENABLE_ASSERTIONS=ON -S "${BASE_SCRATCH_DIR}"/msan/llvm-project/runtimes
+  make -C "${BASE_SCRATCH_DIR}"/msan/build/ "$MAKEJOBS"
 fi
 
 if [[ "${RUN_TIDY}" == "true" ]]; then
-  ${CI_RETRY_EXE} git clone https://github.com/include-what-you-use/include-what-you-use -b master /include-what-you-use
-  git -C /include-what-you-use checkout a138eaac254e5a472464e31d5ec418fe6e6f1fc7
-  cmake -B /iwyu-build/ -G 'Unix Makefiles' -DCMAKE_PREFIX_PATH=/usr/lib/llvm-"${TIDY_LLVM_V}" -S /include-what-you-use
-  make -C /iwyu-build/ install "-j$( nproc )"  # Use nproc, because MAKEJOBS is the default in docker image builds
+  git clone --depth=1 https://github.com/include-what-you-use/include-what-you-use -b clang_16 "${DIR_IWYU}"/include-what-you-use
+  cmake -B "${DIR_IWYU}"/build/ -G 'Unix Makefiles' -DCMAKE_PREFIX_PATH=/usr/lib/llvm-16 -S "${DIR_IWYU}"/include-what-you-use
+  make -C "${DIR_IWYU}"/build/ install "$MAKEJOBS"
 fi
 
 mkdir -p "${DEPENDS_DIR}/SDKs" "${DEPENDS_DIR}/sdk-sources"
@@ -81,7 +69,7 @@ if [ -n "$XCODE_VERSION" ] && [ ! -d "${DEPENDS_DIR}/SDKs/${OSX_SDK_BASENAME}" ]
   OSX_SDK_FILENAME="${OSX_SDK_BASENAME}.tar.gz"
   OSX_SDK_PATH="${DEPENDS_DIR}/sdk-sources/${OSX_SDK_FILENAME}"
   if [ ! -f "$OSX_SDK_PATH" ]; then
-    ${CI_RETRY_EXE} curl --location --fail "${SDK_URL}/${OSX_SDK_FILENAME}" -o "$OSX_SDK_PATH"
+    curl --location --fail "${SDK_URL}/${OSX_SDK_FILENAME}" -o "$OSX_SDK_PATH"
   fi
   tar -C "${DEPENDS_DIR}/SDKs" -xf "$OSX_SDK_PATH"
 fi
@@ -89,11 +77,11 @@ fi
 if [ -n "$ANDROID_HOME" ] && [ ! -d "$ANDROID_HOME" ]; then
   ANDROID_TOOLS_PATH=${DEPENDS_DIR}/sdk-sources/android-tools.zip
   if [ ! -f "$ANDROID_TOOLS_PATH" ]; then
-    ${CI_RETRY_EXE} curl --location --fail "${ANDROID_TOOLS_URL}" -o "$ANDROID_TOOLS_PATH"
+    curl --location --fail "${ANDROID_TOOLS_URL}" -o "$ANDROID_TOOLS_PATH"
   fi
   mkdir -p "$ANDROID_HOME"
   unzip -o "$ANDROID_TOOLS_PATH" -d "$ANDROID_HOME"
-  yes | "${ANDROID_HOME}"/cmdline-tools/bin/sdkmanager --sdk_root="${ANDROID_HOME}" --install "build-tools;${ANDROID_BUILD_TOOLS_VERSION}" "platform-tools" "platforms;android-31" "platforms;android-${ANDROID_API_LEVEL}" "ndk;${ANDROID_NDK_VERSION}"
+  yes | "${ANDROID_HOME}"/cmdline-tools/bin/sdkmanager --sdk_root="${ANDROID_HOME}" --install "build-tools;${ANDROID_BUILD_TOOLS_VERSION}" "platform-tools" "platforms;android-${ANDROID_API_LEVEL}" "ndk;${ANDROID_NDK_VERSION}"
 fi
 
 git config --global ${CFG_DONE} "true"

ci/test/02_run_container.sh

@@ -1,92 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2018-present The Bitcoin Core developers
-# Distributed under the MIT software license, see the accompanying
-# file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-export LC_ALL=C.UTF-8
-export CI_IMAGE_LABEL="bitcoin-ci-test"
-
-set -ex
-
-if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
-  # Export all env vars to avoid missing some.
-  # Though, exclude those with newlines to avoid parsing problems.
-  python3 -c 'import os; [print(f"{key}={value}") for key, value in os.environ.items() if "\n" not in value and "HOME" != key and "PATH" != key and "USER" != key]' | tee /tmp/env
-  # System-dependent env vars must be kept as is. So read them from the container.
-  docker run --rm "${CI_IMAGE_NAME_TAG}" bash -c "env | grep --extended-regexp '^(HOME|PATH|USER)='" | tee --append /tmp/env
-  echo "Creating $CI_IMAGE_NAME_TAG container to run in"
-  DOCKER_BUILDKIT=1 docker build \
-      --file "${BASE_READ_ONLY_DIR}/ci/test_imagefile" \
-      --build-arg "CI_IMAGE_NAME_TAG=${CI_IMAGE_NAME_TAG}" \
-      --build-arg "FILE_ENV=${FILE_ENV}" \
-      --label="${CI_IMAGE_LABEL}" \
-      --tag="${CONTAINER_NAME}" \
-      "${BASE_READ_ONLY_DIR}"
-  docker volume create "${CONTAINER_NAME}_ccache" || true
-  docker volume create "${CONTAINER_NAME}_depends" || true
-  docker volume create "${CONTAINER_NAME}_depends_sources" || true
-  docker volume create "${CONTAINER_NAME}_depends_SDKs_android" || true
-  docker volume create "${CONTAINER_NAME}_previous_releases" || true
-
-  if [ -n "${RESTART_CI_DOCKER_BEFORE_RUN}" ] ; then
-    echo "Restart docker before run to stop and clear all containers started with --rm"
-    podman container rm --force --all  # Similar to "systemctl restart docker"
-
-    # Still prune everything in case the filtered pruning doesn't work, or if labels were not set
-    # on a previous run. Belt and suspenders approach, should be fine to remove in the future.
-    # Prune images used by --external containers (e.g. build containers) when
-    # using podman.
-    echo "Prune all dangling images"
-    podman image prune --force --external
-  fi
-  echo "Prune all dangling $CI_IMAGE_LABEL images"
-  # When detecting podman-docker, `--external` should be added.
-  docker image prune --force --filter "label=$CI_IMAGE_LABEL"
-
-  # shellcheck disable=SC2086
-  CI_CONTAINER_ID=$(docker run --cap-add LINUX_IMMUTABLE $CI_CONTAINER_CAP --rm --interactive --detach --tty \
-                  --mount "type=bind,src=$BASE_READ_ONLY_DIR,dst=$BASE_READ_ONLY_DIR,readonly" \
-                  --mount "type=volume,src=${CONTAINER_NAME}_ccache,dst=$CCACHE_DIR" \
-                  --mount "type=volume,src=${CONTAINER_NAME}_depends,dst=$DEPENDS_DIR/built" \
-                  --mount "type=volume,src=${CONTAINER_NAME}_depends_sources,dst=$DEPENDS_DIR/sources" \
-                  --mount "type=volume,src=${CONTAINER_NAME}_depends_SDKs_android,dst=$DEPENDS_DIR/SDKs/android" \
-                  --mount "type=volume,src=${CONTAINER_NAME}_previous_releases,dst=$PREVIOUS_RELEASES_DIR" \
-                  --env-file /tmp/env \
-                  --name "$CONTAINER_NAME" \
-                  "$CONTAINER_NAME")
-  export CI_CONTAINER_ID
-  export CI_EXEC_CMD_PREFIX="docker exec ${CI_CONTAINER_ID}"
-else
-  echo "Running on host system without docker wrapper"
-  echo "Create missing folders"
-  mkdir -p "${CCACHE_DIR}"
-  mkdir -p "${PREVIOUS_RELEASES_DIR}"
-fi
-
-if [ "$CI_OS_NAME" == "macos" ]; then
-  IN_GETOPT_BIN="$(brew --prefix gnu-getopt)/bin/getopt"
-  export IN_GETOPT_BIN
-fi
-
-CI_EXEC () {
-  $CI_EXEC_CMD_PREFIX bash -c "export PATH=${BINS_SCRATCH_DIR}:${BASE_ROOT_DIR}/ci/retry:\$PATH && cd \"${BASE_ROOT_DIR}\" && $*"
-}
-export -f CI_EXEC
-
-# Normalize all folders to BASE_ROOT_DIR
-CI_EXEC rsync --archive --stats --human-readable "${BASE_READ_ONLY_DIR}/" "${BASE_ROOT_DIR}" || echo "Nothing to copy from ${BASE_READ_ONLY_DIR}/"
-CI_EXEC "${BASE_ROOT_DIR}/ci/test/01_base_install.sh"
-
-# Fixes permission issues when there is a container UID/GID mismatch with the owner
-# of the git source code directory.
-CI_EXEC git config --global --add safe.directory \"*\"
-
-CI_EXEC mkdir -p "${BINS_SCRATCH_DIR}"
-
-CI_EXEC "${BASE_ROOT_DIR}/ci/test/06_script_b.sh"
-
-if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
-  echo "Stop and remove CI container by ID"
-  docker container kill "${CI_CONTAINER_ID}"
-fi

ci/test/04_install.sh

@@ -0,0 +1,112 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2018-2022 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+export LC_ALL=C.UTF-8
+
+if [[ $QEMU_USER_CMD == qemu-s390* ]]; then
+  export LC_ALL=C
+fi
+
+# Create folders that are mounted into the docker
+mkdir -p "${CCACHE_DIR}"
+mkdir -p "${PREVIOUS_RELEASES_DIR}"
+
+export ASAN_OPTIONS="detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1"
+export LSAN_OPTIONS="suppressions=${BASE_ROOT_DIR}/test/sanitizer_suppressions/lsan"
+export TSAN_OPTIONS="suppressions=${BASE_ROOT_DIR}/test/sanitizer_suppressions/tsan:halt_on_error=1:log_path=${BASE_SCRATCH_DIR}/sanitizer-output/tsan"
+export UBSAN_OPTIONS="suppressions=${BASE_ROOT_DIR}/test/sanitizer_suppressions/ubsan:print_stacktrace=1:halt_on_error=1:report_error_type=1"
+env | grep -E '^(BITCOIN_CONFIG|BASE_|QEMU_|CCACHE_|LC_ALL|BOOST_TEST_RANDOM|DEBIAN_FRONTEND|CONFIG_SHELL|(ASAN|LSAN|TSAN|UBSAN)_OPTIONS|PREVIOUS_RELEASES_DIR)' | tee /tmp/env
+if [[ $BITCOIN_CONFIG = *--with-sanitizers=*address* ]]; then # If ran with (ASan + LSan), Docker needs access to ptrace (https://github.com/google/sanitizers/issues/764)
+  CI_CONTAINER_CAP="--cap-add SYS_PTRACE"
+fi
+
+export P_CI_DIR="$PWD"
+export BINS_SCRATCH_DIR="${BASE_SCRATCH_DIR}/bins/"
+
+if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
+  echo "Creating $CI_IMAGE_NAME_TAG container to run in"
+  DOCKER_BUILDKIT=1 ${CI_RETRY_EXE} docker build \
+      --file "${BASE_ROOT_DIR}/ci/test_imagefile" \
+      --build-arg "CI_IMAGE_NAME_TAG=${CI_IMAGE_NAME_TAG}" \
+      --build-arg "FILE_ENV=${FILE_ENV}" \
+      --tag="${CONTAINER_NAME}" \
+      "${BASE_ROOT_DIR}"
+  docker volume create "${CONTAINER_NAME}_ccache" || true
+  docker volume create "${CONTAINER_NAME}_depends" || true
+  docker volume create "${CONTAINER_NAME}_previous_releases" || true
+
+  if [ -n "${RESTART_CI_DOCKER_BEFORE_RUN}" ] ; then
+    echo "Restart docker before run to stop and clear all containers started with --rm"
+    podman container stop --all  # Similar to "systemctl restart docker"
+    echo "Prune all dangling images"
+    docker image prune --force
+  fi
+
+  # shellcheck disable=SC2086
+  CI_CONTAINER_ID=$(docker run $CI_CONTAINER_CAP --rm --interactive --detach --tty \
+                  --mount type=bind,src=$BASE_ROOT_DIR,dst=/ro_base,readonly \
+                  --mount "type=volume,src=${CONTAINER_NAME}_ccache,dst=$CCACHE_DIR" \
+                  --mount "type=volume,src=${CONTAINER_NAME}_depends,dst=$DEPENDS_DIR" \
+                  --mount "type=volume,src=${CONTAINER_NAME}_previous_releases,dst=$PREVIOUS_RELEASES_DIR" \
+                  -w $BASE_ROOT_DIR \
+                  --env-file /tmp/env \
+                  --name $CONTAINER_NAME \
+                  $CONTAINER_NAME)
+  export CI_CONTAINER_ID
+  export CI_EXEC_CMD_PREFIX="docker exec ${CI_CONTAINER_ID}"
+else
+  echo "Running on host system without docker wrapper"
+fi
+
+CI_EXEC () {
+  $CI_EXEC_CMD_PREFIX bash -c "export PATH=${BINS_SCRATCH_DIR}:\$PATH && cd \"$P_CI_DIR\" && $*"
+}
+export -f CI_EXEC
+
+CI_EXEC rsync --archive --stats --human-readable /ci_base_install/ "${BASE_ROOT_DIR}" || echo "/ci_base_install/ missing"
+CI_EXEC "${BASE_ROOT_DIR}/ci/test/01_base_install.sh"
+CI_EXEC rsync --archive --stats --human-readable /ro_base/ "${BASE_ROOT_DIR}" || echo "Nothing to copy from ro_base"
+# Fixes permission issues when there is a container UID/GID mismatch with the owner
+# of the git source code directory.
+CI_EXEC git config --global --add safe.directory \"*\"
+
+CI_EXEC mkdir -p "${BINS_SCRATCH_DIR}"
+
+if [ "$CI_OS_NAME" == "macos" ]; then
+  top -l 1 -s 0 | awk ' /PhysMem/ {print}'
+  echo "Number of CPUs: $(sysctl -n hw.logicalcpu)"
+else
+  CI_EXEC free -m -h
+  CI_EXEC echo "Number of CPUs \(nproc\):" \$\(nproc\)
+  CI_EXEC echo "$(lscpu | grep Endian)"
+fi
+CI_EXEC echo "Free disk space:"
+CI_EXEC df -h
+
+if [ "$RUN_FUZZ_TESTS" = "true" ]; then
+  export DIR_FUZZ_IN=${DIR_QA_ASSETS}/fuzz_seed_corpus/
+  if [ ! -d "$DIR_FUZZ_IN" ]; then
+    CI_EXEC git clone --depth=1 https://github.com/bitcoin-core/qa-assets "${DIR_QA_ASSETS}"
+  fi
+elif [ "$RUN_UNIT_TESTS" = "true" ] || [ "$RUN_UNIT_TESTS_SEQUENTIAL" = "true" ]; then
+  export DIR_UNIT_TEST_DATA=${DIR_QA_ASSETS}/unit_test_data/
+  if [ ! -d "$DIR_UNIT_TEST_DATA" ]; then
+    CI_EXEC mkdir -p "$DIR_UNIT_TEST_DATA"
+    CI_EXEC curl --location --fail https://github.com/bitcoin-core/qa-assets/raw/main/unit_test_data/script_assets_test.json -o "${DIR_UNIT_TEST_DATA}/script_assets_test.json"
+  fi
+fi
+
+CI_EXEC mkdir -p "${BASE_SCRATCH_DIR}/sanitizer-output/"
+
+if [ "$USE_BUSY_BOX" = "true" ]; then
+  echo "Setup to use BusyBox utils"
+  # tar excluded for now because it requires passing in the exact archive type in ./depends (fixed in later BusyBox version)
+  # ar excluded for now because it does not recognize the -q option in ./depends (unknown if fixed)
+  # shellcheck disable=SC1010
+  CI_EXEC for util in \$\(busybox --list \| grep -v "^ar$" \| grep -v "^tar$" \)\; do ln -s \$\(command -v busybox\) "${BINS_SCRATCH_DIR}/\$util"\; done
+  # Print BusyBox version
+  CI_EXEC patch --help
+fi

ci/test/05_before_script.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2018-2022 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+export LC_ALL=C.UTF-8
+
+# Make sure default datadir does not exist and is never read by creating a dummy file
+if [ "$CI_OS_NAME" == "macos" ]; then
+  echo > "${HOME}/Library/Application Support/Bitcoin"
+else
+  CI_EXEC echo \> \$HOME/.bitcoin
+fi
+
+if [ -z "$NO_DEPENDS" ]; then
+  if [[ $CI_IMAGE_NAME_TAG == *centos* ]]; then
+    # CentOS has problems building the depends if the config shell is not explicitly set
+    # (i.e. for libevent a Makefile with an empty SHELL variable is generated, leading to
+    #  an error as the first command is executed)
+    SHELL_OPTS="LC_ALL=en_US.UTF-8 CONFIG_SHELL=/bin/dash"
+  else
+    SHELL_OPTS="CONFIG_SHELL="
+  fi
+  CI_EXEC "$SHELL_OPTS" make "$MAKEJOBS" -C depends HOST="$HOST" "$DEP_OPTS" LOG=1
+fi
+if [ "$DOWNLOAD_PREVIOUS_RELEASES" = "true" ]; then
+  CI_EXEC test/get_previous_releases.py -b -t "$PREVIOUS_RELEASES_DIR"
+fi

ci/test/06_script_a.sh

@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2018-2022 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+export LC_ALL=C.UTF-8
+
+BITCOIN_CONFIG_ALL="--enable-suppress-external-warnings --disable-dependency-tracking"
+if [ -z "$NO_DEPENDS" ]; then
+  BITCOIN_CONFIG_ALL="${BITCOIN_CONFIG_ALL} CONFIG_SITE=$DEPENDS_DIR/$HOST/share/config.site"
+fi
+if [ -z "$NO_WERROR" ]; then
+  BITCOIN_CONFIG_ALL="${BITCOIN_CONFIG_ALL} --enable-werror"
+fi
+
+CI_EXEC "ccache --zero-stats --max-size=$CCACHE_SIZE"
+PRINT_CCACHE_STATISTICS="ccache --version | head -n 1 && ccache --show-stats"
+
+if [ -n "$ANDROID_TOOLS_URL" ]; then
+  CI_EXEC make distclean || true
+  CI_EXEC ./autogen.sh
+  CI_EXEC ./configure "$BITCOIN_CONFIG_ALL" "$BITCOIN_CONFIG" || ( (CI_EXEC cat config.log) && false)
+  CI_EXEC "make $MAKEJOBS && cd src/qt && ANDROID_HOME=${ANDROID_HOME} ANDROID_NDK_HOME=${ANDROID_NDK_HOME} make apk"
+  CI_EXEC "${PRINT_CCACHE_STATISTICS}"
+  exit 0
+fi
+
+BITCOIN_CONFIG_ALL="${BITCOIN_CONFIG_ALL} --enable-external-signer --prefix=$BASE_OUTDIR"
+
+if [ -n "$CONFIG_SHELL" ]; then
+  CI_EXEC "$CONFIG_SHELL" -c "./autogen.sh"
+else
+  CI_EXEC ./autogen.sh
+fi
+
+CI_EXEC mkdir -p "${BASE_BUILD_DIR}"
+export P_CI_DIR="${BASE_BUILD_DIR}"
+
+CI_EXEC "${BASE_ROOT_DIR}/configure" --cache-file=config.cache "$BITCOIN_CONFIG_ALL" "$BITCOIN_CONFIG" || ( (CI_EXEC cat config.log) && false)
+
+CI_EXEC make distdir VERSION="$HOST"
+
+export P_CI_DIR="${BASE_BUILD_DIR}/bitcoin-$HOST"
+
+CI_EXEC ./configure --cache-file=../config.cache "$BITCOIN_CONFIG_ALL" "$BITCOIN_CONFIG" || ( (CI_EXEC cat config.log) && false)
+
+set -o errtrace
+trap 'CI_EXEC "cat ${BASE_SCRATCH_DIR}/sanitizer-output/* 2> /dev/null"' ERR
+
+if [[ ${USE_MEMORY_SANITIZER} == "true" ]]; then
+  # MemorySanitizer (MSAN) does not support tracking memory initialization done by
+  # using the Linux getrandom syscall. Avoid using getrandom by undefining
+  # HAVE_SYS_GETRANDOM. See https://github.com/google/sanitizers/issues/852 for
+  # details.
+  CI_EXEC 'grep -v HAVE_SYS_GETRANDOM src/config/bitcoin-config.h > src/config/bitcoin-config.h.tmp && mv src/config/bitcoin-config.h.tmp src/config/bitcoin-config.h'
+fi
+
+if [[ "${RUN_TIDY}" == "true" ]]; then
+  MAYBE_BEAR="bear --config src/.bear-tidy-config"
+  MAYBE_TOKEN="--"
+fi
+
+CI_EXEC "${MAYBE_BEAR}" "${MAYBE_TOKEN}" make "$MAKEJOBS" "$GOAL" || ( echo "Build failure. Verbose build follows." && CI_EXEC make "$GOAL" V=1 ; false )
+
+CI_EXEC "${PRINT_CCACHE_STATISTICS}"
+CI_EXEC du -sh "${DEPENDS_DIR}"/*/
+CI_EXEC du -sh "${PREVIOUS_RELEASES_DIR}"

ci/test/06_script_b.sh

@@ -1,204 +1,104 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2018-present The Bitcoin Core developers
+# Copyright (c) 2018-2022 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
-set -ex
-
-export ASAN_OPTIONS="detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1"
-export LSAN_OPTIONS="suppressions=${BASE_ROOT_DIR}/test/sanitizer_suppressions/lsan"
-export TSAN_OPTIONS="suppressions=${BASE_ROOT_DIR}/test/sanitizer_suppressions/tsan:halt_on_error=1:log_path=${BASE_SCRATCH_DIR}/sanitizer-output/tsan"
-export UBSAN_OPTIONS="suppressions=${BASE_ROOT_DIR}/test/sanitizer_suppressions/ubsan:print_stacktrace=1:halt_on_error=1:report_error_type=1"
-
-if [ "$CI_OS_NAME" == "macos" ]; then
-  top -l 1 -s 0 | awk ' /PhysMem/ {print}'
-  echo "Number of CPUs: $(sysctl -n hw.logicalcpu)"
-else
-  free -m -h
-  echo "Number of CPUs (nproc): $(nproc)"
-  echo "System info: $(uname --kernel-name --kernel-release)"
-  lscpu
-fi
-echo "Free disk space:"
-df -h
-
-# What host to compile for. See also ./depends/README.md
-# Tests that need cross-compilation export the appropriate HOST.
-# Tests that run natively guess the host
-export HOST=${HOST:-$("$BASE_ROOT_DIR/depends/config.guess")}
-
-(
-  # compact->outputs[i].file_size is uninitialized memory, so reading it is UB.
-  # The statistic bytes_written is only used for logging, which is disabled in
-  # CI, so as a temporary minimal fix to work around UB and CI failures, leave
-  # bytes_written unmodified.
-  # See https://github.com/bitcoin/bitcoin/pull/28359#issuecomment-1698694748
-  echo 'diff --git a/src/leveldb/db/db_impl.cc b/src/leveldb/db/db_impl.cc
-index 65e31724bc..f61b471953 100644
---- a/src/leveldb/db/db_impl.cc
-+++ b/src/leveldb/db/db_impl.cc
-@@ -1028,9 +1028,6 @@ Status DBImpl::DoCompactionWork(CompactionState* compact) {
-       stats.bytes_read += compact->compaction->input(which, i)->file_size;
-     }
-   }
--  for (size_t i = 0; i < compact->outputs.size(); i++) {
--    stats.bytes_written += compact->outputs[i].file_size;
--  }
-
-   mutex_.Lock();
-   stats_[compact->compaction->level() + 1].Add(stats);' | patch -p1
-  git diff
-)
-
-if [ "$RUN_FUZZ_TESTS" = "true" ]; then
-  export DIR_FUZZ_IN=${DIR_QA_ASSETS}/fuzz_seed_corpus/
-  if [ ! -d "$DIR_FUZZ_IN" ]; then
-    ${CI_RETRY_EXE} git clone --depth=1 https://github.com/bitcoin-core/qa-assets "${DIR_QA_ASSETS}"
-  fi
-  (
-    cd "${DIR_QA_ASSETS}"
-    echo "Using qa-assets repo from commit ..."
-    git log -1
-  )
-elif [ "$RUN_UNIT_TESTS" = "true" ] || [ "$RUN_UNIT_TESTS_SEQUENTIAL" = "true" ]; then
-  export DIR_UNIT_TEST_DATA=${DIR_QA_ASSETS}/unit_test_data/
-  if [ ! -d "$DIR_UNIT_TEST_DATA" ]; then
-    mkdir -p "$DIR_UNIT_TEST_DATA"
-    ${CI_RETRY_EXE} curl --location --fail https://github.com/bitcoin-core/qa-assets/raw/main/unit_test_data/script_assets_test.json -o "${DIR_UNIT_TEST_DATA}/script_assets_test.json"
-  fi
-fi
-
-mkdir -p "${BASE_SCRATCH_DIR}/sanitizer-output/"
-
-if [ "$USE_BUSY_BOX" = "true" ]; then
-  echo "Setup to use BusyBox utils"
-  # tar excluded for now because it requires passing in the exact archive type in ./depends (fixed in later BusyBox version)
-  # ar excluded for now because it does not recognize the -q option in ./depends (unknown if fixed)
-  for util in $(busybox --list | grep -v "^ar$" | grep -v "^tar$" ); do ln -s "$(command -v busybox)" "${BINS_SCRATCH_DIR}/$util"; done
-  # Print BusyBox version
-  patch --help
-fi
-
-# Make sure default datadir does not exist and is never read by creating a dummy file
-if [ "$CI_OS_NAME" == "macos" ]; then
-  echo > "${HOME}/Library/Application Support/Bitcoin"
-else
-  echo > "${HOME}/.bitcoin"
-fi
-
-if [ -z "$NO_DEPENDS" ]; then
-  if [[ $CI_IMAGE_NAME_TAG == *centos* ]]; then
-    SHELL_OPTS="CONFIG_SHELL=/bin/dash"
-  else
-    SHELL_OPTS="CONFIG_SHELL="
-  fi
-  bash -c "$SHELL_OPTS make $MAKEJOBS -C depends HOST=$HOST $DEP_OPTS LOG=1"
-fi
-if [ "$DOWNLOAD_PREVIOUS_RELEASES" = "true" ]; then
-  test/get_previous_releases.py -b -t "$PREVIOUS_RELEASES_DIR"
-fi
-
-BITCOIN_CONFIG_ALL="--disable-dependency-tracking"
-if [ -z "$NO_DEPENDS" ]; then
-  BITCOIN_CONFIG_ALL="${BITCOIN_CONFIG_ALL} CONFIG_SITE=$DEPENDS_DIR/$HOST/share/config.site"
-fi
-if [ -z "$NO_WERROR" ]; then
-  BITCOIN_CONFIG_ALL="${BITCOIN_CONFIG_ALL} --enable-werror"
-fi
-
-ccache --zero-stats
-PRINT_CCACHE_STATISTICS="ccache --version | head -n 1 && ccache --show-stats"
-
-if [ -n "$ANDROID_TOOLS_URL" ]; then
-  make distclean || true
-  ./autogen.sh
-  bash -c "./configure $BITCOIN_CONFIG_ALL $BITCOIN_CONFIG" || ( (cat config.log) && false)
-  make "${MAKEJOBS}" && cd src/qt && ANDROID_HOME=${ANDROID_HOME} ANDROID_NDK_HOME=${ANDROID_NDK_HOME} make apk
-  bash -c "${PRINT_CCACHE_STATISTICS}"
-  exit 0
-fi
-
-BITCOIN_CONFIG_ALL="${BITCOIN_CONFIG_ALL} --enable-external-signer --prefix=$BASE_OUTDIR"
-
-if [ -n "$CONFIG_SHELL" ]; then
-  "$CONFIG_SHELL" -c "./autogen.sh"
-else
-  ./autogen.sh
-fi
-
-mkdir -p "${BASE_BUILD_DIR}"
-cd "${BASE_BUILD_DIR}"
-
-bash -c "${BASE_ROOT_DIR}/configure --cache-file=config.cache $BITCOIN_CONFIG_ALL $BITCOIN_CONFIG" || ( (cat config.log) && false)
-
-make distdir VERSION="$HOST"
-
-cd "${BASE_BUILD_DIR}/bitcoin-$HOST"
-
-bash -c "./configure --cache-file=../config.cache $BITCOIN_CONFIG_ALL $BITCOIN_CONFIG" || ( (cat config.log) && false)
-
-if [[ "${RUN_TIDY}" == "true" ]]; then
-  MAYBE_BEAR="bear --config src/.bear-tidy-config"
-  MAYBE_TOKEN="--"
-fi
-
-bash -c "${MAYBE_BEAR} ${MAYBE_TOKEN} make $MAKEJOBS $GOAL" || ( echo "Build failure. Verbose build follows." && make "$GOAL" V=1 ; false )
-
-bash -c "${PRINT_CCACHE_STATISTICS}"
-du -sh "${DEPENDS_DIR}"/*/
-du -sh "${PREVIOUS_RELEASES_DIR}"
-
 if [[ $HOST = *-mingw32 ]]; then
   # Generate all binaries, so that they can be wrapped
-  make "$MAKEJOBS" -C src/secp256k1 VERBOSE=1
-  make "$MAKEJOBS" -C src minisketch/test.exe VERBOSE=1
-  "${BASE_ROOT_DIR}/ci/test/wrap-wine.sh"
+  CI_EXEC make "$MAKEJOBS" -C src/secp256k1 VERBOSE=1
+  CI_EXEC make "$MAKEJOBS" -C src minisketch/test.exe VERBOSE=1
+  CI_EXEC "${BASE_ROOT_DIR}/ci/test/wrap-wine.sh"
+fi
+
+if [ -n "$QEMU_USER_CMD" ]; then
+  # Generate all binaries, so that they can be wrapped
+  CI_EXEC make "$MAKEJOBS" -C src/secp256k1 VERBOSE=1
+  CI_EXEC make "$MAKEJOBS" -C src minisketch/test VERBOSE=1
+  CI_EXEC "${BASE_ROOT_DIR}/ci/test/wrap-qemu.sh"
 fi
 
 if [ -n "$USE_VALGRIND" ]; then
-  "${BASE_ROOT_DIR}/ci/test/wrap-valgrind.sh"
+  CI_EXEC "${BASE_ROOT_DIR}/ci/test/wrap-valgrind.sh"
 fi
 
 if [ "$RUN_UNIT_TESTS" = "true" ]; then
-  bash -c "${TEST_RUNNER_ENV} DIR_UNIT_TEST_DATA=${DIR_UNIT_TEST_DATA} LD_LIBRARY_PATH=${DEPENDS_DIR}/${HOST}/lib make $MAKEJOBS check VERBOSE=1"
+  CI_EXEC "${TEST_RUNNER_ENV}" DIR_UNIT_TEST_DATA="${DIR_UNIT_TEST_DATA}" LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" make "$MAKEJOBS" check VERBOSE=1
 fi
 
 if [ "$RUN_UNIT_TESTS_SEQUENTIAL" = "true" ]; then
-  bash -c "${TEST_RUNNER_ENV} DIR_UNIT_TEST_DATA=${DIR_UNIT_TEST_DATA} LD_LIBRARY_PATH=${DEPENDS_DIR}/${HOST}/lib ${BASE_OUTDIR}/bin/test_bitcoin --catch_system_errors=no -l test_suite"
+  CI_EXEC "${TEST_RUNNER_ENV}" DIR_UNIT_TEST_DATA="${DIR_UNIT_TEST_DATA}" LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" "${BASE_OUTDIR}/bin/test_bitcoin" --catch_system_errors=no -l test_suite
 fi
 
 if [ "$RUN_FUNCTIONAL_TESTS" = "true" ]; then
-  bash -c "LD_LIBRARY_PATH=${DEPENDS_DIR}/${HOST}/lib ${TEST_RUNNER_ENV} test/functional/test_runner.py --ci $MAKEJOBS --tmpdirprefix ${BASE_SCRATCH_DIR}/test_runner/ --ansi --combinedlogslen=99999999 --timeout-factor=${TEST_RUNNER_TIMEOUT_FACTOR} ${TEST_RUNNER_EXTRA} --quiet --failfast"
+  CI_EXEC LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" "${TEST_RUNNER_ENV}" test/functional/test_runner.py --ci "$MAKEJOBS" --tmpdirprefix "${BASE_SCRATCH_DIR}/test_runner/" --ansi --combinedlogslen=99999999 --timeout-factor="${TEST_RUNNER_TIMEOUT_FACTOR}" "${TEST_RUNNER_EXTRA}" --quiet --failfast
 fi
 
 if [ "${RUN_TIDY}" = "true" ]; then
-  cmake -B /tidy-build -DLLVM_DIR=/usr/lib/llvm-"${TIDY_LLVM_V}"/cmake -DCMAKE_BUILD_TYPE=Release -S "${BASE_ROOT_DIR}"/contrib/devtools/bitcoin-tidy
-  cmake --build /tidy-build "$MAKEJOBS"
-  cmake --build /tidy-build --target bitcoin-tidy-tests "$MAKEJOBS"
-
   set -eo pipefail
-  cd "${BASE_BUILD_DIR}/bitcoin-$HOST/src/"
-  ( run-clang-tidy-"${TIDY_LLVM_V}" -quiet -load="/tidy-build/libbitcoin-tidy.so" "${MAKEJOBS}" ) | grep -C5 "error"
-  # Filter out files by regex here, because regex may not be
-  # accepted in src/.bear-tidy-config
-  # Filter out:
-  # * qt qrc and moc generated files
-  jq 'map(select(.file | test("src/qt/qrc_.*\\.cpp$|/moc_.*\\.cpp$") | not))' ../compile_commands.json > tmp.json
-  mv tmp.json ../compile_commands.json
-  cd "${BASE_BUILD_DIR}/bitcoin-$HOST/"
-  python3 "/include-what-you-use/iwyu_tool.py" \
-           -p . "${MAKEJOBS}" \
-           -- -Xiwyu --cxx17ns -Xiwyu --mapping_file="${BASE_BUILD_DIR}/bitcoin-$HOST/contrib/devtools/iwyu/bitcoin.core.imp" \
-           -Xiwyu --max_line_length=160 \
-           2>&1 | tee /tmp/iwyu_ci.out
-  cd "${BASE_ROOT_DIR}/src"
-  python3 "/include-what-you-use/fix_includes.py" --nosafe_headers < /tmp/iwyu_ci.out
-  git --no-pager diff
+  export P_CI_DIR="${BASE_BUILD_DIR}/bitcoin-$HOST/src/"
+  ( CI_EXEC run-clang-tidy-16 -quiet "${MAKEJOBS}" ) | grep -C5 "error"
+  export P_CI_DIR="${BASE_BUILD_DIR}/bitcoin-$HOST/"
+  CI_EXEC "python3 ${DIR_IWYU}/include-what-you-use/iwyu_tool.py"\
+          " src/common/init.cpp"\
+          " src/common/url.cpp"\
+          " src/compat"\
+          " src/dbwrapper.cpp"\
+          " src/init"\
+          " src/kernel"\
+          " src/node/chainstate.cpp"\
+          " src/node/chainstatemanager_args.cpp"\
+          " src/node/mempool_args.cpp"\
+          " src/node/minisketchwrapper.cpp"\
+          " src/node/utxo_snapshot.cpp"\
+          " src/node/validation_cache_args.cpp"\
+          " src/policy/feerate.cpp"\
+          " src/policy/packages.cpp"\
+          " src/policy/settings.cpp"\
+          " src/primitives/transaction.cpp"\
+          " src/random.cpp"\
+          " src/rpc/fees.cpp"\
+          " src/rpc/signmessage.cpp"\
+          " src/test/fuzz/string.cpp"\
+          " src/test/fuzz/txorphan.cpp"\
+          " src/test/fuzz/util/"\
+          " src/test/util/coins.cpp"\
+          " src/uint256.cpp"\
+          " src/util/bip32.cpp"\
+          " src/util/bytevectorhash.cpp"\
+          " src/util/check.cpp"\
+          " src/util/error.cpp"\
+          " src/util/exception.cpp"\
+          " src/util/getuniquepath.cpp"\
+          " src/util/hasher.cpp"\
+          " src/util/message.cpp"\
+          " src/util/moneystr.cpp"\
+          " src/util/serfloat.cpp"\
+          " src/util/spanparsing.cpp"\
+          " src/util/strencodings.cpp"\
+          " src/util/string.cpp"\
+          " src/util/syserror.cpp"\
+          " src/util/threadinterrupt.cpp"\
+          " src/zmq"\
+          " -p . ${MAKEJOBS}"\
+          " -- -Xiwyu --cxx17ns -Xiwyu --mapping_file=${BASE_BUILD_DIR}/bitcoin-$HOST/contrib/devtools/iwyu/bitcoin.core.imp"\
+          " |& tee /tmp/iwyu_ci.out"
+  export P_CI_DIR="${BASE_ROOT_DIR}/src"
+  CI_EXEC "python3 ${DIR_IWYU}/include-what-you-use/fix_includes.py --nosafe_headers < /tmp/iwyu_ci.out"
+  CI_EXEC "git --no-pager diff"
+fi
+
+if [ "$RUN_SECURITY_TESTS" = "true" ]; then
+  CI_EXEC make test-security-check
 fi
 
 if [ "$RUN_FUZZ_TESTS" = "true" ]; then
-  bash -c "LD_LIBRARY_PATH=${DEPENDS_DIR}/${HOST}/lib test/fuzz/test_runner.py ${FUZZ_TESTS_CONFIG} $MAKEJOBS -l DEBUG ${DIR_FUZZ_IN} --empty_min_time=60"
+  CI_EXEC LD_LIBRARY_PATH="${DEPENDS_DIR}/${HOST}/lib" test/fuzz/test_runner.py "${FUZZ_TESTS_CONFIG}" "$MAKEJOBS" -l DEBUG "${DIR_FUZZ_IN}"
+fi
+
+if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
+  echo "Stop and remove CI container by ID"
+  docker container kill "${CI_CONTAINER_ID}"
 fi

ci/test/wrap-qemu.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2018-2022 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+export LC_ALL=C.UTF-8
+
+for b_name in {"${BASE_OUTDIR}/bin"/*,src/secp256k1/*tests,src/minisketch/test{,-verify},src/univalue/{test_json,unitester,object}}; do
+    # shellcheck disable=SC2044
+    for b in $(find "${BASE_ROOT_DIR}" -executable -type f -name "$(basename "$b_name")"); do
+      echo "Wrap $b ..."
+      mv "$b" "${b}_orig"
+      echo '#!/usr/bin/env bash' > "$b"
+      echo "$QEMU_USER_CMD \"${b}_orig\" \"\$@\"" >> "$b"
+      chmod +x "$b"
+    done
+done

ci/test/wrapped-cl.bat

@@ -0,0 +1 @@
+ccache cl %*

ci/test_imagefile

@@ -1,9 +1,3 @@
-# Copyright (c) The Bitcoin Core developers
-# Distributed under the MIT software license, see the accompanying
-# file COPYING or https://opensource.org/license/mit/.
-
-# See ci/README.md for usage.
-
 ARG CI_IMAGE_NAME_TAG
 FROM ${CI_IMAGE_NAME_TAG}
 
@@ -11,6 +5,6 @@ ARG FILE_ENV
 ENV FILE_ENV=${FILE_ENV}
 
 COPY ./ci/retry/retry /usr/bin/retry
-COPY ./ci/test/00_setup_env.sh ./${FILE_ENV} ./ci/test/01_base_install.sh /ci_container_base/ci/test/
+COPY ./ci/test/00_setup_env.sh ./${FILE_ENV} ./ci/test/01_base_install.sh /ci_base_install/ci/test/
 
-RUN ["bash", "-c", "cd /ci_container_base/ && set -o errexit && source ./ci/test/00_setup_env.sh && ./ci/test/01_base_install.sh"]
+RUN ["bash", "-c", "cd /ci_base_install/ && set -o errexit && source ./ci/test/00_setup_env.sh && ./ci/test/01_base_install.sh"]

ci/test_run_all.sh

@@ -1,11 +1,13 @@
 #!/usr/bin/env bash
 #
-# Copyright (c) 2019-present The Bitcoin Core developers
+# Copyright (c) 2019-2020 The Bitcoin Core developers
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 export LC_ALL=C.UTF-8
 
 set -o errexit; source ./ci/test/00_setup_env.sh
-set -o errexit
-"./ci/test/02_run_container.sh"
+set -o errexit; source ./ci/test/04_install.sh
+set -o errexit; source ./ci/test/05_before_script.sh
+set -o errexit; source ./ci/test/06_script_a.sh
+set -o errexit; source ./ci/test/06_script_b.sh

configure.ac

@@ -1,10 +1,10 @@
 AC_PREREQ([2.69])
-define(_CLIENT_VERSION_MAJOR, 26)
+define(_CLIENT_VERSION_MAJOR, 25)
 define(_CLIENT_VERSION_MINOR, 2)
 define(_CLIENT_VERSION_BUILD, 0)
-define(_CLIENT_VERSION_RC, 0)
+define(_CLIENT_VERSION_RC, 1)
 define(_CLIENT_VERSION_IS_RELEASE, true)
-define(_COPYRIGHT_YEAR, 2024)
+define(_COPYRIGHT_YEAR, 2023)
 define(_COPYRIGHT_HOLDERS,[The %s developers])
 define(_COPYRIGHT_HOLDERS_SUBSTITUTION,[[Bitcoin Core]])
 AC_INIT([Bitcoin Core],m4_join([.], _CLIENT_VERSION_MAJOR, _CLIENT_VERSION_MINOR, _CLIENT_VERSION_BUILD)m4_if(_CLIENT_VERSION_RC, [0], [], [rc]_CLIENT_VERSION_RC),[https://github.com/bitcoin/bitcoin/issues],[bitcoin],[https://bitcoincore.org/])
@@ -70,12 +70,11 @@ else
 fi
 AC_PROG_CXX
 
-dnl libtool overrides
+dnl By default, libtool for mingw refuses to link static libs into a dll for
+dnl fear of mixing pic/non-pic objects, and import/export complications. Since
+dnl we have those under control, re-enable that functionality.
 case $host in
   *mingw*)
-     dnl By default, libtool for mingw refuses to link static libs into a dll for
-     dnl fear of mixing pic/non-pic objects, and import/export complications. Since
-     dnl we have those under control, re-enable that functionality.
      lt_cv_deplibs_check_method="pass_all"
 
      dnl Remove unwanted -DDLL_EXPORT from these variables.
@@ -84,18 +83,14 @@ case $host in
      lt_cv_prog_compiler_pic="-DPIC"
      lt_cv_prog_compiler_pic_CXX="-DPIC"
   ;;
-  *darwin*)
-     dnl Because it prints a verbose warning, lld fails the following check
-     dnl for "-Wl,-single_module" from libtool.m4:
-     dnl   # If there is a non-empty error log, and "single_module"
-     dnl   # appears in it, assume the flag caused a linker warning
-     dnl "-single_module" works fine on ld64 and lld, so just bypass the test.
-     dnl Failure to set this to "yes" causes libtool to use a very broken
-     dnl link-line for shared libs.
-     lt_cv_apple_cc_single_mod="yes"
-  ;;
 esac
 
+AC_ARG_WITH([seccomp],
+  [AS_HELP_STRING([--with-seccomp],
+  [enable experimental syscall sandbox feature (-sandbox), default is yes if seccomp-bpf is detected under Linux x86_64])],
+  [seccomp_found=$withval],
+  [seccomp_found=auto])
+
 AC_ARG_ENABLE([c++20],
   [AS_HELP_STRING([--enable-c++20],
   [enable compilation in c++20 mode (disabled by default)])],
@@ -109,6 +104,9 @@ else
 AX_CXX_COMPILE_STDCXX([20], [noext], [mandatory])
 fi
 
+dnl check if additional link flags are required for std::filesystem
+CHECK_FILESYSTEM
+
 dnl Unless the user specified OBJCXX, force it to be the same as CXX. This ensures
 dnl that we get the same -std flags for both.
 m4_ifdef([AC_PROG_OBJCXX],[
@@ -128,8 +126,8 @@ AC_PATH_TOOL([AR], [ar])
 AC_PATH_TOOL([GCOV], [gcov])
 AC_PATH_TOOL([LLVM_COV], [llvm-cov])
 AC_PATH_PROG([LCOV], [lcov])
-dnl Python 3.8 is specified in .python-version and should be used if available, see doc/dependencies.md
-AC_PATH_PROGS([PYTHON], [python3.8 python3.9 python3.10 python3.11 python3.12 python3 python])
+dnl Python 3.7 is specified in .python-version and should be used if available, see doc/dependencies.md
+AC_PATH_PROGS([PYTHON], [python3.7 python3.8 python3.9 python3.10 python3.11 python3.12 python3 python])
 AC_PATH_PROG([GENHTML], [genhtml])
 AC_PATH_PROG([GIT], [git])
 AC_PATH_PROG([CCACHE], [ccache])
@@ -238,10 +236,10 @@ dnl May be useful if warnings from external headers clutter the build output
 dnl too much, so that it becomes difficult to spot Bitcoin Core warnings
 dnl or if they cause a build failure with --enable-werror.
 AC_ARG_ENABLE([suppress-external-warnings],
-  [AS_HELP_STRING([--disable-suppress-external-warnings],
-                  [Do not suppress warnings from external headers (default is to suppress)])],
+  [AS_HELP_STRING([--enable-suppress-external-warnings],
+                  [Suppress warnings from external headers (default is no)])],
   [suppress_external_warnings=$enableval],
-  [suppress_external_warnings=yes])
+  [suppress_external_warnings=no])
 
 AC_ARG_ENABLE([lcov],
   [AS_HELP_STRING([--enable-lcov],
@@ -427,6 +425,12 @@ if test "$enable_werror" = "yes"; then
     AC_MSG_ERROR([enable-werror set but -Werror is not usable])
   fi
   ERROR_CXXFLAGS=$CXXFLAG_WERROR
+
+  dnl -Wreturn-type is broken in GCC for MinGW-w64.
+  dnl https://sourceforge.net/p/mingw-w64/bugs/306/
+  AX_CHECK_COMPILE_FLAG([-Werror=return-type], [], [ERROR_CXXFLAGS="$ERROR_CXXFLAGS -Wno-error=return-type"], [$CXXFLAG_WERROR],
+                        [AC_LANG_SOURCE([[#include <cassert>
+                                          int f(){ assert(false); }]])])
 fi
 
 if test "$CXXFLAGS_overridden" = "no"; then
@@ -471,12 +475,6 @@ fi
 dnl Don't allow extended (non-ASCII) symbols in identifiers. This is easier for code review.
 AX_CHECK_COMPILE_FLAG([-fno-extended-identifiers], [CORE_CXXFLAGS="$CORE_CXXFLAGS -fno-extended-identifiers"], [], [$CXXFLAG_WERROR])
 
-dnl Currently all versions of gcc are subject to a class of bugs, see the
-dnl gccbug_90348 test case (only reproduces on GCC 11 and earlier) and
-dnl https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111843. To work around that, set
-dnl -fstack-reuse=none for all gcc builds. (Only gcc understands this flag)
-AX_CHECK_COMPILE_FLAG([-fstack-reuse=none], [CORE_CXXFLAGS="$CORE_CXXFLAGS -fstack-reuse=none"])
-
 enable_arm_crc=no
 enable_arm_shani=no
 enable_sse42=no
@@ -584,7 +582,7 @@ AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
 CXXFLAGS="$TEMP_CXXFLAGS"
 
 # ARM
-AX_CHECK_COMPILE_FLAG([-march=armv8-a+crc+crypto], [ARM_CRC_CXXFLAGS="-march=armv8-a+crc+crypto"], [], [$CXXFLAG_WERROR])
+AX_CHECK_COMPILE_FLAG([-march=armv8-a+crc], [ARM_CRC_CXXFLAGS="-march=armv8-a+crc"], [], [$CXXFLAG_WERROR])
 AX_CHECK_COMPILE_FLAG([-march=armv8-a+crypto], [ARM_SHANI_CXXFLAGS="-march=armv8-a+crypto"], [], [$CXXFLAG_WERROR])
 
 TEMP_CXXFLAGS="$CXXFLAGS"
@@ -714,8 +712,6 @@ case $host in
        AC_MSG_ERROR([windres not found])
      fi
 
-     CORE_CPPFLAGS="$CORE_CPPFLAGS -DSECP256K1_STATIC"
-
      CORE_CPPFLAGS="$CORE_CPPFLAGS -D_MT -DWIN32 -D_WINDOWS -D_WIN32_WINNT=0x0601 -D_WIN32_IE=0x0501 -DWIN32_LEAN_AND_MEAN"
      dnl Prevent the definition of min/max macros.
      dnl We always want to use the standard library.
@@ -731,10 +727,6 @@ case $host in
 
      dnl We require Windows 7 (NT 6.1) or later
      AX_CHECK_LINK_FLAG([-Wl,--major-subsystem-version -Wl,6 -Wl,--minor-subsystem-version -Wl,1], [CORE_LDFLAGS="$CORE_LDFLAGS -Wl,--major-subsystem-version -Wl,6 -Wl,--minor-subsystem-version -Wl,1"], [], [$LDFLAG_WERROR])
-
-     dnl Avoid the use of aligned vector instructions when building for Windows.
-     dnl See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412.
-     AX_CHECK_COMPILE_FLAG([-Wa,-muse-unaligned-vector-move], [CORE_CXXFLAGS="$CORE_CXXFLAGS -Wa,-muse-unaligned-vector-move"], [], [$CXXFLAG_WERROR])
      ;;
   *darwin*)
      TARGET_OS=darwin
@@ -755,7 +747,7 @@ case $host in
          dnl option to system-ify all /usr/local/include paths without adding it to the list
          dnl of search paths in case it's not already there.
          if test "$suppress_external_warnings" != "no"; then
-           AX_CHECK_PREPROC_FLAG([-Xclang -internal-isystem -Xclang /usr/local/include/], [CORE_CPPFLAGS="$CORE_CPPFLAGS -Xclang -internal-isystem -Xclang /usr/local/include/"], [], [$CXXFLAG_WERROR])
+           AX_CHECK_PREPROC_FLAG([-Xclang -internal-isystem/usr/local/include], [CORE_CPPFLAGS="$CORE_CPPFLAGS -Xclang -internal-isystem/usr/local/include"], [], [$CXXFLAG_WERROR])
          fi
 
          if test "$use_bdb" != "no" && $BREW list --versions berkeley-db@4 >/dev/null && test "$BDB_CFLAGS" = "" && test "$BDB_LIBS" = ""; then
@@ -805,7 +797,7 @@ case $host in
            AC_PATH_TOOL([DSYMUTIL], [dsymutil], [dsymutil])
            AC_PATH_TOOL([INSTALL_NAME_TOOL], [install_name_tool], [install_name_tool])
            AC_PATH_TOOL([OTOOL], [otool], [otool])
-           AC_PATH_PROG([ZIP], [zip], [zip])
+           AC_PATH_PROGS([XORRISOFS], [xorrisofs], [xorrisofs])
 
            dnl libtool will try to strip the static lib, which is a problem for
            dnl cross-builds because strip attempts to call a hard-coded ld,
@@ -925,6 +917,8 @@ if test "$ac_cv_sys_large_files" != "" &&
   CORE_CPPFLAGS="$CORE_CPPFLAGS -D_LARGE_FILES=$ac_cv_sys_large_files"
 fi
 
+AC_SEARCH_LIBS([clock_gettime],[rt])
+
 if test "$enable_gprof" = "yes"; then
     dnl -pg is incompatible with -pie. Since hardening and profiling together doesn't make sense,
     dnl we simply make them mutually exclusive here. Additionally, hardened toolchains may force
@@ -947,6 +941,12 @@ if test "$TARGET_OS" != "windows"; then
   AX_CHECK_COMPILE_FLAG([-fPIC], [PIC_FLAGS="-fPIC"])
 fi
 
+dnl Versions of gcc prior to 12.1 (commit
+dnl https://github.com/gcc-mirror/gcc/commit/551aa75778a4c5165d9533cd447c8fc822f583e1)
+dnl are subject to a bug, see the gccbug_90348 test case and
+dnl https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90348. To work around that, set
+dnl -fstack-reuse=none for all gcc builds. (Only gcc understands this flag)
+AX_CHECK_COMPILE_FLAG([-fstack-reuse=none], [HARDENED_CXXFLAGS="$HARDENED_CXXFLAGS -fstack-reuse=none"])
 if test "$use_hardening" != "no"; then
   use_hardening=yes
   AX_CHECK_COMPILE_FLAG([-Wstack-protector], [HARDENED_CXXFLAGS="$HARDENED_CXXFLAGS -Wstack-protector"])
@@ -956,8 +956,7 @@ if test "$use_hardening" != "no"; then
 
   case $host in
     *mingw*)
-      dnl stack-clash-protection doesn't compile with GCC 10 and earlier.
-      dnl In any case, it is a no-op for Windows.
+      dnl stack-clash-protection doesn't currently work, and likely should just be skipped for Windows.
       dnl See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90458 for more details.
       ;;
     *)
@@ -965,11 +964,6 @@ if test "$use_hardening" != "no"; then
       ;;
   esac
 
-  case $host in
-    *aarch64*)
-      AX_CHECK_COMPILE_FLAG([-mbranch-protection=bti], [HARDENED_CXXFLAGS="$HARDENED_CXXFLAGS -mbranch-protection=bti"])
-    ;;
-  esac
 
   dnl When enable_debug is yes, all optimizations are disabled.
   dnl However, FORTIFY_SOURCE requires that there is some level of optimization, otherwise it does nothing and just creates a compiler warning.
@@ -1006,7 +1000,6 @@ if test "$TARGET_OS" = "darwin"; then
   AX_CHECK_LINK_FLAG([-Wl,-dead_strip], [CORE_LDFLAGS="$CORE_LDFLAGS -Wl,-dead_strip"], [], [$LDFLAG_WERROR])
   AX_CHECK_LINK_FLAG([-Wl,-dead_strip_dylibs], [CORE_LDFLAGS="$CORE_LDFLAGS -Wl,-dead_strip_dylibs"], [], [$LDFLAG_WERROR])
   AX_CHECK_LINK_FLAG([-Wl,-bind_at_load], [HARDENED_LDFLAGS="$HARDENED_LDFLAGS -Wl,-bind_at_load"], [], [$LDFLAG_WERROR])
-  AX_CHECK_LINK_FLAG([-Wl,-fixup_chains], [HARDENED_LDFLAGS="$HARDENED_LDFLAGS -Wl,-fixup_chains"], [], [$LDFLAG_WERROR])
 fi
 
 AC_CHECK_HEADERS([endian.h sys/endian.h byteswap.h sys/select.h sys/prctl.h sys/sysctl.h vm/vm_param.h sys/vmmeter.h sys/resources.h])
@@ -1071,7 +1064,7 @@ AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <malloc.h>]],
 dnl Check for posix_fallocate
 AC_MSG_CHECKING([for posix_fallocate])
 AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
-                   // same as in src/util/fs_helpers.cpp
+                   // same as in src/util/system.cpp
                    #ifdef __linux__
                    #ifdef _POSIX_C_SOURCE
                    #undef _POSIX_C_SOURCE
@@ -1169,16 +1162,17 @@ AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <ctime>]],
 )
 
 dnl Check for different ways of gathering OS randomness
-AC_MSG_CHECKING([for Linux getrandom function])
-AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
-  #include <sys/random.h>]],
- [[ getrandom(nullptr, 32, 0); ]])],
- [ AC_MSG_RESULT([yes]); AC_DEFINE([HAVE_GETRANDOM], [1], [Define this symbol if the Linux getrandom function call is available]) ],
+AC_MSG_CHECKING([for Linux getrandom syscall])
+AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <unistd.h>
+  #include <sys/syscall.h>
+  #include <linux/random.h>]],
+ [[ syscall(SYS_getrandom, nullptr, 32, 0); ]])],
+ [ AC_MSG_RESULT([yes]); AC_DEFINE([HAVE_SYS_GETRANDOM], [1], [Define this symbol if the Linux getrandom system call is available]) ],
  [ AC_MSG_RESULT([no])]
 )
 
-AC_MSG_CHECKING([for getentropy via sys/random.h])
-AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
+AC_MSG_CHECKING([for getentropy via random.h])
+AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <unistd.h>
  #include <sys/random.h>]],
  [[ getentropy(nullptr, 32) ]])],
  [ AC_MSG_RESULT([yes]); AC_DEFINE([HAVE_GETENTROPY_RAND], [1], [Define this symbol if the BSD getentropy system call is available with sys/random.h]) ],
@@ -1407,9 +1401,7 @@ if test "$use_usdt" != "no"; then
   AC_COMPILE_IFELSE([
     AC_LANG_PROGRAM(
       [#include <sys/sdt.h>],
-      [DTRACE_PROBE(context, event);
-       int a, b, c, d, e, f, g;
-       DTRACE_PROBE7(context, event, a, b, c, d, e, f, g);]
+      [DTRACE_PROBE("context", "event");]
     )],
     [AC_MSG_RESULT([yes]); AC_DEFINE([ENABLE_TRACING], [1], [Define to 1 to enable tracepoints for Userspace, Statically Defined Tracing])],
     [AC_MSG_RESULT([no]); use_usdt=no;]
@@ -1538,6 +1530,36 @@ if test "$use_external_signer" != "no"; then
 fi
 AM_CONDITIONAL([ENABLE_EXTERNAL_SIGNER], [test "$use_external_signer" = "yes"])
 
+dnl Do not compile with syscall sandbox support when compiling under the sanitizers.
+dnl The sanitizers introduce use of syscalls that are not typically used in bitcoind
+dnl (such as execve when the sanitizers execute llvm-symbolizer).
+if test "$use_sanitizers" != ""; then
+  AC_MSG_WARN([Specifying --with-sanitizers forces --without-seccomp since the sanitizers introduce use of syscalls not allowed by the bitcoind syscall sandbox (-sandbox=<mode>).])
+  seccomp_found=no
+fi
+if test "$seccomp_found" != "no"; then
+  AC_MSG_CHECKING([for seccomp-bpf (Linux x86-64)])
+  AC_PREPROC_IFELSE([AC_LANG_PROGRAM([[
+      @%:@include <linux/seccomp.h>
+    ]], [[
+      #if !defined(__x86_64__)
+      #  error Syscall sandbox is an experimental feature currently available only under Linux x86-64.
+      #endif
+    ]])],[
+      AC_MSG_RESULT([yes])
+      seccomp_found="yes"
+      AC_DEFINE([USE_SYSCALL_SANDBOX], [1], [Define this symbol to build with syscall sandbox support.])
+    ],[
+      AC_MSG_RESULT([no])
+      seccomp_found="no"
+  ])
+fi
+dnl Currently only enable -sandbox=<mode> feature if seccomp is found.
+dnl In the future, sandboxing could be also be supported with other
+dnl sandboxing mechanisms besides seccomp.
+use_syscall_sandbox=$seccomp_found
+AM_CONDITIONAL([ENABLE_SYSCALL_SANDBOX], [test "$use_syscall_sandbox" != "no"])
+
 dnl Check for reduced exports
 if test "$use_reduce_exports" = "yes"; then
   AX_CHECK_COMPILE_FLAG([-fvisibility=hidden], [CORE_CXXFLAGS="$CORE_CXXFLAGS -fvisibility=hidden"],
@@ -1596,9 +1618,12 @@ dnl ZMQ check
 
 if test "$use_zmq" = "yes"; then
   PKG_CHECK_MODULES([ZMQ], [libzmq >= 4],
-    AC_DEFINE([ENABLE_ZMQ], [1], [Define this symbol to enable ZMQ functions]),
-    [AC_MSG_WARN([libzmq version 4.x or greater not found, disabling])
+    AC_DEFINE([ENABLE_ZMQ], [1], [Define to 1 to enable ZMQ functions]),
+    [AC_DEFINE([ENABLE_ZMQ], [0], [Define to 1 to enable ZMQ functions])
+    AC_MSG_WARN([libzmq version 4.x or greater not found, disabling])
     use_zmq=no])
+else
+  AC_DEFINE_UNQUOTED([ENABLE_ZMQ], [0], [Define to 1 to enable ZMQ functions])
 fi
 
 if test "$use_zmq" = "yes"; then
@@ -1610,8 +1635,6 @@ if test "$use_zmq" = "yes"; then
   esac
 fi
 
-AM_CONDITIONAL([ENABLE_ZMQ], [test "$use_zmq" = "yes"])
-
 dnl libmultiprocess library check
 
 libmultiprocess_found=no
@@ -1806,6 +1829,8 @@ if test "$bitcoin_enable_qt" != "no"; then
   fi
 fi
 
+AM_CONDITIONAL([ENABLE_ZMQ], [test "$use_zmq" = "yes"])
+
 AC_MSG_CHECKING([whether to build test_bitcoin])
 if test "$use_tests" = "yes"; then
   if test "$enable_fuzz" = "yes"; then
@@ -1942,6 +1967,7 @@ AC_CONFIG_FILES([contrib/devtools/split-debug.sh],[chmod +x contrib/devtools/spl
 AM_COND_IF([HAVE_DOXYGEN], [AC_CONFIG_FILES([doc/Doxyfile])])
 AC_CONFIG_LINKS([contrib/devtools/iwyu/bitcoin.core.imp:contrib/devtools/iwyu/bitcoin.core.imp])
 AC_CONFIG_LINKS([contrib/filter-lcov.py:contrib/filter-lcov.py])
+AC_CONFIG_LINKS([contrib/macdeploy/background.tiff:contrib/macdeploy/background.tiff])
 AC_CONFIG_LINKS([src/.bear-tidy-config:src/.bear-tidy-config])
 AC_CONFIG_LINKS([src/.clang-tidy:src/.clang-tidy])
 AC_CONFIG_LINKS([test/functional/test_runner.py:test/functional/test_runner.py])
@@ -1976,6 +2002,7 @@ echo
 echo "Options used to compile and link:"
 echo "  external signer = $use_external_signer"
 echo "  multiprocess    = $build_multiprocess"
+echo "  with experimental syscall sandbox support = $use_syscall_sandbox"
 echo "  with libs       = $build_bitcoin_libs"
 echo "  with wallet     = $enable_wallet"
 if test "$enable_wallet" != "no"; then

contrib/README.md

@@ -35,7 +35,7 @@ Test and Verify Tools
 ### [TestGen](/contrib/testgen) ###
 Utilities to generate test vectors for the data-driven Bitcoin tests.
 
-### [Verify-Binaries](/contrib/verify-binaries) ###
+### [Verify Binaries](/contrib/verifybinaries) ###
 This script attempts to download and verify the signature file SHA256SUMS.asc from bitcoin.org.
 
 Command Line Tools

contrib/devtools/README.md

@@ -83,23 +83,13 @@ A small script to automatically create manpages in ../../doc/man by running the
 This requires help2man which can be found at: https://www.gnu.org/software/help2man/
 
 With in-tree builds this tool can be run from any directory within the
-repository. To use this tool with out-of-tree builds set `BUILDDIR`. For
+repostitory. To use this tool with out-of-tree builds set `BUILDDIR`. For
 example:
 
 ```bash
 BUILDDIR=$PWD/build contrib/devtools/gen-manpages.py
 ```
 
-headerssync-params.py
-=====================
-
-A script to generate optimal parameters for the headerssync module (src/headerssync.cpp). It takes no command-line
-options, as all its configuration is set at the top of the file. It runs many times faster inside PyPy. Invocation:
-
-```bash
-pypy3 contrib/devtools/headerssync-params.py
-```
-
 gen-bitcoin-conf.sh
 ===================
 

contrib/devtools/bitcoin-tidy/CMakeLists.txt

@@ -1,56 +0,0 @@
-cmake_minimum_required(VERSION 3.9)
-
-project(bitcoin-tidy VERSION 1.0.0 DESCRIPTION "clang-tidy checks for Bitcoin Core")
-
-include(GNUInstallDirs)
-
-set(CMAKE_CXX_STANDARD 17)
-set(CMAKE_CXX_STANDARD_REQUIRED True)
-set(CMAKE_CXX_EXTENSIONS False)
-
-# TODO: Figure out how to avoid the terminfo check
-find_package(LLVM REQUIRED CONFIG)
-find_program(CLANG_TIDY_EXE NAMES "clang-tidy-${LLVM_VERSION_MAJOR}" "clang-tidy" HINTS ${LLVM_TOOLS_BINARY_DIR})
-message(STATUS "Found LLVM ${LLVM_PACKAGE_VERSION}")
-message(STATUS "Found clang-tidy: ${CLANG_TIDY_EXE}")
-
-add_library(bitcoin-tidy MODULE bitcoin-tidy.cpp logprintf.cpp)
-target_include_directories(bitcoin-tidy SYSTEM PRIVATE ${LLVM_INCLUDE_DIRS})
-
-# Disable RTTI and exceptions as necessary
-if (MSVC)
-    target_compile_options(bitcoin-tidy PRIVATE /GR-)
-else()
-    target_compile_options(bitcoin-tidy PRIVATE -fno-rtti)
-    target_compile_options(bitcoin-tidy PRIVATE -fno-exceptions)
-endif()
-
-if(CMAKE_HOST_APPLE)
-    # ld64 expects no undefined symbols by default
-    target_link_options(bitcoin-tidy PRIVATE -Wl,-flat_namespace)
-    target_link_options(bitcoin-tidy PRIVATE -Wl,-undefined -Wl,suppress)
-endif()
-
-# Add warnings
-if (MSVC)
-    target_compile_options(bitcoin-tidy PRIVATE /W4)
-else()
-    target_compile_options(bitcoin-tidy PRIVATE -Wall)
-    target_compile_options(bitcoin-tidy PRIVATE -Wextra)
-endif()
-
-if(CMAKE_VERSION VERSION_LESS 3.27)
-    set(CLANG_TIDY_COMMAND "${CLANG_TIDY_EXE}" "--load=${CMAKE_BINARY_DIR}/${CMAKE_SHARED_MODULE_PREFIX}bitcoin-tidy${CMAKE_SHARED_MODULE_SUFFIX}" "-checks=-*,bitcoin-*")
-else()
-    # CLANG_TIDY_COMMAND supports generator expressions as of 3.27
-    set(CLANG_TIDY_COMMAND "${CLANG_TIDY_EXE}" "--load=$<TARGET_FILE:bitcoin-tidy>" "-checks=-*,bitcoin-*")
-endif()
-
-# Create a dummy library that runs clang-tidy tests as a side-effect of building
-add_library(bitcoin-tidy-tests OBJECT EXCLUDE_FROM_ALL example_logprintf.cpp)
-add_dependencies(bitcoin-tidy-tests bitcoin-tidy)
-
-set_target_properties(bitcoin-tidy-tests PROPERTIES CXX_CLANG_TIDY "${CLANG_TIDY_COMMAND}")
-
-
-install(TARGETS bitcoin-tidy LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR})

contrib/devtools/bitcoin-tidy/README

@@ -1,11 +0,0 @@
-# Bitcoin Tidy
-
-Example Usage:
-
-```bash
-cmake -S . -B build -DLLVM_DIR=$(llvm-config --cmakedir) -DCMAKE_BUILD_TYPE=Release
-
-cmake --build build -j$(nproc)
-
-cmake --build build --target bitcoin-tidy-tests -j$(nproc)
-```

contrib/devtools/bitcoin-tidy/bitcoin-tidy.cpp

@@ -1,22 +0,0 @@
-// Copyright (c) 2023 Bitcoin Developers
-// Distributed under the MIT software license, see the accompanying
-// file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-#include "logprintf.h"
-
-#include <clang-tidy/ClangTidyModule.h>
-#include <clang-tidy/ClangTidyModuleRegistry.h>
-
-class BitcoinModule final : public clang::tidy::ClangTidyModule
-{
-public:
-    void addCheckFactories(clang::tidy::ClangTidyCheckFactories& CheckFactories) override
-    {
-        CheckFactories.registerCheck<bitcoin::LogPrintfCheck>("bitcoin-unterminated-logprintf");
-    }
-};
-
-static clang::tidy::ClangTidyModuleRegistry::Add<BitcoinModule>
-    X("bitcoin-module", "Adds bitcoin checks.");
-
-volatile int BitcoinModuleAnchorSource = 0;

contrib/devtools/bitcoin-tidy/example_logprintf.cpp

@@ -1,108 +0,0 @@
-// Copyright (c) 2023 Bitcoin Developers
-// Distributed under the MIT software license, see the accompanying
-// file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-#include <string>
-
-// Test for bitcoin-unterminated-logprintf
-
-enum LogFlags {
-    NONE
-};
-
-enum Level {
-    None
-};
-
-template <typename... Args>
-static inline void LogPrintf_(const std::string& logging_function, const std::string& source_file, const int source_line, const LogFlags flag, const Level level, const char* fmt, const Args&... args)
-{
-}
-
-#define LogPrintLevel_(category, level, ...) LogPrintf_(__func__, __FILE__, __LINE__, category, level, __VA_ARGS__)
-#define LogPrintf(...) LogPrintLevel_(LogFlags::NONE, Level::None, __VA_ARGS__)
-
-#define LogPrint(category, ...) \
-    do {                        \
-        LogPrintf(__VA_ARGS__); \
-    } while (0)
-
-
-class CWallet
-{
-    std::string GetDisplayName() const
-    {
-        return "default wallet";
-    }
-
-public:
-    template <typename... Params>
-    void WalletLogPrintf(const char* fmt, Params... parameters) const
-    {
-        LogPrintf(("%s " + std::string{fmt}).c_str(), GetDisplayName(), parameters...);
-    };
-};
-
-struct ScriptPubKeyMan
-{
-    std::string GetDisplayName() const
-    {
-        return "default wallet";
-    }
-
-    template <typename... Params>
-    void WalletLogPrintf(const char* fmt, Params... parameters) const
-    {
-        LogPrintf(("%s " + std::string{fmt}).c_str(), GetDisplayName(), parameters...);
-    };
-};
-
-void good_func()
-{
-    LogPrintf("hello world!\n");
-}
-void good_func2()
-{
-    CWallet wallet;
-    wallet.WalletLogPrintf("hi\n");
-    ScriptPubKeyMan spkm;
-    spkm.WalletLogPrintf("hi\n");
-
-    const CWallet& walletref = wallet;
-    walletref.WalletLogPrintf("hi\n");
-
-    auto* walletptr = new CWallet();
-    walletptr->WalletLogPrintf("hi\n");
-    delete walletptr;
-}
-void bad_func()
-{
-    LogPrintf("hello world!");
-}
-void bad_func2()
-{
-    LogPrintf("");
-}
-void bad_func3()
-{
-    // Ending in "..." has no special meaning.
-    LogPrintf("hello world!...");
-}
-void bad_func4_ignored()
-{
-    LogPrintf("hello world!"); // NOLINT(bitcoin-unterminated-logprintf)
-}
-void bad_func5()
-{
-    CWallet wallet;
-    wallet.WalletLogPrintf("hi");
-    ScriptPubKeyMan spkm;
-    spkm.WalletLogPrintf("hi");
-
-    const CWallet& walletref = wallet;
-    walletref.WalletLogPrintf("hi");
-
-    auto* walletptr = new CWallet();
-    walletptr->WalletLogPrintf("hi");
-    delete walletptr;
-}

contrib/devtools/bitcoin-tidy/logprintf.cpp

@@ -1,60 +0,0 @@
-// Copyright (c) 2023 Bitcoin Developers
-// Distributed under the MIT software license, see the accompanying
-// file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-#include "logprintf.h"
-
-#include <clang/AST/ASTContext.h>
-#include <clang/ASTMatchers/ASTMatchFinder.h>
-
-
-namespace {
-AST_MATCHER(clang::StringLiteral, unterminated)
-{
-    size_t len = Node.getLength();
-    if (len > 0 && Node.getCodeUnit(len - 1) == '\n') {
-        return false;
-    }
-    return true;
-}
-} // namespace
-
-namespace bitcoin {
-
-void LogPrintfCheck::registerMatchers(clang::ast_matchers::MatchFinder* finder)
-{
-    using namespace clang::ast_matchers;
-
-    /*
-      Logprintf(..., ..., ..., ..., ..., "foo", ...)
-    */
-
-    finder->addMatcher(
-        callExpr(
-            callee(functionDecl(hasName("LogPrintf_"))),
-            hasArgument(5, stringLiteral(unterminated()).bind("logstring"))),
-        this);
-
-    /*
-      auto walletptr = &wallet;
-      wallet.WalletLogPrintf("foo");
-      wallet->WalletLogPrintf("foo");
-    */
-    finder->addMatcher(
-        cxxMemberCallExpr(
-            callee(cxxMethodDecl(hasName("WalletLogPrintf"))),
-            hasArgument(0, stringLiteral(unterminated()).bind("logstring"))),
-        this);
-}
-
-void LogPrintfCheck::check(const clang::ast_matchers::MatchFinder::MatchResult& Result)
-{
-    if (const clang::StringLiteral* lit = Result.Nodes.getNodeAs<clang::StringLiteral>("logstring")) {
-        const clang::ASTContext& ctx = *Result.Context;
-        const auto user_diag = diag(lit->getEndLoc(), "Unterminated format string used with LogPrintf");
-        const auto& loc = lit->getLocationOfByte(lit->getByteLength(), *Result.SourceManager, ctx.getLangOpts(), ctx.getTargetInfo());
-        user_diag << clang::FixItHint::CreateInsertion(loc, "\\n");
-    }
-}
-
-} // namespace bitcoin

contrib/devtools/bitcoin-tidy/logprintf.h

@@ -1,29 +0,0 @@
-// Copyright (c) 2023 Bitcoin Developers
-// Distributed under the MIT software license, see the accompanying
-// file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-#ifndef LOGPRINTF_CHECK_H
-#define LOGPRINTF_CHECK_H
-
-#include <clang-tidy/ClangTidyCheck.h>
-
-namespace bitcoin {
-
-// Warn about any use of LogPrintf that does not end with a newline.
-class LogPrintfCheck final : public clang::tidy::ClangTidyCheck
-{
-public:
-    LogPrintfCheck(clang::StringRef Name, clang::tidy::ClangTidyContext* Context)
-        : clang::tidy::ClangTidyCheck(Name, Context) {}
-
-    bool isLanguageVersionSupported(const clang::LangOptions& LangOpts) const override
-    {
-        return LangOpts.CPlusPlus;
-    }
-    void registerMatchers(clang::ast_matchers::MatchFinder* Finder) override;
-    void check(const clang::ast_matchers::MatchFinder::MatchResult& Result) override;
-};
-
-} // namespace bitcoin
-
-#endif // LOGPRINTF_CHECK_H

contrib/devtools/headerssync-params.py

@@ -1,357 +0,0 @@
-#!/usr/bin/env python3
-# Copyright (c) 2022 Pieter Wuille
-# Distributed under the MIT software license, see the accompanying
-# file COPYING or http://www.opensource.org/licenses/mit-license.php.
-
-"""Script to find the optimal parameters for the headerssync module through simulation."""
-
-from math import log, exp, sqrt
-from datetime import datetime, timedelta
-import random
-
-# Parameters:
-
-# Aim for still working fine at some point in the future. [datetime]
-TIME = datetime(2026, 10, 5)
-
-# Expected block interval. [timedelta]
-BLOCK_INTERVAL = timedelta(seconds=600)
-
-# The number of headers corresponding to the minchainwork parameter. [headers]
-MINCHAINWORK_HEADERS = 804000
-
-# Combined processing bandwidth from all attackers to one victim. [bit/s]
-# 6 Gbit/s is approximately the speed at which a single thread of a Ryzen 5950X CPU thread can hash
-# headers. In practice, the victim's network bandwidth and network processing overheads probably
-# impose a far lower number, but it's a useful upper bound.
-ATTACK_BANDWIDTH = 6000000000
-
-# How much additional permanent memory usage are attackers (jointly) allowed to cause in the victim,
-# expressed as fraction of the normal memory usage due to mainchain growth, for the duration the
-# attack is sustained. [unitless]
-# 0.2 means that attackers, while they keep up the attack, can cause permanent memory usage due to
-# headers storage to grow at 1.2 header per BLOCK_INTERVAL.
-ATTACK_FRACTION = 0.2
-
-# When this is set, the mapping from period size to memory usage (at optimal buffer size for that
-# period) is assumed to be convex. This greatly speeds up the computation, and does not appear
-# to influence the outcome. Set to False for a stronger guarantee to get the optimal result.
-ASSUME_CONVEX = True
-
-# Explanation:
-#
-#  The headerssync module implements a DoS protection against low-difficulty header spam which does
-#  not rely on checkpoints. In short it works as follows:
-#
-#  - (initial) header synchronization is split into two phases:
-#    - A commitment phase, in which headers are downloaded from the peer, and a very compact
-#      commitment to them is remembered in per-peer memory. The commitment phase ends when the
-#      received chain's combined work reaches a predetermined threshold.
-#    - A redownload phase, during which the headers are downloaded a second time from the same peer,
-#      and compared against the commitment constructed in the first phase. If there is a match, the
-#      redownloaded headers are fed to validation and accepted into permanent storage.
-#
-#    This separation guarantees that no headers are accepted into permanent storage without
-#    requiring the peer to first prove the chain actually has sufficient work.
-#
-#  - To actually implement this commitment mechanism, the following approach is used:
-#    - Keep a *1 bit* commitment (constructed using a salted hash function), for every block whose
-#      height is a multiple of {period} plus an offset value. If RANDOMIZE_OFFSET, the offset,
-#      like the salt, is chosen randomly when the synchronization starts and kept fixed afterwards.
-#    - When redownloading, headers are fed through a per-peer queue that holds {bufsize} headers,
-#      before passing them to validation. All the headers in this queue are verified against the
-#      commitment bits created in the first phase before any header is released from it. This means
-#      {bufsize/period} bits are checked "on top of" each header before actually processing it,
-#      which results in a commitment structure with roughly {bufsize/period} bits of security, as
-#      once a header is modified, due to the prevhash inclusion, all future headers necessarily
-#      change as well.
-#
-#  The question is what these {period} and {bufsize} parameters need to be set to. This program
-#  exhaustively tests a range of values to find the optimal choice, taking into account:
-#
-#  - Minimizing the (maximum of) two scenarios that trigger per-peer memory usage:
-#
-#    - When downloading a (likely honest) chain that reaches the chainwork threshold after {n}
-#      blocks, and then redownloads them, we will consume per-peer memory that is sufficient to
-#      store {n/period} commitment bits and {bufsize} headers. We only consider attackers without
-#      sufficient hashpower (as otherwise they are from a PoW perspective not attackers), which
-#      means {n} is restricted to the honest chain's length before reaching minchainwork.
-#
-#    - When downloading a (likely false) chain of {n} headers that never reaches the chainwork
-#      threshold, we will consume per-peer memory that is sufficient to store {n/period}
-#      commitment bits. Such a chain may be very long, by exploiting the timewarp bug to avoid
-#      ramping up difficulty. There is however an absolute limit on how long such a chain can be: 6
-#      blocks per second since genesis, due to the increasing MTP consensus rule.
-#
-#  - Not gratuitously preventing synchronizing any valid chain, however difficult such a chain may
-#    be to construct. In particular, the above scenario with an enormous timewarp-expoiting chain
-#    cannot simply be ignored, as it is legal that the honest main chain is like that. We however
-#    do not bother minimizing the memory usage in that case (because a billion-header long honest
-#    chain will inevitably use far larger amounts of memory than designed for).
-#
-#  - Keep the rate at which attackers can get low-difficulty headers accepted to the block index
-#    negligible. Specifically, the possibility exists for an attacker to send the honest main
-#    chain's headers during the commitment phase, but then start deviating at an attacker-chosen
-#    point by sending novel low-difficulty headers instead. Depending on how high we set the
-#    {bufsize/period} ratio, we can make the probability that such a header makes it in
-#    arbitrarily small, but at the cost of higher memory during the redownload phase. It turns out,
-#    some rate of memory usage growth is expected anyway due to chain growth, so permitting the
-#    attacker to increase that rate by a small factor isn't concerning. The attacker may start
-#    somewhat later than genesis, as long as the difficulty doesn't get too high. This reduces
-#    the attacker bandwidth required at the cost of higher PoW needed for constructing the
-#    alternate chain. This trade-off is ignored here, as it results in at most a small constant
-#    factor in attack rate.
-
-
-# System properties:
-
-# Headers in the redownload buffer are stored without prevhash. [bits]
-COMPACT_HEADER_SIZE = 48 * 8
-
-# How many bits a header uses in P2P protocol. [bits]
-NET_HEADER_SIZE = 81 * 8
-
-# How many headers are sent at once. [headers]
-HEADER_BATCH_COUNT = 2000
-
-# Whether or not the offset of which blocks heights get checksummed is randomized.
-RANDOMIZE_OFFSET = True
-
-# Timestamp of the genesis block
-GENESIS_TIME = datetime(2009, 1, 3)
-
-# Derived values:
-
-# What rate of headers worth of RAM attackers are allowed to cause in the victim. [headers/s]
-LIMIT_HEADERRATE = ATTACK_FRACTION / BLOCK_INTERVAL.total_seconds()
-
-# How many headers can attackers (jointly) send a victim per second. [headers/s]
-NET_HEADERRATE = ATTACK_BANDWIDTH / NET_HEADER_SIZE
-
-# What fraction of headers sent by attackers can at most be accepted by a victim [unitless]
-LIMIT_FRACTION = LIMIT_HEADERRATE / NET_HEADERRATE
-
-# How many headers we permit attackers to cause being accepted per attack. [headers/attack]
-ATTACK_HEADERS = LIMIT_FRACTION * MINCHAINWORK_HEADERS
-
-
-def find_max_headers(when):
-    """Compute the maximum number of headers a valid Bitcoin chain can have at given time."""
-    # When exploiting the timewarp attack, this can be up to 6 per second since genesis.
-    return 6 * ((when - GENESIS_TIME) // timedelta(seconds=1))
-
-
-def lambert_w(value):
-    """Solve the equation x*exp(x)=value (x > 0, value > 0)."""
-    # Initial approximation.
-    approx = max(log(value), 0.0)
-    for _ in range(10):
-        # Newton-Rhapson iteration steps.
-        approx += (value * exp(-approx) - approx) / (approx + 1.0)
-    return approx
-
-
-def attack_rate(period, bufsize, limit=None):
-    """Compute maximal accepted headers per attack in (period, bufsize) configuration.
-
-    If limit is provided, the computation is stopped early when the result is known to exceed the
-    value in limit.
-    """
-
-    max_rate = None
-    max_honest = None
-    # Let the current batch 0 being received be the first one in which the attacker starts lying.
-    # They will only ever start doing so right after a commitment block, but where that is can be
-    # in a number of places. Let honest be the number of honest headers in this current batch,
-    # preceding the forged ones.
-    for honest in range(HEADER_BATCH_COUNT):
-        # The number of headers the attack under consideration will on average get accepted.
-        # This is the number being computed.
-        rate = 0
-
-        # Iterate over the possible alignments of commitments w.r.t. the first batch. In case
-        # the alignments are randomized, try all values. If not, the attacker can know/choose
-        # the alignment, and will always start forging right after a commitment.
-        if RANDOMIZE_OFFSET:
-            align_choices = list(range(period))
-        else:
-            align_choices = [(honest - 1) % period]
-        # Now loop over those possible alignment values, computing the average attack rate
-        # over them by dividing each contribution by len(align_choices).
-        for align in align_choices:
-            # These state variables capture the situation after receiving the first batch.
-            # - The number of headers received after the last commitment for an honest block:
-            after_good_commit = HEADER_BATCH_COUNT - honest + ((honest - align - 1) % period)
-            # - The number of forged headers in the redownload buffer:
-            forged_in_buf = HEADER_BATCH_COUNT - honest
-
-            # Now iterate over the next batches of headers received, adding contributions to the
-            # rate variable.
-            while True:
-                # Process the first HEADER_BATCH_COUNT headers in the buffer:
-                accept_forged_headers = max(forged_in_buf - bufsize, 0)
-                forged_in_buf -= accept_forged_headers
-                if accept_forged_headers:
-                    # The probability the attack has not been detected yet at this point:
-                    prob = 0.5 ** (after_good_commit // period)
-                    # Update attack rate, divided by align_choices to average over the alignments.
-                    rate += accept_forged_headers * prob / len(align_choices)
-                    # If this means we exceed limit, bail out early (performance optimization).
-                    if limit is not None and rate >= limit:
-                        return rate, None
-                    # If the maximal term being added is negligible compared to rate, stop
-                    # iterating.
-                    if HEADER_BATCH_COUNT * prob < 1.0e-16 * rate * len(align_choices):
-                        break
-                # Update state from a new incoming batch (which is all forged)
-                after_good_commit += HEADER_BATCH_COUNT
-                forged_in_buf += HEADER_BATCH_COUNT
-
-        if max_rate is None or rate > max_rate:
-            max_rate = rate
-            max_honest = honest
-
-    return max_rate, max_honest
-
-
-def memory_usage(period, bufsize, when):
-    """How much memory (max,mainchain,timewarp) does the (period,bufsize) configuration need?"""
-
-    # Per-peer memory usage for a timewarp chain that never meets minchainwork
-    mem_timewarp = find_max_headers(when) // period
-    # Per-peer memory usage for being fed the main chain
-    mem_mainchain = (MINCHAINWORK_HEADERS // period) + bufsize * COMPACT_HEADER_SIZE
-    # Maximum per-peer memory usage
-    max_mem = max(mem_timewarp, mem_mainchain)
-
-    return max_mem, mem_mainchain, mem_timewarp
-
-def find_bufsize(period, attack_headers, when, max_mem=None, min_bufsize=1):
-    """Determine how big bufsize needs to be given a specific period length.
-
-    Given a period, find the smallest value of bufsize such that the attack rate against the
-    (period, bufsize) configuration is below attack_headers. If max_mem is provided, and no
-    such bufsize exists that needs less than max_mem bits of memory, None is returned.
-    min_bufsize is the minimal result to be considered."""
-
-    if max_mem is None:
-        succ_buf = min_bufsize - 1
-        fail_buf = min_bufsize
-        # First double iteratively until an upper bound for failure is found.
-        while True:
-            if attack_rate(period, fail_buf, attack_headers)[0] < attack_headers:
-                break
-            succ_buf, fail_buf = fail_buf, 3 * fail_buf - 2 * succ_buf
-    else:
-        # If a long low-work header chain exists that exceeds max_mem already, give up.
-        if find_max_headers(when) // period > max_mem:
-            return None
-        # Otherwise, verify that the maximal buffer size that permits a mainchain sync with less
-        # than max_mem memory is sufficient to get the attack rate below attack_headers. If not,
-        # also give up.
-        max_buf = (max_mem - (MINCHAINWORK_HEADERS // period)) // COMPACT_HEADER_SIZE
-        if max_buf < min_bufsize:
-            return None
-        if attack_rate(period, max_buf, attack_headers)[0] >= attack_headers:
-            return None
-        # If it is sufficient, that's an upper bound to start our search.
-        succ_buf = min_bufsize - 1
-        fail_buf = max_buf
-
-    # Then perform a bisection search to narrow it down.
-    while fail_buf > succ_buf + 1:
-        try_buf = (succ_buf + fail_buf) // 2
-        if attack_rate(period, try_buf, attack_headers)[0] >= attack_headers:
-            succ_buf = try_buf
-        else:
-            fail_buf = try_buf
-    return fail_buf
-
-
-def optimize(when):
-    """Find the best (period, bufsize) configuration."""
-
-    # When period*bufsize = memory_scale, the per-peer memory for a mainchain sync and a maximally
-    # long low-difficulty header sync are equal.
-    memory_scale = (find_max_headers(when) - MINCHAINWORK_HEADERS) / COMPACT_HEADER_SIZE
-    # Compute approximation for {bufsize/period}, using a formula for a simplified problem.
-    approx_ratio = lambert_w(log(4) * memory_scale / ATTACK_HEADERS**2) / log(4)
-    # Use those for a first attempt.
-    print("Searching configurations:")
-    period = int(sqrt(memory_scale / approx_ratio) + 0.5)
-    bufsize = find_bufsize(period, ATTACK_HEADERS, when)
-    mem = memory_usage(period, bufsize, when)
-    best = (period, bufsize, mem)
-    maps = [(period, bufsize), (MINCHAINWORK_HEADERS + 1, None)]
-    print(f"- Initial: period={period}, buffer={bufsize}, mem={mem[0] / 8192:.3f} KiB")
-
-    # Consider all period values between 1 and MINCHAINWORK_HEADERS, except the one just tried.
-    periods = [iv for iv in range(1, MINCHAINWORK_HEADERS + 1) if iv != period]
-    # Iterate, picking a random element from periods, computing its corresponding bufsize, and
-    # then using the result to shrink the period.
-    while True:
-        # Remove all periods whose memory usage for low-work long chain sync exceed the best
-        # memory usage we've found so far.
-        periods = [p for p in periods if find_max_headers(when) // p < best[2][0]]
-        # Stop if there is nothing left to try.
-        if len(periods) == 0:
-            break
-        # Pick a random remaining option for period size, and compute corresponding bufsize.
-        period = periods.pop(random.randrange(len(periods)))
-        # The buffer size (at a given attack level) cannot shrink as the period grows. Find the
-        # largest period smaller than the selected one we know the buffer size for, and use that
-        # as a lower bound to find_bufsize.
-        min_bufsize = max([(p, b) for p, b in maps if p < period] + [(0,0)])[1]
-        bufsize = find_bufsize(period, ATTACK_HEADERS, when, best[2][0], min_bufsize)
-        if bufsize is not None:
-            # We found a (period, bufsize) configuration with better memory usage than our best
-            # so far. Remember it for future lower bounds.
-            maps.append((period, bufsize))
-            mem = memory_usage(period, bufsize, when)
-            assert mem[0] <= best[2][0]
-            if ASSUME_CONVEX:
-                # Remove all periods that are on the other side of the former best as the new
-                # best.
-                periods = [p for p in periods if (p < best[0]) == (period < best[0])]
-            best = (period, bufsize, mem)
-            print(f"- New best: period={period}, buffer={bufsize}, mem={mem[0] / 8192:.3f} KiB")
-        else:
-            # The (period, bufsize) configuration we found is worse than what we already had.
-            if ASSUME_CONVEX:
-                # Remove all periods that are on the other side of the tried configuration as the
-                # best one.
-                periods = [p for p in periods if (p < period) == (best[0] < period)]
-
-    # Return the result.
-    period, bufsize, _ = best
-    return period, bufsize
-
-
-def analyze(when):
-    """Find the best configuration and print it out."""
-
-    period, bufsize = optimize(when)
-    # Compute accurate statistics for the best found configuration.
-    _, mem_mainchain, mem_timewarp = memory_usage(period, bufsize, when)
-    headers_per_attack, _ = attack_rate(period, bufsize)
-    attack_volume = NET_HEADER_SIZE * MINCHAINWORK_HEADERS
-    # And report them.
-    print()
-    print("Optimal configuration:")
-    print()
-    print("//! Store one header commitment per HEADER_COMMITMENT_PERIOD blocks.")
-    print(f"constexpr size_t HEADER_COMMITMENT_PERIOD{{{period}}};")
-    print()
-    print("//! Only feed headers to validation once this many headers on top have been")
-    print("//! received and validated against commitments.")
-    print(f"constexpr size_t REDOWNLOAD_BUFFER_SIZE{{{bufsize}}};"
-          f" // {bufsize}/{period} = ~{bufsize/period:.1f} commitments")
-    print()
-    print("Properties:")
-    print(f"- Per-peer memory for mainchain sync: {mem_mainchain / 8192:.3f} KiB")
-    print(f"- Per-peer memory for timewarp attack: {mem_timewarp / 8192:.3f} KiB")
-    print(f"- Attack rate: {1/headers_per_attack:.1f} attacks for 1 header of memory growth")
-    print(f"  (where each attack costs {attack_volume / 8388608:.3f} MiB bandwidth)")
-
-
-analyze(TIME)

contrib/devtools/security-check.py

@@ -10,7 +10,7 @@ Otherwise the exit status will be 1 and it will log which executables failed whi
 import sys
 from typing import List
 
-import lief
+import lief #type:ignore
 
 def check_ELF_RELRO(binary) -> bool:
     '''
@@ -113,7 +113,7 @@ def check_ELF_control_flow(binary) -> bool:
     main = binary.get_function_address('main')
     content = binary.get_content_from_virtual_address(main, 4, lief.Binary.VA_TYPES.AUTO)
 
-    if content.tolist() == [243, 15, 30, 250]: # endbr64
+    if content == [243, 15, 30, 250]: # endbr64
         return True
     return False
 
@@ -142,7 +142,7 @@ def check_PE_control_flow(binary) -> bool:
 
     content = binary.get_content_from_virtual_address(virtual_address, 4, lief.Binary.VA_TYPES.VA)
 
-    if content.tolist() == [243, 15, 30, 250]: # endbr64
+    if content == [243, 15, 30, 250]: # endbr64
         return True
     return False
 
@@ -158,11 +158,12 @@ def check_MACHO_NOUNDEFS(binary) -> bool:
     '''
     return binary.header.has(lief.MachO.HEADER_FLAGS.NOUNDEFS)
 
-def check_MACHO_FIXUP_CHAINS(binary) -> bool:
+def check_MACHO_LAZY_BINDINGS(binary) -> bool:
     '''
-    Check for use of chained fixups.
+    Check for no lazy bindings.
+    We don't use or check for MH_BINDATLOAD. See #18295.
     '''
-    return binary.has_dyld_chained_fixups
+    return binary.dyld_info.lazy_bind == (0,0)
 
 def check_MACHO_Canary(binary) -> bool:
     '''
@@ -189,7 +190,7 @@ def check_MACHO_control_flow(binary) -> bool:
     '''
     content = binary.get_content_from_virtual_address(binary.entrypoint, 4, lief.Binary.VA_TYPES.AUTO)
 
-    if content.tolist() == [243, 15, 30, 250]: # endbr64
+    if content == [243, 15, 30, 250]: # endbr64
         return True
     return False
 
@@ -213,8 +214,8 @@ BASE_PE = [
 
 BASE_MACHO = [
     ('NOUNDEFS', check_MACHO_NOUNDEFS),
+    ('LAZY_BINDINGS', check_MACHO_LAZY_BINDINGS),
     ('Canary', check_MACHO_Canary),
-    ('FIXUP_CHAINS', check_MACHO_FIXUP_CHAINS),
 ]
 
 CHECKS = {

contrib/devtools/symbol-check.py

@@ -13,7 +13,7 @@ Example usage:
 import sys
 from typing import List, Dict
 
-import lief
+import lief #type:ignore
 
 # Debian 10 (Buster) EOL: 2024. https://wiki.debian.org/LTS
 #
@@ -72,25 +72,6 @@ ELF_INTERPRETER_NAMES: Dict[lief.ELF.ARCH, Dict[lief.ENDIANNESS, str]] = {
     },
 }
 
-ELF_ABIS: Dict[lief.ELF.ARCH, Dict[lief.ENDIANNESS, List[int]]] = {
-    lief.ELF.ARCH.x86_64: {
-        lief.ENDIANNESS.LITTLE: [3,2,0],
-    },
-    lief.ELF.ARCH.ARM: {
-        lief.ENDIANNESS.LITTLE: [3,2,0],
-    },
-    lief.ELF.ARCH.AARCH64: {
-        lief.ENDIANNESS.LITTLE: [3,7,0],
-    },
-    lief.ELF.ARCH.PPC64: {
-        lief.ENDIANNESS.LITTLE: [3,10,0],
-        lief.ENDIANNESS.BIG: [3,2,0],
-    },
-    lief.ELF.ARCH.RISCV: {
-        lief.ENDIANNESS.LITTLE: [4,15,0],
-    },
-}
-
 # Allowed NEEDED libraries
 ELF_ALLOWED_LIBRARIES = {
 # bitcoind and bitcoin-qt
@@ -98,6 +79,7 @@ ELF_ALLOWED_LIBRARIES = {
 'libc.so.6', # C library
 'libpthread.so.0', # threading
 'libm.so.6', # math library
+'librt.so.1', # real-time (clock)
 'libatomic.so.1',
 'ld-linux-x86-64.so.2', # 64-bit dynamic linker
 'ld-linux.so.2', # 32-bit dynamic linker
@@ -157,21 +139,21 @@ PE_ALLOWED_LIBRARIES = {
 'KERNEL32.dll', # win32 base APIs
 'msvcrt.dll', # C standard library for MSVC
 'SHELL32.dll', # shell API
+'USER32.dll', # user interface
 'WS2_32.dll', # sockets
 # bitcoin-qt only
 'dwmapi.dll', # desktop window manager
 'GDI32.dll', # graphics device interface
 'IMM32.dll', # input method editor
-'NETAPI32.dll', # network management
+'NETAPI32.dll',
 'ole32.dll', # component object model
 'OLEAUT32.dll', # OLE Automation API
 'SHLWAPI.dll', # light weight shell API
-'USER32.dll', # user interface
-'USERENV.dll', # user management
-'UxTheme.dll', # visual style
+'USERENV.dll',
+'UxTheme.dll',
 'VERSION.dll', # version checking
 'WINMM.dll', # WinMM audio API
-'WTSAPI32.dll', # Remote Desktop
+'WTSAPI32.dll',
 }
 
 def check_version(max_versions, version, arch) -> bool:
@@ -231,7 +213,7 @@ def check_MACHO_libraries(binary) -> bool:
     return ok
 
 def check_MACHO_min_os(binary) -> bool:
-    if binary.build_version.minos == [11,0,0]:
+    if binary.build_version.minos == [10,15,0]:
         return True
     return False
 
@@ -240,11 +222,6 @@ def check_MACHO_sdk(binary) -> bool:
         return True
     return False
 
-def check_MACHO_ld64(binary) -> bool:
-    if binary.build_version.tools[0].version == [711, 0, 0]:
-        return True
-    return False
-
 def check_PE_libraries(binary) -> bool:
     ok: bool = True
     for dylib in binary.libraries:
@@ -265,25 +242,17 @@ def check_ELF_interpreter(binary) -> bool:
 
     return binary.concrete.interpreter == expected_interpreter
 
-def check_ELF_ABI(binary) -> bool:
-    expected_abi = ELF_ABIS[binary.header.machine_type][binary.abstract.header.endianness]
-    note = binary.concrete.get(lief.ELF.NOTE_TYPES.ABI_TAG)
-    assert note.details.abi == lief.ELF.NOTE_ABIS.LINUX
-    return note.details.version == expected_abi
-
 CHECKS = {
 lief.EXE_FORMATS.ELF: [
     ('IMPORTED_SYMBOLS', check_imported_symbols),
     ('EXPORTED_SYMBOLS', check_exported_symbols),
     ('LIBRARY_DEPENDENCIES', check_ELF_libraries),
     ('INTERPRETER_NAME', check_ELF_interpreter),
-    ('ABI', check_ELF_ABI),
 ],
 lief.EXE_FORMATS.MACHO: [
     ('DYNAMIC_LIBRARIES', check_MACHO_libraries),
     ('MIN_OS', check_MACHO_min_os),
     ('SDK', check_MACHO_sdk),
-    ('LD64', check_MACHO_ld64),
 ],
 lief.EXE_FORMATS.PE: [
     ('DYNAMIC_LIBRARIES', check_PE_libraries),

contrib/devtools/test-security-check.py

@@ -5,7 +5,7 @@
 '''
 Test script for security-check.py
 '''
-import lief
+import lief #type:ignore
 import os
 import subprocess
 from typing import List
@@ -28,7 +28,7 @@ def clean_files(source, executable):
     os.remove(source)
     os.remove(executable)
 
-def call_security_check(cc: str, source: str, executable: str, options) -> tuple:
+def call_security_check(cc, source, executable, options):
     # This should behave the same as AC_TRY_LINK, so arrange well-known flags
     # in the same order as autoconf would.
     #
@@ -119,31 +119,29 @@ class TestSecurityChecks(unittest.TestCase):
         arch = get_arch(cc, source, executable)
 
         if arch == lief.ARCHITECTURES.X86:
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fno-stack-protector', '-Wl,-no_fixup_chains']),
-                (1, executable+': failed NOUNDEFS Canary FIXUP_CHAINS PIE NX CONTROL_FLOW'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fno-stack-protector', '-Wl,-fixup_chains']),
-                (1, executable+': failed NOUNDEFS Canary PIE NX CONTROL_FLOW'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fstack-protector-all', '-Wl,-fixup_chains']),
-                (1, executable+': failed NOUNDEFS PIE NX CONTROL_FLOW'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-fstack-protector-all', '-Wl,-fixup_chains']),
-                (1, executable+': failed NOUNDEFS PIE CONTROL_FLOW'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-fstack-protector-all', '-Wl,-fixup_chains']),
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fno-stack-protector']),
+                (1, executable+': failed NOUNDEFS LAZY_BINDINGS Canary PIE NX CONTROL_FLOW'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-Wl,-allow_stack_execute','-fstack-protector-all']),
+                (1, executable+': failed NOUNDEFS LAZY_BINDINGS PIE NX CONTROL_FLOW'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-flat_namespace','-fstack-protector-all']),
+                (1, executable+': failed NOUNDEFS LAZY_BINDINGS PIE CONTROL_FLOW'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-fstack-protector-all']),
+                (1, executable+': failed LAZY_BINDINGS PIE CONTROL_FLOW'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-bind_at_load','-fstack-protector-all']),
                 (1, executable+': failed PIE CONTROL_FLOW'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-bind_at_load','-fstack-protector-all', '-Wl,-fixup_chains']),
-                (1, executable+': failed PIE CONTROL_FLOW'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-bind_at_load','-fstack-protector-all', '-fcf-protection=full', '-Wl,-fixup_chains']),
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-no_pie','-Wl,-bind_at_load','-fstack-protector-all', '-fcf-protection=full']),
                 (1, executable+': failed PIE'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-pie','-Wl,-bind_at_load','-fstack-protector-all', '-fcf-protection=full', '-Wl,-fixup_chains']),
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-pie','-Wl,-bind_at_load','-fstack-protector-all', '-fcf-protection=full']),
                 (0, ''))
         else:
             # arm64 darwin doesn't support non-PIE binaries, control flow or executable stacks
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-flat_namespace','-fno-stack-protector', '-Wl,-no_fixup_chains']),
-                (1, executable+': failed NOUNDEFS Canary FIXUP_CHAINS'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-flat_namespace','-fno-stack-protector', '-Wl,-fixup_chains']),
-                (1, executable+': failed NOUNDEFS Canary'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-flat_namespace','-fstack-protector-all', '-Wl,-fixup_chains']),
-                (1, executable+': failed NOUNDEFS'))
-            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-bind_at_load','-fstack-protector-all', '-Wl,-fixup_chains']),
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-flat_namespace','-fno-stack-protector']),
+                (1, executable+': failed NOUNDEFS LAZY_BINDINGS Canary'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-flat_namespace','-fstack-protector-all']),
+                (1, executable+': failed NOUNDEFS LAZY_BINDINGS'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-fstack-protector-all']),
+                (1, executable+': failed LAZY_BINDINGS'))
+            self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-bind_at_load','-fstack-protector-all']),
                 (0, ''))
 
 

contrib/devtools/test-symbol-check.py

@@ -121,7 +121,7 @@ class TestSymbolChecks(unittest.TestCase):
                 }
         ''')
 
-        self.assertEqual(call_symbol_check(cc, source, executable, ['-Wl,-platform_version','-Wl,macos', '-Wl,11.0', '-Wl,11.4']),
+        self.assertEqual(call_symbol_check(cc, source, executable, ['-Wl,-platform_version','-Wl,macos', '-Wl,10.15', '-Wl,11.4']),
                 (1, f'{executable}: failed SDK'))
 
     def test_PE(self):

contrib/devtools/test_utxo_snapshots.sh

@@ -1,209 +0,0 @@
-#!/usr/bin/env bash
-# Demonstrate the creation and usage of UTXO snapshots.
-#
-# A server node starts up, IBDs up to a certain height, then generates a UTXO
-# snapshot at that point.
-#
-# The server then downloads more blocks (to create a diff from the snapshot).
-#
-# We bring a client up, load the UTXO snapshot, and we show the client sync to
-# the "network tip" and then start a background validation of the snapshot it
-# loaded. We see the background validation chainstate removed after validation
-# completes.
-#
-# The shellcheck rule SC2086 (quoted variables) disablements are necessary
-# since this rule needs to be violated in order to get bitcoind to pick up on
-# $EARLY_IBD_FLAGS for the script to work.
-
-export LC_ALL=C
-set -e
-
-BASE_HEIGHT=${1:-30000}
-INCREMENTAL_HEIGHT=20000
-FINAL_HEIGHT=$((BASE_HEIGHT + INCREMENTAL_HEIGHT))
-
-SERVER_DATADIR="$(pwd)/utxodemo-data-server-$BASE_HEIGHT"
-CLIENT_DATADIR="$(pwd)/utxodemo-data-client-$BASE_HEIGHT"
-UTXO_DAT_FILE="$(pwd)/utxo.$BASE_HEIGHT.dat"
-
-# Chosen to try to not interfere with any running bitcoind processes.
-SERVER_PORT=8633
-SERVER_RPC_PORT=8632
-
-CLIENT_PORT=8733
-CLIENT_RPC_PORT=8732
-
-SERVER_PORTS="-port=${SERVER_PORT} -rpcport=${SERVER_RPC_PORT}"
-CLIENT_PORTS="-port=${CLIENT_PORT} -rpcport=${CLIENT_RPC_PORT}"
-
-# Ensure the client exercises all indexes to test that snapshot use works
-# properly with indexes.
-ALL_INDEXES="-txindex -coinstatsindex -blockfilterindex=1"
-
-if ! command -v jq >/dev/null ; then
-  echo "This script requires jq to parse JSON RPC output. Please install it."
-  echo "(e.g. sudo apt install jq)"
-  exit 1
-fi
-
-DUMP_OUTPUT="dumptxoutset-output-$BASE_HEIGHT.json"
-
-finish() {
-  echo
-  echo "Killing server and client PIDs ($SERVER_PID, $CLIENT_PID) and cleaning up datadirs"
-  echo
-  rm -f "$UTXO_DAT_FILE" "$DUMP_OUTPUT"
-  rm -rf "$SERVER_DATADIR" "$CLIENT_DATADIR"
-  kill -9 "$SERVER_PID" "$CLIENT_PID"
-}
-
-trap finish EXIT
-
-# Need to specify these to trick client into accepting server as a peer
-# it can IBD from, otherwise the default values prevent IBD from the server node.
-EARLY_IBD_FLAGS="-maxtipage=9223372036854775207 -minimumchainwork=0x00"
-
-server_rpc() {
-  ./src/bitcoin-cli -rpcport=$SERVER_RPC_PORT -datadir="$SERVER_DATADIR" "$@"
-}
-client_rpc() {
-  ./src/bitcoin-cli -rpcport=$CLIENT_RPC_PORT -datadir="$CLIENT_DATADIR" "$@"
-}
-server_sleep_til_boot() {
-  while ! server_rpc ping >/dev/null 2>&1; do sleep 0.1; done
-}
-client_sleep_til_boot() {
-  while ! client_rpc ping >/dev/null 2>&1; do sleep 0.1; done
-}
-server_sleep_til_shutdown() {
-  while server_rpc ping >/dev/null 2>&1; do sleep 0.1; done
-}
-
-mkdir -p "$SERVER_DATADIR" "$CLIENT_DATADIR"
-
-echo "Hi, welcome to the assumeutxo demo/test"
-echo
-echo "We're going to"
-echo
-echo "  - start up a 'server' node, sync it via mainnet IBD to height ${BASE_HEIGHT}"
-echo "  - create a UTXO snapshot at that height"
-echo "  - IBD ${INCREMENTAL_HEIGHT} more blocks on top of that"
-echo
-echo "then we'll demonstrate assumeutxo by "
-echo
-echo "  - starting another node (the 'client') and loading the snapshot in"
-echo "    * first you'll have to modify the code slightly (chainparams) and recompile"
-echo "    * don't worry, we'll make it easy"
-echo "  - observing the client sync ${INCREMENTAL_HEIGHT} blocks on top of the snapshot from the server"
-echo "  - observing the client validate the snapshot chain via background IBD"
-echo
-read -p "Press [enter] to continue" _
-
-echo
-echo "-- Starting the demo. You might want to run the two following commands in"
-echo "   separate terminal windows:"
-echo
-echo "   watch -n0.1 tail -n 30 $SERVER_DATADIR/debug.log"
-echo "   watch -n0.1 tail -n 30 $CLIENT_DATADIR/debug.log"
-echo
-read -p "Press [enter] to continue" _
-
-echo
-echo "-- IBDing the blocks (height=$BASE_HEIGHT) required to the server node..."
-# shellcheck disable=SC2086
-./src/bitcoind -logthreadnames=1 $SERVER_PORTS \
-    -datadir="$SERVER_DATADIR" $EARLY_IBD_FLAGS -stopatheight="$BASE_HEIGHT" >/dev/null
-
-echo
-echo "-- Creating snapshot at ~ height $BASE_HEIGHT ($UTXO_DAT_FILE)..."
-server_sleep_til_shutdown # wait for stopatheight to be hit
-# shellcheck disable=SC2086
-./src/bitcoind -logthreadnames=1 $SERVER_PORTS \
-    -datadir="$SERVER_DATADIR" $EARLY_IBD_FLAGS -connect=0 -listen=0 >/dev/null &
-SERVER_PID="$!"
-
-server_sleep_til_boot
-server_rpc dumptxoutset "$UTXO_DAT_FILE" > "$DUMP_OUTPUT"
-cat "$DUMP_OUTPUT"
-kill -9 "$SERVER_PID"
-
-RPC_BASE_HEIGHT=$(jq -r .base_height < "$DUMP_OUTPUT")
-RPC_AU=$(jq -r .txoutset_hash < "$DUMP_OUTPUT")
-RPC_NCHAINTX=$(jq -r .nchaintx < "$DUMP_OUTPUT")
-RPC_BLOCKHASH=$(jq -r .base_hash < "$DUMP_OUTPUT")
-
-server_sleep_til_shutdown
-
-echo
-echo "-- Now: add the following to CMainParams::m_assumeutxo_data"
-echo "   in src/kernel/chainparams.cpp, and recompile:"
-echo
-echo "   {${RPC_BASE_HEIGHT}, AssumeutxoHash{uint256S(\"0x${RPC_AU}\")}, ${RPC_NCHAINTX}, uint256S(\"0x${RPC_BLOCKHASH}\")},"
-echo
-echo
-echo "-- IBDing more blocks to the server node (height=$FINAL_HEIGHT) so there is a diff between snapshot and tip..."
-# shellcheck disable=SC2086
-./src/bitcoind $SERVER_PORTS -logthreadnames=1 -datadir="$SERVER_DATADIR" \
-    $EARLY_IBD_FLAGS -stopatheight="$FINAL_HEIGHT" >/dev/null
-
-echo
-echo "-- Starting the server node to provide blocks to the client node..."
-# shellcheck disable=SC2086
-./src/bitcoind $SERVER_PORTS -logthreadnames=1 -debug=net -datadir="$SERVER_DATADIR" \
-    $EARLY_IBD_FLAGS -connect=0 -listen=1 >/dev/null &
-SERVER_PID="$!"
-server_sleep_til_boot
-
-echo
-echo "-- Okay, what you're about to see is the client starting up and activating the snapshot."
-echo "   I'm going to display the top 14 log lines from the client on top of an RPC called"
-echo "   getchainstates, which is like getblockchaininfo but for both the snapshot and "
-echo "   background validation chainstates."
-echo
-echo "   You're going to first see the snapshot chainstate sync to the server's tip, then"
-echo "   the background IBD chain kicks in to validate up to the base of the snapshot."
-echo
-echo "   Once validation of the snapshot is done, you should see log lines indicating"
-echo "   that we've deleted the background validation chainstate."
-echo
-echo "   Once everything completes, exit the watch command with CTRL+C."
-echo
-read -p "When you're ready for all this, hit [enter]" _
-
-echo
-echo "-- Starting the client node to get headers from the server, then load the snapshot..."
-# shellcheck disable=SC2086
-./src/bitcoind $CLIENT_PORTS $ALL_INDEXES -logthreadnames=1 -datadir="$CLIENT_DATADIR" \
-    -connect=0 -addnode=127.0.0.1:$SERVER_PORT -debug=net $EARLY_IBD_FLAGS >/dev/null &
-CLIENT_PID="$!"
-client_sleep_til_boot
-
-echo
-echo "-- Initial state of the client:"
-client_rpc getchainstates
-
-echo
-echo "-- Loading UTXO snapshot into client..."
-client_rpc loadtxoutset "$UTXO_DAT_FILE"
-
-watch -n 0.3 "( tail -n 14 $CLIENT_DATADIR/debug.log ; echo ; ./src/bitcoin-cli -rpcport=$CLIENT_RPC_PORT -datadir=$CLIENT_DATADIR getchainstates) | cat"
-
-echo
-echo "-- Okay, now I'm going to restart the client to make sure that the snapshot chain reloads "
-echo "   as the main chain properly..."
-echo
-echo "   Press CTRL+C after you're satisfied to exit the demo"
-echo
-read -p "Press [enter] to continue"
-
-client_sleep_til_boot
-# shellcheck disable=SC2086
-./src/bitcoind $CLIENT_PORTS $ALL_INDEXES -logthreadnames=1 -datadir="$CLIENT_DATADIR" -connect=0 \
-    -addnode=127.0.0.1:$SERVER_PORT "$EARLY_IBD_FLAGS" >/dev/null &
-CLIENT_PID="$!"
-client_sleep_til_boot
-
-watch -n 0.3 "( tail -n 14 $CLIENT_DATADIR/debug.log ; echo ; ./src/bitcoin-cli -rpcport=$CLIENT_RPC_PORT -datadir=$CLIENT_DATADIR getchainstates) | cat"
-
-echo
-echo "-- Done!"

contrib/devtools/utxo_snapshot.sh

@@ -34,7 +34,7 @@ ${BITCOIN_CLI_CALL} invalidateblock "${PIVOT_BLOCKHASH}"
 
 if [[ "${OUTPUT_PATH}" = "-" ]]; then
   (>&2 echo "Generating txoutset info...")
-  ${BITCOIN_CLI_CALL} gettxoutsetinfo | grep hash_serialized_3 | sed 's/^.*: "\(.\+\)\+",/\1/g'
+  ${BITCOIN_CLI_CALL} gettxoutsetinfo | grep hash_serialized_2 | sed 's/^.*: "\(.\+\)\+",/\1/g'
 else
   (>&2 echo "Generating UTXO snapshot...")
   ${BITCOIN_CLI_CALL} dumptxoutset "${OUTPUT_PATH}"

contrib/guix/INSTALL.md

@@ -578,7 +578,7 @@ sudo --login guix pull --commit=<particular-commit>
 ```
 
 `guix pull` is quite a long process (especially if you're using
-`--no-substitutes`). If you encounter build problems, please refer to the
+`--no-substitute`). If you encounter build problems, please refer to the
 [troubleshooting section](#troubleshooting).
 
 Note that running a bare `guix pull` with no commit or branch specified will
@@ -616,7 +616,7 @@ systemctl enable guix-daemon
 systemctl start guix-daemon
 ```
 
-Remember to set `--no-substitutes` in `$libdir/systemd/system/guix-daemon.service` and other customizations if you used them for `guix-daemon-original.service`.
+Remember to set `--no-substitute` in `$libdir/systemd/system/guix-daemon.service` and other customizations if you used them for `guix-daemon-original.service`.
 
 ##### If you installed Guix via the Debian/Ubuntu distribution packages
 

contrib/guix/libexec/build.sh

@@ -52,8 +52,7 @@ BASEPREFIX="${PWD}/depends"
 store_path() {
     grep --extended-regexp "/[^-]{32}-${1}-[^-]+${2:+-${2}}" "${GUIX_ENVIRONMENT}/manifest" \
         | head --lines=1 \
-        | sed --expression='s|\x29*$||' \
-              --expression='s|^[[:space:]]*"||' \
+        | sed --expression='s|^[[:space:]]*"||' \
               --expression='s|"[[:space:]]*$||'
 }
 
@@ -315,7 +314,7 @@ mkdir -p "$DISTSRC"
                     | gzip -9n > "${OUTDIR}/${DISTNAME}-${HOST}-unsigned.tar.gz" \
                     || ( rm -f "${OUTDIR}/${DISTNAME}-${HOST}-unsigned.tar.gz" && exit 1 )
             )
-            make deploy ${V:+V=1} OSX_ZIP="${OUTDIR}/${DISTNAME}-${HOST}-unsigned.zip"
+            make deploy ${V:+V=1} OSX_DMG="${OUTDIR}/${DISTNAME}-${HOST}-unsigned.dmg"
             ;;
     esac
     (

contrib/guix/libexec/codesign.sh

@@ -85,12 +85,11 @@ mkdir -p "$DISTSRC"
             # Apply detached codesignatures to dist/ (in-place)
             signapple apply dist/Bitcoin-Qt.app codesignatures/osx/dist
 
-            # Make a .zip from dist/
-            cd dist/
-            find . -print0 \
-                | xargs -0r touch --no-dereference --date="@${SOURCE_DATE_EPOCH}"
-            find . | sort \
-                | zip -X@ "${OUTDIR}/${DISTNAME}-${HOST}.zip"
+            # Make a DMG from dist/
+            xorrisofs -D -l -V "$(< osx_volname)" -no-pad -r -dir-mode 0755 \
+                      -o "${OUTDIR}/${DISTNAME}-${HOST}.dmg" \
+                      dist \
+                      -- -volume_date all_file_dates ="$SOURCE_DATE_EPOCH"
             ;;
         *)
             exit 1

contrib/guix/libexec/prelude.bash

@@ -51,7 +51,7 @@ fi
 time-machine() {
     # shellcheck disable=SC2086
     guix time-machine --url=https://git.savannah.gnu.org/git/guix.git \
-                      --commit=160f78a4d92205df986ed9efcce7d3aac188cb24 \
+                      --commit=998eda3067c7d21e0d9bb3310d2f5a14b8f1c681 \
                       --cores="$JOBS" \
                       --keep-failed \
                       --fallback \

contrib/guix/manifest.scm

@@ -1,35 +1,44 @@
-(use-modules (gnu packages)
+(use-modules (gnu)
+             (gnu packages)
              (gnu packages autotools)
-             ((gnu packages bash) #:select (bash-minimal))
+             (gnu packages base)
+             (gnu packages bash)
              (gnu packages bison)
-             ((gnu packages certs) #:select (nss-certs))
-             ((gnu packages cmake) #:select (cmake-minimal))
+             (gnu packages certs)
+             (gnu packages cdrom)
+             (gnu packages check)
+             (gnu packages cmake)
              (gnu packages commencement)
              (gnu packages compression)
              (gnu packages cross-base)
+             (gnu packages curl)
              (gnu packages file)
              (gnu packages gawk)
              (gnu packages gcc)
-             ((gnu packages installers) #:select (nsis-x86_64))
-             ((gnu packages linux) #:select (linux-libre-headers-5.15 util-linux))
+             (gnu packages gnome)
+             (gnu packages installers)
+             (gnu packages linux)
              (gnu packages llvm)
              (gnu packages mingw)
              (gnu packages moreutils)
              (gnu packages pkg-config)
-             ((gnu packages python) #:select (python-minimal))
-             ((gnu packages python-build) #:select (python-tomli))
-             ((gnu packages python-crypto) #:select (python-asn1crypto))
-             ((gnu packages tls) #:select (openssl))
-             ((gnu packages version-control) #:select (git-minimal))
+             (gnu packages python)
+             (gnu packages python-crypto)
+             (gnu packages python-web)
+             (gnu packages shells)
+             (gnu packages tls)
+             (gnu packages version-control)
              (guix build-system cmake)
              (guix build-system gnu)
              (guix build-system python)
              (guix build-system trivial)
+             (guix download)
              (guix gexp)
              (guix git-download)
              ((guix licenses) #:prefix license:)
              (guix packages)
-             ((guix utils) #:select (substitute-keyword-arguments)))
+             (guix profiles)
+             (guix utils))
 
 (define-syntax-rule (search-our-patches file-name ...)
   "Return the list of absolute file names corresponding to each
@@ -38,7 +47,41 @@ FILE-NAME found in ./patches relative to the current file."
       ((%patch-path (list (string-append (dirname (current-filename)) "/patches"))))
     (list (search-patch file-name) ...)))
 
-(define building-on (string-append "--build=" (list-ref (string-split (%current-system) #\-) 0) "-guix-linux-gnu"))
+(define (make-ssp-fixed-gcc xgcc)
+  "Given a XGCC package, return a modified package that uses the SSP function
+from glibc instead of from libssp.so. Our `symbol-check' script will complain if
+we link against libssp.so, and thus will ensure that this works properly.
+
+Taken from:
+http://www.linuxfromscratch.org/hlfs/view/development/chapter05/gcc-pass1.html"
+  (package
+    (inherit xgcc)
+    (arguments
+     (substitute-keyword-arguments (package-arguments xgcc)
+       ((#:make-flags flags)
+        `(cons "gcc_cv_libc_provides_ssp=yes" ,flags))))))
+
+(define (make-gcc-rpath-link xgcc)
+  "Given a XGCC package, return a modified package that replace each instance of
+-rpath in the default system spec that's inserted by Guix with -rpath-link"
+  (package
+    (inherit xgcc)
+    (arguments
+     (substitute-keyword-arguments (package-arguments xgcc)
+       ((#:phases phases)
+        `(modify-phases ,phases
+           (add-after 'pre-configure 'replace-rpath-with-rpath-link
+             (lambda _
+               (substitute* (cons "gcc/config/rs6000/sysv4.h"
+                                  (find-files "gcc/config"
+                                              "^gnu-user.*\\.h$"))
+                 (("-rpath=") "-rpath-link="))
+               #t))))))))
+
+(define building-on (string-append (list-ref (string-split (%current-system) #\-) 0) "-guix-linux-gnu"))
+
+(define (explicit-cross-configure package)
+  (package-with-extra-configure-variable package "--build" building-on))
 
 (define (make-cross-toolchain target
                               base-gcc-for-libc
@@ -49,28 +92,28 @@ FILE-NAME found in ./patches relative to the current file."
   (let* ((xbinutils (cross-binutils target))
          ;; 1. Build a cross-compiling gcc without targeting any libc, derived
          ;; from BASE-GCC-FOR-LIBC
-         (xgcc-sans-libc (cross-gcc target
-                                    #:xgcc base-gcc-for-libc
-                                    #:xbinutils xbinutils))
+         (xgcc-sans-libc (explicit-cross-configure (cross-gcc target
+                                                              #:xgcc base-gcc-for-libc
+                                                              #:xbinutils xbinutils)))
          ;; 2. Build cross-compiled kernel headers with XGCC-SANS-LIBC, derived
          ;; from BASE-KERNEL-HEADERS
          (xkernel (cross-kernel-headers target
-                                        #:linux-headers base-kernel-headers
-                                        #:xgcc xgcc-sans-libc
-                                        #:xbinutils xbinutils))
+                                        base-kernel-headers
+                                        xgcc-sans-libc
+                                        xbinutils))
          ;; 3. Build a cross-compiled libc with XGCC-SANS-LIBC and XKERNEL,
          ;; derived from BASE-LIBC
-         (xlibc (cross-libc target
-                            #:libc base-libc
-                            #:xgcc xgcc-sans-libc
-                            #:xbinutils xbinutils
-                            #:xheaders xkernel))
+         (xlibc (explicit-cross-configure (cross-libc target
+                                                      base-libc
+                                                      xgcc-sans-libc
+                                                      xbinutils
+                                                      xkernel)))
          ;; 4. Build a cross-compiling gcc targeting XLIBC, derived from
          ;; BASE-GCC
-         (xgcc (cross-gcc target
-                          #:xgcc base-gcc
-                          #:xbinutils xbinutils
-                          #:libc xlibc)))
+         (xgcc (explicit-cross-configure (cross-gcc target
+                                                    #:xgcc base-gcc
+                                                    #:xbinutils xbinutils
+                                                    #:libc xlibc))))
     ;; Define a meta-package that propagates the resulting XBINUTILS, XLIBC, and
     ;; XGCC
     (package
@@ -94,12 +137,21 @@ chain for " target " development."))
 (define base-gcc gcc-10)
 (define base-linux-kernel-headers linux-libre-headers-5.15)
 
+;; https://gcc.gnu.org/install/configure.html
+(define (hardened-gcc gcc)
+  (package-with-extra-configure-variable (
+    package-with-extra-configure-variable (
+      package-with-extra-configure-variable gcc
+      "--enable-initfini-array" "yes")
+      "--enable-default-ssp" "yes")
+      "--enable-default-pie" "yes"))
+
 (define* (make-bitcoin-cross-toolchain target
                                        #:key
-                                       (base-gcc-for-libc linux-base-gcc)
+                                       (base-gcc-for-libc base-gcc)
                                        (base-kernel-headers base-linux-kernel-headers)
-                                       (base-libc glibc-2.27)
-                                       (base-gcc linux-base-gcc))
+                                       (base-libc (hardened-glibc glibc-2.27))
+                                       (base-gcc (make-gcc-rpath-link (hardened-gcc base-gcc))))
   "Convenience wrapper around MAKE-CROSS-TOOLCHAIN with default values
 desirable for building Bitcoin Core release binaries."
   (make-cross-toolchain target
@@ -108,19 +160,26 @@ desirable for building Bitcoin Core release binaries."
                         base-libc
                         base-gcc))
 
-(define (gcc-mingw-patches gcc)
-  (package-with-extra-patches gcc
-    (search-our-patches "gcc-remap-guix-store.patch"
-                        "vmov-alignment.patch")))
+(define (make-gcc-with-pthreads gcc)
+  (package-with-extra-configure-variable
+    (package-with-extra-patches gcc
+      (search-our-patches "gcc-10-remap-guix-store.patch"))
+    "--enable-threads" "posix"))
+
+(define (make-mingw-w64-cross-gcc cross-gcc)
+  (package-with-extra-patches cross-gcc
+    (search-our-patches "vmov-alignment.patch"
+                        "gcc-broken-longjmp.patch")))
 
 (define (make-mingw-pthreads-cross-toolchain target)
   "Create a cross-compilation toolchain package for TARGET"
   (let* ((xbinutils (cross-binutils target))
          (pthreads-xlibc mingw-w64-x86_64-winpthreads)
-         (pthreads-xgcc (cross-gcc target
-                                    #:xgcc (gcc-mingw-patches mingw-w64-base-gcc)
+         (pthreads-xgcc (make-gcc-with-pthreads
+                         (cross-gcc target
+                                    #:xgcc (make-ssp-fixed-gcc (make-mingw-w64-cross-gcc base-gcc))
                                     #:xbinutils xbinutils
-                                    #:libc pthreads-xlibc)))
+                                    #:libc pthreads-xlibc))))
     ;; Define a meta-package that propagates the resulting XBINUTILS, XLIBC, and
     ;; XGCC
     (package
@@ -140,44 +199,43 @@ chain for " target " development."))
       (home-page (package-home-page pthreads-xgcc))
       (license (package-license pthreads-xgcc)))))
 
-;; While LIEF is packaged in Guix, we maintain our own package,
-;; to simplify building, and more easily apply updates.
-;; Moreover, the Guix's package uses cmake, which caused build
-;; failure; see https://github.com/bitcoin/bitcoin/pull/27296.
+(define (make-nsis-for-gcc-10 base-nsis)
+  (package-with-extra-patches base-nsis
+    (search-our-patches "nsis-gcc-10-memmove.patch"
+                        "nsis-disable-installer-reloc.patch")))
+
+(define (fix-ppc64-nx-default lief)
+  (package-with-extra-patches lief
+    (search-our-patches "lief-fix-ppc64-nx-default.patch")))
+
+;; Our python-lief package can be removed once we are using
+;; guix 83bfdb409787cb2737e68b093a319b247b7858e6 or later.
+;; Note we currently use cmake-minimal.
 (define-public python-lief
   (package
     (name "python-lief")
-    (version "0.13.2")
+    (version "0.12.3")
     (source (origin
               (method git-fetch)
               (uri (git-reference
                     (url "https://github.com/lief-project/LIEF")
                     (commit version)))
               (file-name (git-file-name name version))
-              (modules '((guix build utils)))
-              (snippet
-               '(begin
-                  ;; Configure build for Python bindings.
-                  (substitute* "api/python/config-default.toml"
-                    (("(ninja         = )true" all m)
-                     (string-append m "false"))
-                    (("(parallel-jobs = )0" all m)
-                     (string-append m (number->string (parallel-job-count)))))))
               (sha256
                (base32
-                "0y48x358ppig5xp97ahcphfipx7cg9chldj2q5zrmn610fmi4zll"))))
+                "11i6hqmcjh56y554kqhl61698n9v66j2qk1c1g63mv2w07h2z661"))))
     (build-system python-build-system)
-    (native-inputs (list cmake-minimal python-tomli))
+    (native-inputs (list cmake-minimal))
     (arguments
      (list
       #:tests? #f                  ;needs network
       #:phases #~(modify-phases %standard-phases
-                   (add-before 'build 'change-directory
-                     (lambda _
-                       (chdir "api/python")))
                    (replace 'build
                      (lambda _
-                       (invoke "python" "setup.py" "build"))))))
+                       (invoke
+                        "python" "setup.py" "--sdk" "build"
+                        (string-append
+                         "-j" (number->string (parallel-job-count)))))))))
     (home-page "https://github.com/lief-project/LIEF")
     (synopsis "Library to instrument executable formats")
     (description
@@ -190,16 +248,18 @@ and abstract ELF, PE and MachO formats.")
     (name "osslsigncode")
     (version "2.5")
     (source (origin
-              (method git-fetch)
-              (uri (git-reference
-                    (url "https://github.com/mtrojnar/osslsigncode")
-                    (commit version)))
+              (method url-fetch)
+              (uri (string-append "https://github.com/mtrojnar/"
+                                  name "/archive/" version ".tar.gz"))
               (sha256
                (base32
-                "1j47vwq4caxfv0xw68kw5yh00qcpbd56d7rq6c483ma3y7s96yyz"))))
+                "03by9706gg0an6dn48pljx38vcb76ziv11bgm8ilwsf293x2k4hv"))))
     (build-system cmake-build-system)
     (inputs
      `(("openssl", openssl)))
+    (arguments
+     '(#:configure-flags
+        (list "-DCMAKE_DISABLE_FIND_PACKAGE_CURL=TRUE")))
     (home-page "https://github.com/mtrojnar/osslsigncode")
     (synopsis "Authenticode signing and timestamping tool")
     (description "osslsigncode is a small tool that implements part of the
@@ -235,7 +295,7 @@ thus should be able to compile on most platforms where these exist.")
 (define-public python-oscrypto
   (package
     (name "python-oscrypto")
-    (version "1.3.0")
+    (version "1.2.1")
     (source
      (origin
        (method git-fetch)
@@ -245,7 +305,7 @@ thus should be able to compile on most platforms where these exist.")
        (file-name (git-file-name name version))
        (sha256
         (base32
-         "1v5wkmzcyiqy39db8j2dvkdrv2nlsc48556h73x4dzjwd6kg4q0a"))
+         "1d4d8s4z340qhvb3g5m5v3436y3a71yc26wk4749q64m09kxqc3l"))
        (patches (search-our-patches "oscrypto-hard-code-openssl.patch"))))
     (build-system python-build-system)
     (native-search-paths
@@ -444,7 +504,7 @@ and endian independent.")
     (license license:expat)))
 
 (define-public python-signapple
-  (let ((commit "7a96b4171a360abf0f0f56e499f8f9ed2116280d"))
+  (let ((commit "8a945a2e7583be2665cf3a6a89d665b70ecd1ab6"))
     (package
       (name "python-signapple")
       (version (git-version "0.1" "1" commit))
@@ -457,13 +517,14 @@ and endian independent.")
          (file-name (git-file-name name commit))
          (sha256
           (base32
-           "0aa4k180jnpal15yhncnm3g3z9gzmi7qb25q5l0kaj444a1p2pm4"))))
+           "0fr1hangvfyiwflca6jg5g8zvg3jc9qr7vd2c12ff89pznf38dlg"))))
       (build-system python-build-system)
       (propagated-inputs
        `(("python-asn1crypto" ,python-asn1crypto)
          ("python-oscrypto" ,python-oscrypto)
          ("python-certvalidator" ,python-certvalidator)
          ("python-elfesteem" ,python-elfesteem)
+         ("python-requests" ,python-requests)
          ("python-macholib" ,python-macholib)))
       ;; There are no tests, but attempting to run python setup.py test leads to
       ;; problems, just disable the test
@@ -474,45 +535,16 @@ and endian independent.")
 inspecting signatures in Mach-O binaries.")
       (license license:expat))))
 
-(define-public mingw-w64-base-gcc
-  (package
-    (inherit base-gcc)
-    (arguments
-      (substitute-keyword-arguments (package-arguments base-gcc)
-        ((#:configure-flags flags)
-          `(append ,flags
-            ;; https://gcc.gnu.org/install/configure.html
-            (list "--enable-threads=posix",
-                  building-on)))
-        ((#:make-flags flags)
-          ;; Uses the SSP functions from glibc instead of from libssp.so.
-          ;; Our 'symbol-check' script will complain if we link against libssp.so,
-          ;; and thus will ensure that this works properly.
-          `(cons "gcc_cv_libc_provides_ssp=yes" ,flags))))))
-
-(define-public linux-base-gcc
-  (package
-    (inherit base-gcc)
-    (arguments
-      (substitute-keyword-arguments (package-arguments base-gcc)
-        ((#:configure-flags flags)
-          `(append ,flags
-            ;; https://gcc.gnu.org/install/configure.html
-            (list "--enable-initfini-array=yes",
-                  "--enable-default-ssp=yes",
-                  "--enable-default-pie=yes",
-                  building-on)))
-        ((#:phases phases)
-          `(modify-phases ,phases
-            ;; Given a XGCC package, return a modified package that replace each instance of
-            ;; -rpath in the default system spec that's inserted by Guix with -rpath-link
-            (add-after 'pre-configure 'replace-rpath-with-rpath-link
-             (lambda _
-               (substitute* (cons "gcc/config/rs6000/sysv4.h"
-                                  (find-files "gcc/config"
-                                              "^gnu-user.*\\.h$"))
-                 (("-rpath=") "-rpath-link="))
-               #t))))))))
+;; https://www.gnu.org/software/libc/manual/html_node/Configuring-and-compiling.html
+;; We don't use --disable-werror directly, as that would be passed through to bash,
+;; and cause it's build to fail.
+(define (hardened-glibc glibc)
+  (package-with-extra-configure-variable (
+    package-with-extra-configure-variable (
+      package-with-extra-configure-variable glibc
+      "enable_werror" "no")
+      "--enable-stack-protector" "all")
+      "--enable-bind-now" "yes"))
 
 (define-public glibc-2.27
   (package
@@ -527,32 +559,11 @@ inspecting signatures in Mach-O binaries.")
               (sha256
                (base32
                 "0azpb9cvnbv25zg8019rqz48h8i2257ngyjg566dlnp74ivrs9vq"))
-              (patches (search-our-patches "glibc-2.27-riscv64-Use-__has_include-to-include-asm-syscalls.h.patch"
+              (patches (search-our-patches "glibc-ldd-x86_64.patch"
+                                           "glibc-versioned-locpath.patch"
+                                           "glibc-2.27-riscv64-Use-__has_include-to-include-asm-syscalls.h.patch"
                                            "glibc-2.27-fcommon.patch"
-                                           "glibc-2.27-guix-prefix.patch"
-                                           "glibc-2.27-no-librt.patch"
-                                           "glibc-2.27-powerpc-ldbrx.patch"))))
-    (arguments
-      (substitute-keyword-arguments (package-arguments glibc)
-        ((#:configure-flags flags)
-          `(append ,flags
-            ;; https://www.gnu.org/software/libc/manual/html_node/Configuring-and-compiling.html
-            (list "--enable-stack-protector=all",
-                  "--enable-bind-now",
-                  "--disable-werror",
-                  building-on)))
-    ((#:phases phases)
-        `(modify-phases ,phases
-           (add-before 'configure 'set-etc-rpc-installation-directory
-             (lambda* (#:key outputs #:allow-other-keys)
-               ;; Install the rpc data base file under `$out/etc/rpc'.
-               ;; Otherwise build will fail with "Permission denied."
-               (let ((out (assoc-ref outputs "out")))
-                 (substitute* "sunrpc/Makefile"
-                   (("^\\$\\(inst_sysconfdir\\)/rpc(.*)$" _ suffix)
-                    (string-append out "/etc/rpc" suffix "\n"))
-                   (("^install-others =.*$")
-                    (string-append "install-others = " out "/etc/rpc\n"))))))))))))
+                                           "glibc-2.27-guix-prefix.patch"))))))
 
 (packages->manifest
  (append
@@ -578,7 +589,7 @@ inspecting signatures in Mach-O binaries.")
         xz
         ;; Build tools
         gnu-make
-        libtool
+        libtool-2.4.7
         autoconf-2.71
         automake
         pkg-config
@@ -587,21 +598,21 @@ inspecting signatures in Mach-O binaries.")
         gcc-toolchain-10
         (list gcc-toolchain-10 "static")
         ;; Scripting
-        python-minimal ;; (3.10)
+        python-minimal ;; (3.9)
         ;; Git
         git-minimal
         ;; Tests
-        python-lief)
+        (fix-ppc64-nx-default python-lief))
   (let ((target (getenv "HOST")))
     (cond ((string-suffix? "-mingw32" target)
            ;; Windows
            (list zip
                  (make-mingw-pthreads-cross-toolchain "x86_64-w64-mingw32")
-                 nsis-x86_64
+                 (make-nsis-for-gcc-10 nsis-x86_64)
                  nss-certs
                  osslsigncode))
           ((string-contains target "-linux-")
            (list (make-bitcoin-cross-toolchain target)))
           ((string-contains target "darwin")
-           (list clang-toolchain-15 binutils cmake-minimal python-signapple zip))
+           (list clang-toolchain-10 binutils cmake-minimal xorriso python-signapple))
           (else '())))))

contrib/guix/patches/gcc-broken-longjmp.patch

@@ -0,0 +1,68 @@
+commit eb5698897c52702498938592d7f76e67d126451f
+Author: Eric Botcazou <ebotcazou@adacore.com>
+Date:   Wed May 5 22:48:51 2021 +0200
+
+    Fix PR target/100402
+    
+    This is a regression for 64-bit Windows present from mainline down to the 9
+    branch and introduced by the fix for PR target/99234.  Again SEH, but with
+    a twist related to the way MinGW implements setjmp/longjmp, which turns out
+    to be piggybacked on SEH with recent versions of MinGW, i.e. the longjmp
+    performs a bona-fide unwinding of the stack, because it calls RtlUnwindEx
+    with the second argument initially passed to setjmp, which is the result of
+    __builtin_frame_address (0) in the MinGW header file:
+    
+      define setjmp(BUF) _setjmp((BUF), __builtin_frame_address (0))
+    
+    This means that we directly expose the frame pointer to the SEH machinery
+    here (unlike with regular exception handling where we use an intermediate
+    CFA) and thus that we cannot do whatever we want with it.  The old code
+    would leave it unaligned, i.e. not multiple of 16, whereas the new code
+    aligns it, but this breaks for some reason; at least it appears that a
+    .seh_setframe directive with 0 as second argument always works, so the
+    fix aligns it this way.
+    
+    gcc/
+            PR target/100402
+            * config/i386/i386.c (ix86_compute_frame_layout): For a SEH target,
+            always return the establisher frame for __builtin_frame_address (0).
+    gcc/testsuite/
+            * gcc.c-torture/execute/20210505-1.c: New test.
+
+diff --git a/gcc/config/i386/i386.c b/gcc/config/i386/i386.c
+index 2f838840e96..06ad1b2274e 100644
+--- a/gcc/config/i386/i386.c
++++ b/gcc/config/i386/i386.c
+@@ -6356,12 +6356,29 @@ ix86_compute_frame_layout (void)
+ 	 area, see the SEH code in config/i386/winnt.c for the rationale.  */
+       frame->hard_frame_pointer_offset = frame->sse_reg_save_offset;
+ 
+-      /* If we can leave the frame pointer where it is, do so.  Also, return
++      /* If we can leave the frame pointer where it is, do so; however return
+ 	 the establisher frame for __builtin_frame_address (0) or else if the
+-	 frame overflows the SEH maximum frame size.  */
++	 frame overflows the SEH maximum frame size.
++
++	 Note that the value returned by __builtin_frame_address (0) is quite
++	 constrained, because setjmp is piggybacked on the SEH machinery with
++	 recent versions of MinGW:
++
++	  #    elif defined(__SEH__)
++	  #     if defined(__aarch64__) || defined(_ARM64_)
++	  #      define setjmp(BUF) _setjmp((BUF), __builtin_sponentry())
++	  #     elif (__MINGW_GCC_VERSION < 40702)
++	  #      define setjmp(BUF) _setjmp((BUF), mingw_getsp())
++	  #     else
++	  #      define setjmp(BUF) _setjmp((BUF), __builtin_frame_address (0))
++	  #     endif
++
++	 and the second argument passed to _setjmp, if not null, is forwarded
++	 to the TargetFrame parameter of RtlUnwindEx by longjmp (after it has
++	 built an ExceptionRecord on the fly describing the setjmp buffer).  */
+       const HOST_WIDE_INT diff
+ 	= frame->stack_pointer_offset - frame->hard_frame_pointer_offset;
+-      if (diff <= 255)
++      if (diff <= 255 && !crtl->accesses_prior_frames)
+ 	{
+ 	  /* The resulting diff will be a multiple of 16 lower than 255,
+ 	     i.e. at most 240 as required by the unwind data structure.  */

contrib/guix/patches/glibc-2.27-fcommon.patch

@@ -5,7 +5,7 @@ Date:   Fri May 6 11:03:04 2022 +0100
     build: use -fcommon to retain legacy behaviour with GCC 10
     
     GCC 10 started using -fno-common by default, which causes issues with
-    the powerpc builds using gibc 2.27. A patch was commited to glibc to fix
+    the powerpc builds using gibc 2.24. A patch was commited to glibc to fix
     the issue, 18363b4f010da9ba459b13310b113ac0647c2fcc but is non-trvial
     to backport, and was broken in at least one way, see the followup in
     commit 7650321ce037302bfc2f026aa19e0213b8d02fe6.
@@ -17,8 +17,6 @@ Date:   Fri May 6 11:03:04 2022 +0100
     https://sourceware.org/git/?p=glibc.git;a=commit;h=18363b4f010da9ba459b13310b113ac0647c2fcc
     https://sourceware.org/git/?p=glibc.git;a=commit;h=7650321ce037302bfc2f026aa19e0213b8d02fe6
 
-    This patch can be dropped when we are building with glibc 2.31+.
-
 diff --git a/Makeconfig b/Makeconfig
 index 86a71e5802..aa2166be60 100644
 --- a/Makeconfig

contrib/guix/patches/glibc-2.27-guix-prefix.patch

@@ -5,7 +5,7 @@ In order to be reproducible regardless of the architecture used to build
 the package, map all guix store prefixes to something fixed, e.g. /usr.
 
 We might be able to drop this in favour of using --with-nonshared-cflags
-when we begin using newer versions of glibc.
+when we being using newer versions of glibc.
 
 --- a/Makeconfig
 +++ b/Makeconfig

contrib/guix/patches/glibc-2.27-no-librt.patch

@@ -1,53 +0,0 @@
-This patch can be dropped when we are building with glibc 2.30+.
-
-commit 6e41ef56c9baab719a02f1377b1e7ce7bff61e73
-Author: Florian Weimer <fweimer@redhat.com>
-Date:   Fri Feb 8 10:21:56 2019 +0100
-
-    rt: Turn forwards from librt to libc into compat symbols [BZ #24194]
-    
-    As the  result of commit 6e6249d0b461b952d0f544792372663feb6d792a
-    ("BZ#14743: Move clock_* symbols from librt to libc."), in glibc 2.17,
-    clock_gettime, clock_getres, clock_settime, clock_getcpuclockid,
-    clock_nanosleep were added to libc, and the file rt/clock-compat.c
-    was added with forwarders to the actual implementations in libc.
-    These forwarders were wrapped in
-
-    #if SHLIB_COMPAT (librt, GLIBC_2_2, GLIBC_2_17)
-    
-    so that they are not present for newer architectures (such as
-    powerpc64le) with a 2.17 or later ABI baseline.  But the forwarders
-    were not marked as compatibility symbols.  As a result, on older
-    architectures, historic configure checks such as
-    
-    AC_CHECK_LIB(rt, clock_gettime)
-    
-    still cause linking against librt, even though this is completely
-    unnecessary.  It also creates a needless porting hazard because
-    architectures behave differently when it comes to symbol availability.
-    
-    Reviewed-by: Carlos O'Donell <carlos@redhat.com>
-
-diff --git a/rt/clock-compat.c b/rt/clock-compat.c
-index f816973c05..11e71aa890 100644
---- a/rt/clock-compat.c
-+++ b/rt/clock-compat.c
-@@ -30,14 +30,16 @@
- #if HAVE_IFUNC
- # undef INIT_ARCH
- # define INIT_ARCH()
--# define COMPAT_REDIRECT(name, proto, arglist) libc_ifunc (name, &__##name)
-+# define COMPAT_REDIRECT(name, proto, arglist) libc_ifunc (name, &__##name) \
-+    compat_symbol (librt, name, name, GLIBC_2_2);
- #else
- # define COMPAT_REDIRECT(name, proto, arglist)				      \
-   int									      \
-   name proto								      \
-   {									      \
-     return __##name arglist;						      \
--  }
-+  }									      \
-+  compat_symbol (librt, name, name, GLIBC_2_2);
- #endif
- 
- COMPAT_REDIRECT (clock_getres,

contrib/guix/patches/glibc-2.27-powerpc-ldbrx.patch

@@ -1,245 +0,0 @@
-From 50b0b3c9ff71ffd7ebbd74ae46844c3566478123 Mon Sep 17 00:00:00 2001
-From: "Gabriel F. T. Gomes" <gabrielftg@linux.ibm.com>
-Date: Mon, 27 May 2019 15:21:22 -0300
-Subject: [PATCH] powerpc: Fix build failures with current GCC
-
-Since GCC commit 271500 (svn), also known as the following commit on the
-git mirror:
-
-commit e154242724b084380e3221df7c08fcdbd8460674
-Author: amodra <amodra@138bc75d-0d04-0410-961f-82ee72b054a4>
-Date:   Wed May 22 04:34:26 2019 +0000
-
-    [RS6000] Don't pass -many to the assembler
-
-glibc builds are failing when an assembly implementation does not
-declare the correct '.machine' directive, or when no such directive is
-declared at all.  For example, when a POWER6 instruction is used, but
-'.machine power6' is not declared, the assembler will fail with an error
-similar to the following:
-
-    ../sysdeps/powerpc/powerpc64/power8/strcmp.S: Assembler messages:
-     24 ../sysdeps/powerpc/powerpc64/power8/strcmp.S:55: Error: unrecognized opcode: `cmpb'
-
-This patch adds '.machine powerN' directives where none existed, as well
-as it updates '.machine power7' directives on POWER8 files, because the
-minimum binutils version required to build glibc (binutils 2.25) now
-provides this machine version.  It also adds '-many' to the assembler
-command used to build tst-set_ppr.c.
-
-Tested for powerpc, powerpc64, and powerpc64le, as well as with
-build-many-glibcs.py for powerpc targets.
-
-Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
----
- sysdeps/powerpc/Makefile                      |  5 +++
- sysdeps/powerpc/powerpc64/power4/memcmp.S     |  7 ++++
- sysdeps/powerpc/powerpc64/power7/strncmp.S    |  1 +
- .../powerpc/powerpc64/power8/fpu/s_llround.S  |  1 +
- sysdeps/powerpc/powerpc64/power8/strcasecmp.S | 36 ++++++-------------
- sysdeps/powerpc/powerpc64/power8/strcasestr.S | 14 ++------
- sysdeps/powerpc/powerpc64/power8/strcmp.S     |  1 +
- 7 files changed, 28 insertions(+), 37 deletions(-)
-
-diff --git a/sysdeps/powerpc/Makefile b/sysdeps/powerpc/Makefile
-index 6aa683b03f..23126147df 100644
---- a/sysdeps/powerpc/Makefile
-+++ b/sysdeps/powerpc/Makefile
-@@ -45,6 +45,11 @@ ifeq ($(subdir),misc)
- sysdep_headers += sys/platform/ppc.h
- tests += test-gettimebase
- tests += tst-set_ppr
-+
-+# This test is expected to run and exit with EXIT_UNSUPPORTED on
-+# processors that do not implement the Power ISA 2.06 or greater.
-+# But the test makes use of instructions from Power ISA 2.06 and 2.07.
-+CFLAGS-tst-set_ppr.c += -Wa,-many
- endif
- 
- ifneq (,$(filter %le,$(config-machine)))
-diff --git a/sysdeps/powerpc/powerpc64/power4/memcmp.S b/sysdeps/powerpc/powerpc64/power4/memcmp.S
-index e5319f101f..38dcf4c9a1 100644
---- a/sysdeps/powerpc/powerpc64/power4/memcmp.S
-+++ b/sysdeps/powerpc/powerpc64/power4/memcmp.S
-@@ -26,7 +26,14 @@
- # define MEMCMP memcmp
- #endif
- 
-+#ifndef __LITTLE_ENDIAN__
- 	.machine power4
-+#else
-+/* Little endian is only available since POWER8, so it's safe to
-+   specify .machine as power8 (or older), even though this is a POWER4
-+   file.  Since the little-endian code uses 'ldbrx', power7 is enough. */
-+	.machine power7
-+#endif
- ENTRY_TOCLESS (MEMCMP, 4)
- 	CALL_MCOUNT 3
- 
-diff --git a/sysdeps/powerpc/powerpc64/power7/strncmp.S b/sysdeps/powerpc/powerpc64/power7/strncmp.S
-index 0c7429d19f..10f898c5a3 100644
---- a/sysdeps/powerpc/powerpc64/power7/strncmp.S
-+++ b/sysdeps/powerpc/powerpc64/power7/strncmp.S
-@@ -28,6 +28,7 @@
- 		     const char *s2 [r4],
- 		     size_t size [r5])  */
- 
-+	.machine power7
- ENTRY_TOCLESS (STRNCMP, 5)
- 	CALL_MCOUNT 3
- 
-diff --git a/sysdeps/powerpc/powerpc64/power8/fpu/s_llround.S b/sysdeps/powerpc/powerpc64/power8/fpu/s_llround.S
-index a22fc63bb3..84c76ba0f9 100644
---- a/sysdeps/powerpc/powerpc64/power8/fpu/s_llround.S
-+++ b/sysdeps/powerpc/powerpc64/power8/fpu/s_llround.S
-@@ -26,6 +26,7 @@
- 
- /* long long [r3] llround (float x [fp1])  */
- 
-+	.machine power8
- ENTRY_TOCLESS (__llround)
- 	CALL_MCOUNT 0
- 	frin	fp1,fp1	/* Round to nearest +-0.5.  */
-diff --git a/sysdeps/powerpc/powerpc64/power8/strcasecmp.S b/sysdeps/powerpc/powerpc64/power8/strcasecmp.S
-index 3a2efe2a64..eeacd40c7f 100644
---- a/sysdeps/powerpc/powerpc64/power8/strcasecmp.S
-+++ b/sysdeps/powerpc/powerpc64/power8/strcasecmp.S
-@@ -91,21 +91,7 @@
- 3: \
- 	TOLOWER()
- 
--#ifdef _ARCH_PWR8
--#  define VCLZD_V8_v7	vclzd	v8, v7;
--#  define MFVRD_R3_V1	mfvrd	r3, v1;
--#  define VSUBUDM_V9_V8	vsubudm	v9, v9, v8;
--#  define VPOPCNTD_V8_V8	vpopcntd v8, v8;
--#  define VADDUQM_V7_V8	vadduqm	v9, v7, v8;
--#else
--#  define VCLZD_V8_v7	.long	0x11003fc2
--#  define MFVRD_R3_V1	.long	0x7c230067
--#  define VSUBUDM_V9_V8	.long	0x112944c0
--#  define VPOPCNTD_V8_V8	.long	0x110047c3
--#  define VADDUQM_V7_V8	.long	0x11274100
--#endif
--
--	.machine  power7
-+	.machine  power8
- 
- ENTRY (__STRCASECMP)
- #ifdef USE_AS_STRNCASECMP
-@@ -265,15 +251,15 @@ L(different):
- #ifdef __LITTLE_ENDIAN__
- 	/* Count trailing zero.  */
- 	vspltisb	v8, -1
--	VADDUQM_V7_V8
-+	vadduqm	v9, v7, v8
- 	vandc	v8, v9, v7
--	VPOPCNTD_V8_V8
-+	vpopcntd	v8, v8
- 	vspltb	v6, v8, 15
- 	vcmpequb.	v6, v6, v1
- 	blt	cr6, L(shift8)
- #else
- 	/* Count leading zero.  */
--	VCLZD_V8_v7
-+	vclzd	v8, v7
- 	vspltb	v6, v8, 7
- 	vcmpequb.	v6, v6, v1
- 	blt	cr6, L(shift8)
-@@ -291,7 +277,7 @@ L(skipsum):
- 	/* Merge and move to GPR.  */
- 	vmrglb	v6, v6, v7
- 	vslo	v1, v6, v1
--	MFVRD_R3_V1
-+	mfvrd	r3, v1
- 	/* Place the characters that are different in first position.  */
- 	sldi	rSTR2, rRTN, 56
- 	srdi	rSTR2, rSTR2, 56
-@@ -301,7 +287,7 @@ L(skipsum):
- 	vslo	v6, v5, v8
- 	vslo	v7, v4, v8
- 	vmrghb	v1, v6, v7
--	MFVRD_R3_V1
-+	mfvrd	r3, v1
- 	srdi	rSTR2, rRTN, 48
- 	sldi	rSTR2, rSTR2, 56
- 	srdi	rSTR2, rSTR2, 56
-@@ -320,15 +306,15 @@ L(null_found):
- #ifdef __LITTLE_ENDIAN__
- 	/* Count trailing zero.  */
- 	vspltisb	v8, -1
--	VADDUQM_V7_V8
-+	vadduqm	v9, v7, v8
- 	vandc	v8, v9, v7
--	VPOPCNTD_V8_V8
-+	vpopcntd	v8, v8
- 	vspltb	v6, v8, 15
- 	vcmpequb.	v6, v6, v10
- 	blt	cr6, L(shift_8)
- #else
- 	/* Count leading zero.  */
--	VCLZD_V8_v7
-+	vclzd	v8, v7
- 	vspltb	v6, v8, 7
- 	vcmpequb.	v6, v6, v10
- 	blt	cr6, L(shift_8)
-@@ -343,10 +329,10 @@ L(skipsum1):
- 	vspltisb	v10, 7
- 	vslb	v10, v10, v10
- 	vsldoi	v9, v0, v10, 1
--	VSUBUDM_V9_V8
-+	vsubudm	v9, v9, v8
- 	vspltisb	v8, 8
- 	vsldoi	v8, v0, v8, 1
--	VSUBUDM_V9_V8
-+	vsubudm	v9, v9, v8
- 	/* Shift and remove junk after null character.  */
- #ifdef __LITTLE_ENDIAN__
- 	vslo	v5, v5, v9
-diff --git a/sysdeps/powerpc/powerpc64/power8/strcasestr.S b/sysdeps/powerpc/powerpc64/power8/strcasestr.S
-index 9fc24c29f9..e10f06fd86 100644
---- a/sysdeps/powerpc/powerpc64/power8/strcasestr.S
-+++ b/sysdeps/powerpc/powerpc64/power8/strcasestr.S
-@@ -73,18 +73,8 @@
- 	vor	reg, v8, reg; \
- 	vcmpequb.	v6, reg, v4;
- 
--/* TODO: change these to the actual instructions when the minimum required
--   binutils allows it.  */
--#ifdef _ARCH_PWR8
--#define VCLZD_V8_v7	vclzd	v8, v7;
--#else
--#define VCLZD_V8_v7	.long	0x11003fc2
--#endif
--
- #define	FRAMESIZE	(FRAME_MIN_SIZE+48)
--/* TODO: change this to .machine power8 when the minimum required binutils
--   allows it.  */
--	.machine  power7
-+	.machine  power8
- ENTRY (STRCASESTR, 4)
- 	CALL_MCOUNT 2
- 	mflr	r0			/* Load link register LR to r0.  */
-@@ -291,7 +281,7 @@ L(nullchk1):
- 	vcmpequb.	v6, v0, v7
- 	/* Shift r3 by 16 bytes and proceed.  */
- 	blt	cr6, L(shift16)
--	VCLZD_V8_v7
-+	vclzd	v8, v7
- #ifdef __LITTLE_ENDIAN__
- 	vspltb	v6, v8, 15
- #else
-diff --git a/sysdeps/powerpc/powerpc64/power8/strcmp.S b/sysdeps/powerpc/powerpc64/power8/strcmp.S
-index 15e7351d1b..d592266d1d 100644
---- a/sysdeps/powerpc/powerpc64/power8/strcmp.S
-+++ b/sysdeps/powerpc/powerpc64/power8/strcmp.S
-@@ -31,6 +31,7 @@
-    64K as default, the page cross handling assumes minimum page size of
-    4k.  */
- 
-+	.machine power8
- ENTRY_TOCLESS (STRCMP, 4)
- 	li	r0,0
- 
--- 
-2.41.0

contrib/guix/patches/glibc-2.27-riscv64-Use-__has_include-to-include-asm-syscalls.h.patch

@@ -4,8 +4,6 @@ See also: http://lists.busybox.net/pipermail/buildroot/2020-July/590376.html.
 
 https://sourceware.org/git/?p=glibc.git;a=commit;h=0b9c84906f653978fb8768c7ebd0ee14a47e662e
 
-This patch can be dropped when we are building with glibc 2.28+.
-
 From 562c52cc81a4e456a62e6455feb32732049e9070 Mon Sep 17 00:00:00 2001
 From: "H.J. Lu" <hjl.tools@gmail.com>
 Date: Mon, 31 Dec 2018 09:26:42 -0800

contrib/guix/patches/glibc-ldd-x86_64.patch

@@ -0,0 +1,10 @@
+By default, 'RTDLLIST' in 'ldd' refers to 'lib64/ld-linux-x86-64.so', whereas
+it's in 'lib/' for us.  This patch fixes that.
+
+--- a/sysdeps/unix/sysv/linux/x86_64/ldd-rewrite.sed
++++ b/sysdeps/unix/sysv/linux/x86_64/ldd-rewrite.sed
+@@ -1,3 +1,3 @@
+ /LD_TRACE_LOADED_OBJECTS=1/a\
+ add_env="$add_env LD_LIBRARY_VERSION=\\$verify_out"
+-s_^\(RTLDLIST=\)\(.*lib\)\(\|64\|x32\)\(/[^/]*\)\(-x86-64\|-x32\)\(\.so\.[0-9.]*\)[ 	]*$_\1"\2\4\6 \264\4-x86-64\6 \2x32\4-x32\6"_
++s_^\(RTLDLIST=\)\(.*lib\)\(\|64\|x32\)\(/[^/]*\)\(-x86-64\|-x32\)\(\.so\.[0-9.]*\)[ 	]*$_\1"\2\4\6 \2\4-x86-64\6 \2x32\4-x32\6"_

contrib/guix/patches/glibc-versioned-locpath.patch

@@ -0,0 +1,240 @@
+The format of locale data can be incompatible between libc versions, and
+loading incompatible data can lead to 'setlocale' returning EINVAL at best
+or triggering an assertion failure at worst.  See
+https://lists.gnu.org/archive/html/guix-devel/2015-09/msg00717.html
+for background information.
+
+To address that, this patch changes libc to honor a new 'GUIX_LOCPATH'
+variable, and to look for locale data in version-specific sub-directories of
+that variable.  So, if GUIX_LOCPATH=/foo:/bar, locale data is searched for in
+/foo/X.Y and /bar/X.Y, where X.Y is the libc version number.
+
+That way, a single 'GUIX_LOCPATH' setting can work even if different libc
+versions coexist on the system.
+
+--- a/locale/newlocale.c
++++ b/locale/newlocale.c
+@@ -30,6 +30,7 @@
+ /* Lock for protecting global data.  */
+ __libc_rwlock_define (extern , __libc_setlocale_lock attribute_hidden)
+ 
++extern error_t compute_locale_search_path (char **, size_t *);
+ 
+ /* Use this when we come along an error.  */
+ #define ERROR_RETURN							      \
+@@ -48,7 +49,6 @@ __newlocale (int category_mask, const char *locale, __locale_t base)
+   __locale_t result_ptr;
+   char *locale_path;
+   size_t locale_path_len;
+-  const char *locpath_var;
+   int cnt;
+   size_t names_len;
+ 
+@@ -102,17 +102,8 @@ __newlocale (int category_mask, const char *locale, __locale_t base)
+   locale_path = NULL;
+   locale_path_len = 0;
+ 
+-  locpath_var = getenv ("LOCPATH");
+-  if (locpath_var != NULL && locpath_var[0] != '\0')
+-    {
+-      if (__argz_create_sep (locpath_var, ':',
+-			     &locale_path, &locale_path_len) != 0)
+-	return NULL;
+-
+-      if (__argz_add_sep (&locale_path, &locale_path_len,
+-			  _nl_default_locale_path, ':') != 0)
+-	return NULL;
+-    }
++  if (compute_locale_search_path (&locale_path, &locale_path_len) != 0)
++    return NULL;
+ 
+   /* Get the names for the locales we are interested in.  We either
+      allow a composite name or a single name.  */
+diff --git a/locale/setlocale.c b/locale/setlocale.c
+index ead030d..0c0e314 100644
+--- a/locale/setlocale.c
++++ b/locale/setlocale.c
+@@ -215,12 +215,65 @@ setdata (int category, struct __locale_data *data)
+     }
+ }
+ 
++/* Return in *LOCALE_PATH and *LOCALE_PATH_LEN the locale data search path as
++   a colon-separated list.  Return ENOMEN on error, zero otherwise.  */
++error_t
++compute_locale_search_path (char **locale_path, size_t *locale_path_len)
++{
++  char* guix_locpath_var = getenv ("GUIX_LOCPATH");
++  char *locpath_var = getenv ("LOCPATH");
++
++  if (guix_locpath_var != NULL && guix_locpath_var[0] != '\0')
++    {
++      /* Entries in 'GUIX_LOCPATH' take precedence over 'LOCPATH'.  These
++	 entries are systematically prefixed with "/X.Y" where "X.Y" is the
++	 libc version.  */
++      if (__argz_create_sep (guix_locpath_var, ':',
++			     locale_path, locale_path_len) != 0
++	  || __argz_suffix_entries (locale_path, locale_path_len,
++				    "/" VERSION) != 0)
++	goto bail_out;
++    }
++
++  if (locpath_var != NULL && locpath_var[0] != '\0')
++    {
++      char *reg_locale_path = NULL;
++      size_t reg_locale_path_len = 0;
++
++      if (__argz_create_sep (locpath_var, ':',
++			     &reg_locale_path, &reg_locale_path_len) != 0)
++	goto bail_out;
++
++      if (__argz_append (locale_path, locale_path_len,
++			 reg_locale_path, reg_locale_path_len) != 0)
++	goto bail_out;
++
++      free (reg_locale_path);
++    }
++
++  if (*locale_path != NULL)
++    {
++      /* Append the system default locale directory.  */
++      if (__argz_add_sep (locale_path, locale_path_len,
++			  _nl_default_locale_path, ':') != 0)
++	goto bail_out;
++    }
++
++  return 0;
++
++ bail_out:
++  free (*locale_path);
++  *locale_path = NULL;
++  *locale_path_len = 0;
++
++  return ENOMEM;
++}
++
+ char *
+ setlocale (int category, const char *locale)
+ {
+   char *locale_path;
+   size_t locale_path_len;
+-  const char *locpath_var;
+   char *composite;
+ 
+   /* Sanity check for CATEGORY argument.  */
+@@ -251,17 +304,10 @@ setlocale (int category, const char *locale)
+   locale_path = NULL;
+   locale_path_len = 0;
+ 
+-  locpath_var = getenv ("LOCPATH");
+-  if (locpath_var != NULL && locpath_var[0] != '\0')
++  if (compute_locale_search_path (&locale_path, &locale_path_len) != 0)
+     {
+-      if (__argz_create_sep (locpath_var, ':',
+-			     &locale_path, &locale_path_len) != 0
+-	  || __argz_add_sep (&locale_path, &locale_path_len,
+-			     _nl_default_locale_path, ':') != 0)
+-	{
+-	  __libc_rwlock_unlock (__libc_setlocale_lock);
+-	  return NULL;
+-	}
++      __libc_rwlock_unlock (__libc_setlocale_lock);
++      return NULL;
+     }
+ 
+   if (category == LC_ALL)
+diff --git a/string/Makefile b/string/Makefile
+index 8424a61..f925503 100644
+--- a/string/Makefile
++++ b/string/Makefile
+@@ -38,7 +38,7 @@ routines	:= strcat strchr strcmp strcoll strcpy strcspn		\
+ 		   swab strfry memfrob memmem rawmemchr strchrnul	\
+ 		   $(addprefix argz-,append count create ctsep next	\
+ 				     delete extract insert stringify	\
+-				     addsep replace)			\
++				     addsep replace suffix)		\
+ 		   envz basename					\
+ 		   strcoll_l strxfrm_l string-inlines memrchr		\
+ 		   xpg-strerror strerror_l
+diff --git a/string/argz-suffix.c b/string/argz-suffix.c
+new file mode 100644
+index 0000000..505b0f2
+--- /dev/null
++++ b/string/argz-suffix.c
+@@ -0,0 +1,56 @@
++/* Copyright (C) 2015 Free Software Foundation, Inc.
++   This file is part of the GNU C Library.
++   Contributed by Ludovic Courtès <ludo@gnu.org>.
++
++   The GNU C Library is free software; you can redistribute it and/or
++   modify it under the terms of the GNU Lesser General Public
++   License as published by the Free Software Foundation; either
++   version 2.1 of the License, or (at your option) any later version.
++
++   The GNU C Library is distributed in the hope that it will be useful,
++   but WITHOUT ANY WARRANTY; without even the implied warranty of
++   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
++   Lesser General Public License for more details.
++
++   You should have received a copy of the GNU Lesser General Public
++   License along with the GNU C Library; if not, see
++   <http://www.gnu.org/licenses/>.  */
++
++#include <argz.h>
++#include <errno.h>
++#include <stdlib.h>
++#include <string.h>
++
++
++error_t
++__argz_suffix_entries (char **argz, size_t *argz_len, const char *suffix)
++
++{
++  size_t suffix_len = strlen (suffix);
++  size_t count = __argz_count (*argz, *argz_len);
++  size_t new_argz_len = *argz_len + count * suffix_len;
++  char *new_argz = malloc (new_argz_len);
++
++  if (new_argz)
++    {
++      char *p = new_argz, *entry;
++
++      for (entry = *argz;
++	   entry != NULL;
++	   entry = argz_next (*argz, *argz_len, entry))
++	{
++	  p = stpcpy (p, entry);
++	  p = stpcpy (p, suffix);
++	  p++;
++	}
++
++      free (*argz);
++      *argz = new_argz;
++      *argz_len = new_argz_len;
++
++      return 0;
++    }
++  else
++    return ENOMEM;
++}
++weak_alias (__argz_suffix_entries, argz_suffix_entries)
+diff --git a/string/argz.h b/string/argz.h
+index bb62a31..d276a35 100644
+--- a/string/argz.h
++++ b/string/argz.h
+@@ -134,6 +134,16 @@ extern error_t argz_replace (char **__restrict __argz,
+ 			     const char *__restrict __str,
+ 			     const char *__restrict __with,
+ 			     unsigned int *__restrict __replace_count);
++
++/* Suffix each entry of ARGZ & ARGZ_LEN with SUFFIX.  Return 0 on success,
++   and ENOMEN if memory cannot be allocated.  */
++extern error_t __argz_suffix_entries (char **__restrict __argz,
++				      size_t *__restrict __argz_len,
++				      const char *__restrict __suffix);
++extern error_t argz_suffix_entries (char **__restrict __argz,
++				    size_t *__restrict __argz_len,
++				    const char *__restrict __suffix);
++
+ 
+ /* Returns the next entry in ARGZ & ARGZ_LEN after ENTRY, or NULL if there
+    are no more.  If entry is NULL, then the first entry is returned.  This

contrib/guix/patches/lief-fix-ppc64-nx-default.patch

@@ -0,0 +1,29 @@
+Correct default for Binary::has_nx on ppc64
+
+From the Linux kernel source:
+
+    * This is the default if a program doesn't have a PT_GNU_STACK
+    * program header entry. The PPC64 ELF ABI has a non executable stack
+    * stack by default, so in the absence of a PT_GNU_STACK program header
+    * we turn execute permission off.
+
+This patch can be dropped the next time we update LIEF.
+
+diff --git a/src/ELF/Binary.cpp b/src/ELF/Binary.cpp
+index a90be1ab..fd2d9764 100644
+--- a/src/ELF/Binary.cpp
++++ b/src/ELF/Binary.cpp
+@@ -1084,7 +1084,12 @@ bool Binary::has_nx() const {
+                                        return segment->type() == SEGMENT_TYPES::PT_GNU_STACK;
+                                      });
+   if (it_stack == std::end(segments_)) {
+-    return false;
++    if (header().machine_type() == ARCH::EM_PPC64) {
++      // The PPC64 ELF ABI has a non-executable stack by default.
++      return true;
++    } else {
++      return false;
++    }
+   }
+ 
+   return !(*it_stack)->has(ELF_SEGMENT_FLAGS::PF_X);

contrib/guix/patches/nsis-disable-installer-reloc.patch

@@ -0,0 +1,30 @@
+Patch NSIS so that it's installer stubs, produced at NSIS build time,
+do not contain .reloc sections, which will exist by default when using
+binutils/ld 2.36+.
+
+This ultimately fixes an issue when running the installer with the
+"Force randomization for images (Mandatory ASLR)" setting active.
+
+This patch has not yet been sent upstream, because it's not clear if this
+is the best fix, for the underlying issue, which seems to be that makensis
+doesn't account for .reloc sections when it builds installers.
+
+The existence of a reloc section shouldn't be a problem, and, if anything,
+is actually a requirement for working ASLR. All other Windows binaries we
+produce contain them, and function correctly when under the same
+"Force randomization for images (Mandatory ASLR)" setting.
+
+See:
+https://github.com/bitcoin/bitcoin/issues/25726
+https://sourceforge.net/p/nsis/bugs/1131/
+
+--- a/SCons/Config/gnu
++++ b/SCons/Config/gnu
+@@ -102,6 +102,7 @@ stub_env.Append(LINKFLAGS = ['-mwindows'])          # build windows executables
+ stub_env.Append(LINKFLAGS = ['$NODEFLIBS_FLAG'])    # no standard libraries
+ stub_env.Append(LINKFLAGS = ['$ALIGN_FLAG'])        # 512 bytes align
+ stub_env.Append(LINKFLAGS = ['$MAP_FLAG'])          # generate map file
++stub_env.Append(LINKFLAGS = ['-Wl,--disable-reloc-section'])
+ 
+ conf = FlagsConfigure(stub_env)
+ conf.CheckCompileFlag('-fno-tree-loop-distribute-patterns')  # GCC 10: Don't generate msvcrt!memmove calls (bug #1248)

contrib/guix/patches/nsis-gcc-10-memmove.patch

@@ -0,0 +1,23 @@
+commit f6df41524e703dc471e283e566a48e05a735b7f2
+Author: Anders <anders_k@users.sourceforge.net>
+Date:   Sat Jun 27 23:18:45 2020 +0000
+
+    Don't let GCC 10 generate memmove calls (bug #1248)
+    
+    git-svn-id: https://svn.code.sf.net/p/nsis/code/NSIS/trunk@7189 212acab6-be3b-0410-9dea-997c60f758d6
+
+diff --git a/SCons/Config/gnu b/SCons/Config/gnu
+index bfcb362d..21fa446b 100644
+--- a/SCons/Config/gnu
++++ b/SCons/Config/gnu
+@@ -103,6 +103,10 @@ stub_env.Append(LINKFLAGS = ['$NODEFLIBS_FLAG'])    # no standard libraries
+ stub_env.Append(LINKFLAGS = ['$ALIGN_FLAG'])        # 512 bytes align
+ stub_env.Append(LINKFLAGS = ['$MAP_FLAG'])          # generate map file
+ 
++conf = FlagsConfigure(stub_env)
++conf.CheckCompileFlag('-fno-tree-loop-distribute-patterns')  # GCC 10: Don't generate msvcrt!memmove calls (bug #1248)
++conf.Finish()
++
+ stub_uenv = stub_env.Clone()
+ stub_uenv.Append(CPPDEFINES = ['_UNICODE', 'UNICODE'])
+