mirror of
https://github.com/bitcoin/bitcoin.git
synced 2025-07-07 08:50:26 +02:00
guix: Use --cores instead of --max-jobs
In Guix, there are two flags for controlling parallelism: Note: When I say "derivation," think "package" --cores=n - controls the number of CPU cores to build each derivation. This is the value passed to `make`'s `--jobs=` flag. - defaults to 0: as many cores as is available --max-jobs=n - controls how many derivations can be built in parallel - defaults to 1 Therefore, if set --max-jobs=$MAX_JOBS and don't set --cores, Guix could theoretically spin up $MAX_JOBS * $(nproc) number of threads, and that's no good. So we could either default to --cores=1, --max-jobs=$MAX_JOBS - Pro: --cores=1 means that `make` will be invoked with `-j1`, avoiding problems with package whose build systems and test suites break when running multi-threaded. - Con: There will be times when only 1 or 2 derivations can be built at a time, because the rest of the dependency graph all depend on those 1 or 2 derivations. During these times, the machine will be severely under-utilized. or --cores=$MAX_JOBS, --max-jobs=1 - Pro: We don't encounter prolonged periods of severe under-utilization mentioned above. - Con: Many packages' build systems and test suites break when running multi-threaded. or --cores=1, --max-jobs=1 and let the user override with $ADDITIONAL_GUIX_COMMON_FLAGS
This commit is contained in:
@ -80,6 +80,50 @@ at the end of the `guix pull`)
|
|||||||
export PATH="${HOME}/.config/guix/current/bin${PATH:+:}$PATH"
|
export PATH="${HOME}/.config/guix/current/bin${PATH:+:}$PATH"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Controlling the number of threads used by `guix` build commands
|
||||||
|
|
||||||
|
By default, the scripts under `./contrib/guix` will invoke all `guix` build
|
||||||
|
commands with `--cores="$JOBS"`. Note that `$JOBS` defaults to `$(nproc)` if not
|
||||||
|
specified. However, astute manual readers will also notice that there is a
|
||||||
|
`--max-jobs=` flag (which defaults to 1 if unspecified).
|
||||||
|
|
||||||
|
Here is the difference between `--cores=` and `--max-jobs=`:
|
||||||
|
|
||||||
|
> Note: When I say "derivation," think "package"
|
||||||
|
|
||||||
|
`--cores=`
|
||||||
|
|
||||||
|
- controls the number of CPU cores to build each derivation. This is the value
|
||||||
|
passed to `make`'s `--jobs=` flag.
|
||||||
|
|
||||||
|
`--max-jobs=`
|
||||||
|
|
||||||
|
- controls how many derivations can be built in parallel
|
||||||
|
- defaults to 1
|
||||||
|
|
||||||
|
Therefore, the default is for `guix` build commands to build one derivation at a
|
||||||
|
time, utilizing `$JOBS` threads.
|
||||||
|
|
||||||
|
Specifying the `$JOBS` environment variable will only modify `--cores=`, but you
|
||||||
|
can also modify the value for `--max-jobs=` by specifying
|
||||||
|
`$ADDITIONAL_GUIX_COMMON_FLAGS`. For example, if you have a LOT of memory, you
|
||||||
|
may want to set:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
|
||||||
|
```
|
||||||
|
|
||||||
|
Which allows for a maximum of 8 derivations to be built at the same time, each
|
||||||
|
utilizing `$JOBS` threads.
|
||||||
|
|
||||||
|
Or, if you'd like to avoid spurious build failures caused by issues with
|
||||||
|
parallelism within a single package, but would still like to build multiple
|
||||||
|
packages when the dependency graph allows for it, you may want to try:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export JOBS=1 ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
|
||||||
|
```
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
### As a Tool for Deterministic Builds
|
### As a Tool for Deterministic Builds
|
||||||
@ -125,12 +169,16 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
|
|||||||
the actual SDK (e.g. SDK_PATH=$HOME/Downloads/macOS-SDKs instead of
|
the actual SDK (e.g. SDK_PATH=$HOME/Downloads/macOS-SDKs instead of
|
||||||
$HOME/Downloads/macOS-SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers).
|
$HOME/Downloads/macOS-SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers).
|
||||||
|
|
||||||
* _**MAX_JOBS**_
|
* _**JOBS**_
|
||||||
|
|
||||||
Override the maximum number of jobs to run simultaneously, you might want to
|
Override the number of jobs to run simultaneously, you might want to do so on
|
||||||
do so on a memory-limited machine. This may be passed to `make` as in `make
|
a memory-limited machine. This may be passed to:
|
||||||
--jobs="$MAX_JOBS"` or `xargs` as in `xargs -P"$MAX_JOBS"`. _(defaults to the
|
|
||||||
value of `nproc` outside the container)_
|
- `guix` build commands as in `guix environment --cores="$JOBS"`
|
||||||
|
- `make` as in `make --jobs="$JOBS"`
|
||||||
|
- `xargs` as in `xargs -P"$JOBS"`
|
||||||
|
|
||||||
|
_(defaults to the value of `nproc` outside the container)_
|
||||||
|
|
||||||
* _**SOURCE_DATE_EPOCH**_
|
* _**SOURCE_DATE_EPOCH**_
|
||||||
|
|
||||||
|
@ -134,7 +134,7 @@ done
|
|||||||
|
|
||||||
# Determine the maximum number of jobs to run simultaneously (overridable by
|
# Determine the maximum number of jobs to run simultaneously (overridable by
|
||||||
# environment)
|
# environment)
|
||||||
MAX_JOBS="${MAX_JOBS:-$(nproc)}"
|
JOBS="${JOBS:-$(nproc)}"
|
||||||
|
|
||||||
# Usage: host_to_commonname HOST
|
# Usage: host_to_commonname HOST
|
||||||
#
|
#
|
||||||
@ -152,7 +152,7 @@ host_to_commonname() {
|
|||||||
# Download the depends sources now as we won't have internet access in the build
|
# Download the depends sources now as we won't have internet access in the build
|
||||||
# container
|
# container
|
||||||
for host in $HOSTS; do
|
for host in $HOSTS; do
|
||||||
make -C "${PWD}/depends" -j"$MAX_JOBS" download-"$(host_to_commonname "$host")" ${V:+V=1} ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"}
|
make -C "${PWD}/depends" -j"$JOBS" download-"$(host_to_commonname "$host")" ${V:+V=1} ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"}
|
||||||
done
|
done
|
||||||
|
|
||||||
# Determine the reference time used for determinism (overridable by environment)
|
# Determine the reference time used for determinism (overridable by environment)
|
||||||
@ -164,7 +164,7 @@ time-machine() {
|
|||||||
# shellcheck disable=SC2086
|
# shellcheck disable=SC2086
|
||||||
guix time-machine --url=https://github.com/dongcarl/guix.git \
|
guix time-machine --url=https://github.com/dongcarl/guix.git \
|
||||||
--commit=490e39ff303f4f6873a04bfb8253755bdae1b29c \
|
--commit=490e39ff303f4f6873a04bfb8253755bdae1b29c \
|
||||||
--max-jobs="$MAX_JOBS" \
|
--cores="$JOBS" \
|
||||||
--keep-failed \
|
--keep-failed \
|
||||||
${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \
|
${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \
|
||||||
${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_TIMEMACHINE_FLAGS} \
|
${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_TIMEMACHINE_FLAGS} \
|
||||||
@ -218,7 +218,7 @@ for host in $HOSTS; do
|
|||||||
cat << EOF
|
cat << EOF
|
||||||
INFO: Building commit ${GIT_COMMIT:?not set} for platform triple ${HOST:?not set}:
|
INFO: Building commit ${GIT_COMMIT:?not set} for platform triple ${HOST:?not set}:
|
||||||
...using reference timestamp: ${SOURCE_DATE_EPOCH:?not set}
|
...using reference timestamp: ${SOURCE_DATE_EPOCH:?not set}
|
||||||
...running at most ${MAX_JOBS:?not set} jobs
|
...running at most ${JOBS:?not set} jobs
|
||||||
...from worktree directory: '${PWD}'
|
...from worktree directory: '${PWD}'
|
||||||
...bind-mounted in container to: '/bitcoin'
|
...bind-mounted in container to: '/bitcoin'
|
||||||
...in build directory: '$(distsrc_for_host "$HOST")'
|
...in build directory: '$(distsrc_for_host "$HOST")'
|
||||||
@ -304,12 +304,12 @@ EOF
|
|||||||
${SOURCES_PATH:+--share="$SOURCES_PATH"} \
|
${SOURCES_PATH:+--share="$SOURCES_PATH"} \
|
||||||
${BASE_CACHE:+--share="$BASE_CACHE"} \
|
${BASE_CACHE:+--share="$BASE_CACHE"} \
|
||||||
${SDK_PATH:+--share="$SDK_PATH"} \
|
${SDK_PATH:+--share="$SDK_PATH"} \
|
||||||
--max-jobs="$MAX_JOBS" \
|
--cores="$JOBS" \
|
||||||
--keep-failed \
|
--keep-failed \
|
||||||
${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \
|
${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \
|
||||||
${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_ENVIRONMENT_FLAGS} \
|
${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_ENVIRONMENT_FLAGS} \
|
||||||
-- env HOST="$host" \
|
-- env HOST="$host" \
|
||||||
MAX_JOBS="$MAX_JOBS" \
|
JOBS="$JOBS" \
|
||||||
SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:?unable to determine value}" \
|
SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:?unable to determine value}" \
|
||||||
${V:+V=1} \
|
${V:+V=1} \
|
||||||
${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"} \
|
${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"} \
|
||||||
|
@ -26,7 +26,7 @@ cat << EOF
|
|||||||
Required environment variables as seen inside the container:
|
Required environment variables as seen inside the container:
|
||||||
HOST: ${HOST:?not set}
|
HOST: ${HOST:?not set}
|
||||||
SOURCE_DATE_EPOCH: ${SOURCE_DATE_EPOCH:?not set}
|
SOURCE_DATE_EPOCH: ${SOURCE_DATE_EPOCH:?not set}
|
||||||
MAX_JOBS: ${MAX_JOBS:?not set}
|
JOBS: ${JOBS:?not set}
|
||||||
DISTSRC: ${DISTSRC:?not set}
|
DISTSRC: ${DISTSRC:?not set}
|
||||||
OUTDIR: ${OUTDIR:?not set}
|
OUTDIR: ${OUTDIR:?not set}
|
||||||
EOF
|
EOF
|
||||||
@ -173,7 +173,7 @@ esac
|
|||||||
####################
|
####################
|
||||||
|
|
||||||
# Build the depends tree, overriding variables that assume multilib gcc
|
# Build the depends tree, overriding variables that assume multilib gcc
|
||||||
make -C depends --jobs="$MAX_JOBS" HOST="$HOST" \
|
make -C depends --jobs="$JOBS" HOST="$HOST" \
|
||||||
${V:+V=1} \
|
${V:+V=1} \
|
||||||
${SOURCES_PATH+SOURCES_PATH="$SOURCES_PATH"} \
|
${SOURCES_PATH+SOURCES_PATH="$SOURCES_PATH"} \
|
||||||
${BASE_CACHE+BASE_CACHE="$BASE_CACHE"} \
|
${BASE_CACHE+BASE_CACHE="$BASE_CACHE"} \
|
||||||
@ -267,7 +267,7 @@ mkdir -p "$DISTSRC"
|
|||||||
sed -i.old 's/-lstdc++ //g' config.status libtool src/univalue/config.status src/univalue/libtool
|
sed -i.old 's/-lstdc++ //g' config.status libtool src/univalue/config.status src/univalue/libtool
|
||||||
|
|
||||||
# Build Bitcoin Core
|
# Build Bitcoin Core
|
||||||
make --jobs="$MAX_JOBS" ${V:+V=1}
|
make --jobs="$JOBS" ${V:+V=1}
|
||||||
|
|
||||||
# Perform basic ELF security checks on a series of executables.
|
# Perform basic ELF security checks on a series of executables.
|
||||||
make -C src --jobs=1 check-security ${V:+V=1}
|
make -C src --jobs=1 check-security ${V:+V=1}
|
||||||
@ -344,7 +344,7 @@ mkdir -p "$DISTSRC"
|
|||||||
{
|
{
|
||||||
find "${DISTNAME}/bin" -type f -executable -print0
|
find "${DISTNAME}/bin" -type f -executable -print0
|
||||||
find "${DISTNAME}/lib" -type f -print0
|
find "${DISTNAME}/lib" -type f -print0
|
||||||
} | xargs -0 -n1 -P"$MAX_JOBS" -I{} "${DISTSRC}/contrib/devtools/split-debug.sh" {} {} {}.dbg
|
} | xargs -0 -n1 -P"$JOBS" -I{} "${DISTSRC}/contrib/devtools/split-debug.sh" {} {} {}.dbg
|
||||||
;;
|
;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
|
Reference in New Issue
Block a user