mirror of
https://github.com/bitcoin/bitcoin.git
synced 2025-11-10 05:57:59 +01:00
guix: Use --cores instead of --max-jobs
In Guix, there are two flags for controlling parallelism:
Note: When I say "derivation," think "package"
--cores=n
- controls the number of CPU cores to build each derivation. This is
the value passed to `make`'s `--jobs=` flag.
- defaults to 0: as many cores as is available
--max-jobs=n
- controls how many derivations can be built in parallel
- defaults to 1
Therefore, if set --max-jobs=$MAX_JOBS and don't set --cores, Guix could
theoretically spin up $MAX_JOBS * $(nproc) number of threads, and that's
no good.
So we could either default to --cores=1, --max-jobs=$MAX_JOBS
- Pro: --cores=1 means that `make` will be invoked with `-j1`,
avoiding problems with package whose build systems and test
suites break when running multi-threaded.
- Con: There will be times when only 1 or 2 derivations can be built
at a time, because the rest of the dependency graph all depend
on those 1 or 2 derivations. During these times, the machine
will be severely under-utilized.
or --cores=$MAX_JOBS, --max-jobs=1
- Pro: We don't encounter prolonged periods of
severe under-utilization mentioned above.
- Con: Many packages' build systems and test suites break when running
multi-threaded.
or --cores=1, --max-jobs=1 and let the user override with
$ADDITIONAL_GUIX_COMMON_FLAGS
This commit is contained in:
@@ -80,6 +80,50 @@ at the end of the `guix pull`)
|
||||
export PATH="${HOME}/.config/guix/current/bin${PATH:+:}$PATH"
|
||||
```
|
||||
|
||||
### Controlling the number of threads used by `guix` build commands
|
||||
|
||||
By default, the scripts under `./contrib/guix` will invoke all `guix` build
|
||||
commands with `--cores="$JOBS"`. Note that `$JOBS` defaults to `$(nproc)` if not
|
||||
specified. However, astute manual readers will also notice that there is a
|
||||
`--max-jobs=` flag (which defaults to 1 if unspecified).
|
||||
|
||||
Here is the difference between `--cores=` and `--max-jobs=`:
|
||||
|
||||
> Note: When I say "derivation," think "package"
|
||||
|
||||
`--cores=`
|
||||
|
||||
- controls the number of CPU cores to build each derivation. This is the value
|
||||
passed to `make`'s `--jobs=` flag.
|
||||
|
||||
`--max-jobs=`
|
||||
|
||||
- controls how many derivations can be built in parallel
|
||||
- defaults to 1
|
||||
|
||||
Therefore, the default is for `guix` build commands to build one derivation at a
|
||||
time, utilizing `$JOBS` threads.
|
||||
|
||||
Specifying the `$JOBS` environment variable will only modify `--cores=`, but you
|
||||
can also modify the value for `--max-jobs=` by specifying
|
||||
`$ADDITIONAL_GUIX_COMMON_FLAGS`. For example, if you have a LOT of memory, you
|
||||
may want to set:
|
||||
|
||||
```sh
|
||||
export ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
|
||||
```
|
||||
|
||||
Which allows for a maximum of 8 derivations to be built at the same time, each
|
||||
utilizing `$JOBS` threads.
|
||||
|
||||
Or, if you'd like to avoid spurious build failures caused by issues with
|
||||
parallelism within a single package, but would still like to build multiple
|
||||
packages when the dependency graph allows for it, you may want to try:
|
||||
|
||||
```sh
|
||||
export JOBS=1 ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### As a Tool for Deterministic Builds
|
||||
@@ -125,12 +169,16 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
|
||||
the actual SDK (e.g. SDK_PATH=$HOME/Downloads/macOS-SDKs instead of
|
||||
$HOME/Downloads/macOS-SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers).
|
||||
|
||||
* _**MAX_JOBS**_
|
||||
* _**JOBS**_
|
||||
|
||||
Override the maximum number of jobs to run simultaneously, you might want to
|
||||
do so on a memory-limited machine. This may be passed to `make` as in `make
|
||||
--jobs="$MAX_JOBS"` or `xargs` as in `xargs -P"$MAX_JOBS"`. _(defaults to the
|
||||
value of `nproc` outside the container)_
|
||||
Override the number of jobs to run simultaneously, you might want to do so on
|
||||
a memory-limited machine. This may be passed to:
|
||||
|
||||
- `guix` build commands as in `guix environment --cores="$JOBS"`
|
||||
- `make` as in `make --jobs="$JOBS"`
|
||||
- `xargs` as in `xargs -P"$JOBS"`
|
||||
|
||||
_(defaults to the value of `nproc` outside the container)_
|
||||
|
||||
* _**SOURCE_DATE_EPOCH**_
|
||||
|
||||
|
||||
Reference in New Issue
Block a user