highperfocused/bitcoin
1
0
Fork 0
mirror of https://github.com/bitcoin/bitcoin.git synced 2025-11-10 14:08:40 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge bitcoin/bitcoin#21711: guix: Add full installation and usage documentation

Browse Source 
fac4814106 doc/release-process: Add torrent creation details (Carl Dong)
5d24cc3d82 guix/INSTALL: Guix installs init scripts in libdir (Carl Dong)
5da2ee49d5 guix/INSTALL: Add coreutils/inotify-dir-recreate troubleshooting (Carl Dong)
318c60700b guix: Adapt release-process.md to new Guix process (Carl Dong)
fcab35b229 guix-attest: Produce and sign normalized documents (Carl Dong)
c2541fd0ca guix: Overhaul README (Carl Dong)
46ce6ce378 tree-wide: Rename gitian-keys to builder-keys (Carl Dong)
fc4f8449f3 guix: Update various check_tools lists (Carl Dong)
263220a85c guix: Check for a sane services database (Carl Dong)

Pull request description:

  Based on: #21462

  Keeping the README in one file so that it's easy to search through. Will add more jumping links later so navigation is easier.

  Current TODOs:
  - [x] Shell installer option: prompt user to re-login for `/etc/profile.d` entry to be picked up
  - [x] Binary tarball option: prompt user to create `/etc/profile.d` entry and re-login
  - [x] Fanquake docker option: complete section
  - [x] Arch Linux AUR option: prompt to start `guix-daemon-latest` unit after finishing "optional setup" section
  - [x] Building from source option: Insert dependency tree diagram that I made
  - [x] Building from source option: redo sectioning, kind of a mess right now
  - [x] Optional setup: make clear which parts are only needed if building from source
  - [x] Workaround 1 for GnuTLS: perhaps mention how to remove Guix build farm's key
  - [x] Overall (after everything): Make the links work.

  Note to self: wherever possible, tell user how to check that something is true rather than branching by installation option.

ACKs for top commit:
  fanquake:
    ACK fac4814106 - going to go ahead and merge this now. It's a lot of documentation, and could probably be nit-picked / improved further, however, that can continue over the next few weeks. I'm sure more (backportable) improvements / clarifications will be made while we progress through RCs towards a new release.

Tree-SHA512: dc46c0ecdfc67c7c7743ca26e4a603eb3f54adbf81be2f4c1f4c20577ebb84b5250b9c9ec89c0e9860337ab1c7cff94d7963c603287267deecfe1cd987fa070a
This commit is contained in:
fanquake
2021-07-20 11:07:18 +08:00
parent d542603c5a fac4814106
commit e7441a6a45
9 changed files with 1254 additions and 323 deletions
Download Patch File Download Diff File Expand all files Collapse all files

271
doc/release-process.md
View File

@@ -44,10 +44,6 @@ Release Process
- Make an announcement that translators can start translating for the new version. You can use one of the [previous announcements](https://www.transifex.com/bitcoin/bitcoin/announcements/) as a template.
- Change the auto-update URL for the resource to `master`, e.g. `https://raw.githubusercontent.com/bitcoin/bitcoin/master/src/qt/locale/bitcoin_en.xlf`. (Do this only after the previous steps, to prevent an auto-update from interfering.)
#### After branch-off (on master)
- Update the version of `contrib/gitian-descriptors/*.yml`.
#### After branch-off (on the major release branch)
- Update the versions.
@@ -72,14 +68,14 @@ This will perform a few last-minute consistency checks in the build system files
### First time / New builders
If you're using the automated script (found in [contrib/gitian-build.py](/contrib/gitian-build.py)), then at this point you should run it with the "--setup" command. Otherwise ignore this.
Install Guix using one of the installation methods detailed in
[contrib/guix/INSTALL.md](/contrib/guix/INSTALL.md).
Check out the source code in the following directory hierarchy.
cd /path/to/your/toplevel/build
git clone https://github.com/bitcoin-core/gitian.sigs.git
git clone https://github.com/bitcoin-core/guix.sigs.git
git clone https://github.com/bitcoin-core/bitcoin-detached-sigs.git
git clone https://github.com/devrandom/gitian-builder.git
git clone https://github.com/bitcoin/bitcoin.git
### Write the release notes
@@ -94,110 +90,56 @@ Generate list of authors:
git log --format='- %aN' v(current version, e.g. 0.20.0)..v(new version, e.g. 0.20.1) | sort -fiu
### Setup and perform Gitian builds
### Setup and perform Guix builds
If you're using the automated script (found in [contrib/gitian-build.py](/contrib/gitian-build.py)), then at this point you should run it with the "--build" command. Otherwise ignore this.
Checkout the Bitcoin Core version you'd like to build:
Setup Gitian descriptors:
```sh
pushd ./bitcoin
SIGNER='(your builder key, ie bluematt, sipa, etc)'
VERSION='(new version without v-prefix, e.g. 0.20.0)'
git fetch "v${VERSION}"
git checkout "v${VERSION}"
popd
```
pushd ./bitcoin
export SIGNER="(your Gitian key, ie bluematt, sipa, etc)"
export VERSION=(new version, e.g. 0.20.0)
git fetch
git checkout v${VERSION}
popd
Ensure your guix.sigs are up-to-date if you wish to `guix-verify` your builds
against other `guix-attest` signatures.
Ensure your gitian.sigs are up-to-date if you wish to gverify your builds against other Gitian signatures.
```sh
git -C ./guix.sigs pull
```
pushd ./gitian.sigs
git pull
popd
### Create the macOS SDK tarball: (first time, or when SDK version changes)
Ensure gitian-builder is up-to-date:
Create the macOS SDK tarball, see the [macdeploy
instructions](/contrib/macdeploy/README.md#deterministic-macos-dmg-notes) for
details.
pushd ./gitian-builder
git pull
popd
### Build and attest to build outputs:
### Fetch and create inputs: (first time, or when dependency versions change)
Follow the relevant Guix README.md sections:
- [Performing a build](/contrib/guix/README.md#performing-a-build)
- [Attesting to build outputs](/contrib/guix/README.md#attesting-to-build-outputs)
pushd ./gitian-builder
mkdir -p inputs
wget -O inputs/osslsigncode-2.0.tar.gz https://github.com/mtrojnar/osslsigncode/archive/2.0.tar.gz
echo '5a60e0a4b3e0b4d655317b2f12a810211c50242138322b16e7e01c6fbb89d92f inputs/osslsigncode-2.0.tar.gz' | sha256sum -c
popd
### Verify other builders' signatures to your own. (Optional)
Create the macOS SDK tarball, see the [macdeploy instructions](/contrib/macdeploy/README.md#deterministic-macos-dmg-notes) for details, and copy it into the inputs directory.
Add other builders keys to your gpg keyring, and/or refresh keys: See `../bitcoin/contrib/builder-keys/README.md`.
### Optional: Seed the Gitian sources cache and offline git repositories
NOTE: Gitian is sometimes unable to download files. If you have errors, try the step below.
By default, Gitian will fetch source files as needed. To cache them ahead of time, make sure you have checked out the tag you want to build in bitcoin, then:
pushd ./gitian-builder
make -C ../bitcoin/depends download SOURCES_PATH=`pwd`/cache/common
popd
Only missing files will be fetched, so this is safe to re-run for each build.
NOTE: Offline builds must use the --url flag to ensure Gitian fetches only from local URLs. For example:
pushd ./gitian-builder
./bin/gbuild --url bitcoin=/path/to/bitcoin,signature=/path/to/sigs {rest of arguments}
popd
The gbuild invocations below <b>DO NOT DO THIS</b> by default.
### Build and sign Bitcoin Core for Linux, Windows, and macOS:
pushd ./gitian-builder
./bin/gbuild --num-make 2 --memory 3000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
./bin/gsign --signer "$SIGNER" --release ${VERSION}-linux --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
mv build/out/bitcoin-*.tar.gz build/out/src/bitcoin-*.tar.gz ../
./bin/gbuild --num-make 2 --memory 3000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
./bin/gsign --signer "$SIGNER" --release ${VERSION}-win-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
mv build/out/bitcoin-*-win-unsigned.tar.gz inputs/bitcoin-win-unsigned.tar.gz
mv build/out/bitcoin-*.zip build/out/bitcoin-*.exe ../
./bin/gbuild --num-make 2 --memory 3000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
./bin/gsign --signer "$SIGNER" --release ${VERSION}-osx-unsigned --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
mv build/out/bitcoin-*-osx-unsigned.tar.gz inputs/bitcoin-osx-unsigned.tar.gz
mv build/out/bitcoin-*.tar.gz build/out/bitcoin-*.dmg ../
popd
Build output expected:
1. source tarball (`bitcoin-${VERSION}.tar.gz`)
2. linux 32-bit and 64-bit dist tarballs (`bitcoin-${VERSION}-linux[32|64].tar.gz`)
3. windows 32-bit and 64-bit unsigned installers and dist zips (`bitcoin-${VERSION}-win[32|64]-setup-unsigned.exe`, `bitcoin-${VERSION}-win[32|64].zip`)
4. macOS unsigned installer and dist tarball (`bitcoin-${VERSION}-osx-unsigned.dmg`, `bitcoin-${VERSION}-osx64.tar.gz`)
5. Gitian signatures (in `gitian.sigs/${VERSION}-<linux|{win,osx}-unsigned>/(your Gitian key)/`)
### Verify other gitian builders signatures to your own. (Optional)
Add other gitian builders keys to your gpg keyring, and/or refresh keys: See `../bitcoin/contrib/gitian-keys/README.md`.
Verify the signatures
pushd ./gitian-builder
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-linux ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-unsigned ../bitcoin/contrib/gitian-descriptors/gitian-win.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-unsigned ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml
popd
Follow the relevant Guix README.md sections:
- [Verifying build output attestations](/contrib/guix/README.md#verifying-build-output-attestations)
### Next steps:
Commit your signature to gitian.sigs:
Commit your signature to guix.sigs:
pushd gitian.sigs
git add ${VERSION}-linux/"${SIGNER}"
git add ${VERSION}-win-unsigned/"${SIGNER}"
git add ${VERSION}-osx-unsigned/"${SIGNER}"
git commit -m "Add ${VERSION} unsigned sigs for ${SIGNER}"
git push # Assuming you can push to the gitian.sigs tree
popd
```sh
pushd ./guix.sigs
git add "${VERSION}/${SIGNER}"/noncodesigned.SHA256SUMS{,.asc}
git commit -m "Add ${VERSION} unsigned sigs for ${SIGNER}"
git push # Assuming you can push to the guix.sigs tree
popd
```
Codesigner only: Create Windows/macOS detached signatures:
- Only one person handles codesigning. Everyone else should skip to the next step.
@@ -209,7 +151,7 @@ Codesigner only: Sign the macOS binary:
tar xf bitcoin-osx-unsigned.tar.gz
./detached-sig-create.sh -s "Key ID"
Enter the keychain password and authorize the signature
Move signature-osx.tar.gz back to the gitian host
Move signature-osx.tar.gz back to the guix-build host
Codesigner only: Sign the windows binaries:
@@ -220,93 +162,88 @@ Codesigner only: Sign the windows binaries:
Codesigner only: Commit the detached codesign payloads:
cd ~/bitcoin-detached-sigs
checkout the appropriate branch for this release series
rm -rf *
tar xf signature-osx.tar.gz
tar xf signature-win.tar.gz
git add -A
git commit -m "point to ${VERSION}"
git tag -s v${VERSION} HEAD
git push the current branch and new tag
```sh
pushd ./bitcoin-detached-sigs
# checkout the appropriate branch for this release series
rm -rf ./*
tar xf signature-osx.tar.gz
tar xf signature-win.tar.gz
git add -A
git commit -m "point to ${VERSION}"
git tag -s "v${VERSION}" HEAD
git push the current branch and new tag
popd
```
Non-codesigners: wait for Windows/macOS detached signatures:
- Once the Windows/macOS builds each have 3 matching signatures, they will be signed with their respective release keys.
- Detached signatures will then be committed to the [bitcoin-detached-sigs](https://github.com/bitcoin-core/bitcoin-detached-sigs) repository, which can be combined with the unsigned apps to create signed binaries.
Create (and optionally verify) the signed macOS binary:
Create (and optionally verify) the codesigned outputs:
pushd ./gitian-builder
./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
./bin/gsign --signer "$SIGNER" --release ${VERSION}-osx-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-signed ../bitcoin/contrib/gitian-descriptors/gitian-osx-signer.yml
mv build/out/bitcoin-osx-signed.dmg ../bitcoin-${VERSION}-osx.dmg
popd
Create (and optionally verify) the signed Windows binaries:
pushd ./gitian-builder
./bin/gbuild -i --commit signature=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
./bin/gsign --signer "$SIGNER" --release ${VERSION}-win-signed --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-signed ../bitcoin/contrib/gitian-descriptors/gitian-win-signer.yml
mv build/out/bitcoin-*win64-setup.exe ../bitcoin-${VERSION}-win64-setup.exe
popd
- [Codesigning](/contrib/guix/README.md#codesigning)
Commit your signature for the signed macOS/Windows binaries:
pushd gitian.sigs
git add ${VERSION}-osx-signed/"${SIGNER}"
git add ${VERSION}-win-signed/"${SIGNER}"
git commit -m "Add ${SIGNER} ${VERSION} signed binaries signatures"
git push # Assuming you can push to the gitian.sigs tree
popd
```sh
pushd ./guix.sigs
git add "${VERSION}/${SIGNER}"/all.SHA256SUMS{,.asc}
git commit -m "Add ${SIGNER} ${VERSION} signed binaries signatures"
git push # Assuming you can push to the guix.sigs tree
popd
```
### After 3 or more people have gitian-built and their results match:
### After 3 or more people have guix-built and their results match:
- Create `SHA256SUMS.asc` for the builds, and GPG-sign it:
Combine `all.SHA256SUMS` and `all.SHA256SUMS.asc` into a clear-signed
`SHA256SUMS.asc` message:
```sh
echo -e "-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA256\n\n$(cat all.SHA256SUMS)\n$(cat filename.txt.asc)" > SHA256SUMS.asc
```
Here's an equivalent, more readable command if you're confident that you won't
mess up whitespaces when copy-pasting:
```bash
sha256sum * > SHA256SUMS
cat << EOF > SHA256SUMS.asc
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
$(cat all.SHA256SUMS)
$(cat all.SHA256SUMS.asc)
EOF
```
The list of files should be:
```
bitcoin-${VERSION}-aarch64-linux-gnu.tar.gz
bitcoin-${VERSION}-arm-linux-gnueabihf.tar.gz
bitcoin-${VERSION}-riscv64-linux-gnu.tar.gz
bitcoin-${VERSION}-x86_64-linux-gnu.tar.gz
bitcoin-${VERSION}-osx64.tar.gz
bitcoin-${VERSION}-osx.dmg
bitcoin-${VERSION}.tar.gz
bitcoin-${VERSION}-win64-setup.exe
bitcoin-${VERSION}-win64.zip
```
The `*-debug*` files generated by the gitian build contain debug symbols
for troubleshooting by developers. It is assumed that anyone that is interested
in debugging can run gitian to generate the files for themselves. To avoid
end-user confusion about which file to pick, as well as save storage
space *do not upload these to the bitcoincore.org server, nor put them in the torrent*.
- Upload to the bitcoincore.org server (`/var/www/bin/bitcoin-core-${VERSION}`):
1. The contents of `./bitcoin/guix-build-${VERSION}/output`, except for
`*-debug*` files.
- GPG-sign it, delete the unsigned file:
```
gpg --digest-algo sha256 --clearsign SHA256SUMS # outputs SHA256SUMS.asc
rm SHA256SUMS
```
(the digest algorithm is forced to sha256 to avoid confusion of the `Hash:` header that GPG adds with the SHA256 used for the files)
Note: check that SHA256SUMS itself doesn't end up in SHA256SUMS, which is a spurious/nonsensical entry.
The `*-debug*` files generated by the guix build contain debug symbols
for troubleshooting by developers. It is assumed that anyone that is
interested in debugging can run guix to generate the files for
themselves. To avoid end-user confusion about which file to pick, as well
as save storage space *do not upload these to the bitcoincore.org server,
nor put them in the torrent*.
- Upload zips and installers, as well as `SHA256SUMS.asc` from last step, to the bitcoincore.org server
into `/var/www/bin/bitcoin-core-${VERSION}`
2. The combined clear-signed message you just created `SHA256SUMS.asc`
- A `.torrent` will appear in the directory after a few minutes. Optionally help seed this torrent. To get the `magnet:` URI use:
```bash
transmission-show -m <torrent file>
```
Insert the magnet URI into the announcement sent to mailing lists. This permits
people without access to `bitcoincore.org` to download the binary distribution.
Also put it into the `optional_magnetlink:` slot in the YAML file for
bitcoincore.org.
- Create a torrent of the `/var/www/bin/bitcoin-core-${VERSION}` directory such
that at the top level there is only one file: the `bitcoin-core-${VERSION}`
directory containing everything else. Name the torrent
`bitcoin-${VERSION}.torrent` (note that there is no `-core-` in this name).
Optionally help seed this torrent. To get the `magnet:` URI use:
```sh
transmission-show -m <torrent file>
```
Insert the magnet URI into the announcement sent to mailing lists. This permits
people without access to `bitcoincore.org` to download the binary distribution.
Also put it into the `optional_magnetlink:` slot in the YAML file for
bitcoincore.org.
- Update other repositories and websites for new version
@@ -344,14 +281,14 @@ bitcoincore.org.
- https://code.launchpad.net/~bitcoin-core/bitcoin-core-snap/+git/packaging/+ref/0.xx (Click "Create snap package")
- Name it "bitcoin-core-snap-0.xx"
- Leave owner and series as-is
- Select architectures that are compiled via gitian
- Select architectures that are compiled via guix
- Leave "automatically build when branch changes" unticked
- Tick "automatically upload to store"
- Put "bitcoin-core" in the registered store package name field
- Tick the "edge" box
- Put "0.xx" in the track field
- Click "create snap package"
- Click "Request builds" for every new release on this branch (after updating the snapcraft.yml in the branch to reflect the latest gitian results)
- Click "Request builds" for every new release on this branch (after updating the snapcraft.yml in the branch to reflect the latest guix results)
- Promote release on https://snapcraft.io/bitcoin-core/releases if it passes sanity checks
- This repo