Merge #20560: fuzz: Link all targets once

fa13e1b0c5 build: Add option --enable-danger-fuzz-link-all (MarcoFalke)
44444ba759 fuzz: Link all targets once (MarcoFalke)

Pull request description:

  Currently the linker is invoked more than 150 times when compiling with `--enable-fuzz`. This is problematic for several reasons:

  * It wastes disk space north of 20 GB, as all libraries and sanitizers are linked more than 150 times
  * It wastes CPU time, as the link step can practically not be cached (similar to ccache for object files)
  * It makes it a blocker to compile the fuzz tests by default for non-fuzz builds #19388, for the aforementioned reasons
  * The build file is several thousand lines of code, without doing anything meaningful except listing each fuzz target in a highly verbose manner
  * It makes writing new fuzz tests unnecessarily hard, as build system knowledge is required; Compare that to boost unit tests, which can be added by simply editing an existing cpp file
  * It encourages fuzz tests that re-use the `buffer` or assume the `buffer` to be concatenations of seeds, which increases complexity of seeds and complexity for the fuzz engine to explore; Thus reducing the effectiveness of the affected fuzz targets

  Fixes #20088

ACKs for top commit:
  practicalswift:
    Tested ACK fa13e1b0c5
  sipa:
    ACK fa13e1b0c5. Reviewed the code changes, and tested the 3 different test_runner.py modes (run once, merge, generate). I also tested building with the new --enable-danger-fuzz-link-all

Tree-SHA512: 962ab33269ebd51810924c51266ecc62edd6ddf2fcd9a8c359ed906766f58c3f73c223f8d3cc49f2c60f0053f65e8bdd86ce9c19e673f8c2b3cd676e913f2642

doc/fuzzing.md

@@ -12,7 +12,7 @@ $ CC=clang CXX=clang++ ./configure --enable-fuzz --with-sanitizers=address,fuzze
 # macOS users: If you have problem with this step then make sure to read "macOS hints for
 # libFuzzer" on https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md#macos-hints-for-libfuzzer
 $ make
-$ src/test/fuzz/process_message
+$ FUZZ=process_message src/test/fuzz/fuzz
 # abort fuzzing using ctrl-c
 ```
 
@@ -26,7 +26,7 @@ If you specify a corpus directory then any new coverage increasing inputs will b
 
 ```sh
 $ mkdir -p process_message-seeded-from-thin-air/
-$ src/test/fuzz/process_message process_message-seeded-from-thin-air/
+$ FUZZ=process_message src/test/fuzz/fuzz process_message-seeded-from-thin-air/
 INFO: Seed: 840522292
 INFO: Loaded 1 modules   (424174 inline 8-bit counters): 424174 [0x55e121ef9ab8, 0x55e121f613a6),
 INFO: Loaded 1 PC tables (424174 PCs): 424174 [0x55e121f613a8,0x55e1225da288),
@@ -70,7 +70,7 @@ To fuzz `process_message` using the [`bitcoin-core/qa-assets`](https://github.co
 
 ```sh
 $ git clone https://github.com/bitcoin-core/qa-assets
-$ src/test/fuzz/process_message qa-assets/fuzz_seed_corpus/process_message/
+$ FUZZ=process_message src/test/fuzz/fuzz qa-assets/fuzz_seed_corpus/process_message/
 INFO: Seed: 1346407872
 INFO: Loaded 1 modules   (424174 inline 8-bit counters): 424174 [0x55d8a9004ab8, 0x55d8a906c3a6),
 INFO: Loaded 1 PC tables (424174 PCs): 424174 [0x55d8a906c3a8,0x55d8a96e5288),
@@ -129,7 +129,7 @@ $ make
 # try compiling using: AFL_NO_X86=1 make
 $ mkdir -p inputs/ outputs/
 $ echo A > inputs/thin-air-input
-$ afl/afl-fuzz -i inputs/ -o outputs/ -- src/test/fuzz/bech32
+$ FUZZ=bech32 afl/afl-fuzz -i inputs/ -o outputs/ -- src/test/fuzz/fuzz
 # You may have to change a few kernel parameters to test optimally - afl-fuzz
 # will print an error and suggestion if so.
 ```
@@ -153,7 +153,7 @@ $ cd ..
 $ CC=$(pwd)/honggfuzz/hfuzz_cc/hfuzz-clang CXX=$(pwd)/honggfuzz/hfuzz_cc/hfuzz-clang++ ./configure --enable-fuzz --with-sanitizers=address,undefined
 $ make
 $ mkdir -p inputs/
-$ honggfuzz/honggfuzz -i inputs/ -- src/test/fuzz/process_message
+$ FUZZ=process_message honggfuzz/honggfuzz -i inputs/ -- src/test/fuzz/fuzz
 ```
 
 Read the [Honggfuzz documentation](https://github.com/google/honggfuzz/blob/master/docs/USAGE.md) for more information.