mirror of
https://github.com/bitcoin/bitcoin.git
synced 2025-08-24 06:50:57 +02:00
Merge bitcoin/bitcoin#31144: [IBD] multi-byte block obfuscation
248b6a27c3
optimization: peel align-head and unroll body to 64 bytes (Lőrinc)e7114fc6dc
optimization: migrate fixed-size obfuscation from `std::vector<std::byte>` to `uint64_t` (Lőrinc)478d40afc6
refactor: encapsulate `vector`/`array` keys into `Obfuscation` (Lőrinc)377aab8e5a
refactor: move `util::Xor` to `Obfuscation().Xor` (Lőrinc)fa5d296e3b
refactor: prepare mempool_persist for obfuscation key change (Lőrinc)6bbf2d9311
refactor: prepare `DBWrapper` for obfuscation key change (Lőrinc)0b8bec8aa6
scripted-diff: unify xor-vs-obfuscation nomenclature (Lőrinc)972697976c
bench: make ObfuscationBench more representative (Lőrinc)618a30e326
test: compare util::Xor with randomized inputs against simple impl (Lőrinc)a5141cd39e
test: make sure dbwrapper obfuscation key is never obfuscated (Lőrinc)54ab0bd64c
refactor: commit to 8 byte obfuscation keys (Lőrinc)7aa557a37b
random: add fixed-size `std::array` generation (Lőrinc) Pull request description: This change is part of [[IBD] - Tracking PR for speeding up Initial Block Download](https://github.com/bitcoin/bitcoin/pull/32043) ### Summary Current block obfuscations are done byte-by-byte, this PR batches them to 64 bit primitives to speed up obfuscating bigger memory batches. This is especially relevant now that https://github.com/bitcoin/bitcoin/pull/31551 was merged, having bigger obfuscatable chunks. Since this obfuscation is optional, the speedup measured here depends on whether it's a [random value](https://github.com/bitcoin/bitcoin/pull/31144#issuecomment-2523295114) or [completely turned off](https://github.com/bitcoin/bitcoin/pull/31144#issuecomment-2519764142) (i.e. XOR-ing with 0). ### Changes in testing, benchmarking and implementation * Added new tests comparing randomized inputs against a trivial implementation and performing roundtrip checks with random chunks. * Migrated `std::vector<std::byte>(8)` keys to plain `uint64_t`; * Process unaligned bytes separately and unroll body to 64 bytes. ### Assembly Memory alignment is enforced by a small peel-loop (`std::memcpy` is optimized out on tested platform), with an `std::assume_aligned<8>` check, see the Godbolt listing at https://godbolt.org/z/59EMv7h6Y for details <details> <summary>Details</summary> Target & Compiler | Stride (per hot-loop iter) | Main operation(s) in loop | Effective XORs / iter -- | -- | -- | -- Clang x86-64 (trunk) | 64 bytes | 4 × movdqu → pxor → store | 8 × 64-bit GCC x86-64 (trunk) | 64 bytes | 4 × movdqu/pxor sequence, enabled by 8-way unroll | 8 × 64-bit GCC RV32 (trunk) | 8 bytes | copy 8 B to temp → 2 × 32-bit XOR → copy back | 1 × 64-bit (as 2 × 32-bit) GCC s390x (big-endian 14.2) | 64 bytes | 8 × XC (mem-mem 8-B XOR) with key cached on stack | 8 × 64-bit </details> ### Endianness The only endianness issue was with bit rotation, intended to realign the key if obfuscation halted before full key consumption. Elsewhere, memory is read, processed, and written back in the same endianness, preserving byte order. Since CI lacks a big-endian machine, testing was done locally via Docker. <details> <summary>Details</summary> ```bash brew install podman pigz softwareupdate --install-rosetta podman machine init podman machine start docker run --platform linux/s390x -it ubuntu:latest /bin/bash apt update && apt install -y git build-essential cmake ccache pkg-config libevent-dev libboost-dev libssl-dev libsqlite3-dev python3 && \ cd /mnt && git clone --depth=1 https://github.com/bitcoin/bitcoin.git && cd bitcoin && git remote add l0rinc https://github.com/l0rinc/bitcoin.git && git fetch --all && git checkout l0rinc/optimize-xor && \ cmake -B build && cmake --build build --target test_bitcoin -j$(nproc) && \ ./build/bin/test_bitcoin --run_test=streams_tests ``` </details> ### Measurements (micro benchmarks and full IBDs) > cmake -B build -DBUILD_BENCH=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc/clang -DCMAKE_CXX_COMPILER=g++/clang++ && \ cmake --build build -j$(nproc) && \ build/bin/bench_bitcoin -filter='ObfuscationBench' -min-time=5000 <details> <summary>GNU 14.2.0</summary> > Before: | ns/byte | byte/s | err% | ins/byte | cyc/byte | IPC | bra/byte | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 0.84 | 1,184,138,235.64 | 0.0% | 9.01 | 3.03 | 2.971 | 1.00 | 0.1% | 5.50 | `ObfuscationBench` > After (first optimizing commit): | ns/byte | byte/s | err% | ins/byte | cyc/byte | IPC | bra/byte | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 0.04 | 28,365,698,819.44 | 0.0% | 0.34 | 0.13 | 2.714 | 0.07 | 0.0% | 5.33 | `ObfuscationBench` > and (second optimizing commit): | ns/byte | byte/s | err% | ins/byte | cyc/byte | IPC | bra/byte | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 0.03 | 32,464,658,919.11 | 0.0% | 0.50 | 0.11 | 4.474 | 0.08 | 0.0% | 5.29 | `ObfuscationBench` </details> <details> <summary>Clang 20.1.7</summary> > Before: | ns/byte | byte/s | err% | ins/byte | cyc/byte | IPC | bra/byte | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 0.89 | 1,124,087,330.23 | 0.1% | 6.52 | 3.20 | 2.041 | 0.50 | 0.2% | 5.50 | `ObfuscationBench` > After (first optimizing commit): | ns/byte | byte/s | err% | ins/byte | cyc/byte | IPC | bra/byte | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 0.08 | 13,012,464,203.00 | 0.0% | 0.65 | 0.28 | 2.338 | 0.13 | 0.8% | 5.50 | `ObfuscationBench` > and (second optimizing commit): | ns/byte | byte/s | err% | ins/byte | cyc/byte | IPC | bra/byte | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 0.02 | 41,231,547,045.17 | 0.0% | 0.30 | 0.09 | 3.463 | 0.02 | 0.0% | 5.47 | `ObfuscationBench` </details> i.e. 27.4x faster obfuscation with GCC, 36.7x faster with Clang For other benchmark speedups see https://corecheck.dev/bitcoin/bitcoin/pulls/31144 ------ Running an IBD until 888888 blocks reveals a 4% speedup. <details> <summary>Details</summary> SSD: ```bash COMMITS="8324a00bd4a6a5291c841f2d01162d8a014ddb02 5ddfd31b4158a89b0007cfb2be970c03d9278525"; \ STOP_HEIGHT=888888; DBCACHE=1000; \ CC=gcc; CXX=g++; \ BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \ (for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \ hyperfine \ --sort 'command' \ --runs 1 \ --export-json "$BASE_DIR/ibd-${COMMITS// /-}-$STOP_HEIGHT-$DBCACHE-$CC.json" \ --parameter-list COMMIT ${COMMITS// /,} \ --prepare "killall bitcoind; rm -rf $DATA_DIR/*; git checkout {COMMIT}; git clean -fxd; git reset --hard; \ cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && \ cmake --build build -j$(nproc) --target bitcoind && \ ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; sleep 100" \ --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \ "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=$DBCACHE -blocksonly -printtoconsole=0" ``` > 8324a00bd4 test: Compare util::Xor with randomized inputs against simple impl > 5ddfd31b41 optimization: Xor 64 bits together instead of byte-by-byte ```python Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=1000 -blocksonly -printtoconsole=0 (COMMIT = 8324a00bd4a6a5291c841f2d01162d8a014ddb02) Time (abs ≡): 25033.413 s [User: 33953.984 s, System: 2613.604 s] Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=1000 -blocksonly -printtoconsole=0 (COMMIT = 5ddfd31b4158a89b0007cfb2be970c03d9278525) Time (abs ≡): 24110.710 s [User: 33389.536 s, System: 2660.292 s] Relative speed comparison 1.04 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=1000 -blocksonly -printtoconsole=0 (COMMIT = 8324a00bd4a6a5291c841f2d01162d8a014ddb02) 1.00 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=1000 -blocksonly -printtoconsole=0 (COMMIT = 5ddfd31b4158a89b0007cfb2be970c03d9278525) ``` > HDD: ```bash COMMITS="71eb6eaa740ad0b28737e90e59b89a8e951d90d9 46854038e7984b599d25640de26d4680e62caba7"; \ STOP_HEIGHT=888888; DBCACHE=4500; \ CC=gcc; CXX=g++; \ BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \ (for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \ hyperfine \ --sort 'command' \ --runs 2 \ --export-json "$BASE_DIR/ibd-${COMMITS// /-}-$STOP_HEIGHT-$DBCACHE-$CC.json" \ --parameter-list COMMIT ${COMMITS// /,} \ --prepare "killall bitcoind; rm -rf $DATA_DIR/*; git checkout {COMMIT}; git clean -fxd; git reset --hard; \ cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && cmake --build build -j$(nproc) --target bitcoind && \ ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; sleep 100" \ --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \ "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=$DBCACHE -blocksonly -printtoconsole=0" ``` > 71eb6eaa74 test: compare util::Xor with randomized inputs against simple impl > 46854038e7 optimization: migrate fixed-size obfuscation from `std::vector<std::byte>` to `uint64_t` ```python Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=4500 -blocksonly -printtoconsole=0 (COMMIT = 71eb6eaa740ad0b28737e90e59b89a8e951d90d9) Time (mean ± σ): 37676.293 s ± 83.100 s [User: 36900.535 s, System: 2220.382 s] Range (min … max): 37617.533 s … 37735.053 s 2 runs Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=4500 -blocksonly -printtoconsole=0 (COMMIT = 46854038e7984b599d25640de26d4680e62caba7) Time (mean ± σ): 36181.287 s ± 195.248 s [User: 34962.822 s, System: 1988.614 s] Range (min … max): 36043.226 s … 36319.349 s 2 runs Relative speed comparison 1.04 ± 0.01 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=4500 -blocksonly -printtoconsole=0 (COMMIT = 71eb6eaa740ad0b28737e90e59b89a8e951d90d9) 1.00 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=4500 -blocksonly -printtoconsole=0 (COMMIT = 46854038e7984b599d25640de26d4680e62caba7) ``` </details> ACKs for top commit: achow101: ACK248b6a27c3
maflcko: review ACK248b6a27c3
🎻 ryanofsky: Code review ACK248b6a27c3
. Looks good! Thanks for adapting this and considering all the suggestions. I did leave more comments below but non are important and this looks good as-is Tree-SHA512: ef541cd8a1f1dc504613c4eaa708202e32ae5ac86f9c875e03bcdd6357121f6af0860ef83d513c473efa5445b701e59439d416effae1085a559716b0fd45ecd6
This commit is contained in:
@@ -35,6 +35,7 @@ add_executable(bench_bitcoin
|
||||
mempool_eviction.cpp
|
||||
mempool_stress.cpp
|
||||
merkle_root.cpp
|
||||
obfuscation.cpp
|
||||
parse_hex.cpp
|
||||
peer_eviction.cpp
|
||||
poly1305.cpp
|
||||
@@ -52,7 +53,6 @@ add_executable(bench_bitcoin
|
||||
txorphanage.cpp
|
||||
util_time.cpp
|
||||
verify_script.cpp
|
||||
xor.cpp
|
||||
)
|
||||
|
||||
include(TargetDataSources)
|
||||
|
@@ -4,21 +4,22 @@
|
||||
|
||||
#include <bench/bench.h>
|
||||
#include <random.h>
|
||||
#include <span.h>
|
||||
#include <streams.h>
|
||||
#include <util/obfuscation.h>
|
||||
|
||||
#include <cstddef>
|
||||
#include <vector>
|
||||
|
||||
static void Xor(benchmark::Bench& bench)
|
||||
static void ObfuscationBench(benchmark::Bench& bench)
|
||||
{
|
||||
FastRandomContext frc{/*fDeterministic=*/true};
|
||||
auto data{frc.randbytes<std::byte>(1024)};
|
||||
auto key{frc.randbytes<std::byte>(31)};
|
||||
const Obfuscation obfuscation{frc.randbytes<Obfuscation::KEY_SIZE>()};
|
||||
|
||||
size_t offset{0};
|
||||
bench.batch(data.size()).unit("byte").run([&] {
|
||||
util::Xor(data, key);
|
||||
obfuscation(data, offset++); // mutated differently each time
|
||||
ankerl::nanobench::doNotOptimizeAway(data);
|
||||
});
|
||||
}
|
||||
|
||||
BENCHMARK(Xor, benchmark::PriorityLevel::HIGH);
|
||||
BENCHMARK(ObfuscationBench, benchmark::PriorityLevel::HIGH);
|
@@ -11,6 +11,7 @@
|
||||
#include <streams.h>
|
||||
#include <util/fs.h>
|
||||
#include <util/fs_helpers.h>
|
||||
#include <util/obfuscation.h>
|
||||
#include <util/strencodings.h>
|
||||
|
||||
#include <algorithm>
|
||||
@@ -173,7 +174,7 @@ void CDBBatch::Clear()
|
||||
void CDBBatch::WriteImpl(std::span<const std::byte> key, DataStream& ssValue)
|
||||
{
|
||||
leveldb::Slice slKey(CharCast(key.data()), key.size());
|
||||
ssValue.Xor(dbwrapper_private::GetObfuscateKey(parent));
|
||||
dbwrapper_private::GetObfuscation(parent)(ssValue);
|
||||
leveldb::Slice slValue(CharCast(ssValue.data()), ssValue.size());
|
||||
m_impl_batch->batch.Put(slKey, slValue);
|
||||
}
|
||||
@@ -248,24 +249,14 @@ CDBWrapper::CDBWrapper(const DBParams& params)
|
||||
LogPrintf("Finished database compaction of %s\n", fs::PathToString(params.path));
|
||||
}
|
||||
|
||||
// The base-case obfuscation key, which is a noop.
|
||||
obfuscate_key = std::vector<unsigned char>(OBFUSCATE_KEY_NUM_BYTES, '\000');
|
||||
|
||||
bool key_exists = Read(OBFUSCATE_KEY_KEY, obfuscate_key);
|
||||
|
||||
if (!key_exists && params.obfuscate && IsEmpty()) {
|
||||
// Initialize non-degenerate obfuscation if it won't upset
|
||||
// existing, non-obfuscated data.
|
||||
std::vector<unsigned char> new_key = CreateObfuscateKey();
|
||||
|
||||
// Write `new_key` so we don't obfuscate the key with itself
|
||||
Write(OBFUSCATE_KEY_KEY, new_key);
|
||||
obfuscate_key = new_key;
|
||||
|
||||
LogPrintf("Wrote new obfuscate key for %s: %s\n", fs::PathToString(params.path), HexStr(obfuscate_key));
|
||||
assert(!m_obfuscation); // Needed for unobfuscated Read()/Write() below
|
||||
if (!Read(OBFUSCATION_KEY_KEY, m_obfuscation) && params.obfuscate && IsEmpty()) {
|
||||
// Generate, write and read back the new obfuscation key, making sure we don't obfuscate the key itself
|
||||
Write(OBFUSCATION_KEY_KEY, FastRandomContext{}.randbytes(Obfuscation::KEY_SIZE));
|
||||
Read(OBFUSCATION_KEY_KEY, m_obfuscation);
|
||||
LogInfo("Wrote new obfuscation key for %s: %s", fs::PathToString(params.path), m_obfuscation.HexKey());
|
||||
}
|
||||
|
||||
LogPrintf("Using obfuscation key for %s: %s\n", fs::PathToString(params.path), HexStr(obfuscate_key));
|
||||
LogInfo("Using obfuscation key for %s: %s", fs::PathToString(params.path), m_obfuscation.HexKey());
|
||||
}
|
||||
|
||||
CDBWrapper::~CDBWrapper()
|
||||
@@ -310,25 +301,6 @@ size_t CDBWrapper::DynamicMemoryUsage() const
|
||||
return parsed.value();
|
||||
}
|
||||
|
||||
// Prefixed with null character to avoid collisions with other keys
|
||||
//
|
||||
// We must use a string constructor which specifies length so that we copy
|
||||
// past the null-terminator.
|
||||
const std::string CDBWrapper::OBFUSCATE_KEY_KEY("\000obfuscate_key", 14);
|
||||
|
||||
const unsigned int CDBWrapper::OBFUSCATE_KEY_NUM_BYTES = 8;
|
||||
|
||||
/**
|
||||
* Returns a string (consisting of 8 random bytes) suitable for use as an
|
||||
* obfuscating XOR key.
|
||||
*/
|
||||
std::vector<unsigned char> CDBWrapper::CreateObfuscateKey() const
|
||||
{
|
||||
std::vector<uint8_t> ret(OBFUSCATE_KEY_NUM_BYTES);
|
||||
GetRandBytes(ret);
|
||||
return ret;
|
||||
}
|
||||
|
||||
std::optional<std::string> CDBWrapper::ReadImpl(std::span<const std::byte> key) const
|
||||
{
|
||||
leveldb::Slice slKey(CharCast(key.data()), key.size());
|
||||
@@ -412,9 +384,9 @@ void CDBIterator::Next() { m_impl_iter->iter->Next(); }
|
||||
|
||||
namespace dbwrapper_private {
|
||||
|
||||
const std::vector<unsigned char>& GetObfuscateKey(const CDBWrapper &w)
|
||||
const Obfuscation& GetObfuscation(const CDBWrapper& w)
|
||||
{
|
||||
return w.obfuscate_key;
|
||||
return w.m_obfuscation;
|
||||
}
|
||||
|
||||
} // namespace dbwrapper_private
|
||||
|
@@ -18,7 +18,6 @@
|
||||
#include <optional>
|
||||
#include <stdexcept>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
static const size_t DBWRAPPER_PREALLOC_KEY_SIZE = 64;
|
||||
static const size_t DBWRAPPER_PREALLOC_VALUE_SIZE = 1024;
|
||||
@@ -63,8 +62,7 @@ namespace dbwrapper_private {
|
||||
* Database obfuscation should be considered an implementation detail of the
|
||||
* specific database.
|
||||
*/
|
||||
const std::vector<unsigned char>& GetObfuscateKey(const CDBWrapper &w);
|
||||
|
||||
const Obfuscation& GetObfuscation(const CDBWrapper&);
|
||||
}; // namespace dbwrapper_private
|
||||
|
||||
bool DestroyDB(const std::string& path_str);
|
||||
@@ -166,7 +164,7 @@ public:
|
||||
template<typename V> bool GetValue(V& value) {
|
||||
try {
|
||||
DataStream ssValue{GetValueImpl()};
|
||||
ssValue.Xor(dbwrapper_private::GetObfuscateKey(parent));
|
||||
dbwrapper_private::GetObfuscation(parent)(ssValue);
|
||||
ssValue >> value;
|
||||
} catch (const std::exception&) {
|
||||
return false;
|
||||
@@ -179,7 +177,7 @@ struct LevelDBContext;
|
||||
|
||||
class CDBWrapper
|
||||
{
|
||||
friend const std::vector<unsigned char>& dbwrapper_private::GetObfuscateKey(const CDBWrapper &w);
|
||||
friend const Obfuscation& dbwrapper_private::GetObfuscation(const CDBWrapper&);
|
||||
private:
|
||||
//! holds all leveldb-specific fields of this class
|
||||
std::unique_ptr<LevelDBContext> m_db_context;
|
||||
@@ -187,16 +185,11 @@ private:
|
||||
//! the name of this database
|
||||
std::string m_name;
|
||||
|
||||
//! a key used for optional XOR-obfuscation of the database
|
||||
std::vector<unsigned char> obfuscate_key;
|
||||
//! optional XOR-obfuscation of the database
|
||||
Obfuscation m_obfuscation;
|
||||
|
||||
//! the key under which the obfuscation key is stored
|
||||
static const std::string OBFUSCATE_KEY_KEY;
|
||||
|
||||
//! the length of the obfuscate key in number of bytes
|
||||
static const unsigned int OBFUSCATE_KEY_NUM_BYTES;
|
||||
|
||||
std::vector<unsigned char> CreateObfuscateKey() const;
|
||||
//! obfuscation key storage key, null-prefixed to avoid collisions
|
||||
inline static const std::string OBFUSCATION_KEY_KEY{"\000obfuscate_key", 14}; // explicit size to avoid truncation at leading \0
|
||||
|
||||
//! path to filesystem storage
|
||||
const fs::path m_path;
|
||||
@@ -228,7 +221,7 @@ public:
|
||||
}
|
||||
try {
|
||||
DataStream ssValue{MakeByteSpan(*strValue)};
|
||||
ssValue.Xor(obfuscate_key);
|
||||
m_obfuscation(ssValue);
|
||||
ssValue >> value;
|
||||
} catch (const std::exception&) {
|
||||
return false;
|
||||
|
@@ -31,6 +31,7 @@
|
||||
#include <util/batchpriority.h>
|
||||
#include <util/check.h>
|
||||
#include <util/fs.h>
|
||||
#include <util/obfuscation.h>
|
||||
#include <util/signalinterrupt.h>
|
||||
#include <util/strencodings.h>
|
||||
#include <util/syserror.h>
|
||||
@@ -779,13 +780,13 @@ void BlockManager::UnlinkPrunedFiles(const std::set<int>& setFilesToPrune) const
|
||||
|
||||
AutoFile BlockManager::OpenBlockFile(const FlatFilePos& pos, bool fReadOnly) const
|
||||
{
|
||||
return AutoFile{m_block_file_seq.Open(pos, fReadOnly), m_xor_key};
|
||||
return AutoFile{m_block_file_seq.Open(pos, fReadOnly), m_obfuscation};
|
||||
}
|
||||
|
||||
/** Open an undo file (rev?????.dat) */
|
||||
AutoFile BlockManager::OpenUndoFile(const FlatFilePos& pos, bool fReadOnly) const
|
||||
{
|
||||
return AutoFile{m_undo_file_seq.Open(pos, fReadOnly), m_xor_key};
|
||||
return AutoFile{m_undo_file_seq.Open(pos, fReadOnly), m_obfuscation};
|
||||
}
|
||||
|
||||
fs::path BlockManager::GetBlockPosFilename(const FlatFilePos& pos) const
|
||||
@@ -1123,7 +1124,7 @@ static auto InitBlocksdirXorKey(const BlockManager::Options& opts)
|
||||
{
|
||||
// Bytes are serialized without length indicator, so this is also the exact
|
||||
// size of the XOR-key file.
|
||||
std::array<std::byte, 8> xor_key{};
|
||||
std::array<std::byte, Obfuscation::KEY_SIZE> obfuscation{};
|
||||
|
||||
// Consider this to be the first run if the blocksdir contains only hidden
|
||||
// files (those which start with a .). Checking for a fully-empty dir would
|
||||
@@ -1140,14 +1141,14 @@ static auto InitBlocksdirXorKey(const BlockManager::Options& opts)
|
||||
if (opts.use_xor && first_run) {
|
||||
// Only use random fresh key when the boolean option is set and on the
|
||||
// very first start of the program.
|
||||
FastRandomContext{}.fillrand(xor_key);
|
||||
FastRandomContext{}.fillrand(obfuscation);
|
||||
}
|
||||
|
||||
const fs::path xor_key_path{opts.blocks_dir / "xor.dat"};
|
||||
if (fs::exists(xor_key_path)) {
|
||||
// A pre-existing xor key file has priority.
|
||||
AutoFile xor_key_file{fsbridge::fopen(xor_key_path, "rb")};
|
||||
xor_key_file >> xor_key;
|
||||
xor_key_file >> obfuscation;
|
||||
} else {
|
||||
// Create initial or missing xor key file
|
||||
AutoFile xor_key_file{fsbridge::fopen(xor_key_path,
|
||||
@@ -1157,7 +1158,7 @@ static auto InitBlocksdirXorKey(const BlockManager::Options& opts)
|
||||
"wbx"
|
||||
#endif
|
||||
)};
|
||||
xor_key_file << xor_key;
|
||||
xor_key_file << obfuscation;
|
||||
if (xor_key_file.fclose() != 0) {
|
||||
throw std::runtime_error{strprintf("Error closing XOR key file %s: %s",
|
||||
fs::PathToString(xor_key_path),
|
||||
@@ -1165,20 +1166,20 @@ static auto InitBlocksdirXorKey(const BlockManager::Options& opts)
|
||||
}
|
||||
}
|
||||
// If the user disabled the key, it must be zero.
|
||||
if (!opts.use_xor && xor_key != decltype(xor_key){}) {
|
||||
if (!opts.use_xor && obfuscation != decltype(obfuscation){}) {
|
||||
throw std::runtime_error{
|
||||
strprintf("The blocksdir XOR-key can not be disabled when a random key was already stored! "
|
||||
"Stored key: '%s', stored path: '%s'.",
|
||||
HexStr(xor_key), fs::PathToString(xor_key_path)),
|
||||
HexStr(obfuscation), fs::PathToString(xor_key_path)),
|
||||
};
|
||||
}
|
||||
LogInfo("Using obfuscation key for blocksdir *.dat files (%s): '%s'\n", fs::PathToString(opts.blocks_dir), HexStr(xor_key));
|
||||
return std::vector<std::byte>{xor_key.begin(), xor_key.end()};
|
||||
LogInfo("Using obfuscation key for blocksdir *.dat files (%s): '%s'\n", fs::PathToString(opts.blocks_dir), HexStr(obfuscation));
|
||||
return Obfuscation{obfuscation};
|
||||
}
|
||||
|
||||
BlockManager::BlockManager(const util::SignalInterrupt& interrupt, Options opts)
|
||||
: m_prune_mode{opts.prune_target > 0},
|
||||
m_xor_key{InitBlocksdirXorKey(opts)},
|
||||
m_obfuscation{InitBlocksdirXorKey(opts)},
|
||||
m_opts{std::move(opts)},
|
||||
m_block_file_seq{FlatFileSeq{m_opts.blocks_dir, "blk", m_opts.fast_prune ? 0x4000 /* 16kB */ : BLOCKFILE_CHUNK_SIZE}},
|
||||
m_undo_file_seq{FlatFileSeq{m_opts.blocks_dir, "rev", UNDOFILE_CHUNK_SIZE}},
|
||||
|
@@ -235,7 +235,7 @@ private:
|
||||
|
||||
const bool m_prune_mode;
|
||||
|
||||
const std::vector<std::byte> m_xor_key;
|
||||
const Obfuscation m_obfuscation;
|
||||
|
||||
/** Dirty block index entries. */
|
||||
std::set<CBlockIndex*> m_dirty_blockindex;
|
||||
|
@@ -16,6 +16,7 @@
|
||||
#include <uint256.h>
|
||||
#include <util/fs.h>
|
||||
#include <util/fs_helpers.h>
|
||||
#include <util/obfuscation.h>
|
||||
#include <util/signalinterrupt.h>
|
||||
#include <util/syserror.h>
|
||||
#include <util/time.h>
|
||||
@@ -59,15 +60,17 @@ bool LoadMempool(CTxMemPool& pool, const fs::path& load_path, Chainstate& active
|
||||
try {
|
||||
uint64_t version;
|
||||
file >> version;
|
||||
std::vector<std::byte> xor_key;
|
||||
|
||||
if (version == MEMPOOL_DUMP_VERSION_NO_XOR_KEY) {
|
||||
// Leave XOR-key empty
|
||||
file.SetObfuscation({});
|
||||
} else if (version == MEMPOOL_DUMP_VERSION) {
|
||||
file >> xor_key;
|
||||
Obfuscation obfuscation;
|
||||
file >> obfuscation;
|
||||
file.SetObfuscation(obfuscation);
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
file.SetXor(xor_key);
|
||||
|
||||
uint64_t total_txns_to_load;
|
||||
file >> total_txns_to_load;
|
||||
uint64_t txns_tried = 0;
|
||||
@@ -179,12 +182,13 @@ bool DumpMempool(const CTxMemPool& pool, const fs::path& dump_path, FopenFn mock
|
||||
const uint64_t version{pool.m_opts.persist_v1_dat ? MEMPOOL_DUMP_VERSION_NO_XOR_KEY : MEMPOOL_DUMP_VERSION};
|
||||
file << version;
|
||||
|
||||
std::vector<std::byte> xor_key(8);
|
||||
if (!pool.m_opts.persist_v1_dat) {
|
||||
FastRandomContext{}.fillrand(xor_key);
|
||||
file << xor_key;
|
||||
const Obfuscation obfuscation{FastRandomContext{}.randbytes<Obfuscation::KEY_SIZE>()};
|
||||
file << obfuscation;
|
||||
file.SetObfuscation(obfuscation);
|
||||
} else {
|
||||
file.SetObfuscation({});
|
||||
}
|
||||
file.SetXor(xor_key);
|
||||
|
||||
uint64_t mempool_transactions_to_write(vinfo.size());
|
||||
file << mempool_transactions_to_write;
|
||||
|
@@ -301,6 +301,15 @@ public:
|
||||
return ret;
|
||||
}
|
||||
|
||||
/** Generate fixed-size random bytes. */
|
||||
template <size_t N, BasicByte B = std::byte>
|
||||
std::array<B, N> randbytes() noexcept
|
||||
{
|
||||
std::array<B, N> ret;
|
||||
Impl().fillrand(MakeWritableByteSpan(ret));
|
||||
return ret;
|
||||
}
|
||||
|
||||
/** Generate a random 32-bit integer. */
|
||||
uint32_t rand32() noexcept { return Impl().template randbits<32>(); }
|
||||
|
||||
|
@@ -6,11 +6,11 @@
|
||||
#include <span.h>
|
||||
#include <streams.h>
|
||||
#include <util/fs_helpers.h>
|
||||
#include <util/obfuscation.h>
|
||||
|
||||
#include <array>
|
||||
|
||||
AutoFile::AutoFile(std::FILE* file, std::vector<std::byte> data_xor)
|
||||
: m_file{file}, m_xor{std::move(data_xor)}
|
||||
AutoFile::AutoFile(std::FILE* file, const Obfuscation& obfuscation) : m_file{file}, m_obfuscation{obfuscation}
|
||||
{
|
||||
if (!IsNull()) {
|
||||
auto pos{std::ftell(m_file)};
|
||||
@@ -21,12 +21,12 @@ AutoFile::AutoFile(std::FILE* file, std::vector<std::byte> data_xor)
|
||||
std::size_t AutoFile::detail_fread(std::span<std::byte> dst)
|
||||
{
|
||||
if (!m_file) throw std::ios_base::failure("AutoFile::read: file handle is nullptr");
|
||||
size_t ret = std::fread(dst.data(), 1, dst.size(), m_file);
|
||||
if (!m_xor.empty()) {
|
||||
if (!m_position.has_value()) throw std::ios_base::failure("AutoFile::read: position unknown");
|
||||
util::Xor(dst.subspan(0, ret), m_xor, *m_position);
|
||||
const size_t ret = std::fread(dst.data(), 1, dst.size(), m_file);
|
||||
if (m_obfuscation) {
|
||||
if (!m_position) throw std::ios_base::failure("AutoFile::read: position unknown");
|
||||
m_obfuscation(dst.subspan(0, ret), *m_position);
|
||||
}
|
||||
if (m_position.has_value()) *m_position += ret;
|
||||
if (m_position) *m_position += ret;
|
||||
return ret;
|
||||
}
|
||||
|
||||
@@ -81,7 +81,7 @@ void AutoFile::ignore(size_t nSize)
|
||||
void AutoFile::write(std::span<const std::byte> src)
|
||||
{
|
||||
if (!m_file) throw std::ios_base::failure("AutoFile::write: file handle is nullptr");
|
||||
if (m_xor.empty()) {
|
||||
if (!m_obfuscation) {
|
||||
if (std::fwrite(src.data(), 1, src.size(), m_file) != src.size()) {
|
||||
throw std::ios_base::failure("AutoFile::write: write failed");
|
||||
}
|
||||
@@ -101,9 +101,9 @@ void AutoFile::write(std::span<const std::byte> src)
|
||||
void AutoFile::write_buffer(std::span<std::byte> src)
|
||||
{
|
||||
if (!m_file) throw std::ios_base::failure("AutoFile::write_buffer: file handle is nullptr");
|
||||
if (m_xor.size()) {
|
||||
if (m_obfuscation) {
|
||||
if (!m_position) throw std::ios_base::failure("AutoFile::write_buffer: obfuscation position unknown");
|
||||
util::Xor(src, m_xor, *m_position); // obfuscate in-place
|
||||
m_obfuscation(src, *m_position); // obfuscate in-place
|
||||
}
|
||||
if (std::fwrite(src.data(), 1, src.size(), m_file) != src.size()) {
|
||||
throw std::ios_base::failure("AutoFile::write_buffer: write failed");
|
||||
|
@@ -11,6 +11,7 @@
|
||||
#include <span.h>
|
||||
#include <support/allocators/zeroafterfree.h>
|
||||
#include <util/check.h>
|
||||
#include <util/obfuscation.h>
|
||||
#include <util/overflow.h>
|
||||
#include <util/syserror.h>
|
||||
|
||||
@@ -24,30 +25,8 @@
|
||||
#include <limits>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
namespace util {
|
||||
inline void Xor(std::span<std::byte> write, std::span<const std::byte> key, size_t key_offset = 0)
|
||||
{
|
||||
if (key.size() == 0) {
|
||||
return;
|
||||
}
|
||||
key_offset %= key.size();
|
||||
|
||||
for (size_t i = 0, j = key_offset; i != write.size(); i++) {
|
||||
write[i] ^= key[j++];
|
||||
|
||||
// This potentially acts on very many bytes of data, so it's
|
||||
// important that we calculate `j`, i.e. the `key` index in this
|
||||
// way instead of doing a %, which would effectively be a division
|
||||
// for each byte Xor'd -- much slower than need be.
|
||||
if (j == key.size())
|
||||
j = 0;
|
||||
}
|
||||
}
|
||||
} // namespace util
|
||||
|
||||
/* Minimal stream for overwriting and/or appending to an existing byte vector
|
||||
*
|
||||
* The referenced vector will grow as necessary
|
||||
@@ -265,23 +244,13 @@ public:
|
||||
return (*this);
|
||||
}
|
||||
|
||||
template<typename T>
|
||||
template <typename T>
|
||||
DataStream& operator>>(T&& obj)
|
||||
{
|
||||
::Unserialize(*this, obj);
|
||||
return (*this);
|
||||
}
|
||||
|
||||
/**
|
||||
* XOR the contents of this stream with a certain key.
|
||||
*
|
||||
* @param[in] key The key used to XOR the data in this stream.
|
||||
*/
|
||||
void Xor(const std::vector<unsigned char>& key)
|
||||
{
|
||||
util::Xor(MakeWritableByteSpan(*this), MakeByteSpan(key));
|
||||
}
|
||||
|
||||
/** Compute total memory usage of this object (own memory + any dynamic memory). */
|
||||
size_t GetMemoryUsage() const noexcept;
|
||||
};
|
||||
@@ -402,12 +371,12 @@ class AutoFile
|
||||
{
|
||||
protected:
|
||||
std::FILE* m_file;
|
||||
std::vector<std::byte> m_xor;
|
||||
Obfuscation m_obfuscation;
|
||||
std::optional<int64_t> m_position;
|
||||
bool m_was_written{false};
|
||||
|
||||
public:
|
||||
explicit AutoFile(std::FILE* file, std::vector<std::byte> data_xor={});
|
||||
explicit AutoFile(std::FILE* file, const Obfuscation& obfuscation = {});
|
||||
|
||||
~AutoFile()
|
||||
{
|
||||
@@ -455,7 +424,7 @@ public:
|
||||
bool IsNull() const { return m_file == nullptr; }
|
||||
|
||||
/** Continue with a different XOR key */
|
||||
void SetXor(std::vector<std::byte> data_xor) { m_xor = data_xor; }
|
||||
void SetObfuscation(const Obfuscation& obfuscation) { m_obfuscation = obfuscation; }
|
||||
|
||||
/** Implementation detail, only used internally. */
|
||||
std::size_t detail_fread(std::span<std::byte> dst);
|
||||
|
@@ -9,39 +9,62 @@
|
||||
#include <util/string.h>
|
||||
|
||||
#include <memory>
|
||||
#include <ranges>
|
||||
|
||||
#include <boost/test/unit_test.hpp>
|
||||
|
||||
using util::ToString;
|
||||
|
||||
// Test if a string consists entirely of null characters
|
||||
static bool is_null_key(const std::vector<unsigned char>& key) {
|
||||
bool isnull = true;
|
||||
|
||||
for (unsigned int i = 0; i < key.size(); i++)
|
||||
isnull &= (key[i] == '\x00');
|
||||
|
||||
return isnull;
|
||||
}
|
||||
|
||||
BOOST_FIXTURE_TEST_SUITE(dbwrapper_tests, BasicTestingSetup)
|
||||
|
||||
BOOST_AUTO_TEST_CASE(dbwrapper)
|
||||
{
|
||||
// Perform tests both obfuscated and non-obfuscated.
|
||||
for (const bool obfuscate : {false, true}) {
|
||||
fs::path ph = m_args.GetDataDirBase() / (obfuscate ? "dbwrapper_obfuscate_true" : "dbwrapper_obfuscate_false");
|
||||
CDBWrapper dbw({.path = ph, .cache_bytes = 1 << 20, .memory_only = true, .wipe_data = false, .obfuscate = obfuscate});
|
||||
uint8_t key{'k'};
|
||||
uint256 in = m_rng.rand256();
|
||||
uint256 res;
|
||||
constexpr size_t CACHE_SIZE{1_MiB};
|
||||
const fs::path path{m_args.GetDataDirBase() / "dbwrapper"};
|
||||
|
||||
// Ensure that we're doing real obfuscation when obfuscate=true
|
||||
BOOST_CHECK(obfuscate != is_null_key(dbwrapper_private::GetObfuscateKey(dbw)));
|
||||
Obfuscation obfuscation;
|
||||
std::vector<std::pair<uint8_t, uint256>> key_values{};
|
||||
|
||||
BOOST_CHECK(dbw.Write(key, in));
|
||||
BOOST_CHECK(dbw.Read(key, res));
|
||||
BOOST_CHECK_EQUAL(res.ToString(), in.ToString());
|
||||
// Write values
|
||||
{
|
||||
CDBWrapper dbw{{.path = path, .cache_bytes = CACHE_SIZE, .wipe_data = true, .obfuscate = obfuscate}};
|
||||
BOOST_CHECK_EQUAL(obfuscate, !dbw.IsEmpty());
|
||||
|
||||
// Ensure that we're doing real obfuscation when obfuscate=true
|
||||
obfuscation = dbwrapper_private::GetObfuscation(dbw);
|
||||
BOOST_CHECK_EQUAL(obfuscate, dbwrapper_private::GetObfuscation(dbw));
|
||||
|
||||
for (uint8_t k{0}; k < 10; ++k) {
|
||||
uint8_t key{k};
|
||||
uint256 value{m_rng.rand256()};
|
||||
BOOST_CHECK(dbw.Write(key, value));
|
||||
key_values.emplace_back(key, value);
|
||||
}
|
||||
}
|
||||
|
||||
// Verify that the obfuscation key is never obfuscated
|
||||
{
|
||||
CDBWrapper dbw{{.path = path, .cache_bytes = CACHE_SIZE, .obfuscate = false}};
|
||||
BOOST_CHECK_EQUAL(obfuscation, dbwrapper_private::GetObfuscation(dbw));
|
||||
}
|
||||
|
||||
// Read back the values
|
||||
{
|
||||
CDBWrapper dbw{{.path = path, .cache_bytes = CACHE_SIZE, .obfuscate = obfuscate}};
|
||||
|
||||
// Ensure obfuscation is read back correctly
|
||||
BOOST_CHECK_EQUAL(obfuscation, dbwrapper_private::GetObfuscation(dbw));
|
||||
BOOST_CHECK_EQUAL(obfuscate, dbwrapper_private::GetObfuscation(dbw));
|
||||
|
||||
// Verify all written values
|
||||
for (const auto& [key, expected_value] : key_values) {
|
||||
uint256 read_value{};
|
||||
BOOST_CHECK(dbw.Read(key, read_value));
|
||||
BOOST_CHECK_EQUAL(read_value, expected_value);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -57,7 +80,7 @@ BOOST_AUTO_TEST_CASE(dbwrapper_basic_data)
|
||||
bool res_bool;
|
||||
|
||||
// Ensure that we're doing real obfuscation when obfuscate=true
|
||||
BOOST_CHECK(obfuscate != is_null_key(dbwrapper_private::GetObfuscateKey(dbw)));
|
||||
BOOST_CHECK_EQUAL(obfuscate, dbwrapper_private::GetObfuscation(dbw));
|
||||
|
||||
//Simulate block raw data - "b + block hash"
|
||||
std::string key_block = "b" + m_rng.rand256().ToString();
|
||||
@@ -116,13 +139,13 @@ BOOST_AUTO_TEST_CASE(dbwrapper_basic_data)
|
||||
std::string file_option_tag = "F";
|
||||
uint8_t filename_length = m_rng.randbits(8);
|
||||
std::string filename = "randomfilename";
|
||||
std::string key_file_option = strprintf("%s%01x%s", file_option_tag,filename_length,filename);
|
||||
std::string key_file_option = strprintf("%s%01x%s", file_option_tag, filename_length, filename);
|
||||
|
||||
bool in_file_bool = m_rng.randbool();
|
||||
BOOST_CHECK(dbw.Write(key_file_option, in_file_bool));
|
||||
BOOST_CHECK(dbw.Read(key_file_option, res_bool));
|
||||
BOOST_CHECK_EQUAL(res_bool, in_file_bool);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Test batch operations
|
||||
@@ -231,8 +254,8 @@ BOOST_AUTO_TEST_CASE(existing_data_no_obfuscate)
|
||||
BOOST_CHECK(odbw.Read(key, res2));
|
||||
BOOST_CHECK_EQUAL(res2.ToString(), in.ToString());
|
||||
|
||||
BOOST_CHECK(!odbw.IsEmpty()); // There should be existing data
|
||||
BOOST_CHECK(is_null_key(dbwrapper_private::GetObfuscateKey(odbw))); // The key should be an empty string
|
||||
BOOST_CHECK(!odbw.IsEmpty());
|
||||
BOOST_CHECK(!dbwrapper_private::GetObfuscation(odbw)); // The key should be an empty string
|
||||
|
||||
uint256 in2 = m_rng.rand256();
|
||||
uint256 res3;
|
||||
@@ -269,7 +292,7 @@ BOOST_AUTO_TEST_CASE(existing_data_reindex)
|
||||
// Check that the key/val we wrote with unobfuscated wrapper doesn't exist
|
||||
uint256 res2;
|
||||
BOOST_CHECK(!odbw.Read(key, res2));
|
||||
BOOST_CHECK(!is_null_key(dbwrapper_private::GetObfuscateKey(odbw)));
|
||||
BOOST_CHECK(dbwrapper_private::GetObfuscation(odbw));
|
||||
|
||||
uint256 in2 = m_rng.rand256();
|
||||
uint256 res3;
|
||||
|
@@ -4,9 +4,10 @@
|
||||
|
||||
#include <span.h>
|
||||
#include <streams.h>
|
||||
#include <test/fuzz/FuzzedDataProvider.h>
|
||||
#include <test/fuzz/fuzz.h>
|
||||
#include <test/fuzz/FuzzedDataProvider.h>
|
||||
#include <test/fuzz/util.h>
|
||||
#include <util/obfuscation.h>
|
||||
|
||||
#include <array>
|
||||
#include <cstddef>
|
||||
@@ -18,9 +19,10 @@ FUZZ_TARGET(autofile)
|
||||
{
|
||||
FuzzedDataProvider fuzzed_data_provider{buffer.data(), buffer.size()};
|
||||
FuzzedFileProvider fuzzed_file_provider{fuzzed_data_provider};
|
||||
const auto key_bytes{ConsumeFixedLengthByteVector<std::byte>(fuzzed_data_provider, Obfuscation::KEY_SIZE)};
|
||||
AutoFile auto_file{
|
||||
fuzzed_file_provider.open(),
|
||||
ConsumeRandomLengthByteVector<std::byte>(fuzzed_data_provider),
|
||||
Obfuscation{std::span{key_bytes}.first<Obfuscation::KEY_SIZE>()},
|
||||
};
|
||||
LIMITED_WHILE(fuzzed_data_provider.ConsumeBool(), 100)
|
||||
{
|
||||
|
@@ -4,9 +4,10 @@
|
||||
|
||||
#include <span.h>
|
||||
#include <streams.h>
|
||||
#include <test/fuzz/FuzzedDataProvider.h>
|
||||
#include <test/fuzz/fuzz.h>
|
||||
#include <test/fuzz/FuzzedDataProvider.h>
|
||||
#include <test/fuzz/util.h>
|
||||
#include <util/obfuscation.h>
|
||||
|
||||
#include <array>
|
||||
#include <cstddef>
|
||||
@@ -20,9 +21,10 @@ FUZZ_TARGET(buffered_file)
|
||||
FuzzedDataProvider fuzzed_data_provider{buffer.data(), buffer.size()};
|
||||
FuzzedFileProvider fuzzed_file_provider{fuzzed_data_provider};
|
||||
std::optional<BufferedFile> opt_buffered_file;
|
||||
const auto key_bytes{ConsumeFixedLengthByteVector<std::byte>(fuzzed_data_provider, Obfuscation::KEY_SIZE)};
|
||||
AutoFile fuzzed_file{
|
||||
fuzzed_file_provider.open(),
|
||||
ConsumeRandomLengthByteVector<std::byte>(fuzzed_data_provider),
|
||||
Obfuscation{std::span{key_bytes}.first<Obfuscation::KEY_SIZE>()},
|
||||
};
|
||||
try {
|
||||
auto n_buf_size = fuzzed_data_provider.ConsumeIntegralInRange<uint64_t>(0, 4096);
|
||||
|
@@ -58,7 +58,7 @@ BOOST_AUTO_TEST_CASE(fastrandom_tests_deterministic)
|
||||
BOOST_CHECK_EQUAL(ctx1.rand32(), ctx2.rand32());
|
||||
BOOST_CHECK_EQUAL(ctx1.rand64(), ctx2.rand64());
|
||||
BOOST_CHECK_EQUAL(ctx1.randbits(3), ctx2.randbits(3));
|
||||
BOOST_CHECK(ctx1.randbytes(17) == ctx2.randbytes(17));
|
||||
BOOST_CHECK(std::ranges::equal(ctx1.randbytes<std::byte>(17), ctx2.randbytes<17>())); // check vector/array behavior symmetry
|
||||
BOOST_CHECK(ctx1.rand256() == ctx2.rand256());
|
||||
BOOST_CHECK_EQUAL(ctx1.randbits(7), ctx2.randbits(7));
|
||||
BOOST_CHECK(ctx1.randbytes(128) == ctx2.randbytes(128));
|
||||
|
@@ -8,27 +8,120 @@
|
||||
#include <test/util/random.h>
|
||||
#include <test/util/setup_common.h>
|
||||
#include <util/fs.h>
|
||||
#include <util/obfuscation.h>
|
||||
#include <util/strencodings.h>
|
||||
|
||||
#include <boost/test/unit_test.hpp>
|
||||
|
||||
using namespace std::string_literals;
|
||||
using namespace util::hex_literals;
|
||||
|
||||
BOOST_FIXTURE_TEST_SUITE(streams_tests, BasicTestingSetup)
|
||||
|
||||
// Test that obfuscation can be properly reverted even with random chunk sizes.
|
||||
BOOST_AUTO_TEST_CASE(xor_roundtrip_random_chunks)
|
||||
{
|
||||
auto apply_random_xor_chunks{[&](std::span<std::byte> target, const Obfuscation& obfuscation) {
|
||||
for (size_t offset{0}; offset < target.size();) {
|
||||
const size_t chunk_size{1 + m_rng.randrange(target.size() - offset)};
|
||||
obfuscation(target.subspan(offset, chunk_size), offset);
|
||||
offset += chunk_size;
|
||||
}
|
||||
}};
|
||||
|
||||
for (size_t test{0}; test < 100; ++test) {
|
||||
const size_t write_size{1 + m_rng.randrange(100U)};
|
||||
const std::vector original{m_rng.randbytes<std::byte>(write_size)};
|
||||
std::vector roundtrip{original};
|
||||
|
||||
const auto key_bytes{m_rng.randbool() ? m_rng.randbytes<Obfuscation::KEY_SIZE>() : std::array<std::byte, Obfuscation::KEY_SIZE>{}};
|
||||
const Obfuscation obfuscation{key_bytes};
|
||||
apply_random_xor_chunks(roundtrip, obfuscation);
|
||||
|
||||
const bool key_all_zeros{std::ranges::all_of(
|
||||
std::span{key_bytes}.first(std::min(write_size, Obfuscation::KEY_SIZE)), [](auto b) { return b == std::byte{0}; })};
|
||||
BOOST_CHECK(key_all_zeros ? original == roundtrip : original != roundtrip);
|
||||
|
||||
apply_random_xor_chunks(roundtrip, obfuscation);
|
||||
BOOST_CHECK(original == roundtrip);
|
||||
}
|
||||
}
|
||||
|
||||
// Compares optimized obfuscation against a trivial, byte-by-byte reference implementation
|
||||
// with random offsets to ensure proper handling of key wrapping.
|
||||
BOOST_AUTO_TEST_CASE(xor_bytes_reference)
|
||||
{
|
||||
auto expected_xor{[](std::span<std::byte> target, std::span<const std::byte, Obfuscation::KEY_SIZE> obfuscation, size_t key_offset) {
|
||||
for (auto& b : target) {
|
||||
b ^= obfuscation[key_offset++ % obfuscation.size()];
|
||||
}
|
||||
}};
|
||||
|
||||
for (size_t test{0}; test < 100; ++test) {
|
||||
const size_t write_size{1 + m_rng.randrange(100U)};
|
||||
const size_t key_offset{m_rng.randrange(3 * Obfuscation::KEY_SIZE)}; // Make sure the key can wrap around
|
||||
const size_t write_offset{std::min(write_size, m_rng.randrange(Obfuscation::KEY_SIZE * 2))}; // Write unaligned data
|
||||
|
||||
const auto key_bytes{m_rng.randbool() ? m_rng.randbytes<Obfuscation::KEY_SIZE>() : std::array<std::byte, Obfuscation::KEY_SIZE>{}};
|
||||
const Obfuscation obfuscation{key_bytes};
|
||||
std::vector expected{m_rng.randbytes<std::byte>(write_size)};
|
||||
std::vector actual{expected};
|
||||
|
||||
expected_xor(std::span{expected}.subspan(write_offset), key_bytes, key_offset);
|
||||
obfuscation(std::span{actual}.subspan(write_offset), key_offset);
|
||||
|
||||
BOOST_CHECK_EQUAL_COLLECTIONS(expected.begin(), expected.end(), actual.begin(), actual.end());
|
||||
}
|
||||
}
|
||||
|
||||
BOOST_AUTO_TEST_CASE(obfuscation_hexkey)
|
||||
{
|
||||
const auto key_bytes{m_rng.randbytes<Obfuscation::KEY_SIZE>()};
|
||||
|
||||
const Obfuscation obfuscation{key_bytes};
|
||||
BOOST_CHECK_EQUAL(obfuscation.HexKey(), HexStr(key_bytes));
|
||||
}
|
||||
|
||||
BOOST_AUTO_TEST_CASE(obfuscation_serialize)
|
||||
{
|
||||
const Obfuscation original{m_rng.randbytes<Obfuscation::KEY_SIZE>()};
|
||||
|
||||
// Serialization
|
||||
DataStream ds;
|
||||
ds << original;
|
||||
|
||||
BOOST_CHECK_EQUAL(ds.size(), 1 + Obfuscation::KEY_SIZE); // serialized as a vector
|
||||
|
||||
// Deserialization
|
||||
Obfuscation recovered{};
|
||||
ds >> recovered;
|
||||
|
||||
BOOST_CHECK_EQUAL(recovered.HexKey(), original.HexKey());
|
||||
}
|
||||
|
||||
BOOST_AUTO_TEST_CASE(obfuscation_empty)
|
||||
{
|
||||
const Obfuscation null_obf{};
|
||||
BOOST_CHECK(!null_obf);
|
||||
|
||||
const Obfuscation non_null_obf{"ff00ff00ff00ff00"_hex};
|
||||
BOOST_CHECK(non_null_obf);
|
||||
}
|
||||
|
||||
BOOST_AUTO_TEST_CASE(xor_file)
|
||||
{
|
||||
fs::path xor_path{m_args.GetDataDirBase() / "test_xor.bin"};
|
||||
auto raw_file{[&](const auto& mode) { return fsbridge::fopen(xor_path, mode); }};
|
||||
const std::vector<uint8_t> test1{1, 2, 3};
|
||||
const std::vector<uint8_t> test2{4, 5};
|
||||
const std::vector<std::byte> xor_pat{std::byte{0xff}, std::byte{0x00}};
|
||||
const Obfuscation obfuscation{"ff00ff00ff00ff00"_hex};
|
||||
|
||||
{
|
||||
// Check errors for missing file
|
||||
AutoFile xor_file{raw_file("rb"), xor_pat};
|
||||
BOOST_CHECK_EXCEPTION(xor_file << std::byte{}, std::ios_base::failure, HasReason{"AutoFile::write: file handle is nullpt"});
|
||||
BOOST_CHECK_EXCEPTION(xor_file >> std::byte{}, std::ios_base::failure, HasReason{"AutoFile::read: file handle is nullpt"});
|
||||
BOOST_CHECK_EXCEPTION(xor_file.ignore(1), std::ios_base::failure, HasReason{"AutoFile::ignore: file handle is nullpt"});
|
||||
AutoFile xor_file{raw_file("rb"), obfuscation};
|
||||
BOOST_CHECK_EXCEPTION(xor_file << std::byte{}, std::ios_base::failure, HasReason{"AutoFile::write: file handle is nullptr"});
|
||||
BOOST_CHECK_EXCEPTION(xor_file >> std::byte{}, std::ios_base::failure, HasReason{"AutoFile::read: file handle is nullptr"});
|
||||
BOOST_CHECK_EXCEPTION(xor_file.ignore(1), std::ios_base::failure, HasReason{"AutoFile::ignore: file handle is nullptr"});
|
||||
}
|
||||
{
|
||||
#ifdef __MINGW64__
|
||||
@@ -37,7 +130,7 @@ BOOST_AUTO_TEST_CASE(xor_file)
|
||||
#else
|
||||
const char* mode = "wbx";
|
||||
#endif
|
||||
AutoFile xor_file{raw_file(mode), xor_pat};
|
||||
AutoFile xor_file{raw_file(mode), obfuscation};
|
||||
xor_file << test1 << test2;
|
||||
BOOST_REQUIRE_EQUAL(xor_file.fclose(), 0);
|
||||
}
|
||||
@@ -51,7 +144,7 @@ BOOST_AUTO_TEST_CASE(xor_file)
|
||||
BOOST_CHECK_EXCEPTION(non_xor_file.ignore(1), std::ios_base::failure, HasReason{"AutoFile::ignore: end of file"});
|
||||
}
|
||||
{
|
||||
AutoFile xor_file{raw_file("rb"), xor_pat};
|
||||
AutoFile xor_file{raw_file("rb"), obfuscation};
|
||||
std::vector<std::byte> read1, read2;
|
||||
xor_file >> read1 >> read2;
|
||||
BOOST_CHECK_EQUAL(HexStr(read1), HexStr(test1));
|
||||
@@ -60,7 +153,7 @@ BOOST_AUTO_TEST_CASE(xor_file)
|
||||
BOOST_CHECK_EXCEPTION(xor_file >> std::byte{}, std::ios_base::failure, HasReason{"AutoFile::read: end of file"});
|
||||
}
|
||||
{
|
||||
AutoFile xor_file{raw_file("rb"), xor_pat};
|
||||
AutoFile xor_file{raw_file("rb"), obfuscation};
|
||||
std::vector<std::byte> read2;
|
||||
// Check that ignore works
|
||||
xor_file.ignore(4);
|
||||
@@ -76,7 +169,7 @@ BOOST_AUTO_TEST_CASE(streams_vector_writer)
|
||||
{
|
||||
unsigned char a(1);
|
||||
unsigned char b(2);
|
||||
unsigned char bytes[] = { 3, 4, 5, 6 };
|
||||
unsigned char bytes[] = {3, 4, 5, 6};
|
||||
std::vector<unsigned char> vch;
|
||||
|
||||
// Each test runs twice. Serializing a second time at the same starting
|
||||
@@ -223,34 +316,26 @@ BOOST_AUTO_TEST_CASE(bitstream_reader_writer)
|
||||
|
||||
BOOST_AUTO_TEST_CASE(streams_serializedata_xor)
|
||||
{
|
||||
std::vector<std::byte> in;
|
||||
|
||||
// Degenerate case
|
||||
{
|
||||
DataStream ds{in};
|
||||
ds.Xor({0x00, 0x00});
|
||||
DataStream ds{};
|
||||
Obfuscation{}(ds);
|
||||
BOOST_CHECK_EQUAL(""s, ds.str());
|
||||
}
|
||||
|
||||
in.push_back(std::byte{0x0f});
|
||||
in.push_back(std::byte{0xf0});
|
||||
|
||||
// Single character key
|
||||
{
|
||||
DataStream ds{in};
|
||||
ds.Xor({0xff});
|
||||
const Obfuscation obfuscation{"ffffffffffffffff"_hex};
|
||||
|
||||
DataStream ds{"0ff0"_hex};
|
||||
obfuscation(ds);
|
||||
BOOST_CHECK_EQUAL("\xf0\x0f"s, ds.str());
|
||||
}
|
||||
|
||||
// Multi character key
|
||||
|
||||
in.clear();
|
||||
in.push_back(std::byte{0xf0});
|
||||
in.push_back(std::byte{0x0f});
|
||||
|
||||
{
|
||||
DataStream ds{in};
|
||||
ds.Xor({0xff, 0x0f});
|
||||
const Obfuscation obfuscation{"ff0fff0fff0fff0f"_hex};
|
||||
|
||||
DataStream ds{"f00f"_hex};
|
||||
obfuscation(ds);
|
||||
BOOST_CHECK_EQUAL("\x0f\x00"s, ds.str());
|
||||
}
|
||||
}
|
||||
@@ -563,7 +648,7 @@ BOOST_AUTO_TEST_CASE(buffered_reader_matches_autofile_random_content)
|
||||
const FlatFilePos pos{0, 0};
|
||||
|
||||
const FlatFileSeq test_file{m_args.GetDataDirBase(), "buffered_file_test_random", node::BLOCKFILE_CHUNK_SIZE};
|
||||
const std::vector obfuscation{m_rng.randbytes<std::byte>(8)};
|
||||
const Obfuscation obfuscation{m_rng.randbytes<Obfuscation::KEY_SIZE>()};
|
||||
|
||||
// Write out the file with random content
|
||||
{
|
||||
@@ -618,7 +703,7 @@ BOOST_AUTO_TEST_CASE(buffered_writer_matches_autofile_random_content)
|
||||
|
||||
const FlatFileSeq test_buffered{m_args.GetDataDirBase(), "buffered_write_test", node::BLOCKFILE_CHUNK_SIZE};
|
||||
const FlatFileSeq test_direct{m_args.GetDataDirBase(), "direct_write_test", node::BLOCKFILE_CHUNK_SIZE};
|
||||
const std::vector obfuscation{m_rng.randbytes<std::byte>(8)};
|
||||
const Obfuscation obfuscation{m_rng.randbytes<Obfuscation::KEY_SIZE>()};
|
||||
|
||||
{
|
||||
DataBuffer test_data{m_rng.randbytes<std::byte>(file_size)};
|
||||
|
115
src/util/obfuscation.h
Normal file
115
src/util/obfuscation.h
Normal file
@@ -0,0 +1,115 @@
|
||||
// Copyright (c) 2025-present The Bitcoin Core developers
|
||||
// Distributed under the MIT software license, see the accompanying
|
||||
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
|
||||
#ifndef BITCOIN_UTIL_OBFUSCATION_H
|
||||
#define BITCOIN_UTIL_OBFUSCATION_H
|
||||
|
||||
#include <cstdint>
|
||||
#include <span.h>
|
||||
#include <tinyformat.h>
|
||||
#include <util/strencodings.h>
|
||||
|
||||
#include <array>
|
||||
#include <bit>
|
||||
#include <climits>
|
||||
#include <ios>
|
||||
#include <memory>
|
||||
|
||||
class Obfuscation
|
||||
{
|
||||
public:
|
||||
using KeyType = uint64_t;
|
||||
static constexpr size_t KEY_SIZE{sizeof(KeyType)};
|
||||
|
||||
Obfuscation() { SetRotations(0); }
|
||||
explicit Obfuscation(std::span<const std::byte, KEY_SIZE> key_bytes)
|
||||
{
|
||||
SetRotations(ToKey(key_bytes));
|
||||
}
|
||||
|
||||
operator bool() const { return m_rotations[0] != 0; }
|
||||
|
||||
void operator()(std::span<std::byte> target, size_t key_offset = 0) const
|
||||
{
|
||||
if (!*this) return;
|
||||
|
||||
KeyType rot_key{m_rotations[key_offset % KEY_SIZE]}; // Continue obfuscation from where we left off
|
||||
if (target.size() > KEY_SIZE) {
|
||||
// Obfuscate until 64-bit alignment boundary
|
||||
if (const auto misalign{std::bit_cast<uintptr_t>(target.data()) % KEY_SIZE}) {
|
||||
const size_t alignment{std::min(KEY_SIZE - misalign, target.size())};
|
||||
XorWord(target.first(alignment), rot_key);
|
||||
|
||||
target = {std::assume_aligned<KEY_SIZE>(target.data() + alignment), target.size() - alignment};
|
||||
rot_key = m_rotations[(key_offset + alignment) % KEY_SIZE];
|
||||
}
|
||||
// Aligned obfuscation in 64-byte chunks
|
||||
for (constexpr auto unroll{8}; target.size() >= KEY_SIZE * unroll; target = target.subspan(KEY_SIZE * unroll)) {
|
||||
for (size_t i{0}; i < unroll; ++i) {
|
||||
XorWord(target.subspan(i * KEY_SIZE, KEY_SIZE), rot_key);
|
||||
}
|
||||
}
|
||||
// Aligned obfuscation in 64-bit chunks
|
||||
for (; target.size() >= KEY_SIZE; target = target.subspan(KEY_SIZE)) {
|
||||
XorWord(target.first<KEY_SIZE>(), rot_key);
|
||||
}
|
||||
}
|
||||
XorWord(target, rot_key);
|
||||
}
|
||||
|
||||
template <typename Stream>
|
||||
void Serialize(Stream& s) const
|
||||
{
|
||||
// Use vector serialization for convenient compact size prefix.
|
||||
std::vector<std::byte> bytes{KEY_SIZE};
|
||||
std::memcpy(bytes.data(), &m_rotations[0], KEY_SIZE);
|
||||
s << bytes;
|
||||
}
|
||||
|
||||
template <typename Stream>
|
||||
void Unserialize(Stream& s)
|
||||
{
|
||||
std::vector<std::byte> bytes{KEY_SIZE};
|
||||
s >> bytes;
|
||||
if (bytes.size() != KEY_SIZE) throw std::ios_base::failure(strprintf("Obfuscation key size should be exactly %s bytes long", KEY_SIZE));
|
||||
SetRotations(ToKey(std::span<std::byte, KEY_SIZE>(bytes)));
|
||||
}
|
||||
|
||||
std::string HexKey() const
|
||||
{
|
||||
return HexStr(std::bit_cast<std::array<uint8_t, KEY_SIZE>>(m_rotations[0]));
|
||||
}
|
||||
|
||||
private:
|
||||
// Cached key rotations for different offsets.
|
||||
std::array<KeyType, KEY_SIZE> m_rotations;
|
||||
|
||||
void SetRotations(KeyType key)
|
||||
{
|
||||
for (size_t i{0}; i < KEY_SIZE; ++i) {
|
||||
int key_rotation_bits{int(CHAR_BIT * i)};
|
||||
if constexpr (std::endian::native == std::endian::big) key_rotation_bits *= -1;
|
||||
m_rotations[i] = std::rotr(key, key_rotation_bits);
|
||||
}
|
||||
}
|
||||
|
||||
static KeyType ToKey(std::span<const std::byte, KEY_SIZE> key_span)
|
||||
{
|
||||
KeyType key{};
|
||||
std::memcpy(&key, key_span.data(), KEY_SIZE);
|
||||
return key;
|
||||
}
|
||||
|
||||
static void XorWord(std::span<std::byte> target, KeyType key)
|
||||
{
|
||||
assert(target.size() <= KEY_SIZE);
|
||||
if (target.empty()) return;
|
||||
KeyType raw{};
|
||||
std::memcpy(&raw, target.data(), target.size());
|
||||
raw ^= key;
|
||||
std::memcpy(target.data(), &raw, target.size());
|
||||
}
|
||||
};
|
||||
|
||||
#endif // BITCOIN_UTIL_OBFUSCATION_H
|
Reference in New Issue
Block a user