highperfocused/bitcoin
1
0
Fork 0
mirror of https://github.com/bitcoin/bitcoin.git synced 2026-04-27 07:18:33 +02:00
Code Issues Packages Projects Releases Wiki Activity

Merge #19077: wallet: Add sqlite as an alternative wallet database and use it for new descriptor wallets

Browse Source 
c4a29d0a90 Update wallet_multiwallet.py for descriptor and sqlite wallets (Russell Yanofsky)
310b0fde04 Run dumpwallet for legacy wallets only in  wallet_backup.py (Andrew Chow)
6c6639ac9f Include sqlite3 in documentation (Andrew Chow)
f023b7cac0 wallet: Enforce sqlite serialized threading mode (Andrew Chow)
6173269866 Set and check the sqlite user version (Andrew Chow)
9d3d2d263c Use network magic as sqlite wallet application ID (Andrew Chow)
9af5de3798 Use SQLite for descriptor wallets (Andrew Chow)
9b78f3ce8e walletutil: Wallets can also be sqlite (Andrew Chow)
ac38a87225 Determine wallet file type based on file magic (Andrew Chow)
6045f77003 Implement SQLiteDatabase::MakeBatch (Andrew Chow)
727e6b2a4e Implement SQLiteDatabase::Verify (Andrew Chow)
b4df8fdb19 Implement SQLiteDatabase::Rewrite (Andrew Chow)
010e365906 Implement SQLiteDatabase::TxnBegin, TxnCommit, and TxnAbort (Andrew Chow)
ac5c1617e7 Implement SQLiteDatabase::Backup (Andrew Chow)
f6f9cd6a64 Implement SQLiteBatch::StartCursor, ReadAtCursor, and CloseCursor (Andrew Chow)
bf90e033f4 Implement SQLiteBatch::ReadKey, WriteKey, EraseKey, and HasKey (Andrew Chow)
7aa45620e2 Add SetupSQLStatements (Andrew Chow)
6636a2608a Implement SQLiteBatch::Close (Andrew Chow)
93825352a3 Implement SQLiteDatabase::Close (Andrew Chow)
a0de83372b Implement SQLiteDatabase::Open (Andrew Chow)
3bfa0fe125 Initialize and Shutdown sqlite3 globals (Andrew Chow)
5a488b3d77 Constructors, destructors, and relevant private fields for SQLiteDatabase/Batch (Andrew Chow)
ca8b7e04ab Implement SQLiteDatabaseVersion (Andrew Chow)
7577b6e1c8 Add SQLiteDatabase and SQLiteBatch dummy classes (Andrew Chow)
e87df82580 Add sqlite to travis and depends (Andrew Chow)
54729f3f4e Add libsqlite3 (Andrew Chow)

Pull request description:

  This PR adds a new class `SQLiteDatabase` which is a subclass of `WalletDatabase`. This provides access to a SQLite database that is used to store the wallet records. To keep compatibility with BDB and to complexity of the change down, we don't make use of many SQLite's features. We use it strictly as a key-value store. We create a table `main` which has two columns, `key` and `value` both with the type `blob`.

  For new descriptor wallets, we will create a `SQLiteDatabase` instead of a `BerkeleyDatabase`. There is no requirement that all SQLite wallets are descriptor wallets, nor is there a requirement that all descriptor wallets be SQLite wallets. This allows for existing descriptor wallets to work as well as keeping open the option to migrate existing wallets to SQLite.

  We keep the name `wallet.dat` for SQLite wallets. We are able to determine which database type to use by searching for specific magic bytes in the `wallet.dat` file. SQLite begins it's files with a null terminated string `SQLite format 3`. BDB has `0x00053162` at byte 12 (note that the byte order of this integer depends on the system endianness). So when we see that there is a `wallet.dat` file that we want to open, we check for the magic bytes to determine which database system to use.

  I decided to keep the `wallet.dat` naming to keep things like backup script to continue to function as they won't need to be modified to look for a different file name. It also simplifies a couple of things in the implementation and the tests as `wallet.dat` is something that is specifically being looked for. If we don't want this behavior, then I do have another branch which creates `wallet.sqlite` files instead, but I find that this direction is easier.

ACKs for top commit:
  Sjors:
    re-utACK c4a29d0a90
  promag:
    Tested ACK c4a29d0a90.
  fjahr:
    reACK c4a29d0a90
  S3RK:
    Re-review ACK c4a29d0a90
  meshcollider:
    re-utACK c4a29d0a90
  hebasto:
    re-ACK c4a29d0a90, only rebased since my [previous](https://github.com/bitcoin/bitcoin/pull/19077#pullrequestreview-507743699) review, verified with `git range-diff master d18892dcc c4a29d0a9`.
  ryanofsky:
    Code review ACK c4a29d0a90. I am honestly confused about reasons for locking into `wallet.dat` again when it's so easy now to use a clean format. I assume I'm just very dense, or there's some unstated reason, because the only thing that's been brought up are unrealistic compatibility scenarios (all require actively creating a wallet with non-default descriptor+sqlite option, then trying to using the descriptor+sqlite wallets with old software or scripts and ignoring the results) that we didn't pay attention to with previous PRs like #11687, which did not require any active interfaction.
  jonatack:
    ACK c4a29d0a90, debug builds and test runs after rebase to latest master @ c2c4dbaebd, some manual testing creating, using, unloading and reloading a few different new sqlite descriptor wallets over several node restarts/shutdowns.

Tree-SHA512: 19145732e5001484947352d3175a660b5102bc6e833f227a55bd41b9b2f4d92737bbed7cead64b75b509decf9e1408cd81c185ab1fb4b90561aee427c4f9751c
This commit is contained in:
Samuel Dobson
2020-10-15 19:20:18 +13:00
parent f2e6d14430 c4a29d0a90
commit 8ed37f6c84
34 changed files with 934 additions and 80 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
doc/build-osx.md
View File

@@ -19,7 +19,7 @@ Then install [Homebrew](https://brew.sh).
## Dependencies
```shell
brew install automake berkeley-db4 libtool boost miniupnpc pkg-config python qt libevent qrencode
brew install automake berkeley-db4 libtool boost miniupnpc pkg-config python qt libevent qrencode sqlite
```
If you run into issues, check [Homebrew's troubleshooting page](https://docs.brew.sh/Troubleshooting).
@@ -79,7 +79,7 @@ compiled in `disable-wallet` mode with:
./configure --disable-wallet
```
In this case there is no dependency on Berkeley DB 4.8.
In this case there is no dependency on Berkeley DB 4.8 and SQLite.
Mining is also possible in disable-wallet mode using the `getblocktemplate` RPC call.

11
doc/build-unix.md
View File

@@ -46,6 +46,7 @@ Optional dependencies:
libqrencode | QR codes in GUI | Optional for generating QR codes (only needed when GUI enabled)
univalue | Utility | JSON parsing and encoding (bundled version will be used unless --with-system-univalue passed to configure)
libzmq3 | ZMQ notification | Optional, allows generating ZMQ notifications (requires ZMQ version >= 4.0.0)
sqlite3 | SQLite DB | Wallet storage (only needed when wallet enabled)
For the versions used, see [dependencies.md](dependencies.md)
@@ -91,6 +92,10 @@ pass `--with-incompatible-bdb` to configure.
Otherwise, you can build from self-compiled `depends` (see above).
SQLite is required for the wallet:
sudo apt install libsqlite3-dev
To build Bitcoin Core without wallet, see [*Disable-wallet mode*](/doc/build-unix.md#disable-wallet-mode)
@@ -144,6 +149,10 @@ libqrencode (optional) can be installed with:
sudo dnf install qrencode-devel
SQLite can be installed with:
sudo dnf install sqlite-devel
Notes
-----
The release is built with GCC and then "strip bitcoind" to strip the debug
@@ -238,7 +247,7 @@ disable-wallet mode with:
./configure --disable-wallet
In this case there is no dependency on Berkeley DB 4.8.
In this case there is no dependency on Berkeley DB 4.8 and SQLite.
Mining is also possible in disable-wallet mode using the `getblocktemplate` RPC call.

2
doc/dependencies.md
View File

@@ -21,6 +21,7 @@ These are the dependencies currently used by Bitcoin Core. You can find instruct
| Python (tests) | | [3.5](https://www.python.org/downloads) | | | |
| qrencode | [3.4.4](https://fukuchi.org/works/qrencode) | | No | | |
| Qt | [5.9.8](https://download.qt.io/official_releases/qt/) | [5.5.1](https://github.com/bitcoin/bitcoin/issues/13478) | No | | |
| SQLite | [3.32.1](https://sqlite.org/download.html) | [3.7.17](https://github.com/bitcoin/bitcoin/pull/19077) | | | |
| XCB | | | | | [Yes](https://github.com/bitcoin/bitcoin/blob/master/depends/packages/qt.mk) (Linux only) |
| xkbcommon | | | | | [Yes](https://github.com/bitcoin/bitcoin/blob/master/depends/packages/qt.mk) (Linux only) |
| ZeroMQ | [4.3.1](https://github.com/zeromq/libzmq/releases) | 4.0.0 | No | | |
@@ -33,6 +34,7 @@ Some dependencies are not needed in all configurations. The following are some f
#### Options passed to `./configure`
* MiniUPnPc is not needed with `--with-miniupnpc=no`.
* Berkeley DB is not needed with `--disable-wallet`.
* SQLite is not needed with `--disable-wallet`.
* Qt is not needed with `--without-gui`.
* If the qrencode dependency is absent, QR support won't be added. To force an error when that happens, pass `--with-qrencode`.
* ZeroMQ is needed only with the `--with-zmq` option.

3
doc/files.md
View File

@@ -72,8 +72,9 @@ Subdirectory | File(s) | Description
-------------|-------------------|------------
`database/` | BDB logging files | Part of BDB environment; created at start and deleted on shutdown; a user *must keep it as safe* as personal wallet `wallet.dat`
`./` | `db.log` | BDB error file
`./` | `wallet.dat` | Personal wallet (BDB) with keys and transactions
`./` | `wallet.dat` | Personal wallet with keys and transactions. May be either a Berkeley DB or SQLite database file.
`./` | `.walletlock` | Wallet lock file
`./` | `wallet.dat-journal` | SQLite Rollback Journal file for `wallet.dat`. Usually created at start and deleted on shutdown. A user *must keep it as safe* as the `wallet.dat` file.
1. Each user-defined wallet named "wallet_name" resides in `wallets/wallet_name/` subdirectory.