Merge bitcoin/bitcoin#27286: wallet: Keep track of the wallet's own transaction outputs in memory

215e5999e2 wallet: Remove unused CachedTxGet{Available,Immature}Credit (Ava Chow)
49675de035 wallet: Have GetDebit use the wallet's TXO set (Ava Chow)
17d453cb3a wallet: Recompute wallet TXOs after descriptor migration (Ava Chow)
764016eb22 wallet: Retrieve TXO directly in FetchSelectedInputs (Ava Chow)
c1801b78f1 wallet: Use wallet's TXO set in AvailableCoins (Ava Chow)
dde7cbe105 wallet: Change balance calculation to use m_txos (Ava Chow)
96e7a89c5e wallet: Recalculate the wallet's txos after any imports (Ava Chow)
ae888c38d0 wallet: Exit IsTrustedTx early if wtx is already in trusted_parents (Ava Chow)
ae0876ec42 wallet: Keep track of transaction outputs owned by the wallet (Ava Chow)
0f269bc48c walletdb: Load Txs last (Ava Chow)
5cc32ee2a7 test: Test for balance update due to untracked output becoming spendable (Ava Chow)
8222341d4f wallet: MarkDirty after AddWalletDescriptor (Ava Chow)
e02f2d331c bench: Have AvailableCoins benchmark include a lot of unrelated utxos (Ava Chow)

Pull request description:

  Currently, the wallet is not actually aware about its own transaction outputs. Instead, it will iterate all of the transactions stored in `mapWallet`, and then all of the outputs of those transactions, in order to figure out what belongs to it for the purposes of coin selection and balance calculation. For balance calculation, there is caching that results in it only iterating all of the transactions, but not all of the outputs. However when the cache is dirty, everything is iterated. This is especially problematic for wallets that have a lot of transactions, or transactions that have a lot of unrelated outputs (as may occur with coinjoins or batched payments).

  This PR helps to resolve this issue by making the wallet track all of the outputs that belong to it in a new member `m_txos`. Note that this includes outputs that may have already been spent. Both balance calculation (`GetBalance`) and coin selection (`AvailableCoins`) are updated to iterate `m_txos`. This is generally faster since it ignores all of the unrelated outputs, and it is not slower as in the worst case of wallets containing only single output transactions, it's exactly the same number of outputs iterated.

  `m_txos` is memory only, and it is populated during wallet loading. When each transaction is loaded, all of its outputs are checked to see if it is `IsMine`, and if so, an entry added to `m_txos`. When new transactions are received, the same procedure is done.

  Since imports can change the `IsMine` status of a transaction (although they can only be "promoted" from watchonly to spendable), all of the import RPCs will be a bit slower as they re-iterate all transactions and all outputs to update `m_txos`.

  Each output in `m_txos` is stored in a new `WalletTXO` class. This class contains references to the parent `CWalletTx` and the `CTxOut` itself. It also caches the `IsMine` value of the txout. This should be safe as `IsMine` should not change unless there are imports. This allows us to have additional performance improvements in places that use these `WalletTXO`s as they can use the cached `IsMine` rather than repeatedly calling `IsMine` which can be expensive.

  The existing `WalletBalance` benchmark demonstrates the performance improvement that this PR makes. The existing `WalletAvailableCoins` benchmark doesn't as all of the outputs used in that benchmark belong to the test wallet. I've updated that benchmark to have a bunch of unrelated outputs in each transaction so that the difference is demonstrated.

  This is part of a larger project to have the wallet actually track and store a set of its UTXOs.

  Built on #24914 as it requires loading the txs last in order for `m_txos` to be built correctly.

  ***

  ## Benchmarks:

  Master:

  |               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
  |       34,590,013.00 |               28.91 |    0.0% |  812,669,269.00 |  148,360,642.50 |  5.478 |  18,356,853.00 |    0.2% |      0.76 | `WalletAvailableCoins`
  |            3,193.46 |          313,139.91 |    0.4% |       96,868.06 |       13,731.82 |  7.054 |      26,238.01 |    0.1% |      0.01 | `WalletBalanceClean`
  |           26,871.18 |           37,214.59 |    3.3% |      768,179.50 |      115,544.39 |  6.648 |     154,171.09 |    0.1% |      0.01 | `WalletBalanceDirty`
  |            3,177.30 |          314,732.47 |    0.2% |       96,868.06 |       13,646.20 |  7.099 |      26,238.01 |    0.1% |      0.01 | `WalletBalanceMine`
  |               10.73 |       93,186,952.53 |    0.1% |          157.00 |           46.14 |  3.403 |          36.00 |    0.0% |      0.01 | `WalletBalanceWatch`
  |      590,497,920.00 |                1.69 |    0.1% |12,761,692,005.00 |2,536,899,595.00 |  5.030 | 129,124,399.00 |    0.7% |      6.50 | `WalletCreateEncrypted`
  |      182,929,529.00 |                5.47 |    0.0% |4,199,271,397.00 |  785,477,302.00 |  5.346 |  75,363,377.00 |    1.1% |      2.01 | `WalletCreatePlain`
  |          699,337.20 |            1,429.93 |    0.7% |   18,054,294.00 |    3,005,072.20 |  6.008 |     387,756.60 |    0.3% |      0.04 | `WalletCreateTxUseOnlyPresetInputs`
  |       32,068,583.80 |               31.18 |    0.5% |  562,026,110.00 |  137,457,635.60 |  4.089 |  90,667,459.40 |    0.3% |      1.78 | `WalletCreateTxUsePresetInputsAndCoinSelection`
  |               36.62 |       27,306,578.40 |    0.5% |          951.00 |          157.05 |  6.056 |         133.00 |    0.0% |      0.01 | `WalletIsMineDescriptors`
  |               35.00 |       28,569,989.42 |    0.7% |          937.00 |          150.33 |  6.233 |         129.00 |    0.0% |      0.01 | `WalletIsMineMigratedDescriptors`
  |      203,284,889.00 |                4.92 |    0.0% |4,622,691,895.00 |  872,875,275.00 |  5.296 |  90,345,002.00 |    1.2% |      1.02 | `WalletLoadingDescriptors`
  |    1,165,766,084.00 |                0.86 |    0.0% |24,139,316,211.00 |5,005,218,705.00 |  4.823 |2,664,455,775.00 |    0.1% |      1.17 | `WalletMigration`

  PR:

  |               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
  |       33,975,750.50 |               29.43 |    0.1% |  794,719,150.50 |  145,763,550.00 |  5.452 |  16,036,630.50 |    0.2% |      0.75 | `WalletAvailableCoins`
  |            2,442.01 |          409,498.46 |    0.2% |       60,782.04 |       10,500.60 |  5.788 |       9,492.01 |    0.3% |      0.01 | `WalletBalanceClean`
  |            2,763.12 |          361,909.21 |    0.2% |       61,493.05 |       11,859.48 |  5.185 |       9,625.01 |    0.2% |      0.01 | `WalletBalanceDirty`
  |            2,347.98 |          425,898.72 |    0.3% |       60,782.04 |       10,082.73 |  6.028 |       9,492.01 |    0.2% |      0.01 | `WalletBalanceMine`
  |               11.67 |       85,654,630.36 |    0.2% |          176.00 |           50.18 |  3.508 |          40.00 |    0.0% |      0.01 | `WalletBalanceWatch`
  |      590,119,519.00 |                1.69 |    0.1% |12,754,398,258.00 |2,534,998,522.00 |  5.031 | 129,078,027.00 |    0.7% |      6.50 | `WalletCreateEncrypted`
  |      183,124,790.00 |                5.46 |    0.1% |4,199,212,926.00 |  786,323,886.00 |  5.340 |  75,354,437.00 |    1.1% |      2.02 | `WalletCreatePlain`
  |          669,643.00 |            1,493.33 |    0.1% |   17,213,904.20 |    2,877,336.40 |  5.983 |     394,292.80 |    0.3% |      0.04 | `WalletCreateTxUseOnlyPresetInputs`
  |       26,205,987.80 |               38.16 |    0.8% |  365,551,340.80 |  112,376,905.20 |  3.253 |  65,684,276.20 |    0.4% |      1.44 | `WalletCreateTxUsePresetInputsAndCoinSelection`
  |               34.75 |       28,778,846.38 |    0.1% |          937.00 |          149.41 |  6.271 |         129.00 |    0.0% |      0.01 | `WalletIsMineDescriptors`
  |               29.91 |       33,428,072.85 |    0.2% |          920.00 |          128.63 |  7.152 |         126.00 |    0.0% |      0.01 | `WalletIsMineMigratedDescriptors`
  |      202,437,985.00 |                4.94 |    0.1% |4,626,686,256.00 |  869,439,274.00 |  5.321 |  90,961,305.00 |    1.1% |      1.02 | `WalletLoadingDescriptors`
  |    1,158,394,152.00 |                0.86 |    0.0% |24,143,589,972.00 |4,971,946,380.00 |  4.856 |2,665,355,654.00 |    0.1% |      1.16 | `WalletMigration`

ACKs for top commit:
  davidgumberg:
    untested reACK 215e599
  murchandamus:
    reACK 215e5999e2
  ishaanam:
    reACK 215e5999e2
  w0xlt:
    reACK 215e5999e2

Tree-SHA512: d6b929de56f67930678db654e46f15fb71008390189c701a026b2d76af8f14a7c9769e49835ce7e2b6515d2934a77aad8de0b1a82231a2e1de5337de25db9629

test/functional/wallet_balance.py

@@ -4,15 +4,18 @@
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 """Test the wallet balance RPC methods."""
 from decimal import Decimal
+import time
 
 from test_framework.address import ADDRESS_BCRT1_UNSPENDABLE as ADDRESS_WATCHONLY
 from test_framework.blocktools import COINBASE_MATURITY
+from test_framework.descriptors import descsum_create
 from test_framework.test_framework import BitcoinTestFramework
 from test_framework.util import (
     assert_equal,
     assert_is_hash_string,
     assert_raises_rpc_error,
 )
+from test_framework.wallet_util import get_generate_key
 
 
 def create_transactions(node, address, amt, fees):
@@ -279,5 +282,30 @@ class WalletTest(BitcoinTestFramework):
         assert_equal(tx_info['lastprocessedblock']['height'], prev_height)
         assert_equal(tx_info['lastprocessedblock']['hash'], prev_hash)
 
+        self.log.info("Test that the balance is updated by an import that makes an untracked output in an existing tx \"mine\"")
+        default = self.nodes[0].get_wallet_rpc(self.default_wallet_name)
+        self.nodes[0].createwallet("importupdate")
+        wallet = self.nodes[0].get_wallet_rpc("importupdate")
+
+        import_key1 = get_generate_key()
+        import_key2 = get_generate_key()
+        wallet.importdescriptors([{"desc": descsum_create(f"wpkh({import_key1.privkey})"), "timestamp": "now"}])
+
+        amount = 15
+        default.send([{import_key1.p2wpkh_addr: amount},{import_key2.p2wpkh_addr: amount}])
+        self.generate(self.nodes[0], 1)
+        # Mock the time forward by 1 day so that "now" will exclude the block we just mined
+        self.nodes[0].setmocktime(int(time.time()) + 86400)
+        # Mine 11 blocks to move the MTP past the block we just mined
+        self.generate(self.nodes[0], 11, sync_fun=self.no_op)
+
+        balances = wallet.getbalances()
+        assert_equal(balances["mine"]["trusted"], amount)
+
+        # Don't rescan to make sure that the import updates the wallet txos
+        wallet.importdescriptors([{"desc": descsum_create(f"wpkh({import_key2.privkey})"), "timestamp": "now"}])
+        balances = wallet.getbalances()
+        assert_equal(balances["mine"]["trusted"], amount * 2)
+
 if __name__ == '__main__':
     WalletTest(__file__).main()