8098dea06944f9de8b285f44958eb98761f133ee test: Add mempool_updatefromblock.py (Hennadii Stepanov)
Pull request description:
This PR adds a new test for mempool update of transaction descendants/ancestors information (count, size) when transactions have been re-added from a disconnected block to the mempool.
It could be helpful for working on PRs like #17925, #18191.
ACKs for top commit:
ariard:
ACK 8098dea
Tree-SHA512: 7e808fa8df8d7d7a7dbdc3f79361049b49c7bce9b58fd5539b28c9636bedac747695537e500d7ed68dc8bdb80167ad3f1c01086f7551691405d2ba2e38ef1d06
66fe7b1a98c03f690dcf60d359baac124658aeae test: added test for upgradewallet RPC (Harris)
Pull request description:
This PR adds tests for the newly merged *upgradewallet* RPC.
Additionally, it expands `test_framework/util.py` by adding the function `adjust_bitcoin_conf_for_pre_17` to support nodes that don't parse configuration sections.
This test uses two older node versions, v0.15.2 and v0.16.3, to create older wallet versions to be used by `upgradewallet`.
Fixes https://github.com/bitcoin/bitcoin/issues/18767
Top commit has no ACKs.
Tree-SHA512: bb72ff1e829e2c3954386cc308842820ef0828a4fbb754202b225a8748f92d4dcc5ec77fb146bfd5484a5c2f29ce95adf9f3fb4483437088ff3ea4a8d2c442c1
50fc4df6c4e8a84bdda13ade7bed7a2131796f00 [mempool] Persist unbroadcast set to mempool.dat (Amiti Uttarwar)
297a1785360c4db662a7f3d3ade7b6b503258d39 [test] Integration tests for unbroadcast functionality (Amiti Uttarwar)
6851502472d3625416f0e7796e9f2a0379d14d49 [refactor/test] Extract P2PTxInvStore into test framework (Amiti Uttarwar)
dc1da48dc5e5526215561311c184a8cbc345ecdc [wallet] Update the rebroadcast frequency to be ~1/day. (Amiti Uttarwar)
e25e42f20a3aa39651fbc1f9fa3df1a49f1f5868 [p2p] Reattempt initial send of unbroadcast transactions (Amiti Uttarwar)
7e93eecce3bc5a1b7bb0284e06f9e2e69454f5ba [util] Add method that returns random time in milliseconds (Amiti Uttarwar)
89eeb4a3335f8e871cc3f5286af4546dff66172a [mempool] Track "unbroadcast" transactions (Amiti Uttarwar)
Pull request description:
This PR introduces mempool tracking of unbroadcast transactions and periodic reattempts at initial broadcast. This is a part of the rebroadcast project, and a standalone privacy win.
The current rebroadcast logic is terrible for privacy because 1. only the source wallet rebroadcasts transactions and 2. it does so quite frequently. In the current system, if a user submits a transaction that does not immediately get broadcast to the network (eg. they are offline), this "rebroadcast" behavior is the safety net that can actually serve as the initial broadcast. So, keeping the attempts frequent is important for initial delivery within a reasonable timespan.
This PR aims to improve # 2 by reducing the wallet rebroadcast frequency to ~1/day from ~1/15 min. It achieves this by separating the notion of initial broadcast from rebroadcasts. With these changes, the mempool tracks locally submitted transactions & periodically reattempts initial broadcast. Transactions submitted via the wallet or RPC are added to an "unbroadcast" set & are removed when a peer sends a `getdata` request, or the transaction is removed from the mempool. Every 10-15 minutes, the node reattempts an initial broadcast. This enables reducing the wallet rebroadcast frequency while ensuring the transactions will be propagated to the network.
For privacy improvements around # 1, please see #16698.
Thank you to gmaxwell for the idea of how to break out this subset of functionality (https://github.com/bitcoin/bitcoin/pull/16698#issuecomment-571399346)
ACKs for top commit:
fjahr:
Code review ACK 50fc4df6c4e8a84bdda13ade7bed7a2131796f00
MarcoFalke:
ACK 50fc4df6c4e8a84bdda13ade7bed7a2131796f00, I think this is ready for merge now 👻
amitiuttarwar:
The current tip `50fc4df` currently has 6 ACKs on it, so I've opened #18807 to address the last bits.
jnewbery:
utACK 50fc4df6c4e8a84bdda13ade7bed7a2131796f00.
ariard:
Code Review ACK 50fc4df (minor points no need to invalid other ACKs)
robot-visions:
ACK 50fc4df6c4e8a84bdda13ade7bed7a2131796f00
sipa:
utACK 50fc4df6c4e8a84bdda13ade7bed7a2131796f00
naumenkogs:
utACK 50fc4df
Tree-SHA512: 2dd935d645d5e209f8abf87bfaa3ef0e4492705ce7e89ea64279cb27ffd37f4727fa94ad62d41be331177332f8edbebf3c7f4972f8cda10dd951b80a28ab3c0f
7b2b06dfe3061b5ab4a283245930e2f7773eb3ef tests: Add missing sync_all to wallet_importdescriptors.py (Andrew Chow)
Pull request description:
node1 will sometimes do sendtoaddress before it has received a funding transaction which will cause the test to fail. sync_all to ensure it gets the transaction first.
Fixes#18800
ACKs for top commit:
instagibbs:
ACK 7b2b06d The wallet endpoint right after is indeed node 1.
Tree-SHA512: b610a771d062b5f955cd70b34337577a1ab8dacbf4be20aa74e1e8234495b0be9faff138eb1713f29decb5574446e0583e221bc2c9a6eea13611b422ea3a296a
fa301fec966b77c54d02ac54ae7d726629adbfd5 test: Fix wallet_bumpfee intermittent error (MarcoFalke)
Pull request description:
Remove incorrect and undocumented `connect_nodes(self.nodes[0], 1)`.
Issue is that transactions are re-relayed (going full circle) between the two nodes, that have two connections between each other.
https://travis-ci.org/github/bitcoin/bitcoin/jobs/679201559#L6992
Also fix some pep8 while touching the file
This bug has been introduced by accident in c1dde3a949b36ce9c2155777b3fa1372e7ed97d8
ACKs for top commit:
achow101:
ACK fa301fec966b77c54d02ac54ae7d726629adbfd5
Tree-SHA512: a6565ca30dbe44b02e3f58f159d2515c2ea4a74030118fafc1a3391ce980a4b6d4505dcf51315fda24843f72550a7dea7407b877b3b796883dd73d3b6f009e6f
node1 will sometimes do sendtoaddress before it has received a funding
transaction which will cause the test to fail. sync_all to ensure it
gets the transaction first.
223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow)
869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow)
cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow)
886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow)
3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow)
388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow)
1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow)
f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen)
ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow)
1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow)
82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow)
b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow)
8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow)
72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow)
84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow)
bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow)
d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow)
f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow)
586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow)
f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow)
a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow)
bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow)
58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow)
e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow)
46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow)
4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow)
d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow)
953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow)
2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow)
46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow)
ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow)
741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow)
2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow)
db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow)
78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow)
834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow)
d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow)
3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow)
6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow)
aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow)
96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow)
6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow)
06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow)
Pull request description:
Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`.
Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each.
Descriptors can also be imported with a new `importdescriptors` RPC.
Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things.
A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`).
ACKs for top commit:
Sjors:
utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed)
jonatack:
Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82.
fjahr:
re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82
instagibbs:
light re-ACK 223588b
meshcollider:
Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82
Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986
Test the test_framework, but don't use test_framework objects or functions to test itself
Use python unittest library and put test_framework's unit tests inside their respective files
Add the filename to TEST_FRAMEWORK_MODULES in test_runner
Aggregate all test_framework tests into one TestSuite to run before the functional tests in test_runner
Delete framework_test_script, move test_bn2vch to script.py and add to TEST_FRAMEWORK_MODULES in test_runner
fa8b9b5d1f48ad95eecf47ebbd7bf374777fc621 test: Fix intermittent failure in wallet_importmulti (MarcoFalke)
Pull request description:
The wallet is async, so after generating a block, we must call `syncwithvalidationinterfacequeue`. Otherwise the timestamp will be of the previous block.
https://travis-ci.org/github/bitcoin/bitcoin/jobs/677685073#L2648
ACKs for top commit:
promag:
ACK fa8b9b5d1f48ad95eecf47ebbd7bf374777fc621.
Tree-SHA512: c21f9912aabbe22019d4ac9d0da06d6e46ef7f2a84d2781110e04c9836eb0ecf90a22cf2bae7f608be611670d17b20600135d1c5e5404aa1e762839816285fb4
as defined in PEP 3135:
"The new syntax:
super()
is equivalent to:
super(__class__, <firstarg>)
where __class__ is the class that the method was defined in, and <firstarg> is
the first parameter of the method (normally self for instance methods, and cls
for class methods)."
2495110012fc0cb7142a4536791dd79da5cc3e44 test: add coverage for -rpcwallet cli option (Jon Atack)
Pull request description:
The bitcoin-cli `-rpcwallet=` option is an essential RPC/CLI option when more than one wallet is loaded (see `bitcoin-cli -help | grep -A5 rpcwallet` or `src/bitcoin-cli.cpp::L61`) and it currently has no test coverage.
It is not only used by users, but also by the test framework and ~10 test files via `get_wallet_rpc()`.
This PR adds coverage, while simultaneously improving the `-getinfo` coverage when multiple wallets are loaded. This is similar to the test coverage that would be added in #18594.
ACKs for top commit:
robot-visions:
ACK 2495110012fc0cb7142a4536791dd79da5cc3e44
Tree-SHA512: caaa8b99fb8fa481ab2c6b2a287ed29720bb4553c3f66657462c44fa2990acaaf36cabeaaf81408678e5fdce4e105d729dd94b5ed8588dd1a6f2cb03fc25acf3
c4027e735072c3de4b4ffb20eecd7187ff36bad7 refactor: test: use wait_for_getdata() in p2p_compactblocks.py (Sebastian Falbesoner)
Pull request description:
The method `wait_for_getdata()` was recently changed to be more precise by waiting for a specified list of hashes, instead of only matching _any_ `getdata` message (see Issue #18614 and PR #18690). This PR replaces the remaining occurences of manual inspection of `last_messages` with this call.
ACKs for top commit:
robot-visions:
ACK c4027e735072c3de4b4ffb20eecd7187ff36bad7
Tree-SHA512: e10b346742f235b6ee2ef1f32f7fd74406c1a277389f020fb9913a93e94cc9530e1e9414872b83c9d2ae652ebce2b09b2c8c8372260c1afb4e0e54fbf7a935b0
fa262712ca0981cb0ee68cd3dd99a214a20dcbf1 test: Check submitblock return values (MarcoFalke)
Pull request description:
Add `assert_equal` in some tests to check the `submitblock` return value
ACKs for top commit:
robot-visions:
ACK fa262712ca0981cb0ee68cd3dd99a214a20dcbf1
Tree-SHA512: 25d9effe82a4f6852184b9ac848f96336cc2cafb0bb07edb2792f00cd363f0759575bc9c164dd62f64425d3754028b4acd0675600c07d51277aa80bf66c6f960
8f5dc8800aeb524eee2fa2451cd22883b7b2bfec test: display command line options passed to send_cli() in debug log (Jon Atack)
Pull request description:
as per https://github.com/bitcoin/bitcoin/pull/18691#discussion_r411382589, and revert two cli calls changed in #18691 from rpc commands back to command line options (these were the only occurrences).
ACKs for top commit:
MarcoFalke:
ACK 8f5dc8800aeb524eee2fa2451cd22883b7b2bfec
Tree-SHA512: fcb3eca00aa4099066028c90d5e50a02e074366e09a17f5f5b937d9f7562dd054ff65681aa0ad4c94f6de1e98b1e2b9ac4cd084ddc297010253989a80483b1b9
fae98668d150bae7a68167749ae134894cb1140e test: Fix intermittent error in mempool_reorg (MarcoFalke)
Pull request description:
Example: https://travis-ci.org/github/bitcoin/bitcoin/jobs/677689899#L4717
Also speed up tx relay and fix two pep8 errors while touching the file anyway.
ACKs for top commit:
vasild:
utACK fae9866
Tree-SHA512: 23a7894e71ad0e1a59c74c73643708fca21b505fa4e980038d554294063fd63c396669eefb233ffdffb0083968e51b702c643cb449df8f656dd8345a20f33907
Check that...
- mempool tracks & reattempts delivery of a transaction where a GETDATA hasn't
been requested by a peer yet.
- transaction delivery is not attempted again after GETDATA is received.
- transaction is removed from the unbroadcast set when its removed from the
mempool.
Since the mempool unbroadcast mechanism handles the reattempts for initial
broadcast, the wallet rebroadcast attempts can be much less frequent
(previously ~1/30 min)
Adds a --descriptors option globally to the test framework. This will
make the test create and use descriptor wallets. However some tests may
not work with this.
Some tests are modified to work with --descriptors and run with that
option in test_runer:
* wallet_basic.py
* wallet_encryption.py
* wallet_keypool.py
* wallet_keypool_topup.py
* wallet_labels.py
* wallet_avoidreuse.py
RPCOverloadWrapper overloads some deprecated or disabled RPCs with
an implementation using other RPCs to avoid having a ton of code churn
around replacing those RPCs.
9f5608c2893f89cd56c7c548b748996199e0da1d test: check for matching object hashes in wait_for_getdata (Danny Lee)
Pull request description:
Previously, `wait_for_getdata` only looked for the presence of a recent `"getdata"` message. Additionally checking the object hashes inside the message should make tests involving `wait_for_getdata` more robust.
`p2p_sendheaders.py` already overrides `wait_for_getdata` do this check; we can use the same approach consistently across all tests that call `wait_for_getdata`.
This PR is progress towards #18614 , but closing that issue would also involve some additional changes to `wait_for_getheaders`.
ACKs for top commit:
theStack:
ACK 9f5608c2893f89cd56c7c548b748996199e0da1d 🍻
Tree-SHA512: 8e7f95881c19631db014d4bb2399fea0d14686a32542f6ca3b60809744b0d684eac4e4c107c87143991f3cd0c2d4ab09d0c17486239768a9b40bee25f2e4d54a
3c21db7b78fb0a27a0ea61b64eede8af0972791f [test] add 8 error messages to feature_segwit and change version to type (Gloria Zhao)
Pull request description:
Followup to [this](https://github.com/bitcoin/bitcoin/pull/15169/files#r303673472) comment on functional test feature_segwit.py verifying that unsigned witness transactions are invalid.
(1) Changes 8 error messages from "mandatory-script-verify-flag" to "non-mandatory-script-verify-flag" and with more specific error messages.
(2) Edits comments that incorrectly describe the test, namely that the `v` variable corresponds to using P2WSH versus P2WPKH, not witness versions.
ACKs for top commit:
MarcoFalke:
ACK 3c21db7b78 🍾
Tree-SHA512: 3734ea3762667636c4fb20f5285634ab94d6b3527b7390fcc5e41b4582829dfe0099beabeaed42098613d168ede3385a6ffcd73989d1fa9dbd18004f5e9cf083
P2WPKH witness program without signature -> throws "hash mismatch" error
P2WSH witness program without signature -> throws "empty witness" error
same errors for P2SH_P2WPKH and P2SH_P2WSH respectively when passed redeemScript but no signature
P2SH_P2WPKH and P2SH_P2WSH with no signature fail with "Operation not valid with current stack size" when not signed due to missing input
change VER to TYPE and constants WIT_V0 to P2WPKH=0 and WIT_V1 to P2WSH=1
fac2fc4dd8a28b99e17c57e4ab6580a3231f1d0a test: Increase debugging to hunt down mempool_reorg intermittent failure (MarcoFalke)
Pull request description:
Top commit has no ACKs.
Tree-SHA512: 4094b44afaa623e58b69f8d0332e60f0150b9ae2fd8bb265210d85546d887672ab8a3435cd9b086be14f69ab5b17e0f9fae06bd8aec1e7947ca766dd72b577c4
a9ecbdfcaa15499644d16e9c8ad2c63dfc45b37b test: add more inactive filter tests to p2p_filter.py (Sebastian Falbesoner)
5eae034996b340c19cebab9efb6c89d20fe051ef net: limit BIP37 filter lifespan (active between 'filterload' and 'filterclear') (Sebastian Falbesoner)
Pull request description:
This PR fixes https://github.com/bitcoin/bitcoin/issues/18483. On the master branch, there is currently _always_ a BIP37 filter set for every peer: if not a specific filter is set through a `filterload` message, a default match-everything filter is instanciated and pointed to via the `CBloomFilter` default constructor; that happens both initially, when the containing structure `TxRelay` is constructed:
c0b389b335/src/net.h (L812)
and after a loaded filter is removed again through a `filterclear` message:
c0b389b335/src/net_processing.cpp (L3201)
The behaviour was introduced by commit 37c6389c5a (an intentional covert fix for [CVE-2013-5700](https://github.com/bitcoin/bitcoin/pull/18515), according to gmaxwell).
This default match-everything filter leads to some unintended side-effects:
1. `getdata` request for filtered blocks (i.e. type `MSG_FILTERED_BLOCK`) are always responded to with `merkleblock`s, even if no filter was set by the peer, see issue #18483 (strictly speaking, this is a violation of BIP37) c0b389b335/src/net_processing.cpp (L1504-L1507)
2. if a peer sends a `filteradd` message without having loaded a filter via `filterload` before, the intended increasing of the banscore never happens (triggered if `bad` is set to true, a few lines below) c0b389b335/src/net_processing.cpp (L3182-L3186)
This PR basically activates the `else`-branch code paths for all checks of `pfilter` again (on the master branch, they are dead code) by limiting the pointer's lifespan: instead of always having a filter set, the `pfilter` is only pointing to a `CBloomFilter`-instance after receiving a `filterload` message and the instance is destroyed again (and the pointer nullified) after receiving a `filterclear` message.
Here is a before/after comparison in behaviour:
| code part / scenario | master branch | PR branch |
| --------------------------------------------- | ---------------------------------- | ---------------------------------------------------- |
| `getdata` processing for `MSG_FILTERED_BLOCK` | always responds with `merkleblock` | only responds if filter was set via `filterload` |
| `filteradd` processing, no filter was loaded | nothing | peer's banscore increases by 100 (i.e. disconnect) |
On the other code parts where `pfilter` is checked there is no change in the logic behaviour (except that `CBloomFilter::IsRelevantAndUpdate()` is unnecessarily called and immediately returned in the master branch).
Note that the default constructor of `CBloomFilter` is only used for deserializing the received `filterload` message and nowhere else. The PR also contains a functional test checking that sending `getdata` for filtered blocks is ignored by the node if no bloom filter is set.
ACKs for top commit:
MarcoFalke:
re-ACK a9ecbdfcaa, only change is in test code 🕙
Tree-SHA512: 1a656a6d74ccaf628e7fdca063ba63fbab2089e0b6d0a11be9bbd387c2ee6d3230706ff8ffc1a55711481df3d4547137dd7c9d9184d89eaa43ade4927792d0b6
by calling wait_for_cookie_credentials() to ensure the cookie file is written
and auth credentials available for testing the CLI -rpcwait option before the
RPC connection is up.
9df32e820d83aa74e2f175d8d63b5666b8b4ef0e scripted-diff: test: replace command with msgtype (Sebastian Falbesoner)
Pull request description:
This is a follow-up PR to https://github.com/bitcoin/bitcoin/pull/18533, which changed the naming of `strCommand` to `msg_type` in the network processing code. The same approach is done here for the function test framework, to get rid of the wrong "command" terminology for network mesage types. (Commands are usually used in the CLI or RPC context, so using the same name in the network message context would only be confusing.)
The commit was created through the following steps:
1. search for all occurences of the string "command" within the folder `test/functional`
```git grep -i command test/functional > command_finds```
2. manually sort out all false-positives, i.e. occurences of "command" which describe commands in the correct sense (mostly CLI or RPC related, also some with Socks5)
3. put the remaining occurences into a scripted-diff (a quite simple one, actually) that renames "command" to "msgtype" in the concerned files.
The name `msgtype` was intentionally chosen without the underscore `_` as classes beginning with `msg_` define concrete types of messages.
ACKs for top commit:
MarcoFalke:
ACK 9df32e820d83aa74e2f175d8d63b5666b8b4ef0e . Makes sense that tests use the same naming as Bitcoin Core. See `NetMsgType` here: https://doxygen.bitcoincore.org/namespace_net_msg_type.html
Tree-SHA512: cd0ee08a382910b7f10ce583acdaf4f8a39f9ba4a22434a914415727eedd98bac538de9bf6633574d5eb86f62558bc8dcb638a3289d99b04f8481f34e7a9a0c7
0d32d661481f099af572e7a08a50e17bcc165c44 Remove -upgradewallet startup option (Andrew Chow)
92263cce5b6c6b66296dadda5f29724611db0160 Add upgradewallet RPC (Andrew Chow)
1e48796c99b63aa8fa8451ce7b0c20759ea43500 Make UpgradeWallet a member function of CWallet (Andrew Chow)
c988f27937bc79c90f4eed48552c72f1b66dc044 Have UpgradeWallet take the version to upgrade to and an error message out parameter (Andrew Chow)
183323712398e26ddcf3a9dc048aaa9900a91f5a Only run UpgradeWallet if the wallet needs to be upgraded (Andrew Chow)
9c16b1735f8e530ce68d678e9ca0eceb2ceb3520 Move wallet upgrading to its own function (Andrew Chow)
Pull request description:
`-upgradewallet` is largely incompatible with many recent wallet features and versions. For example, it was disabled if multiple wallets were used and would not work with encrypted wallets that were being upgraded to HD.
This PR does away with the old method of upgrading upon startup and instead allows users to upgrade their wallets via an `upgradewallet` RPC. This does largely the same thing as the old `-upgradewallet` option but because the wallet is loaded, it can be unlocked to upgrade to HD. Furthermore it is compatible with multiwallet as it works on the individual wallet that is specified by the RPC.
ACKs for top commit:
meshcollider:
Code review ACK 0d32d661481f099af572e7a08a50e17bcc165c44
darosior:
ACK 0d32d661481f099af572e7a08a50e17bcc165c44
MarcoFalke:
ACK 0d32d661481f099af572e7a08a50e17bcc165c44 🚵
Tree-SHA512: b425bf6f5d605e26506889d63c780895482f07cbc086193218e031e8504d3072d41e90d65cd41bcc98ee4c1eb048954bc5d4ac85435f7394892373aac89a3b0a
fa03713e133e3017112fdd5c278e0c8643054578 test: Properly raise FailedToStartError when rpc shutdown before warmup finished (take 2) (MarcoFalke)
Pull request description:
actually (?) fix#18561
See most recent traceback https://travis-ci.org/github/bitcoin/bitcoin/jobs/674668692#L7062
I believe the reason the error is still there is that ConnectionResetError is derived from OSError:
ConnectionResetError(ConnectionError(OSError))
And IOError is an alias for OSError since python 3.3, see https://docs.python.org/3/library/exceptions.html#IOError
So fix that by renaming IOError to the alias OSError and move the less specific catch clause down a few lines.
ACKs for top commit:
jonatack:
ACK fa03713e133e3017112fdd5c278e0c8643054578
Tree-SHA512: 6e5b214ed9101bf8ebe7472dcc1f9e9d128e2575c93ec00c8d0774ae1a9b52a8c2a653a45a0eab8d881570b08dd5ffeddf5aca88a10438c366e1f633253cb0b5
