Commit Graph

141 Commits

Author SHA1 Message Date
MarcoFalke
fa3f100010 test: Use msg_generic in p2p_ping.py 2025-06-30 14:50:55 +02:00
dergoegge
7dc43ea503 test: Add msgtype to msg_generic slots 2025-06-30 11:13:11 +01:00
Ava Chow
b3bb4031ab Merge bitcoin/bitcoin#32540: rest: fetch spent transaction outputs by blockhash
c48846ec41 doc: add release notes for #32540 (Roman Zeyde)
d4e212e8a6 rest: fetch spent transaction outputs by blockhash (Roman Zeyde)

Pull request description:

  Today, it is possible to fetch a block's spent prevouts in order to build an external index by using the `/rest/block/BLOCKHASH.json` endpoint. However, its performance is low due to JSON serialization overhead.

  We can significantly optimize it by adding a new [REST API](https://github.com/bitcoin/bitcoin/blob/master/doc/REST-interface.md) endpoint, using a binary response format (returning a collection of spent txout lists, one per each block transaction):

  ```
  $ BLOCKHASH=00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054

  $ ab -k -c 1 -n 100 http://localhost:8332/rest/block/$BLOCKHASH.json
  Document Length:        13278152 bytes
  Requests per second:    3.53 [#/sec] (mean)
  Time per request:       283.569 [ms] (mean)

  $ ab -k -c 1 -n 10000 http://localhost:8332/rest/spenttxouts/$BLOCKHASH.bin
  Document Length:        195591 bytes
  Requests per second:    254.47 [#/sec] (mean)
  Time per request:       3.930 [ms] (mean)
  ```

  Currently, this PR is being used and tested by Bindex[^1].

  This PR would allow to improve the performance of external indexers such as electrs[^2], ElectrumX[^3], Fulcrum[^4] and Blockbook[^5].

  [^1]: https://github.com/romanz/bindex-rs
  [^2]: https://github.com/romanz/electrs (also [blockstream.info](https://github.com/Blockstream/electrs) and [mempool.space](https://github.com/mempool/electrs) forks)
  [^3]: https://github.com/spesmilo/electrumx
  [^4]: https://github.com/cculianu/Fulcrum
  [^5]: https://github.com/trezor/blockbook

ACKs for top commit:
  maflcko:
    re-ACK c48846ec41 📶
  TheCharlatan:
    Re-ACK c48846ec41
  achow101:
    ACK c48846ec41

Tree-SHA512: cf423541be90d6615289760494ae849b7239b69427036db6cc528ac81df10900f514471d81a460125522c5ffa31e9747ddfca187a1f93151e4ae77fe773c6b7b
2025-06-27 14:44:41 -07:00
Sebastian Falbesoner
472f3770ae scripted-diff: test: rename CTransaction .getwtxid() -> wtxid_hex for consistency
-BEGIN VERIFY SCRIPT-

sed -i "s|def getwtxid|@property\n    def wtxid_hex|g" ./test/functional/test_framework/messages.py
sed -i "s|getwtxid()|wtxid_hex|g" $(git grep -l getwtxid)

-END VERIFY SCRIPT-
2025-06-11 00:52:25 +02:00
Sebastian Falbesoner
81af4334e8 test: rename CTransaction .sha256 -> .txid_int for consistency
Note that we unfortunately can't use a scripted diff here, as the same
property name is also used for `CBlockHeader`/`CBlock` instances.
2025-06-11 00:52:25 +02:00
Sebastian Falbesoner
ce83924237 test: rename CTransaction .rehash()/.hash -> .txid_hex for consistency
Note that we unfortunately can't use a scripted diff here, as the same
property and method name is also used for `CBlockHeader`/`CBlock` instances.
2025-06-11 00:49:10 +02:00
Sebastian Falbesoner
e9cdaefb0a test: introduce and use CTransaction .wtxid_int property
This commits removes the `.calc_sha256` method from the CTransaction
and introduces a property `.wtxid_int` property as replacement.
2025-06-09 17:28:28 +02:00
Sebastian Falbesoner
9b3dce24a3 test: remove bare CTransaction .rehash()/.calc_sha256() calls
Since the previous commit, CTransaction object calls to the
methods `.rehash()` and `.calc_sha256()` are effectively no-ops
if the returned value is not used, so we can just remove them.
2025-06-09 17:28:24 +02:00
Sebastian Falbesoner
a2724e3ea3 test: remove txid caching in CTransaction class
Rather than txids (represented by the fields `.sha256` and `.hash`)
being stateful, simply compute them on-the-fly. This ensures that
the correct values are always returned and takes the burden of
rehashing from test writers, making the code shorter overall.
In a first step, the fields are kept at the same name with @property
functions as drop-in replacements, for a minimal diff. In later commits,
the names are changed to be more descriptive and indicating the return
type of the txid.
2025-06-09 16:55:04 +02:00
Roman Zeyde
d4e212e8a6 rest: fetch spent transaction outputs by blockhash
Today, it is possible to fetch a block's spent prevouts in order to
build an external index by using the `/rest/block/HASH.json` endpoint.
However, its performance is low due to JSON serialization overhead.

We can significantly optimize it by adding a new REST endpoint, using
a binary response format:

```
$ BLOCKHASH=00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054

$ ab -k -c 1 -n 100 http://localhost:8332/rest/block/$BLOCKHASH.json
Document Length:        13278152 bytes
Requests per second:    3.53 [#/sec] (mean)
Time per request:       283.569 [ms] (mean)

$ ab -k -c 1 -n 10000 http://localhost:8332/rest/spentoutputs/$BLOCKHASH.bin
Document Length:        195591 bytes
Requests per second:    254.47 [#/sec] (mean)
Time per request:       3.930 [ms] (mean)
```

Currently, this PR is being used and tested by Bindex:

 * https://github.com/romanz/bindex-rs

This PR would allow to improve the performance of external indexers
such as electrs, ElectrumX, Fulcrum and Blockbook:

 * https://github.com/romanz/electrs (also https://github.com/Blockstream/electrs and https://github.com/mempool/electrs)
 * https://github.com/spesmilo/electrumx
 * https://github.com/cculianu/Fulcrum
 * https://github.com/trezor/blockbook
2025-06-07 20:17:50 +03:00
Greg Sanders
9f36962b07 policy: uncap datacarrier by default
Datacarrier output script sizes and output counts are now
uncapped by default.

To avoid introducing another startup argument, we modify the
OP_RETURN accounting to "budget" the spk sizes.

If a user has set a custom default, this results in that
budget being spent over the sum of all OP_RETURN outputs'
scripts in the transaction, no longer capping the number
of OP_RETURN outputs themselves. This should allow a
superset of current behavior while respecting the passed
argument in terms of total arbitrary data storage.

Co-authored-by: Anthony Towns <aj@erisian.com.au>
2025-05-30 10:12:38 -04:00
Ava Chow
06439a14c8 Merge bitcoin/bitcoin#31953: rpc: Allow fullrbf fee bump in (psbt)bumpfee
fa86190e6e rpc: Allow fullrbf fee bump (MarcoFalke)

Pull request description:

  The RPCs (psbt)bumpfee, and the GUI, reject fee bumps when BIP 125 signalling is absent in the transaction even when the mempool and other RPCs allow them. Fix the confusion by allowing the fee bump.

  This is done after fullrbf is always on (https://github.com/bitcoin/bitcoin/pull/30592)

ACKs for top commit:
  1440000bytes:
    reACK fa86190e6e
  achow101:
    ACK fa86190e6e
  w0xlt:
    ACK fa86190e6e
  rkrux:
    reACK fa86190e6e
  glozow:
    ACK fa86190e6e

Tree-SHA512: b2ffe8dcadbe71e9be767a16cf8aa0bf383c2de7aa1aee9438d125f444e24f3f7e4f02ddb28981bd3b8b645b6a24a407b4ad6bb0b21946ae637e78f6386e05bf
2025-04-21 13:25:52 -07:00
MarcoFalke
fa86190e6e rpc: Allow fullrbf fee bump
Also, fix the incorrect documention of the 'replaceable' RPC argument
with respect to sequence number handling. The docs were incorrect
before, so the fix could be extracted, but it seems fine to include here
as well.
2025-04-17 13:12:26 +02:00
Sjors Provoost
d424bd5941 test: drop unused testnet3 magic bytes 2025-03-26 12:29:04 +01:00
Sjors Provoost
8cfc09fafe test: cover testnet4 magic in assumeutxo.py
Replace testnet3 and use MAGIC_BYTES constants.
2025-03-26 12:29:04 +01:00
Sebastian Falbesoner
a7911ed101 test: introduce VARINT (de)serialization routines 2025-03-04 10:08:12 -05:00
ismaelsadeeq
3eaa0a3b66 miner: init: add -blockreservedweight startup option
- Prevent setting the value of `-blockreservedweight` below
  a safety value of 2000.
2025-02-04 11:53:11 -05:00
ismaelsadeeq
2c7d90a6d6 miner: bugfix: fix duplicate weight reservation in block assembler
- This commit renamed coinbase_max_additional_weight to block_reserved_weight.

- Also clarify that the reservation is for block header, transaction count
  and coinbase transaction.
2025-02-04 11:53:03 -05:00
MarcoFalke
fae76393bd test: Avoid F541 (f-string without any placeholders) 2024-12-05 08:39:09 +01:00
Martin Zumsande
ec5e294e4b test: fix constructor of msg_tx
In python, if the default value is a mutable object (here: a class)
its shared over all instances, so that one instance being changed
would affect others to be changed as well.
This was likely the source of various intermittent bugs in the
functional tests.
2024-07-30 17:49:07 -04:00
Ava Chow
429ec1aaaa refactor: Rename CTransaction::nVersion to version
In order to ensure that the change of nVersion to a uint32_t in the
previous commit has no effect, rename nVersion to version in this commit
so that reviewers can easily spot if a spot was missed or if there is a
check somewhere whose semantics have changed.
2024-06-07 13:55:23 -04:00
Ava Chow
27e70f1f5b consensus: Store transaction nVersion as uint32_t
Given that the use of a transaction's nVersion is always as an unsigned
int, it doesn't make sense to store it as signed and then cast it to
unsigned.
2024-06-07 12:40:21 -04:00
Sebastian Falbesoner
c4a67d396d test: p2p: check disconnect due to lack of desirable service flags 2024-03-11 15:23:09 +01:00
MarcoFalke
facafa90f7 test: Fix CPartialMerkleTree.nTransactions signedness 2024-02-01 13:18:40 +01:00
glozow
78c06a38c4 Merge bitcoin/bitcoin#29067: test: Treat msg_version.relay as unsigned, Remove struct packing in messages.py
55556a64a8 test: Remove struct import from messages.py (MarcoFalke)
fa3fa86dda scripted-diff: test: Use int from_bytes and to_bytes over struct packing (MarcoFalke)
fafc0d68ee test: Use int from_bytes and to_bytes over struct packing (MarcoFalke)
fa3886b7c6 test: Treat msg_version.relay as unsigned (MarcoFalke)

Pull request description:

  `struct` has many issues in messages.py:

  * For unpacking, it requires to specify the length a second time, even when it is already clear from the `f.read(num_bytes)` context.
  * For unpacking, it is designed to support a long format string and returning a tuple of many values. However, except for 3 instances in `messages.py`, usually only a single value is unpacked and all those cases require an `[0]` access.
  * For packing and unpacking of a single value, the format string consists of characters that may be confusing and may need to be looked up in the documentation, as opposed to using easy to understand self-documenting code.

  I presume the above issues lead to accidentally treat `msg_version.relay` as a "signed bool", which is fine, but confusing.

  Fix all issues by using the built-in `int` helpers `to_bytes` and `from_bytes` via a scripted diff.

  Review notes:

  * `struct.unpack` throws an error if the number of bytes passed is incorrect. `int.from_bytes` doesn't know about "missing" bytes and treats an empty byte array as `int(0)`. "Extraneous" bytes should never happen, because all `read` calls are limited in this file. If it is important to keep this error behavior, a helper `int_from_stream(stream, num_bytes, bytes, byteorder, *, **kwargs)` can be added, which checks the number of bytes read from the stream.
  * For `struct.pack` and `int.to_bytes` the error behavior is the same, although the error messages are not identical.

ACKs for top commit:
  stickies-v:
    ACK 55556a64a8
  theStack:
    re-ACK 55556a64a8

Tree-SHA512: 1cef8cdfd763fb424ed4b8be850a834b83fd0ef47fbea626a29784eb4f4832d44e42c4fe05b298b6070a933ef278b0222289a9955a97c86707e091e20bbb247a
2024-01-30 12:00:47 +00:00
MarcoFalke
55556a64a8 test: Remove struct import from messages.py 2024-01-29 11:12:15 +01:00
MarcoFalke
fa3fa86dda scripted-diff: test: Use int from_bytes and to_bytes over struct packing
-BEGIN VERIFY SCRIPT-
 sed -i --regexp-extended 's!struct.unpack\("(|<|>)B", (.*)\)\[0\]!int.from_bytes(\2, "little")!g'               ./test/functional/test_framework/messages.py
 sed -i --regexp-extended 's!struct.unpack\("<(H|I|Q)", (.*)\)\[0\]!int.from_bytes(\2, "little")!g'              ./test/functional/test_framework/messages.py
 sed -i --regexp-extended 's!struct.unpack\("<(h|i|q)", (.*)\)\[0\]!int.from_bytes(\2, "little", signed=True)!g' ./test/functional/test_framework/messages.py
 sed -i --regexp-extended 's!struct.unpack\(">(H|I|Q)", (.*)\)\[0\]!int.from_bytes(\2, "big")!g'                 ./test/functional/test_framework/messages.py

 sed -i --regexp-extended 's!struct.pack\("<?B", (.*)\)!\1.to_bytes(1, "little")!g'             ./test/functional/test_framework/messages.py
 sed -i --regexp-extended 's!struct.pack\("<I", (.*)\)!\1.to_bytes(4, "little")!g'              ./test/functional/test_framework/messages.py
 sed -i --regexp-extended 's!struct.pack\("<i", (.*)\)!\1.to_bytes(4, "little", signed=True)!g' ./test/functional/test_framework/messages.py
 sed -i --regexp-extended 's!struct.pack\("<Q", (.*)\)!\1.to_bytes(8, "little")!g'              ./test/functional/test_framework/messages.py
 sed -i --regexp-extended 's!struct.pack\("<q", (.*)\)!\1.to_bytes(8, "little", signed=True)!g' ./test/functional/test_framework/messages.py
 sed -i --regexp-extended 's!struct.pack\(">H", (.*)\)!\1.to_bytes(2, "big")!g'                 ./test/functional/test_framework/messages.py
-END VERIFY SCRIPT-
2024-01-29 11:11:54 +01:00
MarcoFalke
fafc0d68ee test: Use int from_bytes and to_bytes over struct packing
This is done in prepration for the scripted diff, which can not deal
with the 0 literal int.
2024-01-29 11:10:59 +01:00
MarcoFalke
fa3886b7c6 test: Treat msg_version.relay as unsigned
The C++ code treats bool as uint8_t, so the python tests should as well.

This also allows to simplify the code, because converting an empty byte
array to int gives int(0).

>>> int.from_bytes(b'')
0
2024-01-29 11:09:35 +01:00
stratospher
8d6c848a48 [test] Move MAGIC_BYTES to messages.py
This avoids circular dependency happening when importing MAGIC_BYTES.
Before,
	p2p.py <--import for EncryptedP2PState-- v2_p2p.py
	  |					    ^
	  |				            |
	  └---------import for MAGIC_BYTES----------┘
Now, MAGIC_BYTES are kept separately in messages.py

Co-authored-by: Martin Zumsande <mzumsande@gmail.com>
2024-01-23 22:04:55 +05:30
Andrew Chow
962ea5c525 Merge bitcoin/bitcoin#28374: test: python cryptography required for BIP 324 functional tests
c534c08710 [test/crypto] Add FSChaCha20Poly1305 AEAD python implementation (stratospher)
c2a458f1c2 [test/crypto] Add FSChaCha20 python implementation (stratospher)
c4ea5f6288 [test/crypto] Add RFC 8439's ChaCha20Poly1305 AEAD (stratospher)
9fc6e0355e [test/crypto] Add Poly1305 python implementation (stratospher)
fec2ca6c9a [test/crypto] Use chacha20_block function in `data_to_num3072` (stratospher)
0cde60da3a [test/crypto] Add ChaCha20 python implementation (stratospher)
69d3f50ab6 [test/crypto] Add HMAC-based Key Derivation Function (HKDF) (stratospher)
08a4a56cbc [test] Move test framework crypto functions to crypto/ (stratospher)

Pull request description:

  split off from #24748 to keep commits related to cryptography and functional test framework changes separate.

  This PR adds python implementation and unit tests for HKDF, ChaCha20, Poly1305, ChaCha20Poly1305 AEAD, FSChaCha20 and FSChaCha20Poly1305 AEAD.

  They're based on cc177ab7bc/bip-0324/reference.py for easy review.

ACKs for top commit:
  sipa:
    utACK c534c08710
  achow101:
    ACK c534c08710
  theStack:
    re-ACK c534c08710

Tree-SHA512: 08a0a422d2937eadcf0edfede37e535e6bc4c2e4b192441bbf9bc26dd3f03fa3388effd22f0527c55af173933d0b50e5b2b3d36f2b62d0aca3098728ef06970e
2023-11-07 16:48:57 -05:00
Pieter Wuille
abf343b320 net: advertise NODE_P2P_V2 if CLI arg -v2transport is on
Co-authored-by: Dhruv Mehta <856960+dhruv@users.noreply.github.com>
2023-10-02 18:09:53 -04:00
stratospher
08a4a56cbc [test] Move test framework crypto functions to crypto/ 2023-09-10 23:16:39 +05:30
brunoerg
80f64a3d40 test: add support for all networks in CAddress in messages.py
Also removes TorV2 from messages.py
See https://github.com/bitcoin/bitcoin/pull/22050

Co-authored-by: Matthew Zipkin <pinheadmz@gmail.com>
2023-07-10 10:07:40 -04:00
Sebastian Falbesoner
96bf0bca4a test: simplify uint256 (de)serialization routines
These routines look fancy, but do nothing more than converting between
byte objects of length 32 to/from integers in little endian byte order
and can be replaced by simple one-liners, using the int.{from,to}_bytes
methods (available since Python 3.2).
2023-04-23 02:47:20 +02:00
Hennadii Stepanov
306ccd4927 scripted-diff: Bump copyright headers
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-

Commits of previous years:
- 2021: f47dda2c58
- 2020: fa0074e2d8
- 2019: aaaaad6ac9
2022-12-24 23:49:50 +00:00
Gleb Naumenko
a60f729e29 p2p: Drop roles from sendtxrcncl
This feature was currently redundant (although could have provided
more flexibility in the future), and already been causing confusion.
2022-11-10 09:21:57 +02:00
Gleb Naumenko
cfcef60779 test: Add functional tests for sendtxrcncl from inbound 2022-10-17 12:36:13 +03:00
Andrew Chow
3405f3eed5 test: Test that an unconfirmed not-in-mempool chain is rebroadcast
The test checks that parent txs are broadcast before child txs.

The previous behavior is that the rebroadcasting would simply iterate mapWallet. As
mapWallet is a std::unsorted_map, the child can sometimes come before the parent and thus
be rebroadcast in the wrong order and fail the test.
2022-08-29 12:41:50 -04:00
w0xlt
8b3d2bbd0d test: add tests for datacarrier and datacarriersize options
Co-authored-by: Sebastian Falbesoner <sebastian.falbesoner@gmail.com>
2022-08-11 12:05:09 -03:00
Sebastian Falbesoner
b4a5ab96b4 test: refactor: deduplicate DEFAULT_{ANCESTOR,DESCENDANT}_LIMIT constants 2022-08-09 15:22:38 +02:00
Sebastian Falbesoner
7c0dfec2dd refactor: move from_binary helper from signet miner to test framework
Can be easily reviewed with `--color-moved=dimmed-zebra`.
2022-07-19 15:40:51 +02:00
MacroFake
fa0404dbb7 scripted-diff: [test] Rename BIP125_SEQUENCE_NUMBER to MAX_BIP125_RBF_SEQUENCE
-BEGIN VERIFY SCRIPT-
 sed -i 's:BIP125_SEQUENCE_NUMBER:MAX_BIP125_RBF_SEQUENCE:g' $(git grep -l BIP125_SEQUENCE_NUMBER ./test)
-END VERIFY SCRIPT-
2022-07-12 18:49:08 +02:00
Sebastian Falbesoner
5dc6d92077 test: make BIP157 messages default-constructible (MESSAGEMAP compatibility)
In order to deserialize received or read messages via lookup in
MESSAGEMAP (e.g.: `t = MESSAGEMAP[msgtype]()`), the messages must have a
default constructor, i.e. there needs to be the possibility to
initialize them with zero arguments.
2022-05-13 13:53:25 +02:00
Sebastian Falbesoner
f30041c914 test: create txs with current nVersion (2) by default
This enables testing of BIP68 without the need of explicitly
setting nVersion to 2. This is e.g. useful for transactions
created with MiniWallet.
2022-01-11 16:22:27 +01:00
Sebastian Falbesoner
2f79786822 test: refactor: add constant for sequence number SEQUENCE_FINAL 2022-01-11 15:28:28 +01:00
MarcoFalke
fa1b63c018 test: Replace hashlib.new with named constructor 2021-12-09 14:36:06 +01:00
MarcoFalke
fac23c2114 scripted-diff: Bump copyright headers
The previous diff touched most files in ./test/, so bump the headers to
avoid having to touch them again for a bump later.

-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./test/
-END VERIFY SCRIPT-
2021-11-10 11:10:24 +01:00
Zero-1729
021daedfa1 refactor: replace remaining binascii method calls 2021-08-16 19:30:56 +01:00
MarcoFalke
ad0fc453cc Merge bitcoin/bitcoin#16333: test: Set BIP34Height = 2 for regtest
222290f543 test: Set BIP34Height = 2 for regtest (MarcoFalke)
fac90c55be test: Create all blocks with version 4 or higher (MarcoFalke)

Pull request description:

  BIP34 is active on the current tip of mainnet, so all miners must obey it. It would be nice if it also was active in fresh regtest instances from the earliest time possible.

  I changed the BIP34 height to `2`, so that the block at height=1 may be used to mine a duplicate coinbase. (Needed to test mainnet behaviour)

  This pull is done in two commits:

  *  test: Create all blocks with version 4 or higher:
     Now that BIP34 is activated earlier, we need to create blocks with a higher version number. Just bump it to 4 instead of 2 to avoid having to bump it again later.

  *  test: Set BIP34Height = 2 for regtest:
     This fixes the BIP34 implementation in the tests (to match the one of the Core codebase) and updates the tests where needed

ACKs for top commit:
  ajtowns:
    ACK 222290f543
  jonatack:
    ACK 222290f543 tested and reviewed rebased to current master 5e213822f8
  theStack:
    Tested ACK 222290f543

Tree-SHA512: d69c637a62a64b8e87de8c7f0b305823d8f4d115c1852514b923625dbbcf9a4854b5bb3771ff41702ebf47c4c182a4442c6d7c0b9f282c95a34b83e56a73939b
2021-08-03 10:10:43 +02:00