9c20859b5f
61020b36c5doc: add release note for #35182 replace libevent HTTP server (Matthew Zipkin)39e9099da5logging: deprecate libevent category (Matthew Zipkin)8c1eea0777http: remove libevent usage from this subsystem (Matthew Zipkin)e427c227fafuzz: switch http_libevent::HTTPRequest to http_bitcoin::HTTPRequest (Matthew Zipkin)21c7542cf8http: switch servers from libevent to bitcoin (Matthew Zipkin)cbb8d1fb33HTTPServer: disconnect after idle timeout (-rpcservertimeout) (Matthew Zipkin)e5f242eef3HTTPServer: implement control methods to match legacy API (Matthew Zipkin)2ca645c2e4refactor: split HTTPBindAddresses into config parse and libevent setup (Matthew Zipkin)fec6b6bca8refactor: split http_request_cb into libevent callback and dispatch (Matthew Zipkin)f946ff5a0bAdd helper methods to HTTPRequest to match original API (Matthew Zipkin)dd11b5e01bdefine HTTP request methods at module level outside of class (Matthew Zipkin)7ee7df988eHTTPServer: use a queue to pipeline requests from each connected client (Matthew Zipkin)5ef1b80a09Allow http workers to send data optimistically as an optimization (Matthew Zipkin)a69bb9e1e6HTTPServer: disconnect clients (Matthew Zipkin)cdf71998e5HTTPServer: compose and send replies to connected clients (Matthew Zipkin)6734bcdeffHTTPserver: support "chunked" Transfer-Encoding (Matthew Zipkin)80e1cfe5a2HTTPServer: read requests from connected clients (Matthew Zipkin)3c5226ab96HTTPServer: start an I/O loop in a new thread and accept connections (Matthew Zipkin)4ef4ebdc0chttp: Introduce HTTPRemoteClient class (Matthew Zipkin)a85286c5c7HTTPServer: generate sequential Ids for each newly accepted connection (Matthew Zipkin)5a3aa1af28HTTPServer: implement and test AcceptConnection() (Matthew Zipkin)f5bc018948http: Introduce HTTPServer class and implement binding to listening socket (Matthew Zipkin)9463e98781http: Implement HTTPRequest class (Matthew Zipkin)ad50aa4a0fhttp: Implement HTTPResponse class (Matthew Zipkin)68b5d289d1http: Implement HTTPHeaders class (Matthew Zipkin)89c54ae4cbhttp: enclose libevent-dependent code in a namespace (Matthew Zipkin)5aa3629b48util/string: LineReader should only trim \r or \r\n (Matthew Zipkin)0cdbb191b5util/string: use string_view in LineReader (Matthew Zipkin)881d4b6c75test: cover common HTTP attacks and common malformed requests (Matthew Zipkin) Pull request description: Continued from #32061. This is a major component of [removing libevent as a dependency of the project](https://github.com/bitcoin/bitcoin/issues/31194), by replacing the HTTP server used for RPC and REST with one implemented entirely within the Bitcoin Core codebase. The new `HTTPServer` class runs its own I/O thread, handling socket connections with code based on #30988, but tailored specifically for HTTP. Commit strategy: - Isolate the existing libevent-based HTTP server in a namespace `http_libevent` - Implement HTTP in a new namespace `http_bitcoin` (classes like `HTTPRequest`, `HTTPClient`, etc...) - Switch bitcoind from the libevent server to the new server - Clean up (delete `http_libevent`) ## RFC compliance I ran https://www.http-probe.com/ against v31.0 release and this PR branch at commit ddefe4263b42. While both HTTP servers failed several of the very picky compliance tests, the only differences in test results between the two branches is an *improvement* in the test results of the PR branch. Raw test results: - [v31.0](https://gist.github.com/pinheadmz/bb0debd49e8556417e9f0072a4463b31) - [PR Branch](https://gist.github.com/pinheadmz/7267cdd90ff9deee8034788b5d1148de) Test result comparison: - https://gist.github.com/pinheadmz/08a5814345e02bf1ac1304e8e5b163a1 - Summary: `Changed results: 33 (Fail→Pass: 32, Pass→Fail: 0, other: 1)` ## Fuzz testing ### libfuzzer - 4 days x8 cores at ddefe4263b - generated corpus: https://github.com/pinheadmz/qa-assets/tree/httprequest-pr35182-ddefe4263b - coverage report: https://thebitcoinblockclock.com/demo/libfuzzer_pr35182_ddefe4263b/index.html ### fuzzamoto - 4 days x8 cores at commit ddefe4263b with a [patched fuzzer](https://github.com/pinheadmz/fuzzamoto/tree/http) - coverage report: https://thebitcoinblockclock.com/demo/fuzzamoto_pr35182_ddefe4263b/coverage/bitcoin/index.html ## Integration testing: I am testing the new HTTP server by forking projects that integrate with bitcoin via HTTP and running their integration tests with bitcoind built from this branch (on Github actions). I will continue adding integrations over time, and re-running these CI tests as this branch gets rebased: - ✅ [lnd](https://github.com/pinheadmz/lnd/pull/7) master at `f21262dd6` tested with this branch at `ddefe4263b` - ✅ [NodeJS `rpc-bitcoin`](https://github.com/pinheadmz/rpc-bitcoin/pull/4) master at `eb0c5da7d1094629cb4a381733cb01c9bb9a4608` tested with this branch at `ddefe4263b` - ✅ [electrs](https://github.com/pinheadmz/electrs/pull/5) master at `d81dc35ae8f71c22b96cf23ee279b69683f29919` tested with this branch at `ddefe4263b` - ✅ [eclair](https://github.com/pinheadmz/eclair/pull/4) master at `2dda79468a8b69a2acf7962cdac63245f7cc3ee8` tested with this branch at `ddefe4263b` - ✅ [corepc](https://github.com/pinheadmz/corepc/pull/4) master at `818e5f2929ea5113176150b2d9f7c0248992d99c` tested with this branch at `ddefe4263b` - ✅ [bitcoinjs-lib](https://github.com/pinheadmz/bitcoinjs-lib/pull/4) master at `ab9fad5978bc1f4fb6542d1cde903d4427c3344e` tested with this branch at `ddefe4263b` - ✅ [NodeJS `bitcoin-core`](https://github.com/pinheadmz/bitcoin-core/pull/2) master at `24f3d0be7ab73089542de42cc8e31903727dbdf8` tested with this branch at `ddefe4263b` ## Benchmarks To get a general idea of server speed vs. master I ran the functional test suite on all three platforms, three trials each, from the first and last commit of this branch. The first commit represents master, but adds a few extra functional tests, so the overall suite is the same in the last commit. [Full results](https://docs.google.com/spreadsheets/d/1X8JI3k_mjKm9qUYNCV14TezTUBGOFTqHW-mucZOYRn4/edit?usp=sharing) Summary: Tested at 37ff331f60 There appears to be a 1-2% slow down on Ubuntu and MacOS but a 5% speed up on Windows || Ubuntu - master | Ubuntu - branch | MacOS - master | Macos - branch | Windows - master | Windows - branch | |-|-|-|-|-|-|-| |AVERAGE| 253| 260| 187| 190| 521| 493| |% diff to master| | 2.766798419| | 1.604278075| | -5.37428023 | ACKs for top commit: vasild: ACK61020b36c5theStack: re-ACK61020b36c5Tree-SHA512: d39f782b9e6c7d67fdba5638fe9a1018b83062668c8b2ab0aa3fe363801f3c36a0ae97b8da0ea0395195e03a2c248a5d1511e5fa27c438c9bd7fca5c83d1ae4c
Bitcoin Core integration/staging tree
For an immediately usable, binary version of the Bitcoin Core software, see https://bitcoincore.org/en/download/.
What is Bitcoin Core?
Bitcoin Core connects to the Bitcoin peer-to-peer network to download and fully validate blocks and transactions. It also includes a wallet and graphical user interface, which can be optionally built.
Further information about Bitcoin Core is available in the doc folder.
License
Bitcoin Core is released under the terms of the MIT license. See COPYING for more information or see https://opensource.org/license/MIT.
Development Process
The master branch is regularly built (see doc/build-*.md for instructions) and tested, but it is not guaranteed to be
completely stable. Tags are created
regularly from release branches to indicate new official, stable release versions of Bitcoin Core.
The https://github.com/bitcoin-core/gui repository is used exclusively for the development of the GUI. Its master branch is identical in all monotree repositories. Release branches and tags do not exist, so please do not fork that repository unless it is for development reasons.
The contribution workflow is described in CONTRIBUTING.md and useful hints for developers can be found in doc/developer-notes.md.
Testing
Testing and code review is the bottleneck for development; we get more pull requests than we can review and test on short notice. Please be patient and help out by testing other people's pull requests, and remember this is a security-critical project where any mistake might cost people lots of money.
Automated Testing
Developers are strongly encouraged to write unit tests for new code, and to
submit new unit tests for old code. Unit tests can be compiled and run
(assuming they weren't disabled during the generation of the build system) with: ctest. Further details on running
and extending unit tests can be found in /src/test/README.md.
There are also regression and integration tests, written
in Python.
These tests can be run (if the test dependencies are installed) with: build/test/functional/test_runner.py
(assuming build is your build directory).
The CI (Continuous Integration) systems make sure that every pull request is tested on Windows, Linux, and macOS. The CI must pass on all commits before merge to avoid unrelated CI failures on new pull requests.
Manual Quality Assurance (QA) Testing
Changes should be tested by somebody other than the developer who wrote the code. This is especially important for large or high-risk changes. It is useful to add a test plan to the pull request description if testing the changes is not straightforward.
Translations
Changes to translations as well as new translations can be submitted to Bitcoin Core's Transifex page.
Translations are periodically pulled from Transifex and merged into the git repository. See the translation process for details on how this works.
Important: We do not accept translation changes as GitHub pull requests because the next pull from Transifex would automatically overwrite them again.