2971fd030f
12781db [Tests] check specific validation error in miner tests (Sjors Provoost) Pull request description: ## Problem `BOOST_CHECK_THROW` merely checks that some `std::runtime_error` is thrown, but not which one. Here's an example of how this can cause a test to pass when a developer introduces a consensus bug. The test for the sigops limit assumes that `CreateNewBlock` fails with `bad-blk-sigops`. However it can also fail with bad-txns-vout-negative, if a naive developer lowers `BLOCKSUBSIDY` to `1*COIN`. ## Solution `BOOST_CHECK_EXCEPTION` allows an additional predicate function. This commit uses this for all exceptions that are checked for in `miner_tets.cpp`: * `bad-blk-sigops` * `bad-cb-multiple` * `bad-txns-inputs-missingorspent` * `block-validation-failed` If the function throws a different error, the test will fail. Although the message produced by Boost is a bit [confusing](http://boost.2283326.n4.nabble.com/Test-BOOST-CHECK-EXCEPTION-error-message-still-vague-tt4683257.html#a4683554), it does show which error was actually thrown. Here's what the above `1*COIN` bug would result in: <img width="1134" alt="schermafbeelding 2017-09-02 om 23 42 29" src="https://user-images.githubusercontent.com/10217/29998976-815cabce-9038-11e7-9c46-f5f6cfb0ca7d.png"> ## Other considerations A more elegant solution in my opinion would be to subclass `std::runtime_error` for each `INVALID_TRANSACTION` type, but this would involve touching consensus code. I put the predicates in `test_bitcoin.h` because I assume they can be reused in other test files. However [serialize_tests.cpp](https://github.com/bitcoin/bitcoin/blob/v0.15.0rc3/src/test/serialize_tests.cpp#L245) also uses `BOOST_CHECK_EXCEPTION` and it defines the predicate in the test file itself. Instead of four `IsRejectInvalidReasonX(std::runtime_error const& e)` functions, I'd prefer something reusable like `bool IsRejectInvalidReason(String reason)(std::runtime_error const& e)`, which would be used like `BOOST_CHECK_EXCEPTION(functionThatThrows(), std::runtime_error, IsRejectInvalidReason("bad-blk-sigops")`. I couldn't figure out how to do that in C++. Tree-SHA512: e364f19b4ac19f910f6e8d6533357f57ccddcbd9d53dcfaf923d424d2b9711446d6f36da193208b35788ca21863eadaa7becd9ad890334d334bccf8c2e63dee1
Compiling/running unit tests
Unit tests will be automatically compiled if dependencies were met in ./configure
and tests weren't explicitly disabled.
After configuring, they can be run with make check.
To run the bitcoind tests manually, launch src/test/test_bitcoin. To recompile
after a test file was modified, run make and then run the test again. If you
modify a non-test file, use make -C src/test to recompile only what's needed
to run the bitcoind tests.
To add more bitcoind tests, add BOOST_AUTO_TEST_CASE functions to the existing
.cpp files in the test/ directory or add new .cpp files that
implement new BOOST_AUTO_TEST_SUITE sections.
To run the bitcoin-qt tests manually, launch src/qt/test/test_bitcoin-qt
To add more bitcoin-qt tests, add them to the src/qt/test/ directory and
the src/qt/test/test_main.cpp file.
Running individual tests
test_bitcoin has some built-in command-line arguments; for example, to run just the getarg_tests verbosely:
test_bitcoin --log_level=all --run_test=getarg_tests
... or to run just the doubledash test:
test_bitcoin --run_test=getarg_tests/doubledash
Run test_bitcoin --help for the full list.
Note on adding test cases
The sources in this directory are unit test cases. Boost includes a unit testing framework, and since bitcoin already uses boost, it makes sense to simply use this framework rather than require developers to configure some other framework (we want as few impediments to creating unit tests as possible).
The build system is setup to compile an executable called test_bitcoin
that runs all of the unit tests. The main source file is called
test_bitcoin.cpp. To add a new unit test file to our test suite you need
to add the file to src/Makefile.test.include. The pattern is to create
one test file for each class or source file for which you want to create
unit tests. The file naming convention is <source_filename>_tests.cpp
and such files should wrap their tests in a test suite
called <source_filename>_tests. For an example of this pattern,
examine uint256_tests.cpp.
For further reading, I found the following website to be helpful in explaining how the boost unit test framework works: http://www.alittlemadness.com/2009/03/31/c-unit-testing-with-boosttest/.