e8fa0a3d2025509fcddc59fc618e91371542cf87 Fix WSL file locking by using flock instead of fcntl (Samuel Dobson)
Pull request description:
Fixes#18622
A bug in WSL means that fcntl does not exclusively lock files, allowing multiple instances of bitcoin to use the same datadir. If we instead use flock, it works correctly. Passes Travis, but testing on some OS variety would be sensible.
From what I can tell, flock and fcntl don't work with each other on linux, so it would still be possible to run a node with this code change and a node before it with the same datadir (this isn't true for Mac/FreeBSD). flock also doesn't support NFS on MacOS and linux<2.6.12 while fcntl did. See here for example: https://gavv.github.io/articles/file-locks/
If changing to flock for all systems is inadvisable, it would also be possible to just detect WSL and use flock when on that platform to avoid the bug.
ACKs for top commit:
laanwj:
Code review ACK e8fa0a3d2025509fcddc59fc618e91371542cf87
Tree-SHA512: ca1009e171970101f1dc2332c5e998717aee00eebc80bb586b826927a74bd0d4c94712e46d1396821bc30533d76deac391b6e1c406c406865661f57fa062c702
Replace with RPC request reference to new WalletContext struct similar to the
existing NodeContext struct and reference.
This PR is a followup to 25ad2c623af30056ffb36dcd203a52edda2b170f
https://github.com/bitcoin/bitcoin/pull/18740 removing the g_rpc_node global.
Some later PRs will follow this up and move more wallet globals to the
WalletContext struct.
Co-authored-by: João Barbosa <joao.paulo.barbosa@gmail.com>
5478d6c099e76fe070703cc5383cba7b91468b0f logging: thread safety annotations (Anthony Towns)
e685ca19928eec4e687c66f5edfcfff085a42c27 util/system.cpp: add thread safety annotations for dir_locks (Anthony Towns)
a7887899480db72328784009181d93904e6d479d test/checkqueue_tests: thread safety annotations (Anthony Towns)
479c5846f7477625ec275fbb8a076c7ef157172b rpc/blockchain.cpp: thread safety annotations for latestblock (Anthony Towns)
8b5af3d4c1270267ad85e78f661bf8fab06f3aad net: fMsgProcWake use LOCK instead of lock_guard (Anthony Towns)
de7c5f41aba860751ef7824245e6d9d5088a1200 wallet/wallet.h: Remove mutexScanning which was only protecting a single atomic bool (Anthony Towns)
c3cf2f55013c4ea1c1ef4a878fc7ff8e92f2c42d rpc/blockchain.cpp: Remove g_utxosetscan mutex that is only protecting a single atomic variable (Anthony Towns)
Pull request description:
In a few cases we need to use `std::mutex` rather than the sync.h primitives. But `std::lock_guard<std::mutex>` doesn't include the clang thread safety annotations unless you also use clang's C library, which means you can't indicate when variables should be guarded by `std::mutex` mutexes.
This adds an annotated version of `std::lock_guard<std::mutex>` to threadsafety.h to fix that, and modifies places where `std::mutex` is used to take advantage of the annotations.
It's based on top of #16112, and turns the thread safety comments included there into annotations.
It also changes the RAII classes in wallet/wallet.h and rpc/blockchain.cpp to just use the atomic<bool> flag for synchronisation rather than having a mutex that doesn't actually guard anything as well.
ACKs for top commit:
MarcoFalke:
ACK 5478d6c099e76fe070703cc5383cba7b91468b0f 🗾
hebasto:
re-ACK 5478d6c099e76fe070703cc5383cba7b91468b0f, only renamed s/`MutexGuard`/`LockGuard`/, and dropped the commit "test/util_threadnames_tests: add thread safety annotations" since the [previous](https://github.com/bitcoin/bitcoin/pull/16127#pullrequestreview-414184113) review.
ryanofsky:
Code review ACK 5478d6c099e76fe070703cc5383cba7b91468b0f. Thanks for taking suggestions! Only changes since last review are dropping thread rename test commit d53072ec730d8eec5a5b72f7e65a54b141e62b19 and renaming mutex guard to lock guard
Tree-SHA512: 7b00d31f6f2b5a222ec69431eb810a74abf0542db3a65d1bbad54e354c40df2857ec89c00b4a5e466c81ba223267ca95f3f98d5fbc1a1d052a2c3a7d2209790a
We want to supply well-known vars to ./configure scripts to do with as
they please. However, we do _not_ want to override these well-known vars
at make-time as certain build systems expect a self-mangled version of
these well-known vars.
For example, freetype and bdb will prepend `libtool --mode=compile' to
CC and CXX, which, if we override CC on the command line at make-time,
will break the build.
Previously, we specified the target-os in the toolset (and sometimes
used the wrong command line flags), now we have a clear separation,
which is favored by ./bootstrap.sh and ./b2.
This means that all supported OSes will specify the correct target-os=
and toolset= on the command line.
b2 will pickup our user-config.jam just fine, however, bootstrap.sh has
its own toolset autodetect mechanism, which doesn't GAF about our
user-config.jam
c57f03ce1741b38af448bec7b22ab9f8ac21f067 refactor: Replace const char* to std::string (Calvin Kim)
Pull request description:
Rationale: Addresses #19000
Some functions should be returning std::string instead of const char*.
This commit changes that.
Main benefits/reasoning:
1. The functions never return nullptr, so returning a string makes code at call sites easier to review (reviewers don't have to read the source code to verify that a nullptr is never returned)
2. All call sites convert to string anyway
ACKs for top commit:
MarcoFalke:
re-ACK c57f03ce17 (no changes since previous review) 🚃
Empact:
Fair enough, Code Review ACK c57f03ce17
practicalswift:
ACK c57f03ce1741b38af448bec7b22ab9f8ac21f067 -- patch looks correct
hebasto:
re-ACK c57f03ce1741b38af448bec7b22ab9f8ac21f067
Tree-SHA512: 9ce99bb38fe399b54844315048204cafce0f27fd8f24cae357fa7ac6f5d8094d57bbf5f5c1f5878a65f2d35e4a3f95d527eb17f49250b690c591c0df86ca84fd
4c825792dd9f4eaf4936c3e376ac7a5c177528e2 Remove outdated comment about DER encoding (Elichai Turkel)
Pull request description:
This comment got me confused about the status of BIP66 (Thanks jnewbery for explaining)
The comment was added in: https://github.com/bitcoin/bitcoin/pull/3843
But in https://github.com/bitcoin/bitcoin/pull/5713 strict DER encoding was enforced in consensus,
and is now it's buried and enforced by the height of the block here: 4af01b37d4/src/validation.cpp (L1889)
P.S. This is also quite confusing: 4af01b37d4/src/validation.cpp (L1560-L1563) But seems to be intentional: 4af01b37d4/src/validation.cpp (L1510-L1511)
ACKs for top commit:
laanwj:
ACK 4c825792dd9f4eaf4936c3e376ac7a5c177528e2
Tree-SHA512: 7afbbae84ed4ecfaa0a273ae024b14f2b7ffe65307f078086fe0b5b645c57722bc2952fb15d167d9e4fa5b052d1d0ac6e5e33f57e8fc881c0ea611d352bccc1e
1c91ffefcf54d22f805c86318ea32d23b9d20c96 doc : add link to readme.md in the first section (pad)
Pull request description:
I have searched how to do it in this doc for some time :-(
I think it might help other newbies interested in building with visual studio.
ACKs for top commit:
hebasto:
ACK 1c91ffefcf54d22f805c86318ea32d23b9d20c96, a new link works as expected :)
Tree-SHA512: 42ef3ba374bced9b4ab0010fe8c30de06f59ff8a84f8e02f8a91f33e7e403cf91d624fc7df3f45096df53171a90b9ff60277969cc30f1357d92094ad72ca9d53
84ae0578b6c68dda145ca65fef510ce0fdac0d7b Add release notes about salvage changes (Andrew Chow)
ea337f2d0318a860f695698cfb3aa91c03ded858 Move RecoverKeysOnlyFilter into RecoverDataBaseFile (Andrew Chow)
9ea2d258b46e8a9776100633585ed0feede5c2a4 Move RecoverDatabaseFile and RecoverKeysOnlyFilter into salvage.{cpp/h} (Andrew Chow)
b426c7764d26e280e1f814cf36e050743c45cd12 Make BerkeleyBatch::Recover and WalletBatch::RecoverKeysOnlyFilter standalone (Andrew Chow)
2741774214168eb287c7066d6823afe5e570381d Expose a version of ReadKeyValue and use it in RecoverKeysOnlyFilter (Andrew Chow)
ced95d0e43389fe62b5d30fcc7c42dbca0e88242 Move BerkeleyEnvironment::Salvage into BerkeleyBatch::Recover (Andrew Chow)
07250b8dcebe2b97ed0fd900ad35cba4091b8ecf walletdb: remove fAggressive from Salvage (Andrew Chow)
8ebcbc85c652665b78dcfd2ad55fa67cafd42c73 walletdb: don't automatically salvage when corruption is detected (Andrew Chow)
d321046f4bb4887742699c586755a21f3a2edbe1 wallet: remove -salvagewallet (Andrew Chow)
cdd955e580dff99f3fa440494ed2b348f7f094af Add basic test for bitcoin-wallet salvage (Andrew Chow)
c87770915b88d195d264b58111c64142b1965cfa wallettool: Add a salvage command (Andrew Chow)
Pull request description:
Removes the `-salvagewallet` startup option and adds a `salvage` command to the `bitcoin-wallet` tool. As such, `-salvagewallet` is removed. Additionally, the automatic salvage that is done if the wallet file fails to load is removed.
Lastly the salvage code entirely is moved out entirely into `bitcoin-wallet` from `walletdb.{cpp/h}` and `db.{cpp/h}`.
ACKs for top commit:
jonatack:
ACK 84ae0578b6c68dda145ca65fef510ce0fdac0d7b feedback taken, and compared to my previous review, the bitcoin-wallet salvage command now seems to run and it exits without raising. The new test passes at both 9454105 and 84ae057 so as a sanity check I'd agree there is room for improvement, if possible.
MarcoFalke:
re-ACK 84ae0578b6 🏉
Empact:
Code Review ACK 84ae0578b6
ryanofsky:
Code review ACK 84ae0578b6c68dda145ca65fef510ce0fdac0d7b. Lot of small changes since previous review: added verify step before salvage, added basic test in new commit, removed unused scanstate variable and warnings parameter, tweaked various comments and strings, moved fsuccess variable declaration
meshcollider:
Concept / light code review ACK 84ae0578b6c68dda145ca65fef510ce0fdac0d7b
Tree-SHA512: 05be116b56ecade1c58faca1728c8fe4b78f0a082dbc2544a3f7507dd155f1f4f39070bd1fe90053444384337bc48b97149df5c1010230d78f8ecc08e69d93af
This only changes network serialization. Disk serialization does not
include the filter_type and is defined in
ReadFilterFromDisk()/WriteFilterToDisk().
fa80b4788bbe3ef00c5d767c0d89ba9809d8707c test: Remove global wait_until from p2p_getdata (MarcoFalke)
999922baed3a80b581ce46daa01c4cbca4fcbfd8 test: Default mininode.wait_until timeout to 60s (MarcoFalke)
fab47375fe0bdec1e557e087fdb0707c4dfa7cc2 test: pep-8 p2p_getdata.py (MarcoFalke)
Pull request description:
Using the global wait_until makes it impossible to adjust the timeout based on the hardware the test is running on.
Fix that by using the mininode member function.
So for example, `./test/functional/p2p_getdata.py --timeout-factor=0.04` gives a timeout of 2.4 seconds.
ACKs for top commit:
laanwj:
ACK fa80b4788bbe3ef00c5d767c0d89ba9809d8707c
Tree-SHA512: ebb1b7860a64451de2b8ee9a0966faddb13b84af711f6744e8260d7c9bc0b382e8fb259897df5212190821e850ed30d4d5c2d7af45a97f207fd4511b06b6674a
71f016c6eb42e1ac2c905e04ba4d20c2009e533f Remove old serialization primitives (Pieter Wuille)
92beff15d3ae2646c00bd78146d7592a7097ce9c Convert LimitedString to formatter (Pieter Wuille)
ef17c03e074b6c3f185afa4eff572ba687c2a171 Convert wallet to new serialization (Pieter Wuille)
65c589e45e8b8914698a0fd25cd5aafdda30869c Convert Qt to new serialization (Pieter Wuille)
Pull request description:
This is the final step 🥳 of the serialization improvements extracted from #10785.
It converts the LimitedString wrapper to a new-style formatter, and updates the wallet and Qt code to use the new serialization framework. Finally all remaining old primitives are removed.
ACKs for top commit:
jonatack:
ACK 71f016c6eb42e1ac2 reviewed diff, builds/tests/re-fuzzed.
laanwj:
Code review ACK 71f016c6eb42e1ac2c905e04ba4d20c2009e533f
Tree-SHA512: d952194bc73259f6510bd4ab1348a1febbbf9862af30f905991812fb0e1f23f15948cdb3fc662be54d648e8f6d95b11060055d2e7a8c2cb5bf008224870b1ea1
90eb027204f5a9d7c00fa97d4112243bd37a9012 doc: Add and fix comments about never destroyed objects (Hennadii Stepanov)
26c093a9957756f3743c2347fe0abd90f81159c4 Replace thread_local g_lockstack with a mutex-protected map (Hennadii Stepanov)
58e6881bc5be002e8ddbc9b75422c0deae66a2df refactor: Refactor duplicated code into LockHeld() (Hennadii Stepanov)
f511f61dda4e860079153d5e51d64658cc265283 refactor: Add LockPair type alias (Hennadii Stepanov)
8d8921abd35c3ac1b8ebacb11de8e1bbc7b28d66 refactor: Add LockStackItem type alias (Hennadii Stepanov)
458992b06d80eb568141f60a33d38e12e894e27a Prevent UB in DeleteLock() function (Hennadii Stepanov)
Pull request description:
Tracking our instrumented mutexes (`Mutex` and `RecursiveMutex` types) requires that all involved objects should not be destroyed until after their last use. On master (ec79b5f86b22ad8f77c736f9bb76c2e4d7faeaa4) we have two problems related to the object destroying order:
- the function-local `static` `lockdata` object that is destroyed at [program exit](https://en.cppreference.com/w/cpp/utility/program/exit)
- the `thread_local` `g_lockstack` that is destroyed at [thread exit](https://en.cppreference.com/w/cpp/language/destructor)
Both cases could cause UB at program exit in so far as mutexes are used in other static object destructors.
Fix#18824
ACKs for top commit:
MarcoFalke:
re-ACK 90eb027204, only change is new doc commit 👠
ryanofsky:
Code review ACK 90eb027204f5a9d7c00fa97d4112243bd37a9012 because all the changes look correct and safe. But I don't know the purpose of commit 26c093a9957756f3743c2347fe0abd90f81159c4 "Replace thread_local g_lockstack with a mutex-protected map (5/6)." It seems like it could have a bad impact on debug performance, and the commit message and PR description don't give a reason for the change.
Tree-SHA512: 99f29157fd1278994e3f6eebccedfd9dae540450f5f8b980518345a89d56b635f943a85b20864cef087027fd0fcdb4880b659ef59bfe5626d110452ae22031c6
fa756928c3f455943086051c5fe1d5bb09962248 rpc: Make gettxoutsetinfo/GetUTXOStats interruptible (MarcoFalke)
fa7fc5a8e0fcf9ca81e84b3631f18ae40502be60 rpc: factor out RpcInterruptionPoint from dumptxoutset (MarcoFalke)
Pull request description:
Make it interruptible, so that shutdown doesn't block for up to one hour.
Fixes (partially) #13217
ACKs for top commit:
Empact:
Code Review ACK fa756928c3
laanwj:
Code review ACK fa756928c3f455943086051c5fe1d5bb09962248
Tree-SHA512: 298261e0ff7d79fab542b8f6828cc0ac451cbafe396d5f0816c9d36437faba1330f5c4cb2a25c5540e202bfb9783da6ec858bd453056ce488d21e36335d3d42c
f9b22e3bdb54acb2f830b3ebbad47ff17dfb5781 tests: Add fuzzing harness for CCoinsViewCache (practicalswift)
Pull request description:
Add fuzzing harness for `CCoinsViewCache`.
See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the [Bitcoin Core fuzzing corpus repo](https://github.com/bitcoin-core/qa-assets).
Happy fuzzing :)
ACKs for top commit:
MarcoFalke:
ACK f9b22e3bdb 📫
Tree-SHA512: 4fa79aab683875eef128b672cf199909c86e4d2ed7c406f006fa27a546dafc9cb0061c4de5e660e622458072f1dab69dbf6b6b03d5b863f81c5710bf4cee6c0c
5308c97ccaf0955e5840956bc1636108a43e6f46 [test] Add test for cfheaders (Jim Posen)
f6b58c150686e90bc4952976e488b1605f3ae02a [net processing] Message handling for getcfheaders. (Jim Posen)
3bdc7c2d3977a7864aacea80bffc4df7f37cac51 [doc] Add comment for m_headers_cache (John Newbery)
Pull request description:
Support `getcfheaders` requests when `-peerblockfilters` is set.
Does not advertise compact filter support in version messages.
ACKs for top commit:
jkczyz:
ACK 5308c97ccaf0955e5840956bc1636108a43e6f46
MarcoFalke:
re-ACK 5308c97cca , only change is doc related 🗂
theStack:
ACK 5308c97ccaf0955e5840956bc1636108a43e6f46 🚀
Tree-SHA512: 240fc654f6f634c191d9f7628b6c4801f87ed514a1dd55c7de5d454d4012d1c09509a2d5a246bc7da445cd920252b4cd56a493c060cdb207b04af4ffe53b95f7
69bfcac27a83440092bc6e61904ded910ed4baf4 gui: update Qt base translations for macOS release (fanquake)
Pull request description:
These haven't been updated since their addition, so this updates the list that
controls which qt base translations are bundled with the macOS binary, to all the
languages that are available with qt 5.9.8.
This could probably be improved in some way, however qt updates are infrequent,
and I didn't want to spend any more time looking at this. Also given that no-one
seems to have noticed and/or reported this it wouldn't seem high-priority.
Could be backported to 0.20.1.
Master:
This PR:
ACKs for top commit:
hebasto:
ACK 69bfcac27a83440092bc6e61904ded910ed4baf4, tested on macOS 10.15.
Tree-SHA512: df142fb16097deb514e72e005b73aafc4eb4ff0c17e423ba5040a3ec6874020a733e1c5259a88923580e71ef73c16222aed28f482b8c270a544a85b745a7b327
Returning by const value is only meaningful in a specific circumstance around
user defined types. In this case, the const is not enforcing any restrictions
on the call site, so is misleading.
there were two calls to disconnect_nodes that were no-ops. fixed one & removed
the other & added assertions to confirm node has no connections when creating
the unbroadcast transaction.
