In this commit, we update the SQLStore's ForEachNodeCacheable and
ForEachNodeCached methods to use batch collection for node feature bits.
This results in the following performance gains:
```
name old time/op new time/op delta
ForEachNodeCacheable-native-sqlite-10 184ms ± 2% 145ms ±10% -21.45% (p=0.000 n=10+10)
ForEachNodeCacheable-native-postgres-10 697ms ± 8% 51ms ± 4% -92.68% (p=0.000 n=9+10)
```
In this commit, we add a makefile helper that can be used to add the
"-short" flag to the "go test" command when running unit tests via "make
unit ...".
Tests can then be expanded with a `testing.Short()` check to potentially
skip the test in short mode. This is useful for if a dev wants to
quickly run most of the tests in a package but would like to opt out of
running the longer form tests such as stress tests or rapid generation
tests.
This commit adds this check to two graph/db tests.
This commit is a pure refactor. Here all we are doing is removing the
old `pageSize` constant used in the SQLStore code and instead using the
value provided in the QueryConfig struct.
We rename this helper along the config types & helper types for it
because the word "page" is used more often in the context of paging
through results using an offset and limit whereas this helper is
specifically used to split up the slice in queries of the form
"WHERE x in []slice". We do this rename so that there is mimimal
confusion in contexts where we use batching along with actual paging.
The config struct is also renamed to QueryConfig in preparation for it
holding more config options.
Here, we add a new `buildNodeWithBatchData` helper method that can be
used to construct a `models.LightningNode` object using pre-fetched
batch data. The existing `buildNode` method is then adjusted to use this
new helper.
In this commit, we add the queries that will be needed to batch-fetch
the data of a set of nodes. The logic for using these new queries is
also added but not used yet.
In this commit, we remove the LEFT JOIN query that was used for fetching
a nodes addresses. The reason it was used before was to ensure that we'd
get an empty address list if the node did exist but had no addresses.
This was for the purposes of the `AddrsForNode` method since it needs to
return false/true to indicate if the given node exists.
This commit adds a helper test that is similar to TestPopulateDBs except
for the fact that it uses the MigrateGraphToSQL function directly to
migrate a local kvdb-sql graph to a native SQL one.
This is a prep commit that just adds all the boilerplate connection
logic for connecting to various graph DBs. These will be used in
upcoming commits which will add tests and benchmarks.
Use the new `SLICES` directive to add a DeleteChannels query which takes
a set of DB channel IDs. Then replace all our calls to DeleteChannel
with a paginated call to DeleteChannels.
This commit adds a new GetChannelsByOutpoints query which takes a slice
of outpoint strings. This lets us then update PruneGraph to use
paginated calls to GetChannelsByOutpoints instead of making one DB call
per outpoint.
In preparation for tests where we will want to spin up SQL DBs many
times, we do some refactoring so that it is easy to re-use postgres
fixtures since those are expensive to spin up.
This commit passes no-op reset callbacks to various places where they
are needed in the KVStore. Upcoming commits will replace the no-ops by
adding reset params to the methods that require it.
