We put this new schema update into the main line and change the
versions of the schema updates which are currently only available
in dev builds. The schemas need to be chronological therefore we
also need to rename the file numbers.
This commit adds a BenchmarkFindOptimalSQLQueryConfig test in the
graph/db package which runs ForEachNode and ForEachChannel queries
against a local backend using various different values for the sql
QueryConfig struct. This is done to determine good default values to
use for the config options for sqlite vs postgres.
Now that the SQL query config values are configurable, we add some
validation to make sure that the user doesnt set a max batch size that
is larger than the limits for sqlite/postgres that have been determined
by the TestSQLSliceQueries test.
Here, we make the sql query option params (batch size and pagination
size) configurable. The defaults for SQLite vs Postgres are still the
same but will be changed in an upcoming commit.
The following performance gains were measured using the new benchmark
test.
```
name old time/op new time/op delta
ChanUpdatesInHorizon-native-sqlite-10 18.5s ± 3% 2.0s ± 5% -89.11% (p=0.000 n=9+9)
ChanUpdatesInHorizon-native-postgres-10 59.0s ± 3% 0.8s ±10% -98.65% (p=0.000 n=10+9)
```
In this commit we add a new helper method in the sqldb package:
ExecuteCollectAndBatchWithSharedDataQuery. This can be used to paginate
through items in the database while at the same time performing batch
data collection for those items.
We rename this helper along the config types & helper types for it
because the word "page" is used more often in the context of paging
through results using an offset and limit whereas this helper is
specifically used to split up the slice in queries of the form
"WHERE x in []slice". We do this rename so that there is mimimal
confusion in contexts where we use batching along with actual paging.
The config struct is also renamed to QueryConfig in preparation for it
holding more config options.
In this commit, we add the queries that will be needed to batch-fetch
the data of a set of nodes. The logic for using these new queries is
also added but not used yet.
In this commit, we remove the LEFT JOIN query that was used for fetching
a nodes addresses. The reason it was used before was to ensure that we'd
get an empty address list if the node did exist but had no addresses.
This was for the purposes of the `AddrsForNode` method since it needs to
return false/true to indicate if the given node exists.
Use the new `SLICES` directive to add a DeleteChannels query which takes
a set of DB channel IDs. Then replace all our calls to DeleteChannel
with a paginated call to DeleteChannels.
This commit adds a new GetChannelsByOutpoints query which takes a slice
of outpoint strings. This lets us then update PruneGraph to use
paginated calls to GetChannelsByOutpoints instead of making one DB call
per outpoint.
Here, a new query (GetChannelsByOutpoints) is added which makes use of
the /*SLICE:outpoints*/ directive & added workaround. This is then used
in a test to demonstrate how the ExecutePagedQuery helper can be used to
wrap a query like this such that calls are done in pages.
The query that has been added will also be used by live code paths in an
upcoming commit.
Along with a test for it. This helper will allow us to easily create a
pagination wrapper for queries that will make use of the new
/*SLICE:<field_name>*/ directive. The next commit will add a test
showing this.
This commit plugs in the graph kvdb-to-sql migration for builds
containing the `test_native_sql` tag. This will allow us to perform
local tests and write itests for the migration without exposing it to
the production release build.
Here we start using the newly added message_flags and channel_flags
columns of the channel_policies table. The test added previoulsy to
demonstrate the bug is now updated to show that the bug has been fixed.
We need to explicitly store the entire bitfield types since we may have
channel_updates with bitfields containing bits we just dont need or
understand but we still need to store the entire bitfield so that the
reconstructed announcement remains valid.
This commit only adds the new columns but does not use them yet. NOTE:
this is ok since the migration adding this schema is not available in
the production build yet.
In this commit, a new `test_native_sql` build flag is defined. If this
build flag is used along with the `--use-native-sql` config option, then
the SQLStore implementation of the graphdb.V1Store will be initialised.
This is then used to run our itest suite against the new SQLStore graph
implementation.
NOTE that this only works for new nodes currently - the migration from
kv-to-sql is yet to be implemeneted.
