Commit Graph

40 Commits

Author SHA1 Message Date
Elle Mouton
3b60d33ac8 sqlc: add ListChannelsForNodeIDs batch query
Add a ListChannelsForNodeIDs query which will let us fetch all channels
belonging to a set of nodes.
2025-08-05 08:00:19 +02:00
Elle Mouton
f39edae6e3 graph/db: use new batch helpers for edge loading
Use the new batch helpers to replace the existing logic for loading an
edge.
2025-07-31 07:19:14 +02:00
Elle Mouton
8ad5f633bc sqldb: add channel data batch queries
Also add the calling logic for these queries. This logic is not yet
used.
2025-07-31 07:19:14 +02:00
Elle Mouton
0dc0d320f8 graph/db+sqldb: queries and logic for batch fetching node data
In this commit, we add the queries that will be needed to batch-fetch
the data of a set of nodes. The logic for using these new queries is
also added but not used yet.
2025-07-30 19:29:21 +02:00
Elle Mouton
5a1184c664 graph/db+sqldb: remove LEFT JOIN for fetching node addresses
In this commit, we remove the LEFT JOIN query that was used for fetching
a nodes addresses. The reason it was used before was to ensure that we'd
get an empty address list if the node did exist but had no addresses.
This was for the purposes of the `AddrsForNode` method since it needs to
return false/true to indicate if the given node exists.
2025-07-30 18:11:01 +02:00
Elle Mouton
43db6683d2 graph/db+sqldb: only fetch what is needed for graph cache
Add a ListChannelsWithPoliciesForCachePaginated query that only fetches
the channel/policy fields that the cache requires.
2025-07-30 18:11:01 +02:00
Elle Mouton
ddc0e95eda graph/db+sqldb: delete channels in batches
Use the new `SLICES` directive to add a DeleteChannels query which takes
a set of DB channel IDs. Then replace all our calls to DeleteChannel
with a paginated call to DeleteChannels.
2025-07-22 17:17:23 +02:00
Elle Mouton
e269d57ffa sqldb+graph/db: use pagination for FetchChanInfos 2025-07-22 17:17:11 +02:00
Elle Mouton
88e9a21d63 sqldb+graph/db: update FilterKnownChanIDs to use pagination
Remove a TODO by making use of the new sqldb.ExecutePagedQuery to fetch
channels in batches rather than one by one.
2025-07-22 17:16:59 +02:00
Elle Mouton
f72c48b283 graph/db+sqldb: pass set of outpoints to SQL
This commit adds a new GetChannelsByOutpoints query which takes a slice
of outpoint strings. This lets us then update PruneGraph to use
paginated calls to GetChannelsByOutpoints instead of making one DB call
per outpoint.
2025-07-22 17:16:49 +02:00
Elle Mouton
f0d2d1fd0a sqldb: demonstrate the use of ExecutePagedQuery
Here, a new query (GetChannelsByOutpoints) is added which makes use of
the /*SLICE:outpoints*/ directive & added workaround. This is then used
in a test to demonstrate how the ExecutePagedQuery helper can be used to
wrap a query like this such that calls are done in pages.

The query that has been added will also be used by live code paths in an
upcoming commit.
2025-07-22 17:16:38 +02:00
Elle Mouton
74b70a5108 sqldb+graph/db: prefix graph SQL objects with "graph_"
This makes it more clear what each table is for especially when viewed
with other invoice/payments tables.
2025-07-15 18:40:16 +02:00
Elle Mouton
03ede9ccef graph/db: migrate prune log
This commit expands the `MigrateGraphToSQL` to include migration of the
prune log.
2025-07-11 09:15:47 +02:00
Oliver Gugger
500808fadd Merge pull request #10010 from ellemouton/sqlGraphUpdates
graph/db: various misc updates
2025-07-01 12:40:50 +02:00
Elle Mouton
4a05e5a226 sqldb/sqlc: add message and channe flags to channel_policies table
We need to explicitly store the entire bitfield types since we may have
channel_updates with bitfields containing bits we just dont need or
understand but we still need to store the entire bitfield so that the
reconstructed announcement remains valid.

This commit only adds the new columns but does not use them yet. NOTE:
this is ok since the migration adding this schema is not available in
the production build yet.
2025-07-01 10:12:54 +02:00
Elle Mouton
f1b7ccc6b2 sqldb+graph/db: prune graph nodes in a single query
Update the pruneGraphNodes routine to prune the graph nodes in a single
query instead of two.
2025-06-30 18:41:28 +02:00
Elle Mouton
933ab3c6b7 graph/db+sqldb: impl PutClosedScid and IsClosedScid 2025-06-25 13:26:52 +02:00
Elle Mouton
d46552f5ad graph/db+sqldb: implement AddEdgeProof
And run `TestAddEdgeProof` against the SQL backends.
2025-06-25 13:26:51 +02:00
Elle Mouton
e875183c4f sqldb+graph/db: impl DisconnectBlockAtHeight
Which lets us run `TestDisconnectBlockAtHeight` and
`TestStressTestChannelGraphAPI` against our SQL backends.
2025-06-25 11:22:03 +02:00
Elle Mouton
9dd0361ed0 graph/db+sqldb: impl PruneGraph, PruneTip, ChannelView
Which lets us run `TestGraphPruning` and `TestBatchedAddChannelEdge`
against our SQL backends.
2025-06-25 11:22:00 +02:00
Elle Mouton
2da701cc4f graph/db+sqldb: impl PruneGraphNodes
Which lets us run `TestChannelEdgePruningUpdateIndexDeletion` against
our SQL backends.
2025-06-25 10:48:35 +02:00
Elle Mouton
eec89362a5 graph/db+sqldb: impl DisabledChannelIDs
Which lets us run `TestDisabledChannelIDs` against our SQL DB backends.
2025-06-24 21:04:36 +02:00
Elle Mouton
f1da3812de graph/db+sqldb: impl IsPublicNode
Which lets us run `TestNodeIsPublic` against our SQL DB backends.
Note that we need to tweak the tests a little bit so that
`AddLightningNode` for the same node is always called with a newer
LastUpdate time else it will fail the SQL constraint that only allows
the upsert if the update is newer than the persisted one.
2025-06-24 21:04:36 +02:00
Elle Mouton
13bf6a549f graph/db+sqldb: implement HasChannelEdge and ChannelID
And run `TestEdgeInfoUpdates` against our SQL backends.
2025-06-24 21:04:35 +02:00
Elle Mouton
4fad4a7023 graph/db+sqldb: implement FetchChannelEdgesByOutpoint/SCID
And run `TestEdgeInsertionDeletion` against our SQL backends.
2025-06-24 21:04:35 +02:00
Elle Mouton
2a6e6683eb graph/db+sqldb: implement DeleteChannelEdges
This lets us run TestGraphZombieIndex against the SQL backends.
2025-06-24 19:02:45 +02:00
Elle Mouton
00b6e0204c graph/db+sqldb: implement various zombie index methods
Here we implement the SQLStore methods:
- MarkEdgeZombie
- MarkEdgeLive
- IsZombieEdge
- NumZombies

These will be tested in the next commit as one more method
implementation is required.
2025-06-24 19:02:14 +02:00
Elle Mouton
3687171cd5 sqldb+graph/db: implement FilterChannelRange
This lets us run `TestFilterChannelRange` against the SQL backends.
2025-06-24 12:17:52 +02:00
Elle Mouton
ff84fa1cb2 graph/db+sqldb: impl ForEachNodeCached and ForEachChannel
Which let's us run `TestGraphTraversal` against our SQL backends.
2025-06-24 12:17:52 +02:00
Elle Mouton
39e521e12b graph/db+sqldb: implement ChanUpdatesInHorizon
Add `ChanUpdatesInHorizon` method to the SQLStore. This lets us run
`TestChanUpdatesInHorizon` against our SQL backends.
2025-06-24 12:17:52 +02:00
Elle Mouton
0607982886 graph/db: implement ForEachNodeDirectedChannel and ForEachNodeCacheable
Here we add the `ForEachNodeDirectedChannel` and `ForEachNodeCacheable`
SQLStore implementations which then lets us run
`TestGraphTraversalCacheable` and `TestGraphCacheForEachNodeChannel`
against SQL backends.
2025-06-18 16:55:00 +02:00
Elle Mouton
8af32951c7 graph/db+sqldb: implement ForEachNode
In this commit the `ForEachNode` method is added to the SQLStore. With
this, the `TestGraphCacheTraversal` unit test can be run against SQL
backends.
2025-06-18 16:54:59 +02:00
Elle Mouton
f89e3ceced graph/db+sqldb: implement ForEachSourceNodeChannel
In this commit, the ForEachSourceNodeChannel implementation of the
SQLStore is added. Since this is the first method of the SQLStore that
fetches channel and policy info, it also adds all the helpers that are
required to do so. These will be re-used in upcoming commits as more
"For"-type methods are added.

With this implementation, we convert the `TestForEachSourceNodeChannel`
such that it is run against SQL backends.
2025-06-18 08:43:56 +02:00
Elle Mouton
c327988bb3 graph/db+sqldb: implement UpdateEdgePolicy
In this commit, the various SQL queries are defined that we will need in
order to implement the SQLStore UpdateEdgePolicy method. Channel
policies can be "replaced" and so we use the upsert pattern for them
with the rule that any new channel policy must have a timestamp greater
than the previous one we persisted.

As is done for the KVStore implementation of the method, we use the
batch scheduler for this method.
2025-06-17 13:43:07 +02:00
Elle Mouton
547b836180 sqldb+graph/db: fix UpsertNode bug
Account for the last_update field being null.
2025-06-02 17:29:03 +02:00
Elle Mouton
cf542458bd graph/db+sqldb: add HighestChanID to SQLStore 2025-06-02 14:53:18 +02:00
Elle Mouton
d93d104a66 graph/db+sqldb: implement AddChannelEdge on SQLStore
In this commit, the `AddChannelEdge` method of the SQLStore is
implemented. Like the KVStore implementation, it makes use of the
available channel `batch.Scheduler` and also updates the reject and
channel caches.

This then lets us convert the following 2 unit tests to run against the
SQL backends:
- TestPartialNode
- TestAddChannelEdgeShellNodes
2025-06-02 14:53:18 +02:00
Elle Mouton
0064d33cda sqldb+graph/db: source nodes table, queries and CRUD
In this commit, we add the `source_nodes` table. It points to entries in
the `nodes` table. This table will store one entry per protocol version
that we are announcing a node_announcement on.

With this commit, we can run the TestSourceNode unit test against our
SQL backends.
2025-05-27 18:36:59 +02:00
Elle Mouton
86d48390ca sqldb+graph/db: implement SQLStore.NodeUpdatesInHorizon
In this commit we add the necessary SQL queries and then implement the
SQLStore's NodeUpdatesInHorizon method. This lets us run the
TestNodeUpdatesInHorizon unit tests against SQL backends.
2025-05-27 18:36:59 +02:00
Elle Mouton
d1f7cce68b sqldb+graphdb: SQL methods for Node CRUD
In this commit, we add the various sqlc queries that we need in order
to implement the following V1Store methods:

- AddLightningNode
- FetchLightningNode
- HasLightningNode
- AddrsForNode
- DeleteLightningNode
- FetchNodeFeatures

These are implemented by SQLStore which then lets us use the SQLStore
backend for the following unit tests:

- TestNodeInsertionAndDeletion
- TestLightningNodePersistence
2025-05-27 18:36:59 +02:00