Commit Graph

133 Commits

Author SHA1 Message Date
Elle Mouton
eb67757a07 graph/db: gracefully handle duplicate channel policy announcements 2025-10-06 11:37:39 +02:00
Elle Mouton
abb9654856 graph/db: gracefully handle duplicate node announcements
It can happen that we are handling 2 of the same node announcements in
the same batch transaction. In that case, our `UpsertNode` conflict
assertion may fail. We need to handle this gracefully.
2025-10-06 11:28:33 +02:00
Elle Mouton
b8abe130a5 multi: rename lnwire.NodeAnnouncement
In preparation for adding a NodeAnnouncement2 struct along with a
NodeAnnouncement interface, this commit renames the existing
NodeAnnouncment struct to NodeAnnouncement1.
2025-10-01 13:13:32 +02:00
Olaoluwa Osuntokun
f3a8fd842d graph+discovery: update graph/db gossip backlog interfaces to use iter.Seq2
This lets us emit a rich error if things fail when first creating the
iterator, or if any of the yield attempts fail.
2025-09-26 17:01:11 -07:00
Olaoluwa Osuntokun
32528afa0a graph/db: add tests for iterator implementations 2025-09-26 16:58:41 -07:00
Olaoluwa Osuntokun
069888b51a graph/db: convert ChanUpdatesInHorizon to use iterators
In this commit, we refactor the ChanUpdatesInHorizon method to return
an iterator instead of a slice. This change significantly reduces
memory usage when dealing with large result sets by allowing callers to
process items incrementally rather than loading everything into memory
at once.
2025-09-26 16:57:11 -07:00
Olaoluwa Osuntokun
1d6d54e5db graph/db: convert NodeUpdatesInHorizon to use iterators
In this commit, we refactor the NodeUpdatesInHorizon method to return
an iterator instead of a slice. This change significantly reduces
memory usage when dealing with large result sets by allowing callers to
process items incrementally rather than loading everything into memory
at once.

The new implementation uses Go 1.23's iter.Seq type to provide a
standard iterator interface. The method now supports configurable batch
sizes through functional options, allowing fine-tuned control over
memory usage and performance characteristics.

Rather than reading all the entries from disk into memory (before this
commit, we did consult the cache for most entries, skipping the disk
hits), we now expose a chunked iterator instead.

We also make the process of filtering out public nodes first class. This
saves many newly created db transactions later.
2025-09-26 16:56:41 -07:00
Elle Mouton
6e98b336f6 graph/db+channeldb: rename test helper 2025-09-03 10:14:35 +02:00
Elle Mouton
330f697937 graph: rename HasLightningNode
to HasNode
2025-09-03 10:14:35 +02:00
Elle Mouton
7a1b548e07 graph/db: rename DeleteLightningNode
to DeleteNode
2025-09-03 10:14:35 +02:00
Elle Mouton
cd3bd05810 multi: rename FetchLightningNode
to FetchNode
2025-09-03 10:14:35 +02:00
Elle Mouton
060219780b multi: rename AddLightningNode methods
to AddNode
2025-09-03 10:14:35 +02:00
Elle Mouton
c663a557c4 multi: rename models.LightningNode to models.Node 2025-09-03 10:14:35 +02:00
Elle Mouton
6ef80db746 graph/db+chanbackup: dns add encoding/decoding for persistence 2025-09-03 01:11:35 +00:00
Elle Mouton
4bd2bbca27 graph/db: update test opaque addr to be valid
The first byte of an opaque addr must be one that we dont understand
yet. We do this update in preparation for doing an on-the-fly parse of
persisted opaque addrs to see if they contain addrs that we now support.
For this to work, the first byte cant be 0x01 since this maps to a known
address.
2025-09-03 01:11:35 +00:00
Boris Nagaev
dee8ad3754 multi: context.Background() -> t.Context()
Use the new feature of Go 1.24, fix linter warnings.

This change was produced by:
 - running golangci-lint run --fix
 - sed 's/context.Background/t.Context/' -i `git grep -l context.Background | grep test.go`
 - manually fixing broken tests
 - itest, lntest: use ht.Context() where ht or hn is available
 - in HarnessNode.Stop() we keep using context.Background(), because it is
   called from a cleanup handler in which t.Context() is canceled already.
2025-08-30 14:13:44 -03:00
Elle Mouton
8aa8c7cc42 multi: remove the NodeRTx interface type 2025-08-06 09:56:21 +02:00
Elle Mouton
5727bfa688 graph/db: let ForEachNodeCached maybe fetch node addresses
Here we adjust the ForEachNodeCached graph DB method to pass in a node's
addresses into the provided call-back if requested. This will allow us
to improve the performance of node/channel iteration in the autopilot
subserver.
2025-08-06 09:50:54 +02:00
Elle Mouton
1a60edbd33 make+graph/db: add -short helper and skip tests in short mode
In this commit, we add a makefile helper that can be used to add the
"-short" flag to the "go test" command when running unit tests via "make
unit ...".

Tests can then be expanded with a `testing.Short()` check to potentially
skip the test in short mode. This is useful for if a dev wants to
quickly run most of the tests in a package but would like to opt out of
running the longer form tests such as stress tests or rapid generation
tests.

This commit adds this check to two graph/db tests.
2025-08-05 06:34:38 +02:00
Elle Mouton
83f6dc4983 multi: reset for ForEachNodeDirectedChannel 2025-07-15 11:25:10 +02:00
Elle Mouton
e17fd58a1d graph/db: pass reset to ForEachNodeCacheable 2025-07-15 11:25:09 +02:00
Elle Mouton
21ac7ef6f4 multi: add reset to ForEachNode 2025-07-15 11:25:09 +02:00
Elle Mouton
c32bf642d2 multi: pass reset to ForEachNodeCached 2025-07-15 11:23:27 +02:00
Elle Mouton
e5fbca8299 multi: let ForEachNodeChannel take a reset param 2025-07-15 11:23:27 +02:00
Elle Mouton
16126d7703 graph/db: expose reset in ForEachSourceNodeChannel 2025-07-15 11:23:27 +02:00
Elle Mouton
88261834e5 graph/db: expose reset in ForEachChannel
Add a `reset` call-back param to the V1Store ForEachChannel method.
2025-07-15 11:23:26 +02:00
Elle Mouton
36f8a66b5a graph/db: remove temporary test skip
The skip is no longer needed.
2025-07-10 14:54:04 +02:00
Oliver Gugger
8ee661f18f Merge pull request #10043 from ellemouton/ctx3
multi: add context.Context param to more graphdb.V1Store methods
2025-07-10 13:40:51 +02:00
Elle Mouton
3d5e852c8c graph/db: let ForEachNodeCached take a context 2025-07-08 15:11:02 +02:00
Elle Mouton
1e804a3d5f graph/db: let ForEachNode take a context 2025-07-08 15:11:02 +02:00
Elle Mouton
f05ef2db97 graph/db: let ForEachNodeCacheable take a context 2025-07-08 15:10:30 +02:00
Elle Mouton
046e106a17 graph/db: let ForEachChannel take a context 2025-07-08 15:10:30 +02:00
Elle Mouton
f8d3a3f3ba graph/db: let ForEachNodeChannel take a context 2025-07-08 15:10:30 +02:00
Elle Mouton
67c0f54e13 graph/db+server: let ForEachSourceNodeChannel take a context 2025-07-08 15:10:30 +02:00
Elle Mouton
61c1d258c3 graph/db: factor out helper code
Here we factor out some of the crud code in TestEdgePolicyMissingMaxHtcl
so that we can re-use it later on.
2025-07-08 14:30:14 +02:00
Elle Mouton
2f2845dfc0 refactor+multi: use *lnwire.FeatureVector for ChannelEdgeInfo features
In this commit, we move the serialisation details of a channel's
features to the DB layer and change the `models` field to instead use a
more useful `*lnwire.FeatureVector` type.

This makes the features easier to work with and moves the serialisation
to where it is actually used.
2025-07-01 17:02:07 +02:00
Oliver Gugger
500808fadd Merge pull request #10010 from ellemouton/sqlGraphUpdates
graph/db: various misc updates
2025-07-01 12:40:50 +02:00
Elle Mouton
e4137a3f14 graph/db: fix ChanUpdate message/channel flag bug
Here we start using the newly added message_flags and channel_flags
columns of the channel_policies table. The test added previoulsy to
demonstrate the bug is now updated to show that the bug has been fixed.
2025-07-01 10:12:54 +02:00
Elle Mouton
fa32f97c05 graph/db: unit test to demonstrate SQL chan policies bug
This commit adds a new graph store unit test that demonstrates that
there is currently a bug in the SQL version of the graph store where
unknown message or channel flags in a channel_update are not persisted
correctly. This will be fixed in an upcoming commit.
2025-07-01 10:12:53 +02:00
Elle Mouton
64fc92abe7 graph/db: update nextUpdateTime in tests
Let it return time.Time instead. Also, make use of the helper in a few
more places.
2025-07-01 10:04:03 +02:00
Elle Mouton
714e528a3a graph/db: fix address fetching error 2025-06-30 18:40:25 +02:00
Elle Mouton
272a2db16c graph/db: demonstrate SQL node addresses fetch bug
This commit demonstrates that currently the Addresses field of a
LightningNode is not populated correctly if the node has no addresses if
the SQL graph store is being used. This will be fixed in an upcoming
commit.
2025-06-30 18:40:21 +02:00
Elle Mouton
64110cf459 graph/db: let FetchChannelEdgesByID behave as promised
The comment of FetchChannelEdgesByID says that if the ErrZombieEdge
error is returned, then the ChannelEdgeInfo return parameter will also
still be populated and returned (ie, wont be nil). This commit updates
the SQLStore implementation of FetchChannelEdgesByID to do this. This is
needed to prevent nil dereference panics at any call-sites that depend
on the method working as it describes.
2025-06-30 11:04:35 +02:00
Elle Mouton
cc0be0bca2 graph/db: let IsZombieEdge return an error 2025-06-26 10:19:51 +02:00
Elle Mouton
3ccaacd16f multi: rename MakeTestGraphNew to MakeTestGraph 2025-06-26 10:12:30 +02:00
Elle Mouton
c06036d2b4 multi: use MakeTestGraphNew for all remaining unit tests
And delete the old MakeTestGraph function.

Here we need to update some of our unit tests to ensure that any updates
to existing nodes or channel policies have newer timestamps. This is so
that we don't violate our SQL DB constraints that prevent updates to a
node or channel policy record if the new update is not newer (has a
newer LastUpdate time) than the currently presisted record).
2025-06-26 10:12:27 +02:00
Elle Mouton
933ab3c6b7 graph/db+sqldb: impl PutClosedScid and IsClosedScid 2025-06-25 13:26:52 +02:00
Elle Mouton
d46552f5ad graph/db+sqldb: implement AddEdgeProof
And run `TestAddEdgeProof` against the SQL backends.
2025-06-25 13:26:51 +02:00
Elle Mouton
e875183c4f sqldb+graph/db: impl DisconnectBlockAtHeight
Which lets us run `TestDisconnectBlockAtHeight` and
`TestStressTestChannelGraphAPI` against our SQL backends.
2025-06-25 11:22:03 +02:00
Elle Mouton
9dd0361ed0 graph/db+sqldb: impl PruneGraph, PruneTip, ChannelView
Which lets us run `TestGraphPruning` and `TestBatchedAddChannelEdge`
against our SQL backends.
2025-06-25 11:22:00 +02:00