It can happen that we are handling 2 of the same node announcements in
the same batch transaction. In that case, our `UpsertNode` conflict
assertion may fail. We need to handle this gracefully.
In this commit, the lnwire.NodeAnnouncement2 type is defined. This will
be used to represent the `node_announcement_2` message used in the
Gossip 2 (1.75) protocol.
We leave a TODO that should be addressed after a discussion at the spec
meeting. For now, having the incorrect TLV type is not a problem since
this ChannelUpdate2 type is not used in production.
In preparation for adding a NodeAnnouncement2 struct along with a
NodeAnnouncement interface, this commit renames the existing
NodeAnnouncment struct to NodeAnnouncement1.
Add a new OutPoint type that wraps wire.OutPoint and provides TLV
encoding/decoding capabilities through the tlv.RecordProducer interface.
This enables OutPoint to be used in TLV streams of messages.
In this commit, we update the mockChannelGraphTimeSeries to implement
the new iterator-based UpdatesInHorizon interface. The mock maintains
its existing behavior of receiving messages through a channel and
returning them to the caller, but now wraps this in an iterator
function.
The implementation creates an iterator that pulls the entire message
slice from the mock's response channel, then yields each message
individually. This preserves the test semantics while conforming to the
new interface, ensuring all existing tests continue to pass without
modification.
In this commit, we update ApplyGossipFilter to leverage the new
iterator-based UpdatesInHorizon method. The key innovation here is using
iter.Pull2 to create a pull-based iterator that allows us to check if
any updates exist before launching the background goroutine.
This approach provides several benefits over the previous implementation.
First, we avoid the overhead of launching a goroutine when there are no
updates to send, which was previously unavoidable without materializing
the entire result set. Second, we maintain lazy loading throughout the
sending process, only pulling messages from the database as they're
needed for transmission.
The implementation uses Pull2 to peek at the first message, determining
whether to proceed with sending updates. If updates exist, ownership of
the iterator is transferred to the goroutine, which continues pulling
and sending messages until exhausted. This design ensures memory usage
remains bounded regardless of the number of updates being synchronized.
In this commit, we complete the iterator conversion work started in PR
10128 by threading the iterator pattern through to the higher-level
UpdatesInHorizon method. This change converts the method from returning
a fully materialized slice of messages to returning a lazy iterator that
yields messages on demand.
The new signature uses iter.Seq2 to allow error propagation during
iteration, eliminating the need for a separate error return value. This
approach enables callers to handle errors as they occur during iteration
rather than failing upfront.
The implementation now lazily processes channel and node updates,
yielding them as they're generated rather than accumulating them in
memory. This maintains the same ordering guarantees (channels before
nodes) while significantly reducing memory pressure when dealing with
large update sets during gossip synchronization.
In this commit, we update all callers of NodeUpdatesInHorizon and
ChanUpdatesInHorizon to use the new iterator-based APIs. The changes
use fn.Collect to maintain existing behavior while benefiting from the
memory efficiency of iterators when possible.
In this commit, we update the SQL store implementation to support the
new iterator-based API for ChanUpdatesInHorizon. This includes adding
SQL query pagination support and helper functions for efficient batch
processing.
The SQL implementation uses cursor-based pagination with configurable
batch sizes, allowing efficient iteration over large result sets without
loading everything into memory. The query is optimized to use indexes
effectively and minimize database round trips.
New SQL query GetChannelsByPolicyLastUpdateRange is updated to support:
- Cursor-based pagination using (max_update_time, id) compound cursor
- Configurable batch sizes via MaxResults parameter
- Efficient batch caching with updateChanCacheBatch helper
In this commit, we update the SQL store implementation to support the
new iterator-based API for NodeUpdatesInHorizon. This includes adding a
new SQL query that supports efficient pagination through result sets.
The SQL implementation uses cursor-based pagination with configurable
batch sizes, allowing efficient iteration over large result sets without
loading everything into memory. The query is optimized to use indexes
effectively and minimize database round trips.
New SQL query GetNodesByLastUpdateRange is updated to support:
* Cursor-based pagination using (last_update, pub_key) compound cursor
* Optional filtering for public nodes only
* Configurable batch sizes via MaxResults parameter
In this commit, we refactor the ChanUpdatesInHorizon method to return
an iterator instead of a slice. This change significantly reduces
memory usage when dealing with large result sets by allowing callers to
process items incrementally rather than loading everything into memory
at once.
In this commit, we refactor the NodeUpdatesInHorizon method to return
an iterator instead of a slice. This change significantly reduces
memory usage when dealing with large result sets by allowing callers to
process items incrementally rather than loading everything into memory
at once.
The new implementation uses Go 1.23's iter.Seq type to provide a
standard iterator interface. The method now supports configurable batch
sizes through functional options, allowing fine-tuned control over
memory usage and performance characteristics.
Rather than reading all the entries from disk into memory (before this
commit, we did consult the cache for most entries, skipping the disk
hits), we now expose a chunked iterator instead.
We also make the process of filtering out public nodes first class. This
saves many newly created db transactions later.
In this commit, we introduce a new options pattern for configuring
iterator behavior in the graph database. This includes configuration
for batch sizes when iterating over channel and node updates, as well
as an option to filter for public nodes only.
The new functional options pattern allows callers to customize iterator
behavior without breaking existing APIs. Default batch sizes are set to
1000 entries for both channel and node updates, which provides a good
balance between memory usage and performance.
In this commit, we introduce a new utility function `Collect` to the fn
package. This function drains all elements from an iterator and returns
them as a slice. This is particularly useful when transitioning from
iterator-based APIs to code that expects slices, allowing for gradual
migration to the new iterator patterns.
The fn module's go.mod is also updated to require Go 1.23, which is
necessary for the built-in iter.Seq type support.
The replace directive will be removed once the fn package changes are
merged and a new version is tagged.
Add the new RPC method that looks up the base scid for a given alias.
Given the previous stepping stones this commit is fairly simple, we just
call into the alias manager and return the lookup result.
We add an extra option to the AddLocalAlias method which only controls
whether we store a reverse lookup from the alias back to the base scid
it corresponds to. The previous flag "gossip" is still maintained, and
in a way supercedes the new flag (it will also store the base scid
lookup even if the base lookup flag isn't set). The only call that sets
this option is the XAddLocalChanAlias RPC endpoint, where we want to
make sure that a reverse lookup is stored in the alias manager in order
to later expose it via the new RPC method.
In this commit, we add some docs that explain how to use tools like
heap escape analysis and memory profiling to fully eliminate allocations
in a sample program.
This guide is meant to help devs/contributors use Go's excellent perf
tools to zoom in on an optimization problem.
In this commit, we eliminate the final allocation that takes place when
we write out messages. Once again this was escaping to the heap, so we
make it an attribute on the Machine struct, which allows pure static
allocation.
```
goos: darwin
goarch: arm64
pkg: github.com/lightningnetwork/lnd/brontide
cpu: Apple M4 Max
BenchmarkWriteMessage-16 25840 46376 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25646 46672 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25874 46391 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25512 46427 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25760 46309 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25789 46520 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25602 46619 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25766 46464 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25820 46487 ns/op 2 B/op 0 allocs/op
BenchmarkWriteMessage-16 25634 46553 ns/op 2 B/op 0 allocs/op
PASS
ok github.com/lightningnetwork/lnd/brontide 16.907s
```
In this commit, we update the itest to check for precedence of the
config addresses over the persisted addresses.
We did not check for address persistence after restart (without the
extraArgs `--externalip`) because in `lntest/node/config.goL300`,
`GenArg` by default sets the `--externalip` flag, which makes
the config addrs to always take precedence over the persisted
addrs, for the tests.
