In this commit, we complete the integration of the asynchronous
timestamp range queue by modifying ProcessRemoteAnnouncement to use
the new queuing mechanism instead of calling ApplyGossipFilter
synchronously.
This change ensures that when a peer sends a GossipTimestampRange
message, it is queued for asynchronous processing rather than
blocking the gossiper's main message processing loop. The modification
prevents the peer's readHandler from blocking on potentially slow
gossip filter operations, maintaining connection stability during
periods of high synchronization activity.
If the queue is full when attempting to enqueue a message, we log
a warning but return success to prevent peer disconnection. This
design choice prioritizes connection stability over guaranteed
delivery of every gossip filter request, which is acceptable since
peers can always resend timestamp range messages if needed.
In this commit, we introduce an asynchronous processing queue for
GossipTimestampRange messages in the GossipSyncer. This change addresses
a critical issue where the gossiper could block indefinitely when
processing timestamp range messages during periods of high load.
Previously, when a peer sent a GossipTimestampRange message, the
gossiper would synchronously call ApplyGossipFilter, which could block
on semaphore acquisition, database queries, and rate limiting. This
synchronous processing created a bottleneck where the entire peer
message processing pipeline would stall, potentially causing timeouts
and disconnections.
The new design adds a timestampRangeQueue channel with a capacity of 1
message and a dedicated goroutine for processing these messages
asynchronously. This follows the established pattern used for other
message types in the syncer. When the queue is full, we drop messages
and log a warning rather than blocking indefinitely, providing graceful
degradation under extreme load conditions.
In this commit, we make the gossip filter semaphore capacity configurable
through a new FilterConcurrency field. This change allows node operators
to tune the number of concurrent gossip filter applications based on
their node's resources and network position.
The previous hard-coded limit of 5 concurrent filter applications could
become a bottleneck when multiple peers attempt to synchronize
simultaneously. By making this value configurable via the new
gossip.filter-concurrency option, operators can increase this limit
for better performance on well-resourced nodes or maintain conservative
values on resource-constrained systems.
We keep the default value at 5 to maintain backward compatibility and
avoid unexpected resource usage increases for existing deployments. The
sample configuration file is updated to document this new option.
This commit is a pure refactor. Here all we are doing is removing the
old `pageSize` constant used in the SQLStore code and instead using the
value provided in the QueryConfig struct.
We rename this helper along the config types & helper types for it
because the word "page" is used more often in the context of paging
through results using an offset and limit whereas this helper is
specifically used to split up the slice in queries of the form
"WHERE x in []slice". We do this rename so that there is mimimal
confusion in contexts where we use batching along with actual paging.
The config struct is also renamed to QueryConfig in preparation for it
holding more config options.
Here, we add a new `buildNodeWithBatchData` helper method that can be
used to construct a `models.LightningNode` object using pre-fetched
batch data. The existing `buildNode` method is then adjusted to use this
new helper.
In this commit, we add the queries that will be needed to batch-fetch
the data of a set of nodes. The logic for using these new queries is
also added but not used yet.
In this commit, we remove the LEFT JOIN query that was used for fetching
a nodes addresses. The reason it was used before was to ensure that we'd
get an empty address list if the node did exist but had no addresses.
This was for the purposes of the `AddrsForNode` method since it needs to
return false/true to indicate if the given node exists.
This commit adds a helper test that is similar to TestPopulateDBs except
for the fact that it uses the MigrateGraphToSQL function directly to
migrate a local kvdb-sql graph to a native SQL one.
This is a prep commit that just adds all the boilerplate connection
logic for connecting to various graph DBs. These will be used in
upcoming commits which will add tests and benchmarks.
This script compares a release branch against a source branch
(e.g. master) to verify that all cherry-picked commits are
unmodified. It first attempts fast matching using normalized
patch hashes.
If no exact match is found, it falls back to a fuzzy matching
mechanism:
- Filters source commits by matching author and commit subject
- Compares normalized diffs using diff -u
- Selects the closest match based on line difference count
Useful for verifying cherry-picks or rebased commits during
release processes. Supports scan and compare limits for
performance.
Added itest "reorg_notifications" which tests that RegisterSpendNtfn behaves as
expected during a reorg. A reorg notification is produced after a reorg affects
the block which has produced a spending notification for this registration.
