From 83024585bb133f8149ca62175ff4ae6729ae2280 Mon Sep 17 00:00:00 2001 From: yyforyongyu Date: Fri, 19 Apr 2024 23:05:57 +0800 Subject: [PATCH] sweep: add README --- sweep/README.md | 213 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 213 insertions(+) create mode 100644 sweep/README.md diff --git a/sweep/README.md b/sweep/README.md new file mode 100644 index 000000000..487ebdcdd --- /dev/null +++ b/sweep/README.md @@ -0,0 +1,213 @@ +# Sweep + +`sweep` is a subservice that handles sweeping UTXOs back to `lnd`'s wallet. Its +main purpose is to sweep back the outputs resulting from a force close +transaction, although users can also call `BumpFee` to feed new unconfirmed +inputs to be handled by the sweeper. + +In order to sweep economically, the sweeper needs to understand the time +sensitivity and max fees that can be used when sweeping the inputs. This means +each input must come with a deadline and a fee budget, which can be set via the +RPC request or the config, otherwise the default values will be used. Once +offered to the sweeper, when a new block arrives, inputs with the same deadline +will be batched into a single sweeping transaction to minimize the cost. + +The sweeper will publish this transaction and monitor it for potential fee +bumping, a process that won’t exit until the sweeping transaction is confirmed, +or the specified budget has been used up. + +## Understanding Budget and Deadline + +There are two questions when spending a UTXO - how much fees to pay and what +the confirmation target is, which gives us the concepts of budget and deadline. +This is especially important when sweeping the outputs of a force close +transaction - some of the outputs are time-sensitive, and may result in fund +loss if not confirmed in time. On the other hand, we don’t want to pay more +than what we can get back - if a sweeping transaction spends more than what is +meant to be swept, we are losing money due to fees. + +To properly handle the case, the concept `budget` and `deadline` have been +introduced to `lnd` since `v0.18.0` - for each new sweeping request, the +sweeper requires the caller to specify a deadline and a budget so it can make +economic decisions. A fee function is then created based on the budget and +deadline, which proposes a fee rate to use for the sweeping transaction. When a +new block arrives, unless the transaction is confirmed or the budget is used +up, the sweeper will perform a fee bump on it via RBF. + +## Package Structure + +On a high level, a UTXO is offered to the sweeper via `SweepInput`. The sweeper +keeps track of the pending inputs. When a new block arrives, it asks the +`UtxoAggregator` to group all the pending inputs into batches via +`ClusterInputs`. Each batch is an `InputSet`, and is sent to the `Bumper`. The +`Bumper` creates a `FeeFunction` and a sweeping transaction using the +`InputSet`, and monitors its confirmation status. Every time it's not confirmed +when a new block arrives, the `Bumper` will perform an RBF by calling +`IncreaseFeeRate` on the `FeeFunction`. + +```mermaid +flowchart LR + subgraph SweepInput + UTXO1-->sweeper + UTXO2-->sweeper + UTXO3-->sweeper + UTXO["..."]-->sweeper + sweeper + end + + subgraph ClusterInputs + sweeper-->UtxoAggregator + UtxoAggregator-->InputSet1 + UtxoAggregator-->InputSet2 + UtxoAggregator-->InputSet["..."] + end + + subgraph Broadcast + InputSet1-->Bumper + InputSet2-->Bumper + InputSet-->Bumper + end + + subgraph IncreaseFeeRate + FeeFunction-->Bumper + end + + block["new block"] ==> ClusterInputs +``` + +#### `UtxoAggregator` and `InputSet` + +`UtxoAggregator` is an interface that handles the batching of inputs. +`BudgetAggregator` implements this interface by grouping inputs with the same +deadline together. Inputs with the same deadline express the same time +sensitivity so it makes sense to sweep them in the same transaction. Once +grouped, inputs in each batch are sorted based on their budgets. The only +exception is inputs with `ExclusiveGroup` flag set, which will be swept alone. + +Once the batching is finished, an `InputSet` is returned, which is an interface +used to decide whether a wallet UTXO is needed or not when creating the +sweeping transaction. `BudgetInputSet` implements this interface by checking +the sum of the output values from these inputs against the sum of their +budgets - if the total budget cannot be covered, one or more wallet UTXOs are +needed. + +For instance, when anchor output is swept to perform a CPFP, one or more wallet +UTXOs are likely to be used to meet the specified budget, which is also the +case when sweeping second-level HTLC transactions. However, if the sweeping +transaction also contains other to-be-swept inputs, a wallet UTXO is no longer +needed if their values can cover the total budget. + +#### `Bumper` + +`Bumper` is a transaction creator, publisher, and monitor that works on an +`InputSet`. Once a sweeping transaction is created using the `InputSet`, the +`Bumper` will monitor its confirmation status and attempt an RBF if the +transaction is not confirmed in the next block. It relies on the `FeeFunction` +to determine the new fee rate every block, and this new fee rate may or may not +meet the BIP 125 fee requirements - in that case, the `Bumper` will try to +perform an RBF again in the coming blocks. + +`TxPublisher` implements the `Bumper` interface. When a transaction is created +for the first time, unless its budget has been used up, `TxPublisher` will +guarantee that the initial publish meets the RBF requirements. + +#### `FeeFunction` + +`FeeFunction` is an interface that specifies a function over a starting fee +rate, an ending fee rate, and a width (the deadline delta). It's used by the +`Bumper` to suggest a new fee rate for bumping the sweeping transaction. + +`LinearFeeFunction` implements this interface using a linear function - it +calculates a fee rate delta using `(ending_fee_rate - starting_fee_rate) / +deadline`, and increases the fee rate by this delta value everytime a new block +arrives. Once the deadline is passed, `LinearFeeFunction` will cap its +returning fee rate at the ending fee rate. + +The starting fee rate is the estimated fee rate from the fee estimator, which +is the result from calling `estimatesmartfee`(`bitcoind`), +`estimatefee`(`btcd`), or `feeurl` depending on the config. This fee estimator +is called using the deadline as the conf target, and the returned fee rate is +used as the starting fee rate. This behavior can be overridden by setting the +`--sat_per_vbyte` via `bumpfee` cli when fee bumping a specific input, which +allows users to bypass the fee estimation and set the starting fee rate +directly. + +The ending fee rate is the value from dividing the budget by the size of the +sweeping transaction, and capped at the `--sweeper.maxfeerate`. The ending fee +rate can be overridden by setting the `--budget` via `bumpfee` cli. + +For instance, suppose `lnd` is using `bitcoind` as its fee estimator, and an +input with a deadline of 1000 blocks and a budget of 200,000 sats is being +swept in a transaction that has a size of 500 vbytes, the fee function will be +initialized with: + +- a starting fee rate of 10 sat/vB, which is the result from calling + `estimatesmartfee 1000`. +- an ending fee rate of 400 sat/vB, which is the result of `200,000/500`. +- a fee rate delta of 390 sat/kvB, which is the result of `(400 - 10) / 500 * + 1000`. + +## Sweeping Outputs from a Force Close Transaction + +A force close transaction may have the following outputs: + +- Commit outputs, which are the `to_local` and `to_remote` outputs. +- HTLC outputs, which are the `incoming_htlc` and `outgoing_htlc` outputs. +- Anchor outputs, which are the local and remote anchor outputs. + +#### Sweeping Commit Outputs + +The only output we can spend is the `to_local` output. Because it can only be +spent using our signature, there’s no time pressure here. By default, the +sweeper will use a deadline of 1008 blocks as the confirmation target for +non-time-sensitive outputs. To overwrite the default, users can specify a +value using the config `--sweeper.nodeadlineconftarget`. + +To specify the budget, users can use `--sweeper.budget.tolocal` to set the max +allowed fees in sats, or use `--sweeper.budget.tolocalratio` to set a +proportion of the `to_local` value to be used as the budget. + +#### Sweeping HTLC Outputs + +When facing a local force close transaction, HTLCs are spent in a two-stage +setup - the first stage is to spend the outputs using pre-signed HTLC +success/timeout transactions, the second stage is to spend the outputs from +these success/timeout transactions. All these outputs are automatically handled +by `lnd`. In specific, +- For an incoming HTLC in stage one, the deadline is specified using its CLTV + from the timeout path. This output is time-sensitive. +- For an outgoing HTLC in stage one, the deadline is derived from its + corresponding incoming HTLC’s CLTV. This output is time-sensitive. +- For both incoming and outgoing HTLCs in stage two, because they can only be + spent by us, there is no time pressure to confirm them under a deadline. + +When facing a remote force close transaction, HTLCs can be directly spent from +the commitment transaction, and both incoming and outgoing HTLCs are +time-sensitive. + +By default, `lnd` will use 50% of the HTLC value as its budget. To customize +it, users can specify `--sweeper.budget.deadlinehtlc` and +`--sweeper.budget.deadlinehtlcratio` for time-sensitive HTLCs, and +`--sweeper.budget.nodeadlinehtlc` and `--sweeper.budget.nodeadlinehtlcratio` +for non-time-sensitive sweeps. + +#### Sweeping Anchor Outputs + +An anchor output is a special output that functions as “anchor” to speed up the +unconfirmed force closing transaction via CPFP. If the force close transaction +doesn't contain any HTLCs, the anchor output is generally uneconomical to sweep +and will be ignored. However, if the force close transaction does contain +time-sensitive outputs (HTLCs), the anchor output will be swept to CPFP the +transaction and accelerate the force close process. + +For CPFP-purpose anchor sweeping, the deadline is the closest deadline value of +all the HTLCs on the force close transaction. The budget, however, cannot be a +ratio of the anchor output because the value is too small to contribute +meaningful fees (330 sats). Since its purpose is to accelerate the force close +transaction so the time-sensitive outputs can be swept, the budget is actually +drawn from what we call “value under protection”, which is the sum of all HTLC +outputs minus the sum of their budgets. By default, 50% of this value is used +as the budget, to customize it, either use +`--sweeper.budget.anchorcpfp` to specify sats, or use +`--sweeper.budget.anchorcpfpratio` to specify a ratio. +