sweep: add README

sweep/README.md

@@ -0,0 +1,213 @@
+# Sweep
+
+`sweep` is a subservice that handles sweeping UTXOs back to `lnd`'s wallet. Its
+main purpose is to sweep back the outputs resulting from a force close
+transaction, although users can also call `BumpFee` to feed new unconfirmed
+inputs to be handled by the sweeper.
+
+In order to sweep economically, the sweeper needs to understand the time
+sensitivity and max fees that can be used when sweeping the inputs. This means
+each input must come with a deadline and a fee budget, which can be set via the
+RPC request or the config, otherwise the default values will be used. Once
+offered to the sweeper, when a new block arrives, inputs with the same deadline
+will be batched into a single sweeping transaction to minimize the cost.
+
+The sweeper will publish this transaction and monitor it for potential fee
+bumping, a process that won’t exit until the sweeping transaction is confirmed,
+or the specified budget has been used up.
+
+## Understanding Budget and Deadline
+
+There are two questions when spending a UTXO - how much fees to pay and what
+the confirmation target is, which gives us the concepts of budget and deadline.
+This is especially important when sweeping the outputs of a force close
+transaction - some of the outputs are time-sensitive, and may result in fund
+loss if not confirmed in time. On the other hand, we don’t want to pay more
+than what we can get back - if a sweeping transaction spends more than what is
+meant to be swept, we are losing money due to fees.
+
+To properly handle the case, the concept `budget` and `deadline` have been
+introduced to `lnd` since `v0.18.0` - for each new sweeping request, the
+sweeper requires the caller to specify a deadline and a budget so it can make
+economic decisions. A fee function is then created based on the budget and
+deadline, which proposes a fee rate to use for the sweeping transaction. When a
+new block arrives, unless the transaction is confirmed or the budget is used
+up, the sweeper will perform a fee bump on it via RBF.
+
+## Package Structure
+
+On a high level, a UTXO is offered to the sweeper via `SweepInput`. The sweeper
+keeps track of the pending inputs. When a new block arrives, it asks the
+`UtxoAggregator` to group all the pending inputs into batches via
+`ClusterInputs`. Each batch is an `InputSet`, and is sent to the `Bumper`. The
+`Bumper` creates a `FeeFunction` and a sweeping transaction using the
+`InputSet`, and monitors its confirmation status. Every time it's not confirmed
+when a new block arrives, the `Bumper` will perform an RBF by calling
+`IncreaseFeeRate` on the `FeeFunction`.
+
+```mermaid
+flowchart LR
+        subgraph SweepInput
+        UTXO1-->sweeper
+      UTXO2-->sweeper
+        UTXO3-->sweeper
+        UTXO["..."]-->sweeper
+        sweeper
+    end
+
+    subgraph ClusterInputs
+        sweeper-->UtxoAggregator
+      UtxoAggregator-->InputSet1
+        UtxoAggregator-->InputSet2
+        UtxoAggregator-->InputSet["..."]
+    end
+
+    subgraph Broadcast
+            InputSet1-->Bumper
+            InputSet2-->Bumper
+            InputSet-->Bumper
+    end
+
+    subgraph IncreaseFeeRate
+        FeeFunction-->Bumper
+    end
+
+        block["new block"] ==> ClusterInputs
+```
+
+#### `UtxoAggregator` and `InputSet`
+
+`UtxoAggregator` is an interface that handles the batching of inputs.
+`BudgetAggregator` implements this interface by grouping inputs with the same
+deadline together. Inputs with the same deadline express the same time
+sensitivity so it makes sense to sweep them in the same transaction. Once
+grouped, inputs in each batch are sorted based on their budgets. The only
+exception is inputs with `ExclusiveGroup` flag set, which will be swept alone.
+
+Once the batching is finished, an `InputSet` is returned, which is an interface
+used to decide whether a wallet UTXO is needed or not when creating the
+sweeping transaction. `BudgetInputSet` implements this interface by checking
+the sum of the output values from these inputs against the sum of their
+budgets - if the total budget cannot be covered, one or more wallet UTXOs are
+needed.
+
+For instance, when anchor output is swept to perform a CPFP, one or more wallet
+UTXOs are likely to be used to meet the specified budget, which is also the
+case when sweeping second-level HTLC transactions. However, if the sweeping
+transaction also contains other to-be-swept inputs, a wallet UTXO is no longer
+needed if their values can cover the total budget.
+
+#### `Bumper`
+
+`Bumper` is a transaction creator, publisher, and monitor that works on an
+`InputSet`. Once a sweeping transaction is created using the `InputSet`, the
+`Bumper` will monitor its confirmation status and attempt an RBF if the
+transaction is not confirmed in the next block. It relies on the `FeeFunction`
+to determine the new fee rate every block, and this new fee rate may or may not
+meet the BIP 125 fee requirements - in that case, the `Bumper` will try to
+perform an RBF again in the coming blocks.
+
+`TxPublisher` implements the `Bumper` interface. When a transaction is created
+for the first time, unless its budget has been used up, `TxPublisher` will
+guarantee that the initial publish meets the RBF requirements.
+
+#### `FeeFunction`
+
+`FeeFunction` is an interface that specifies a function over a starting fee
+rate, an ending fee rate, and a width (the deadline delta). It's used by the
+`Bumper` to suggest a new fee rate for bumping the sweeping transaction.
+
+`LinearFeeFunction` implements this interface using a linear function - it
+calculates a fee rate delta using `(ending_fee_rate - starting_fee_rate) /
+deadline`, and increases the fee rate by this delta value everytime a new block
+arrives. Once the deadline is passed, `LinearFeeFunction` will cap its
+returning fee rate at the ending fee rate.
+
+The starting fee rate is the estimated fee rate from the fee estimator, which
+is the result from calling `estimatesmartfee`(`bitcoind`),
+`estimatefee`(`btcd`), or `feeurl` depending on the config. This fee estimator
+is called using the deadline as the conf target, and the returned fee rate is
+used as the starting fee rate. This behavior can be overridden by setting the
+`--sat_per_vbyte` via `bumpfee` cli when fee bumping a specific input, which
+allows users to bypass the fee estimation and set the starting fee rate
+directly.
+
+The ending fee rate is the value from dividing the budget by the size of the
+sweeping transaction, and capped at the `--sweeper.maxfeerate`. The ending fee
+rate can be overridden by setting the `--budget` via `bumpfee` cli.
+
+For instance, suppose `lnd` is using `bitcoind` as its fee estimator, and an
+input with a deadline of 1000 blocks and a budget of 200,000 sats is being
+swept in a transaction that has a size of 500 vbytes, the fee function will be
+initialized with:
+
+- a starting fee rate of 10 sat/vB, which is the result from calling
+  `estimatesmartfee 1000`.
+- an ending fee rate of 400 sat/vB, which is the result of `200,000/500`.
+- a fee rate delta of 390 sat/kvB, which is the result of `(400 - 10) / 500 *
+  1000`.
+
+## Sweeping Outputs from a Force Close Transaction
+
+A force close transaction may have the following outputs:
+
+- Commit outputs, which are the  `to_local` and `to_remote` outputs.
+- HTLC outputs, which are the  `incoming_htlc` and `outgoing_htlc` outputs.
+- Anchor outputs, which are the local and remote anchor outputs.
+
+#### Sweeping Commit Outputs
+
+The only output we can spend is the `to_local` output. Because it can only be
+spent using our signature, there’s no time pressure here. By default, the
+sweeper will use a deadline of 1008 blocks as the confirmation target for
+non-time-sensitive outputs.  To overwrite the default, users can specify a
+value using the config `--sweeper.nodeadlineconftarget`.
+
+To specify the budget, users can use `--sweeper.budget.tolocal` to set the max
+allowed fees in sats, or use `--sweeper.budget.tolocalratio` to set a
+proportion of the `to_local` value to be used as the budget.
+
+#### Sweeping HTLC Outputs
+
+When facing a local force close transaction, HTLCs are spent in a two-stage
+setup - the first stage is to spend the outputs using pre-signed HTLC
+success/timeout transactions, the second stage is to spend the outputs from
+these success/timeout transactions. All these outputs are automatically handled
+by `lnd`. In specific,
+- For an incoming HTLC in stage one, the deadline is specified using its CLTV
+  from the timeout path. This output is time-sensitive.
+- For an outgoing HTLC in stage one, the deadline is derived from its
+  corresponding incoming HTLC’s CLTV. This output is time-sensitive.
+- For both incoming and outgoing HTLCs in stage two, because they can only be
+  spent by us, there is no time pressure to confirm them under a deadline.
+
+When facing a remote force close transaction, HTLCs can be directly spent from
+the commitment transaction, and both incoming and outgoing HTLCs are
+time-sensitive.
+
+By default, `lnd` will use 50% of the HTLC value as its budget. To customize
+it, users can specify `--sweeper.budget.deadlinehtlc` and
+`--sweeper.budget.deadlinehtlcratio` for time-sensitive HTLCs, and
+`--sweeper.budget.nodeadlinehtlc` and `--sweeper.budget.nodeadlinehtlcratio`
+for non-time-sensitive sweeps.
+
+#### Sweeping Anchor Outputs
+
+An anchor output is a special output that functions as “anchor” to speed up the
+unconfirmed force closing transaction via CPFP. If the force close transaction
+doesn't contain any HTLCs, the anchor output is generally uneconomical to sweep
+and will be ignored. However, if the force close transaction does contain
+time-sensitive outputs (HTLCs), the anchor output will be swept to CPFP the
+transaction and accelerate the force close process.
+
+For CPFP-purpose anchor sweeping, the deadline is the closest deadline value of
+all the HTLCs on the force close transaction. The budget, however, cannot be a
+ratio of the anchor output because the value is too small to contribute
+meaningful fees (330 sats). Since its purpose is to accelerate the force close
+transaction so the time-sensitive outputs can be swept, the budget is actually
+drawn from what we call “value under protection”, which is the sum of all HTLC
+outputs minus the sum of their budgets. By default, 50% of this value is used
+as the budget, to customize it, either use
+`--sweeper.budget.anchorcpfp` to specify sats, or use
+`--sweeper.budget.anchorcpfpratio` to specify a ratio.
+