In this document, we'll go over the various built-in mechanisms for recovering
funds from lnd due to any sort of data loss, or malfunction. Coins in lnd
can exist in one of two pools: on-chain or off-chain. On-chain funds are
outputs under the control of lnd that can be spent immediately, and without
any auxiliary data. Off-chain funds on the other hand exist within a 2-of-2
multi-sig output typically referred to as a payment channel. Depending on the
exact nature of operation of a given lnd node, one of these pools of funds
may be empty.
Fund recovery for lnd will require two pieces of data:
- Your 24-word cipher seed
- Your encrypted Static Channel Backup file (or the raw data)
If one is only attempting to recover on chain funds, then only the first item is required.
The SCB file is encrypted using a key derived from the user's seed. As a result, it cannot be used in isolation.
When a new lnd node is created, it's given a 24-word seed phrase, called an
cipher seed.
The two seed formats look similar, but the only commonality they share are
using the same default English dictionary. A valid seed phrase obtained over
the CLI lncli create command looks something like:
!!!YOU MUST WRITE DOWN THIS SEED TO BE ABLE TO RESTORE THE WALLET!!!
---------------BEGIN LND CIPHER SEED---------------
1. ability 2. noise 3. lift 4. document
5. certain 6. month 7. shoot 8. perfect
9. matrix 10. mango 11. excess 12. turkey
13. river 14. pitch 15. fluid 16. rack
17. drill 18. text 19. buddy 20. pool
21. soul 22. fatal 23. ship 24. jelly
---------------END LND CIPHER SEED-----------------
!!!YOU MUST WRITE DOWN THIS SEED TO BE ABLE TO RESTORE THE WALLET!!!
During the creation process, users are first prompted to enter a wallet password:
Input wallet password:
Confirm wallet password:
This password is used to encrypt the wallet on disk, which includes any derived master private keys or public key data.
Users can also optionally enter a second passphrase which we call the cipher seed passphrase:
Your cipher seed can optionally be encrypted.
Input your passphrase if you wish to encrypt it (or press enter to proceed without a cipher seed passphrase):
If specified, then this will be used to encrypt the cipher seed itself. The cipher seed format is unique in that the 24-word phrase is actually a ciphertext. As a result, there's no standard word list as any arbitrary encoding can be used. If a passphrase is specified, then the cipher seed you write down is actually an encryption of the entropy used to generate the BIP 32 root key for the wallet. Unlike a BIP 39 24-word phrase, the cipher seed is able to detect incorrect passphrase. BIP 39 on the other hand, will instead silently decrypt to a new (likely empty) wallet.
The initial entry point to trigger recovery of on-chain funds in the command
line is the lncli create command.
⛰ lncli create
Next, one can enter a new wallet password to encrypt any newly derived keys as a result of the recovery process.
Input wallet password:
Confirm wallet password:
Once a new wallet password has been obtained, the user will be prompted for their existing cipher seed:
Input your 24-word mnemonic separated by spaces: ability noise lift document certain month shoot perfect matrix mango excess turkey river pitch fluid rack drill text buddy pool soul fatal ship jelly
If a cipher seed passphrase was used when the seed was created, it MUST be entered now:
Input your cipher seed passphrase (press enter if your seed doesn't have a passphrase):
Finally, the user has an option to choose a recovery window:
Input an optional address look-ahead used to scan for used keys (default 2500):
The recovery window is a metric that the on-chain rescanner will use to
determine when all the "used" addresses have been found. If the recovery window
is two, lnd will fail to find funds in any addresses generated after the point
in which two consecutive addresses were generated but never used. If an lnd
on-chain wallet was extensively used, then users may want to increase the
default value.
If all the information provided was valid, then you'll be presented with the seed again:
!!!YOU MUST WRITE DOWN THIS SEED TO BE ABLE TO RESTORE THE WALLET!!!
---------------BEGIN LND CIPHER SEED---------------
1. ability 2. noise 3. lift 4. document
5. certain 6. month 7. shoot 8. perfect
9. matrix 10. mango 11. excess 12. turkey
13. river 14. pitch 15. fluid 16. rack
17. drill 18. text 19. buddy 20. pool
21. soul 22. fatal 23. ship 24. jelly
---------------END LND CIPHER SEED-----------------
!!!YOU MUST WRITE DOWN THIS SEED TO BE ABLE TO RESTORE THE WALLET!!!
lnd successfully initialized!
In lnd's logs, you should see something along the lines of (irrelevant lines skipped):
[INF] LNWL: Opened wallet
[INF] LTND: Wallet recovery mode enabled with address lookahead of 2500 addresses
[INF] LNWL: RECOVERY MODE ENABLED -- rescanning for used addresses with recovery_window=2500
[INF] CHBU: Updating backup file at test_lnd3/data/chain/bitcoin/simnet/channel.backup
[INF] CHBU: Swapping old multi backup file from test_lnd3/data/chain/bitcoin/simnet/temp-dont-use.backup to test_lnd3/data/chain/bitcoin/simnet/channel.backup
[INF] LNWL: Seed birthday surpassed, starting recovery of wallet from height=748 hash=3032830c812a4a6ea305d8ead13b52e9e69d6400ff3c997970b6f76fbc770920 with recovery-window=2500
[INF] LNWL: Scanning 1 blocks for recoverable addresses
[INF] LNWL: Recovered addresses from blocks 748-748
[INF] LNWL: Started rescan from block 3032830c812a4a6ea305d8ead13b52e9e69d6400ff3c997970b6f76fbc770920 (height 748) for 800 addresses
[INF] LNWL: Catching up block hashes to height 748, this might take a while
[INF] LNWL: Done catching up block hashes
[INF] LNWL: Finished rescan for 800 addresses (synced to block 3032830c812a4a6ea305d8ead13b52e9e69d6400ff3c997970b6f76fbc770920, height 748)
That final line indicates the rescan is complete! If not all funds have
appeared, then the user may need to repeat the process with a higher recovery
window. Depending on how old the wallet is (the cipher seed stores the wallet's
birthday!) and how many addresses were used, the rescan may take anywhere from
a few minutes to a few hours. To track the recovery progress, one can use the
command lncli getrecoveryinfo. When finished, the following is returned,
{
"recovery_mode": true,
"recovery_finished": true,
"progress": 1
}
If the rescan wasn't able to complete fully (lnd was shutdown for example),
then from lncli unlock, it's possible to restart the rescan from where it
left off with the --recovery-window argument:
⛰ lncli unlock --recovery_window=2500
Note that if this argument is not specified, then the wallet will not re-enter the recovery mode and may miss funds during the portion of the rescan.
The recovery methods described above assume a clean slate for a node, so
there's no existing UTXO or key data in the node's database. However, there're
times when an existing node may want to manually rescan the chain. We have
a tool for that! The tool is called
dropwtxmgr.
It can be installed with the following command:
⛰ go get -v -u github.com/btcsuite/btcwallet/cmd/dropwtxmgr
The dropwtxmgr tool will reset the best synced height of the wallet back to
its birthday, or genesis if the birthday isn't known (for some older wallets).
In order to run the tool, you must first shutdown lnd. Once lnd is
shutdown, the rescan can be initiated with the following commands:
⛰ cp $HOME/.lnd/data/chain/bitcoin/mainnet/wallet.db $HOME/wallet.db # Copy the existing databse just in case!
⛰ dropwtxmgr --db=$HOME/.lnd/data/chain/bitcoin/mainnet/wallet.db
Once the above command returns (if it hangs for a while, then lnd may not
actually be shutdown, so double check!), lnd can be restarted. After it's
restarted, then the wallet should being rescanning. An entry resembling the
following will show up in the logs once it's complete:
[INF] LNWL: Finished rescan for 800 addresses (synced to block 3032830c812a4a6ea305d8ead13b52e9e69d6400ff3c997970b6f76fbc770920, height 748)
After version v0.6-beta of lnd, the daemon now ships with a new feature
called Static Channel Backups (SCBs). We call these static as they only need
to be obtained once: when the channel is created. From there on, a backup is
good until the channel is closed. The backup contains all the information we
need to initiate the Data Loss Protection (DLP) feature in the protocol, which
ultimately leads to us recovering the funds from the channel on-chain. This
is a foolproof safe backup mechanism.
We say safe, as care has been taken to ensure that there are no foot guns in
this method of backing up channels, vs doing things like rsyncing or copying
the channel.db file periodically. Those methods can be dangerous as one never
knows if they have the latest state of a channel or not. Instead, we aim to
provide a simple, safe method to allow users to recover the settled funds in
their channels in the case of partial or complete data loss. The backups
themselves are encrypted using a key derived from the user's seed, this way we
protect privacy of the users channels in the back up state, and ensure that a
random node can't attempt to import another user's channels.
Given a valid SCB, the user will be able to recover funds that are fully settled within their channels. By "fully settled" we mean funds that are in the base commitment outputs, and not HTLCs. We can only restore these funds as right after the channel is created, as we have all the data required to make a backup, but lack information about the future HTLCs that the channel will process.
There are multiple ways of obtaining SCBs from lnd. The most commonly used
method will likely be via the channels.backup file that's stored on-disk
alongside the rest of the chain data. This is a special file that contains SCB
entries for all currently open channels. Each time a channel is opened or
closed, this file is updated on disk in a safe manner (atomic file rename). As
a result, unlike the channel.db file, it's always safe to copy this file
for backup at ones desired location. The default location on Linux is:
~/.lnd/data/chain/bitcoin/mainnet/channel.backup
An example of using file system level notification to copy the backup to a distinct volume/partition/drive can be found here.
Another way to obtain SCBS for all or a target channel is via the new
exportchanbackup lncli command:
⛰ lncli --network=simnet exportchanbackup --chan_point=29be6d259dc71ebdf0a3a0e83b240eda78f9023d8aeaae13c89250c7e59467d5:0
{
"chan_point": "29be6d259dc71ebdf0a3a0e83b240eda78f9023d8aeaae13c89250c7e59467d5:0",
"chan_backup": "02e7b423c8cf11038354732e9696caff9d5ac9720440f70a50ca2b9fcef5d873c8e64d53bdadfe208a86c96c7f31dc4eb370a02631bb02dce6611c435753a0c1f86c9f5b99006457f0dc7ee4a1c19e0d31a1036941d65717a50136c877d66ec80bb8f3e67cee8d9a5cb3f4081c3817cd830a8d0cf851c1f1e03fee35d790e42d98df5b24e07e6d9d9a46a16352e9b44ad412571c903a532017a5bc1ffe1369c123e1e17e1e4d52cc32329aa205d73d57f846389a6e446f612eeb2dcc346e4590f59a4c533f216ee44f09c1d2298b7d6c"
}
⛰ lncli --network=simnet exportchanbackup --all
{
"chan_points": [
"29be6d259dc71ebdf0a3a0e83b240eda78f9023d8aeaae13c89250c7e59467d5:0"
],
"multi_chan_backup": "fd73e992e5133aa085c8e45548e0189c411c8cfe42e902b0ee2dec528a18fb472c3375447868ffced0d4812125e4361d667b7e6a18b2357643e09bbe7e9110c6b28d74f4f55e7c29e92419b52509e5c367cf2d977b670a2ff7560f5fe24021d246abe30542e6c6e3aa52f903453c3a2389af918249dbdb5f1199aaecf4931c0366592165b10bdd58eaf706d6df02a39d9323a0c65260ffcc84776f2705e4942d89e4dbefa11c693027002c35582d56e295dcf74d27e90873699657337696b32c05c8014911a7ec8eb03bdbe526fe658be8abdf50ab12c4fec9ddeefc489cf817721c8e541d28fbe71e32137b5ea066a9f4e19814deedeb360def90eff2965570aab5fedd0ebfcd783ce3289360953680ac084b2e988c9cbd0912da400861467d7bb5ad4b42a95c2d541653e805cbfc84da401baf096fba43300358421ae1b43fd25f3289c8c73489977592f75bc9f73781f41718a752ab325b70c8eb2011c5d979f6efc7a76e16492566e43d94dbd42698eb06ff8ad4fd3f2baabafded"
}
⛰ lncli --network=simnet exportchanbackup --all --output_file=channels.backup
As shown above, a user can either: specify a specific channel to backup, backup all existing channels, or backup directly to an on-disk file. All backups use the same format.
Using the gRPC interace directly, a new call:
SubscribeChannelBackups.
This call allows users to receive a new notification each time the underlying
SCB state changes. This can be used to implement more complex backup
schemes, compared to the file system notification based approach.
If a node is being created from scratch, then it's possible to pass in an
existing SCB using the lncli create or lncli unlock commands:
⛰ lncli create -multi_file=channels.backup
Alternatively, the restorechanbackup command can be used if lnd has already
been created at the time of SCB restoration:
⛰ lncli restorechanbackup -h
NAME:
lncli restorechanbackup - Restore an existing single or multi-channel static channel backup
USAGE:
lncli restorechanbackup [command options] [--single_backup] [--multi_backup] [--multi_file=]
CATEGORY:
Channels
DESCRIPTION:
Allows a user to restore a Static Channel Backup (SCB) that was
obtained either via the exportchanbackup command, or from lnd's
automatically manged channels.backup file. This command should be used
if a user is attempting to restore a channel due to data loss on a
running node restored with the same seed as the node that created the
channel. If successful, this command will allows the user to recover
the settled funds stored in the recovered channels.
The command will accept backups in one of three forms:
* A single channel packed SCB, which can be obtained from
exportchanbackup. This should be passed in hex encoded format.
* A packed multi-channel SCB, which couples several individual
static channel backups in single blob.
* A file path which points to a packed multi-channel backup within a
file, using the same format that lnd does in its channels.backup
file.
OPTIONS:
--single_backup value a hex encoded single channel backup obtained from exportchanbackup
--multi_backup value a hex encoded multi-channel backup obtained from exportchanbackup
--multi_file value the path to a multi-channel back up file
Once the process has been initiated, lnd will proceed to:
- Given the set of channels to recover, the server will then will insert a series of "channel shells" into the database. These contain only the information required to initiate the DLP (data loss protection) protocol and nothing more. As a result, they're marked as "recovered" channels in the database, and we'll disallow trying to use them for any other process.
- Once the channel shell is recovered, the chanbackup package will attempt to insert a LinkNode that contains all prior addresses that we were able to reach the peer at. During the process, we'll also insert the edge for that channel (only in the outgoing direction) into the database as well.
- lnd will then start up, and as usual attempt to establish connections to
all peers that we have channels open with. If
lndis already running, then a new persistent connection attempt will be initiated. - Once we connect with a peer, we'll then initiate the DLP protocol. The remote peer will discover that we've lost data, and then immediately force close their channel. Before they do though, they'll send over the channel reestablishment handshake message which contains the unrevoked commitment point which we need to derive keys (will be fixed in BOLT 1.1 by making the key static) to sweep our funds.
- Once the commitment transaction confirms, given information within the SCB we'll re-derive all keys we need, and then sweep the funds.