Peer storage for nodes to distribute small encrypted blobs.

[adi2011]

This pull request proposes peer storage, inspired by @t-bast's peer backup storage. We create two separate messages, which users can use to distribute encrypted 'blobs' of up to 64 KB to their peers.

Users can distribute a 'blob' and update it whenever their configuration changes by sending a 'peer_storage' message. In return, they will always receive a 'your_peer_storage' message, which will serve as an acknowledgment.

Nodes can also decide whether they want to provide this service in exchange for some sats.

You can check out the implementation in CLN here

Thank you so much, @rustyrussell, for all the help and guidance!

[saubyk]

This part is unclear to me. Based on my reading, there are two instances when a node (running option_provide_storage) will send your_peer_storage message:

• When it gets a peer_storage message and has stored the data (like an ack)
• When it reconnects with the peer for which it has stored data (to ensure that peer has a chance to update stale data)

The point about "regularly" sending your_peer_storage seem to be specified loosely.

• What's the right frequency to send this to your peer?
• Does it create a spam vector for your peer?

[rustyrussell]

Agreed. You don't need to specify this, since you already said in requirements that "MAY send peer_storage whenever necessary (usually, everytime their configuration changes)".

And I made a suggestion for spam above...

[adi2011]

Yes, the two instances that you mentioned are the only time when a peer needs to send your_peer_storage.

[rustyrussell]

Agreed. You don't need to specify this, since you already said in requirements that "MAY send peer_storage whenever necessary (usually, everytime their configuration changes)".

And I made a suggestion for spam above...

[saubyk]

Ack

[t-bast]

As stated during the spec meeting, I think this should be moved to the bLIP repository: https://github.com/lightning/blips
This feels like an extension/add-on, not something that really needs to be in the BOLTs (especially if we're trying to somewhat reduce complexity here).
I wanted to move my encrypted storage proposal to a bLIP, but didn't do it because I was rather waiting for someone else to work on it first, which is happening now.

[adi2011]

I think in order to make peer storage reliable, it should have maximum coverage across the network. This approach attempts to address the backup issue, which is essential for enhancing the usability of the Lightning Network. Many small node operators may be reluctant to invest in RAID systems to protect their nodes; thus, this could serve as one of the safety nets for the network. That is why we should consider adding it to the main spec imo.

[GeorgeTsagk]

(e.g. might get paid for the service)

Could provide a bit more context here w.r.t being paid for the purpose of storing blobs

Is this why if it has an open channel with the sender is a requirement?

[adi2011]

Sure, this needs more explanation, I'll add to it when I get back to my laptop.

[ProofOfKeags]

Is this why if it has an open channel with the sender is a requirement?

I'd imagine this also helps with spam prevention. Since in order to do a drive-by to store data you'd have to commit capital.

[adi2011]

Open channel is not a requirement. Some nodes might provide this service in exchange for some sats that is what might get paid for the service means.

[vincenzopalazzo]

I concur with @t-bast that this feature could be a BLIP.

Since this is not a one-size-fits-all solution, we can create different BLIPs for various backup use cases (if any others are proposed in the future).

If we want to add this to the BOLTs, we should address some concerns like this one ElementsProject/lightning#6652

[ellemouton]

dont think i fully understand why we need the option_want_storage feature bit..?

isnt it enough that im connecting to a node with option_provide_storage and sending peer_storage to it? Or is there a case where someone would want to seek out a node advertising option_want_storage?

[ProofOfKeags]

I'm inclined to agree. I think in practice it is very likely that people will implement this protocol wholesale or not at all (it's pretty simple). That said, I can imagine scenarios where you may specifically not want to provide this feature but still would like to consume it. At 65k per peer though, idk what the big deal would be with providing it anyway.

[adi2011]

Big nodes with large number of peers might not be interested in giving this service to all of their peers.

[ProofOfKeags]

Are you suggesting they will only want to offer it to some of their peers or rather in that situation they wouldn't want to offer it to any of them?

[ellemouton]

Big nodes with large number of peers might not be interested in giving this service to all of their peers.

I dont think having/not having option_want_storage changes this though?

[ellemouton]

the only reason you would advertise option_want_storage is so that nodes offering the service could seek you out. but that doesnt really make sense to me. Makes more sense that if you want storage - you would seek out nodes that can offer it to you.

[ellemouton]

should we not use even types for these messages? since we are only sending these if the peer advertises option_provide_storage

[ProofOfKeags]

does IOKTBO apply to message types?? I thought it was just TLVs

[ellemouton]

yeah it applies to messages too afaiu: https://github.com/lightning/bolts/blob/master/01-messaging.md#lightning-message-format

[ProofOfKeags]

I don't think this applies to the low message type numbers though afaict. If it did then things like accept_channel would be treated the same right? I would support using even messages, no issue with that at all, but I think I'm not quite understanding the intricacies of IOKTBO

[ellemouton]

yeah i think for some of the very first messages this was sort of overlooked but it wasnt a problem since nothing would work if a node didnt understand something fundamental like accept_channel

[TheBlueMatt]

IOKTBO definitely applies to all messages, its just in the very initial protocol there's no distinction cause everyone supports everything :). Its definitely nice for these to be odd because then nodes can send peer_storage without bothering to check if their peer supports the feature at all.

[TheBlueMatt]

Lets not respond with the full 64KiB blob every time data gets stored. That seems incredibly wasteful. If you want an ACK, lets include an ID.

[ProofOfKeags]

we can also ACK with a sha256 of the data provided.

[TheBlueMatt]

Don't make me hash the data, either. Honestly, I'm not really sure if we need to have an ACK at all....its useful if you trust the counterparty and want to use it as a robust data store (but, even then, I think only LDK has support for fully async persistence?, and I'm not convinced our users would prefer this over a traditional storage layer) but I'm not sure that's worth doing here when the whole design space is around "you may persist, if you want, but peers shouldn't rely on it". Maybe @t-bast has other thoughts?

[t-bast]

I agree with @TheBlueMatt here, our proposal for this feature (#881) does not have an ACK because I didn't find that useful.

[adi2011]

It serves as assurance to the sender, although not foolproof, that it won't be excessively wasteful. Additionally, updating peer_storage won't be a frequent occurrence.

[adi2011]

The sole reason for ACK was to assure the sender, is their any other efficient method to do this?

[t-bast]

But does it really assure the sender of anything meaningful? If used to store every channel state change (which is what we've been doing for years), we don't want to pause the channel's operation waiting for an ACK, it adds unnecessary complexity and risk of force-closing. This is mostly being used by nodes who will regularly disconnect (e.g. mobile phones), who have an opportunity of checking the back-up returned on every connection reestablishment, which is IMO enough assurance that the storing node behaves correctly?

[ProofOfKeags]

I'd like to remind us that we didn't need ACKs for every channel update either, and yet we are nudging ourselves towards a simplified update protocol. Granted, the purpose of this is far less consequential than ACK'ing the channel updates themselves, so we can afford to be like "🤷🏻‍♂️ who knows if they got it, we'll just assume so".

My take here is that at minimum it helps with debugging, and the cost of the ACK seems rather trivial so avoiding it seems silly. Yes, it doesn't provide much, we can always send back the packet, or the hash, or some ID, and then just yeet the data into oblivion, but does it really cost us much to just ACK it?

[TheBlueMatt]

The sole reason for ACK was to assure the sender, is their any other efficient method to do this?

To echo @t-bast's comment - to assure the sender of what? And what should the sender do with that assurance. If a given message is not actually handled by a peer/it doesn't change their behavior in some meaningful way, we should drop it.

I'd like to remind us that we didn't need ACKs for every channel update either,

Not sure what you're referring to - do you mean we don't need revoke_and_ack? We absolutely do, the protocol doesn't work without it.

My take here is that at minimum it helps with debugging, and the cost of the ACK seems rather trivial so avoiding it seems silly. Yes, it doesn't provide much, we can always send back the packet, or the hash, or some ID, and then just yeet the data into oblivion, but does it really cost us much to just ACK it?

Yes, more protocol. We have a lot of protocol, anywhere we can have just a bit less protocol is great. Its not clear to me why an ACK will help with debugging - a peer can always give us an old blob, whether they ACK'ed it or not, and if you get an old state you need to go try to figure out if the state is old and which one it is. You can always check if there was a ping/pong after the sent storage message and use that as an ack/they received it indication.

[adi2011]

Removed ACK as a requirement. Still mentioned in the rationale so that implementations can decide for themselves.

[thomash-acinq]

There is no word here about what this will be used for. If we intend to use it to store critical data that will allow us to recover from a loss of channel state on our side, then we should make that explicit.
For that use-case, it's important that we get the latest blob and we should ensure that an honest peer always gives us the latest blob.

[thomash-acinq]

This seem to imply that peer_storage_retrieval is a reply to peer_storage but the requirements below only say that peer_storage_retrieval must be sent after a reconnection, not as a reply to a peer_storage message.

[thomash-acinq]

I think you should expect your peer to send you the latest data and check against the latest. However you shouldn't rely on it because your peer may be trying to cheat you. If the data your peer sent you back is not the latest you should mark them as unreliable.

[thomash-acinq]

The problem with rate limiting like this combined with no ACK is that you can't guarantee that an honest peer will store the latest data which I think is a very desirable property.

[thomash-acinq]

In that case, we want "peer_storage_retrieval MUST sent before channel_reestablish", but there is nothing problematic about sending it again after channel_reestablish, it's just unnecessary.

[Roasbeef]

There is no word here about what this will be used for. If we intend to use it to store critical data that will allow us to recover from a loss of channel state on our side, then we should make that explicit.

I think that's by design, at the surface level, it's just a blob that you send to the other party and can maybe retrieve in the future. Any higher layer that uses this feature should make minimal assumptions.

For that use-case, it's important that we get the latest blob and we should ensure that an honest peer always gives us the latest blob.

There's no way to ensure that. All we can rely on is a best effort attempt by the peer to maybe retrieve the blob, or not.

[t-bast]

ACK e8e6056