BREAKING: Remove Secure-Join-Fingerprint header from "vc-contact-confirm", "vg-member-added" messages. Remove "vc-contact-confirm-received", "vg-member-added-received" messages from Securejoin protocol.

[iequidoo]

BREAKING: Remove Secure-Join-Fingerprint header from "vc-contact-confirm", "vg-member-added" messages (#3836)

It's a compatibility crutch for old clients (< 1.107.0), they require it in the subject
messages. Currently DC sends Autocrypt key gossips instead, they're better because knowing a key
allows not to only trust the peer, but also encrypt to it. Before it was a problem for other devices
on a joining side going online after a successful Securejoin setup -- they didn't have a joiner's
key to encrypt to it. So, we decided not to complicate the Securejoin with sending keys instead, but
rely on the Autocrypt.

Also there's a PR to the Countermitm doc documenting when gossips in 'vc-request-with-auth' are
needed:

Bob's own key fingerprint ``Bob_FP``,
the second challenge ``AUTH`` from step 1 and
optionally an Autocrypt-Gossip header for Alice's
Autocrypt key in order for a second device of Bob
to learn Alice's verified key.

BREAKING: Remove "vc-contact-confirm-received", "vg-member-added-received" messages from Securejoin protocol (#3836)

They are an overkill and do not solve any problem. Also they aren't documented anywhere. And all
necessary tests were added before and they work w/o the removed code.

AFAIU, they were added for Alice (joiner) to be sure that Bob (joinee) has Alice two-way verified,
i.e. Bob can add Alice to other verified groups because Bob knows that Alice trusts them. But that
means Alice should retry sending vc-contact-confirm/vg-member-added messages until getting
*-received message from Bob. But then better let Bob retry sending *-auth-required until getting
vc-contact-confirm/vg-member-added from Alice. And if Alice sees no more retries from Bob, either
Bob has got vc-contact-confirm/vg-member-added from Alice or there are communication problems. But
the latter can't be solved anyway by adding extra messages to the protocol. Note that there are no
retries currently, instead we rely on that Bob will eventually receive a message from Alice and thus
know that Alice verified them. But i could miss some details why they were added, so just in case,
citing from #3836:

@iequidoo: Btw, can somebody explain the purpose of
vg-member-added-received/vc-contact-confirm-received messages in the protocol? They look
excessive and for me it's like a try to solve that problem with two friends that want to meet
and drink some beer but only if each of them is sure that their beermate is also going to the
meeting :) Even if Bob didn't receive vg-member-added message (e.g. because of mail delivery
problems), we can consider Bob joined -- Bob can try sending messages to the group and that must
work afaiu.

@flub: Yes, this is a weird state. It is currently the difference between an internal state
"un-idirectional verified" and the exposed state "bi-directional verified". The latter (bi-)
means both know that both have each other verified, in the former (uni-) only one of them knows
this and the other hasn't figured this out yet.

IIRC the last time this was discussed the revelation (at least to me) was that the main
practical difference between these two is that bi-directional allows you to add the other person
to a verified group, while if you only have them uni-directional verified you can not do they
since they don't trust you enough (IIRC, there should be a cryptpad somewhere with this written
down).

@link2xt: When Bob receives vc-auth-required, he already has Alice one-way verified. When Alice
receives vc-request-with-auth, she has Bob two-way verified (she has verified Bob's key and she
know Bob has verified her), but Bob still does not know about this. When Bob receives
vc-contact-confirm (or vg-member-added) he sets Alice state to "two-way verified".

The question is what happens if vc-contact-confirm/vg-member-added is lost. In this case Alice
has Bob two-way verified, while Bob has Alice only one-way verified. Because of this, Bob cannot
safely add Alice to any verified group, because Bob thinks maybe Alice has not received
vc-request-with-auth and has Bob completely unverified. However, Alice still can add Bob to
verified groups and if Bob later receives any message from Alice in a verified group, he sets
Alice to two-way verified, so eventually both Alice and Bob converge to a two-way verified
state. This is not how it is currently implemented, but this was the idea during the discussion
with @flub and @missytake.

But vg-member-added-received/vc-contact-confirm-received is an overkill and can be removed IMO,
it does not solve any problem.

But for this to be merged, most of clients should be updated to the following commit for compatibility reasons:

commit 5f883a44458cbc76d0ae49ef86bb3ae51bc2ab84
Author:     iequidoo <[email protected]>
CommitDate: 2023-01-12 15:13:30 -0300

    Prepare to remove "vc-contact-confirm-received", "vg-member-added-received" messages from Securejoin
    protocol

i.e. to the core 1.107.0.
It is here in Desktop: deltachat/deltachat-desktop@93644aa
and here in Android: deltachat/deltachat-android@699d97c

[Hocuri]

Could you write a short summary of why they were added (in case you know this, otherwise write that you don't know), what's the advantage of removing them (Probably it's reducing code complexity and network traffic, which is great already 🎉 but it'd be nice to have it explicitly stated, also to know whether there are additional advantages), and other important points discussed in #3836 if there are any?

But for this to be merged, most of clients should be updated to the following commit for compatibility reasons:

What happens if they aren't updated?

It is here in Desktop: deltachat/deltachat-desktop@93644aa

Thanks for directly providing the links to the releases! My feeling is that we should wait about a year (until Jan 2024), since Desktop doesn't have any auto-update mechanism, so there probably are quite some old clients in use. Depends on how bad it is if one client is old, and how big the advantage of removing is, though.

[flub]

Hum

• This is pretty quick to remove the support for this isn't it? I don't know if we have a policy but IIRC users start to be nagged to upgrade after about 1 year, so I guess you need to wait for at least 1 year?
• It was kind of nice that you could read the securejoin protocol and get the fingerprint without having to understand the autocrypt gossip stuff. This makes the dependence stronger, and thus increases the number of things you need to understand to work on it. On the other hand it duplicates information and that's also confusing. I realise I may have said something that led to this... I don't always say smart things... of course this could also be one of those.

[iequidoo]

• Agree, let's wait for 1 year, made it a draft.
• Maybe it would be nice for the Securejoin to not depend on Autocrypt things, but anyway knowing a joiner's fingerpint isn't sufficient for other devices on a joining side to encrypt to the joiner, a key (gossip) is needed. There's also the corresponding PR to the Securejoin documentation: Document when gossips in 'vc-request-with-auth' are needed nextleap-project/countermitm#85 . So, i think better not to complicate the Securejoin protocol with passing keys and rely on the Autocrypt. CC @hpk42

[iequidoo]

Could you write a short summary of why they were added (in case you know this, otherwise write that you don't know), what's the advantage of removing them (Probably it's reducing code complexity and network traffic, which is great already tada but it'd be nice to have it explicitly stated, also to know whether there are additional advantages), and other important points discussed in #3836 if there are any?

I don't know all details why they were added, so just citing from #3836:

@iequidoo: Btw, can somebody explain the purpose of vg-member-added-received/vc-contact-confirm-received messages in the protocol? They look excessive and for me it's like a try to solve that problem with two friends that want to meet and drink some beer but only if each of them is sure that their beermate is also going to the meeting :) Even if Bob didn't receive vg-member-added message (e.g. because of mail delivery problems), we can consider Bob joined -- Bob can try sending messages to the group and that must work afaiu.

@flub: Yes, this is a weird state. It is currently the difference between an internal state "un-idirectional verified" and the exposed state "bi-directional verified". The latter (bi-) means both know that both have each other verified, in the former (uni-) only one of them knows this and the other hasn't figured this out yet.

IIRC the last time this was discussed the revelation (at least to me) was that the main practical difference between these two is that bi-directional allows you to add the other person to a verified group, while if you only have them uni-directional verified you can not do they since they don't trust you enough (IIRC, there should be a cryptpad somewhere with this written down).

@link2xt: When Bob receives vc-auth-required, he already has Alice one-way verified. When Alice receives vc-request-with-auth, she has Bob two-way verified (she has verified Bob's key and she know Bob has verified her), but Bob still does not know about this. When Bob receives vc-contact-confirm (or vg-member-added) he sets Alice state to "two-way verified".

The question is what happens if vc-contact-confirm/vg-member-added is lost. In this case Alice has Bob two-way verified, while Bob has Alice only one-way verified. Because of this, Bob cannot safely add Alice to any verified group, because Bob thinks maybe Alice has not received vc-request-with-auth and has Bob completely unverified. However, Alice still can add Bob to verified groups and if Bob later receives any message from Alice in a verified group, he sets Alice to two-way verified, so eventually both Alice and Bob converge to a two-way verified state. This is not how it is currently implemented, but this was the idea during the discussion with @flub and @missytake.

But vg-member-added-received/vc-contact-confirm-received is an overkill and can be removed IMO, it does not solve any problem.

What happens if they aren't updated?

At least the multi-device setup will be broken for the joining side because other devices (if not updated) learn joiner's fingerprint from vg-member-added-received/vc-contact-confirm-received messages. But to be sure it's the only problem, tests are needed of course.

Thanks for directly providing the links to the releases! My feeling is that we should wait about a year (until Jan 2024), since Desktop doesn't have any auto-update mechanism, so there probably are quite some old clients in use. Depends on how bad it is if one client is old, and how big the advantage of removing is, though.

Ok, let it be one year. Will mark this PR as a draft then. Created it just not to forget about this and see how many merge conflicts it creates (actually the code has changed a bit since Jan, so even now i had to put in a small effort to drop these crutches)

[hpk42]

@iequidoo could you provide more summary/reasoning of what this PR does?
I guess i could try to follow all links and use my known knowledge of the history of SecureJoin to understand but it's much better if you as the one who does the PR write precisely why you are doing the PR, and why it is ok to remove the message etc.

[iequidoo]

Tried to add all reasoning for removal of the subject stuff to the commit message and the PR description. But as i can't know all details on why it was done so, just cited a part of the discussion. If there still will be questions, will try to improve the commit message.

[missytake]

@link2xt [...] However, Alice still can add Bob to verified groups and if Bob later receives any message from Alice in a verified group, he sets Alice to two-way verified, so eventually both Alice and Bob converge to a two-way verified state. This is not how it is currently implemented[...]. But vg-member-added-received/vc-contact-confirm-received is an overkill and can be removed IMO, it does not solve any problem.

I think it makes sense to first find other ways to mark a contact as two-way verified, before we remove vg-member-added-received/vc-contact-confirm-received, e.g. because you see messages of them in verified groups. Until then they might still prevent bugs with verified groups where two-side-verification didn't complete.

[missytake]

What would also be interesting: find a way to communicate when two-way verification dropped. e.g. each message, Alice tells Bob "I have verified you btw", and Bob replies the next time with "I also have still verified you" or so. And if Bob at some point doesn't answer with "I also have still verified you", we recommend Alice to re-verify Bob and deny Alice the option to add Bob to verified groups. We wouldn't need to show Bob anything, at this point Bob couldn't trust such a message anyway, and if it's important, Alice can tell Bob to re-verify.

[hpk42]

On Thu, Apr 27, 2023 at 08:53 -0700, iequidoo wrote: Tried to add all reasoning for removal of the subject stuff to the commit message and the PR description. But as i can't know all details on why it was done so, just cited a part of the discussion. If there still will be questions, will try to improve the commit message.

Thanks for updating the issue text although i'd have prefered *your* one-paragraph summary instead of quoting messages from various people :) I have no doubt that we could optimize the last "confirmation" message away and the PR looks easy enough. Leaving away the confirm message was already discussed at the time of the COUNTERMITM protocol design. I guess we at the time thought it can not hurt and improves overall certainty whereas in fact it introduces new error-states and is conceptually superflous. I am less sure about the repercussions on released Delta Chat apps. Then again, verified groups are still kind of experimental and not neccessarily widely used yet. If i see it correctly there are three things: 1. Bob marks Alice as verified when sending `request_with_auth` 2. Not sending Secure-Join-Fingerprint but rely on gossip 3. Stop sending "confirm" (last step) messages So what about a PR that does 1. and 2. and we merge that quickly? And then Step 3 could be prepared but merged in DC 1.40 or 1.42? (sidenote: end-user facing we progress with even numbers as stable releases and odd numbers as in-between-beta releases so DC 1.38 is the next and 1.40 is the second next end-user facing DC apps release, probably in 3-5 months).

[iequidoo]

If i see it correctly there are three things:

1. Bob marks Alice as verified when sending request_with_auth

2. Not sending Secure-Join-Fingerprint but rely on gossip

3. Stop sending "confirm" (last step) messages

So what about a PR that does 1. and 2. and we merge that quickly?

"1" is already merged -- it's 39601be (it's 1.107.1). But as for 2, i think we should wait a bit here and do it with 3. Gossips were also added in 1.107.1, but we can't remove the Secure-Join-Fingerprint header from vg-member-added messages because before d644988 (1.107.0) it was mandatory for them, the Securejoin will fail for such old clients.

And i think that maybe indeed, waiting for 1 year since 1.107.1 is too long because the verified groups are "kind of experimental" and had several bugs for earlier versions (multi-device usage was in a half-working state f.e.), maybe half a year is sufficient.

UPD: So, the commit here does 2 and 3 actually. Will split it into two then.

[iequidoo]

Splitted into two commits and added my own understanding to the commit message on why *-received messages were added to the protocol and what better to do instead (retries on the joinee side). But left the citations from the discussion just in case if i miss smth

[iequidoo]

I think it makes sense to first find other ways to mark a contact as two-way verified, before we remove vg-member-added-received/vc-contact-confirm-received, e.g. because you see messages of them in verified groups. Until then they might still prevent bugs with verified groups where two-side-verification didn't complete.

vg-member-added-received/vc-contact-confirm-received do nothing useful if Alice don't retry
sending vg-member-added/vc-contact-confirm. Alice already knows that Bob trusts them because Bob already sent them a *-with-auth message, so Alice can add Bob to other verified groups. But Alice thinks that maybe Bob can't add them to verified groups because Bob didn't receive vg-member-added/vc-contact-confirm. That's why Alice should do retries. But i don't see that we do retries currently, so, removing *-received messages has no effect. Further, better let Bob retry sending *-with-auth messages if they didn't get any response, adding new message types to the protocol just complicates it w/o any profit

[hpk42]

As we have other important verified chats work (#4188 ) being merged, we should not add more vectors of breakage in the next few months.
Unlikely we can merge this breaking PR anytime soon, moving it to resurrection.
Later in 2023 we might revisit verified chat things more systematically .