@talSofer need a better understanding of the requirements. Do we need to rotate or replace the admin creds?

@talSofer need a better understanding of the requirements. Do we need to rotate or replace the admin creds?

@talSofer I'd like to add to Itai's question some context:

We have replace credentials already, it's just not documented but pretty straight forward (and easy!).
The downside of replace (single pair of credentials allowed) is down-time to lakeFS users while doing this until all the users update to use the new credential set.

Rotate does not exist and it means allowing 2 pairs of credentials and allowing users to essentially rotating them on the fly and gradually replacing for clients using one or the other.

A note about lost credentials: we don't have a way of overriding credentials unless you delete the user but, to delete the user you need the credentials (so essentially you're locked out). I think this is an acceptable tradeoff in terms of security. Otherwise anyone with access to lakeFS server binary will be able to replace the user.

With that in mind, when I created the issue I wasn't aware of replace.
We can simply update the documentation for replace in lakeFS or extend and develop rotation.

WDYT?

@Isan-Rivkin thanks for elaborating on this.
I prefer to do credentials replacement, but have two clarification questions:

1. What's the current (undocumented) way of doing this?
2. Does your note about lost credentials mean we can only replace the secret access key, and not the access key?

@Isan-Rivkin thanks for elaborating on this. I prefer to do credentials replacement, but have two clarification questions:

1. What's the current (undocumented) way of doing this?

Delete User > Recreate

# Authentication request 
1. lakectl auth users delete --id admin
# Non-Authenticated request optional: pass accessKeyID+secretKeyID
2. lakefs superuser --user-name admin

• Between step 1 and 2 one needs to stop lakeFS server if using local blockstore type due to technical badger limitation.

2. Does your note about lost credentials mean we can only replace the secret access key, and not the access key?

Here's the possible situations (If Lost secret key ID: Nothing you can do locked out):

1. Create new user (same/different user name) with auto generated access key id and secret.
2. Create new user (same/different user name) with pre-defined (using flags) access key Id and access key Secret

There's a BUG discovered now by me and @yonipeleg33:
in our sequence that will keep you locked out even if you didn't lose your initial key pair, when calling the superuser command without passing secret key:

# deleted the previous user - as expected
lakectl auth users delete --id admin
# no more users now, pass only key-id, code allows:
lakefs superuser --user-name whatever --access-key-id <SOME-KEY-ID>`

In such case you'll get locked-out of the server (CC @N-o-Z please keep me honest).
The reason is because when calling superuser here's what happens in the code:

1. lakectl: try auth.CreateUser # succeeds since no users exist 
2. lakectl: if accessKey != "" -> call auth.AddCredentials
3. auth service: run importUserCredentials() # fails because listUserCredentials fails. 
4. resulting in error "no credentials found for user"

Edit: I didn't notice @Isan-Rivkin already commented, so my comment is basically a duplicate of his: #8444 (comment)

What's the current (undocumented) way of doing this?

1. Run this command to delete the existing user:

   $ lakectl auth users delete --id <user-id>
   

2. Shutdown the lakeFS server
   This is required because lakeFS caches the credentials even after they are removed.
   Especially required when running with local kv, as only one write connection is allowed to badger (our local kv db)
3. Run this command to create a new user:

   $ lakefs superuser --user-name <user-id>
   

   You can specify access key and secret explicitly by passing `--access-key-id --secret-access-key .

Does your note about lost credentials mean we can only replace the secret access key, and not the access key?

No, you can (and should) replace both access and secret keys using the method shown above.

Note that this operation is risky; Between running (1) and (2) you are in a state where you don't have any user.
Also, see #8490 for another possible footgun.

@talSofer Please advise on how to proceed from here.
Our (me and @Isan-Rivkin's) suggestion:

1. Document the steps to replace keys somewhere in our docs, with a proper warning message(s) 🙈
2. Prioritize keys rotation (and [Bug]: Creating a user using the superuser command can end up in an unrecoverable state #8490) separately from this issue.

@Isan-Rivkin and @yonipeleg33 thanks for explaining things clearly.

Our (me and @Isan-Rivkin's) suggestion:

1. Document the steps to replace keys somewhere in our docs, with a proper warning message(s) 🙈
2. Prioritize keys rotation (and [Bug]: Creating a user using the superuser command can end up in an unrecoverable state #8490) separately from this issue.

I agree with how you are suggesting to proceed.