Added S3 user management #127

UtkarshBhatthere · 2023-05-09T14:12:54Z

Added CLI to Create, Delete, List and Get S3 users for rgw

ubuntu@microcephtest:~$ sudo microceph s3 list
+---+-------------+
| # |    NAME     |
+---+-------------+
| 1 | test4       |
+---+-------------+
| 2 | testUser100 |
+---+-------------+
| 3 | test        |
+---+-------------+
| 4 | test2       |
+---+-------------+
| 5 | test5       |
+---+-------------+
ubuntu@microcephtest:~$ sudo microceph s3 get testUser100
+-------------+----------------------+------------------------------------------+
|    NAME     |      ACCESS KEY      |                  SECRET                  |
+-------------+----------------------+------------------------------------------+
| testUser100 | 9YI4KEEEZ333ES6Z5PST | iNPJDugy2l1GlnA8TWCWNguncQcb0f6glIjaj1Wu |
+-------------+----------------------+------------------------------------------+
ubuntu@microcephtest:~$ sudo microceph s3 create utkarsh --access-key=yo --secret=doesitwork?
+---------+------------+-------------+
|  NAME   | ACCESS KEY |   SECRET    |
+---------+------------+-------------+
| utkarsh | yo         | doesitwork? |
+---------+------------+-------------+
ubuntu@microcephtest:~$ sudo microceph s3 delete utkarsh
ubuntu@microcephtest:~$ sudo microceph s3 list
+---+-------------+
| # |    NAME     |
+---+-------------+
| 1 | test4       |
+---+-------------+
| 2 | testUser100 |
+---+-------------+
| 3 | test        |
+---+-------------+
| 4 | test2       |
+---+-------------+
| 5 | test5       |
+---+-------------+

UtkarshBhatthere · 2023-05-09T14:14:56Z

Added implementation, will add tests for the same.

UtkarshBhatthere · 2023-05-26T12:04:16Z

The PR lacks test, will remove from drafts once I have added them.

sabaini

Heya,

this looks very useful.

Some comments after a first quick look:

User interface: we talked about this a bit but jftr. -- think the microceph s3 [create|delete|...] ... might be a bit confusing as it's not clear that this is about creating or deleting users. Maybe could do another subcommand e.g. microceph s3 user [create|delete|...] ... or rename the subcommand to something like microceph s3-user [create|delete|...] ...

REST API: it's recommended to give every entity in a REST api their own URI, and have the REST verbs (GET/PUT/DELETE/...) act on those entities

For this API the entities would be users. It's a convention (though not required by REST) to have directories map to collections of entities.

A mapping to URIs could therefore look something like this, with an example user fooUser

GET /s3/users # list user collection
POST /s3/users # create a new user in collection with data from the req. body

GET /s3/users/fooUser # get details for fooUser (no data required)
PUT /s3/users/fooUser # update fooUser with data from req. body
DELETE /s3/users/fooUser # delete fooUser (no data required)

Note there's no POST /s3/users/fooUser as POST is for creating new entities.

Thanks!

UtkarshBhatthere · 2023-07-28T14:08:17Z

@sabaini I have modified the APIs to be using ""services/rgw/user"".

sabaini

Heya Utkarsh, thanks for the update, added a comment about REST API structure

sabaini · 2023-08-31T08:21:22Z

microceph/api/s3.go

+	Get:    rest.EndpointAction{Handler: cmdS3Get, ProxyTarget: true},
+	Put:    rest.EndpointAction{Handler: cmdS3Put, ProxyTarget: true},
+	Delete: rest.EndpointAction{Handler: cmdS3Delete, ProxyTarget: true},
+}


For optimal RESTy-ness I'd like to suggest an API structure like this:

GET /services/rgw/user # returns a list of users
GET /services/rgw/user/{uid} # returns info about a specific user

POST /services/rgw/user # create a new user

PUT /services/rgw/user/{name} # update existing user

DELETE /services/rgw/user/{name} # delete user

Something like this can be used to specify a path with a {uid} var:

var s3Cmd = rest.Endpoint{ Path: "services/rgw/user/{name}", Get: rest.EndpointAction{Handler: cmdS3GetSingle, ProxyTarget: true}, Put: ... func cmdS3GetSingle(s *state.State, r *http.Request) response.Response { var err error var name string name, err = url.PathUnescape(mux.Vars(r)["name"]) if err != nil { return response.BadRequest(err) } // do something with user name user = ceph.GetS3User(name) ...

sabaini · 2023-08-31T08:26:35Z

microceph/client/s3.go

+	defer cancel()
+
+	ret := ""
+	err := c.Query(queryCtx, "GET", api.NewURL().Path("services", "rgw", "user"), user, &ret)


With the API structure change I've suggested above the URL would be built with something like

err = c.Query(queryCtx, "GET", api.NewURL().Path("services", "rgw", "user", strconv.FormatInt(user.name, 10)), nil, nil)

UtkarshBhatthere · 2023-09-04T07:56:02Z

Thanks for the detailed review @sabaini 🙏🏼 . I will implement your RESTy suggestions in and make the PR ready.

Signed-off-by: Utkarsh Bhatt <[email protected]>

sabaini

Hey @UtkarshBhatthere did a quick once over, added some suggestions mostly around API RESTfulness

sabaini · 2024-01-12T16:57:52Z

.github/workflows/q2q-candidate-upgrade.yml

@@ -50,7 +54,7 @@ jobs:
      run: ~/actionutils.sh headexec enable_rgw

    - name: Exercise RGW
-      run: ~/actionutils.sh headexec testrgw
+      run: ~/actionutils.sh headexec testrgw_old


Naming nit: maybe a more descriptive name instead of testrgw_old

sabaini · 2024-01-12T16:59:43Z

.github/workflows/q2q-candidate-upgrade.yml

+        # boto3 for appS3 test script.
+        sudo python -m pip install --upgrade pip
+        sudo pip install boto3
+        ~/actionutils.sh setup_lxd


I see these lines being repeated
Maybe could refactor the actionutils.sh a bit to have a setup_host function instead of setup_lxd that handles all deps

sabaini · 2024-01-12T17:01:45Z

docs/how-to/s3-user.rst

+Manage S3 users on MicroCeph
+=============================
+
+MicroCeph provides an easy to use interface for creating, viewing and deleting s3 users for interfacing with the RGW endpoint.


Suggest: capitalize S3 here and below (proper noun)

sabaini · 2024-01-12T17:02:06Z

docs/how-to/s3-user.rst

+=============================
+
+MicroCeph provides an easy to use interface for creating, viewing and deleting s3 users for interfacing with the RGW endpoint.
+This enables smooth and easy access to Object Storage.


Suggest: s/Object Storage/object storage/

sabaini · 2024-01-12T17:05:11Z

docs/how-to/s3-user.rst

+    | 2 | testUser    |
+    +---+-------------+
+
+3. Get details of a an s3 user (optionally use --json flag to get complete details):


Typo, s/a an/an/

sabaini · 2024-01-12T17:09:17Z

microceph/api/client_s3.go

+	"github.com/canonical/microcluster/state"
+)
+
+// /1.0/resources endpoint.


Copy/paste error :-)

sabaini · 2024-01-12T17:11:37Z

microceph/api/client_s3.go

+	}
+}
+
+func cmdClientS3Put(s *state.State, r *http.Request) response.Response {


For maximum RESTfulness we should be using a POST to the /client/s3 url to create a user

sabaini · 2024-01-12T17:16:08Z

microceph/api/client_s3.go

+	Delete: rest.EndpointAction{Handler: cmdClientS3Delete, ProxyTarget: true},
+}
+
+func cmdClientS3Get(s *state.State, r *http.Request) response.Response {


RESTfulness:

retrieving a list of users should be a GET to /client/s3

retrieving a single user should be a GET to /client/s3/{userid}

See the /disks/{osdid} handler in microceph/api/disks.go for an example on how to implement a handler for a path variable i.e. {userid}

sabaini · 2024-01-12T17:18:34Z

microceph/api/client_s3.go

+	return response.SyncResponse(true, output)
+}
+
+func cmdClientS3Delete(s *state.State, r *http.Request) response.Response {


RESTfulness: this should be using a DELETE against /client/s3/{userid} (also cf. above for getting users)

sabaini · 2024-01-12T17:22:37Z

microceph/cmd/microceph/s3-user.go

@@ -0,0 +1 @@
+package main


I guess this is not actually used?

UtkarshBhatthere marked this pull request as draft May 9, 2023 14:14

UtkarshBhatthere self-assigned this May 9, 2023

UtkarshBhatthere force-pushed the s3Interface branch 4 times, most recently from 0feaf24 to 6e91346 Compare May 12, 2023 08:15

UtkarshBhatthere marked this pull request as ready for review May 26, 2023 11:25

UtkarshBhatthere force-pushed the s3Interface branch from 6e91346 to 3d1e59d Compare May 26, 2023 12:03

UtkarshBhatthere marked this pull request as draft May 26, 2023 12:03

UtkarshBhatthere force-pushed the s3Interface branch 5 times, most recently from 6633df4 to 933e165 Compare July 11, 2023 11:35

UtkarshBhatthere mentioned this pull request Jul 11, 2023

Added test script for RGW S3 interface. #117

Closed

UtkarshBhatthere force-pushed the s3Interface branch 8 times, most recently from 4bb0de4 to d0b881b Compare July 12, 2023 09:07

UtkarshBhatthere marked this pull request as ready for review July 12, 2023 09:33

sabaini requested changes Jul 13, 2023

View reviewed changes

UtkarshBhatthere force-pushed the s3Interface branch 3 times, most recently from 3aa3465 to 88f45a5 Compare July 13, 2023 09:14

UtkarshBhatthere force-pushed the s3Interface branch 2 times, most recently from 51da731 to e1374d6 Compare July 28, 2023 14:07

UtkarshBhatthere force-pushed the s3Interface branch from e1374d6 to 5496249 Compare August 14, 2023 12:05

UtkarshBhatthere force-pushed the s3Interface branch 2 times, most recently from c46d992 to a31fb6c Compare August 29, 2023 06:45

sabaini requested changes Aug 31, 2023

View reviewed changes

UtkarshBhatthere marked this pull request as draft September 4, 2023 07:55

UtkarshBhatthere force-pushed the s3Interface branch 12 times, most recently from d693ae0 to 99ce437 Compare December 13, 2023 05:25

UtkarshBhatthere added 2 commits January 2, 2024 10:56

Added S3 user management

06d35f0

Signed-off-by: Utkarsh Bhatt <[email protected]>

Changed CLI and endpoint interfaces.

89fe07b

Signed-off-by: Utkarsh Bhatt <[email protected]>

UtkarshBhatthere force-pushed the s3Interface branch from 99ce437 to 89fe07b Compare January 2, 2024 05:27

UtkarshBhatthere force-pushed the s3Interface branch 2 times, most recently from d14c3ba to 10cb50d Compare January 11, 2024 12:29

Fixed CI Issues

a163bc8

Signed-off-by: Utkarsh Bhatt <[email protected]>

UtkarshBhatthere force-pushed the s3Interface branch from 10cb50d to a163bc8 Compare January 11, 2024 12:30

sabaini requested changes Jan 12, 2024

View reviewed changes

UtkarshBhatthere force-pushed the main branch from 3204396 to 0863c8b Compare August 13, 2024 19:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added S3 user management #127

Added S3 user management #127

UtkarshBhatthere commented May 9, 2023 •

edited

Loading

UtkarshBhatthere commented May 9, 2023

UtkarshBhatthere commented May 26, 2023

sabaini left a comment

UtkarshBhatthere commented Jul 28, 2023

sabaini left a comment

sabaini Aug 31, 2023

sabaini Aug 31, 2023

UtkarshBhatthere commented Sep 4, 2023

sabaini left a comment

sabaini Jan 12, 2024

sabaini Jan 12, 2024

sabaini Jan 12, 2024

sabaini Jan 12, 2024

sabaini Jan 12, 2024

sabaini Jan 12, 2024

sabaini Jan 12, 2024

sabaini Jan 12, 2024

sabaini Jan 12, 2024

sabaini Jan 12, 2024

Added S3 user management #127

Are you sure you want to change the base?

Added S3 user management #127

Conversation

UtkarshBhatthere commented May 9, 2023 • edited Loading

UtkarshBhatthere commented May 9, 2023

UtkarshBhatthere commented May 26, 2023

sabaini left a comment

Choose a reason for hiding this comment

UtkarshBhatthere commented Jul 28, 2023

sabaini left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

UtkarshBhatthere commented Sep 4, 2023

sabaini left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

UtkarshBhatthere commented May 9, 2023 •

edited

Loading