Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backend deletion #447

Merged
merged 19 commits into from
Nov 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
a9181c5
Add accounts.{is_deleting,next_deletion_attempt_at}
SuperSandro2000 Oct 21, 2024
11c75ef
Mark accounts for deletion instead of complaining about things to delete
SuperSandro2000 Oct 21, 2024
cd3db03
Consider an accounts deletion status
SuperSandro2000 Oct 21, 2024
81577a3
Update EnforceManagedAccountsJob to reflect that it also updates acco…
SuperSandro2000 Oct 21, 2024
e7d0649
Add DeleteAccountsJob by re-using most bits from EnforceManagedAccoun…
SuperSandro2000 Oct 22, 2024
0ba7c9c
Mark account for deletion in enforceManagedAccount
SuperSandro2000 Oct 22, 2024
2f9b231
Update tests with all the changes
SuperSandro2000 Oct 21, 2024
5a48a5b
Almost drop unused metadata field, almost drop in_maintenance in the …
SuperSandro2000 Oct 22, 2024
c3a5e0c
Prevent any API requests on accounts that are in deleting
SuperSandro2000 Oct 29, 2024
b08d061
Drop kinda duplicated audit event
SuperSandro2000 Nov 11, 2024
43070b9
Don't add new occurences of metadata, in_maintenance, reject them in …
SuperSandro2000 Nov 11, 2024
65968ff
Handle nested manifests
SuperSandro2000 Nov 11, 2024
235985c
Test deletion with imageList
SuperSandro2000 Nov 11, 2024
1f71c6e
internal/api/keppel: remove leftovers of obsolete tests
majewsky Nov 12, 2024
c3cafaa
fix whitespace
majewsky Nov 12, 2024
dc27b0e
review: task metric names should be plural
majewsky Nov 12, 2024
72f93ce
remove fixed number of retries on recursive manifest deletion
majewsky Nov 12, 2024
7af4f4f
fix unreachable branch
majewsky Nov 12, 2024
8e6c77a
finish implementation of complex deletion test
majewsky Nov 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cmd/janitor/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ func run(cmd *cobra.Command, args []string) {
janitor := tasks.NewJanitor(cfg, fd, sd, icd, db, amd, auditor)
go janitor.AccountFederationAnnouncementJob(nil).Run(ctx)
go janitor.AbandonedUploadCleanupJob(nil).Run(ctx)
go janitor.DeleteAccountsJob(nil).Run(ctx)
go janitor.EnforceManagedAccountsJob(nil).Run(ctx)
go janitor.ManifestGarbageCollectionJob(nil).Run(ctx)
go janitor.BlobMountSweepJob(nil).Run(ctx)
Expand Down
15 changes: 4 additions & 11 deletions docs/api-spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,6 @@ On success, returns 200 and a JSON response body like this:
{
"name": "firstaccount",
"auth_tenant_id": "firsttenant",
"metadata": {},
"rbac_policies": [
{
"match_repository": "library/.*",
Expand Down Expand Up @@ -109,9 +108,6 @@ On success, returns 200 and a JSON response body like this:
{
"name": "secondaccount",
"auth_tenant_id": "secondtenant",
"metadata": {
"priority": "just an example"
},
"rbac_policies": [],
"replication": {
"strategy": "on_first_use",
Expand Down Expand Up @@ -142,8 +138,7 @@ The following fields may be returned:
| `accounts[].gc_policies[].time_constraint.oldest`<br>`accounts[].gc_policies[].time_constraint.newest` | integer or omitted | If set, the GC policy only applies to at most that many images within each repository, specifically to those that are oldest/newest ones when ordered by the timestamp attribute specified in the `time_constraint.on` key. These constraints are forbidden for policies with action "delete" to ensure that GC runs are idempotent. |
| `accounts[].gc_policies[].time_constraint.older_than`<br>`accounts[].gc_policies[].time_constraint.newer_than` | duration or omitted | If set, the GC policy only applies to at most images whose timestamp (as selected by the `time_constraint.on` key) is older/newer than the given age. Durations are given as a JSON object with the keys `value` (integer) and `unit` (string), e.g. `{"value": 4, "unit": "d"}` for 4 days. The units `s` (second), `m` (minute), `h` (hour), `d` (day), `w` (7 days) and `y` (365 days) are understood. |
| `accounts[].gc_policies[].action` | string | One of: `delete` (to delete matching images) or `protect` (to not delete matching images, even if another policy with a lower priority would want to). |
| `accounts[].in_maintenance` | bool | Whether this account is in maintenance mode. [See below](#maintenance-mode) for details. |
| `accounts[].metadata` | object of strings | Free-form metadata maintained by the user. The contents of this field are not interpreted by Keppel, but may trigger special behavior in applications using this API. |
| `accounts[].state` | string | The state of the account. Only shown when there is a specific state to report. [See below](#account-state) for possible values and details. |
| `accounts[].rbac_policies` | list of objects | Policies for rule-based access control (RBAC) to repositories in this account. RBAC policies are evaluated in addition to the permissions granted by the auth tenant. |
| `accounts[].rbac_policies[].match_cidr` | string | The RBAC policy applies to requests which originate from an IP address that matches the CIDR. |
| `accounts[].rbac_policies[].match_repository` | string | The RBAC policy applies to all repositories in this account whose name matches this regex. The leading account name and slash is stripped from the repository name before matching. The notes on regexes below apply. |
Expand Down Expand Up @@ -204,18 +199,16 @@ The following fields are shown on accounts configured with this strategy:

Note that the `accounts[].replication.upstream.password` field is omitted from GET responses for security reasons.

### Maintenance mode
### Account state

When `accounts[].in_maintenance` is true, the following differences in behavior apply to this account:
When `accounts[].state` is `deleting`, the following differences in behavior apply to this account:

- For primary accounts (i.e. accounts that are not replicas), no new blobs or manifests may be pushed. Only pulling and
deleting are allowed.
- For replica accounts, no new blobs or manifests will be replicated. Pulling is still allowed, but it becomes possible
to delete blobs and manifests.

Maintenance mode is a significant part of the account deletion workflow: Sending a DELETE request on an account is only
allowed while the account is in maintenance mode, and the caller must have deleted all manifests from the account before
attempting to DELETE it.
Sending a DELETE request on an account moves it into `state = "deleting"` and schedules the deletion of everything that belongs to the account, including manifests and blobs.

## GET /keppel/v1/accounts/:name

Expand Down
31 changes: 24 additions & 7 deletions internal/api/keppel/accounts.go
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,22 @@ func (a *API) handlePutAccount(w http.ResponseWriter, r *http.Request) {
http.Error(w, `malformed attribute "account.name" in request body is not allowed here`, http.StatusUnprocessableEntity)
return
}
// ... transfer it here into the struct, to make the below code simpler
// ... or state ...
if req.Account.State != "" {
http.Error(w, `malformed attribute "account.state" in request body is not allowed here`, http.StatusUnprocessableEntity)
return
}
// ... or in_maintenance ...
if req.Account.InMaintenance {
http.Error(w, `malformed attribute "account.in_maintenance" in request body is not allowed here`, http.StatusUnprocessableEntity)
return
}
// ... or metadata ...
if req.Account.Metadata != nil && len(*req.Account.Metadata) > 0 {
http.Error(w, `malformed attribute "account.metadata" in request body is not allowed here`, http.StatusUnprocessableEntity)
return
}
// ... and transfer the name here into the struct, to make the below code simpler
req.Account.Name = models.AccountName(mux.Vars(r)["account"])

// check permission to create account
Expand Down Expand Up @@ -158,18 +173,20 @@ func (a *API) handleDeleteAccount(w http.ResponseWriter, r *http.Request) {
return
}

resp, err := a.processor().DeleteAccount(r.Context(), *account, keppel.AuditContext{
if account.IsDeleting {
w.WriteHeader(http.StatusNoContent)
return
}

err := a.processor().MarkAccountForDeletion(*account, keppel.AuditContext{
UserIdentity: authz.UserIdentity,
Request: r,
})
if respondwith.ErrorText(w, err) {
return
}
if resp == nil {
w.WriteHeader(http.StatusNoContent)
} else {
respondwith.JSON(w, http.StatusConflict, resp)
}

w.WriteHeader(http.StatusNoContent)
}

func (a *API) handlePostAccountSublease(w http.ResponseWriter, r *http.Request) {
Expand Down
Loading