Skip to content
This repository has been archived by the owner on May 10, 2024. It is now read-only.

Implement POST /api/stable/repository/collection/{collectionId}/version #164

Closed
blackandred opened this issue Feb 5, 2022 · 9 comments
Closed

Comments

@blackandred
Copy link
Contributor

blackandred commented Feb 5, 2022

Implement a file upload

Best would be to not save the file to temporary directory, and stream to storage immediately, while being able to analyze the stream on the fly (e.g. if there are GPG headers present).

File upload should:

  • Respect quota limits. With "additional space" in case when upload exceeds the quota a little bit (difference between total backups size and max collection size)
  • Should use a storage adapter to use local filesystem, s3, gcs or other cloud storage
  • Check permissions
  • Rotate uploaded versions (mark old as inactive and delete from storage) AFTER SUCCESSFUL UPLOAD
  • Interface for rotating backups with two implementations: fifo and fifo-plus-older (selectabe rotation strategies)
  • Respect optional "backup windows" to allow only uploading in selected time
  • If user is using a "temporary token" then this token needs to be deactivated RIGHT AFTER SUCCESSFUL UPLOAD
  • Only people with role uploader in given collection (or system-wide with uploader or systemAdmin) can upload
  • Security: Allow for optional validation if file is GPG encrypted (best would be to early exit if there are no GPG headers. Also check if there is a GPG footer at the end of upload)
@blackandred
Copy link
Contributor Author

blackandred commented Feb 5, 2022

Example backup strategy:

strategy: fifo-plus-older
strategySpec:
        keepLastOlderNotMoreThan: 5d
        maxOlderCopies: 2

@blackandred
Copy link
Contributor Author

Related: #156

@blackandred
Copy link
Contributor Author

Related: #97

@blackandred
Copy link
Contributor Author

Related: #122

@blackandred
Copy link
Contributor Author

Related: #127

@blackandred
Copy link
Contributor Author

Related: #140

@blackandred
Copy link
Contributor Author

blackandred commented Feb 16, 2022

GPG validation

curl -s -X POST -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE5NjA0MDQ5NzcsImxvZ2luIjoiYWRtaW4iLCJvcmlnX2lhdCI6MTY0NTA0NDk3N30.uedO-28NdW7KQg9H2sqXkM_PzdgBTlD_iaxc65YXi5A' -F "file=@Makefile" 'http://localhost:8080/api/stable/repository/collection/iwa-ait/version'
{
    "data": {},
    "error": "cannot upload version. cannot upload file, cannot copy stream, error: first chunk of uploaded data does not contain a valid GPG header",
    "status": false
}

blackandred added a commit that referenced this issue Feb 18, 2022
blackandred added a commit that referenced this issue Feb 18, 2022
blackandred added a commit that referenced this issue Feb 20, 2022
blackandred added a commit that referenced this issue Feb 20, 2022
blackandred added a commit that referenced this issue Feb 20, 2022
blackandred added a commit that referenced this issue Feb 20, 2022
blackandred added a commit that referenced this issue Feb 20, 2022
@blackandred
Copy link
Contributor Author

blackandred commented Feb 22, 2022

Locking support

{
    "data": {},
    "error": "cannot upload to same collection in parallel",
    "status": false
}

@blackandred
Copy link
Contributor Author

Implemented.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant