[content-service] download s3 content using s5cmd

[kylos101]

Description

Use s5cmd to download s3 content

Summary generated by Copilot

🤖 Generated by Copilot at 61aa5c9

Improve S3 backup download speed and reliability by using s5cmd in content-service and ws-daemon.

Related Issue(s)

Fixes ENG-884

How to test

Start workspaces from prebuilds in dev-internal, compare with dogfood. dogfood should take more than 2 times longer to start a workspace from a prebuild. Use the Gitpod mono repo project.

Refer to this gist to see how the related parameters were decided on for s5cmd.

Documentation

Preview status

gitpod:summary

Build Options

Build

• /werft with-werft
  Run the build with werft instead of GHA
• leeway-no-cache
• /werft no-test
  Run Leeway with --dont-test

Publish

• /werft publish-to-npm
• /werft publish-to-jb-marketplace

Installer

• analytics=segment
• with-dedicated-emulation
• workspace-feature-flags
  Add desired feature flags to the end of the line above, space separated

Preview Environment / Integration Tests

• /werft with-local-preview
  If enabled this will build install/preview
• /werft with-preview
• /werft with-large-vm
• /werft with-gce-vm
  If enabled this will create the environment on GCE infra
• with-integration-tests=all
  Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh. If enabled, with-preview and with-large-vm will be enabled.
• with-monitoring

/hold

[kylos101]

👋 @aledbf I am going to deploy this branch to dev-internal, and pin the cell to it.

edit: I'll do some isolated testing, to observe performance. For example, when the maximum # of workspaces are starting on one node. If it works well, we can merge and ship to dogfood.

[kylos101]

👋 @aledbf I am going to deploy this branch to dev-internal, and pin the cell to it.

edit: I'll do some isolated testing, to observe performance. For example, when the maximum # of workspaces are starting on one node. If it works well, we can merge and ship to dogfood.

edited: @aledbf ☝️

[kylos101]

dev-internal metrics with s5cmd (2.25s snapshot download):

cat /workspace/.gitpod/ready
{"source":"from-prebuild","metrics":[{"type":"snapshot","duration":9250783765,"size":2253484032},{"type":"prebuild","duration":10256348384,"size":2253484032}]}

dogfood metrics with AWS SDK (11.6s snapshot download, for the same prebuild):

cat /workspace/.gitpod/ready
{"source":"from-other","metrics":[{"type":"snapshot","duration":11624917355,"size":2225299456},{"type":"prebuild","duration":12751727933,"size":2225905664}]}

🚀

[kylos101]

I am seeing some errors on workspace restart and stop:

Oh, no! Something went wrong!
Please try reloading the page. If the issue continues, please [get in touch](mailto:[email protected]?Subject=Gitpod%20Dashboard%20Error&Body=%0A%0AError%3A%20%5Bresource_exhausted%5D%20too%20many%20requests%20If%20this%20error%20is%20unexpected%2C%20please%20quote%20the%20request%20ID%20%27%27%20when%20reaching%20out%20to%20Gitpod%20Support.%0ACode%3A%208).

Reload
Code: 8
[resource_exhausted] too many requests If this error is unexpected, please quote the request ID '' when reaching out to Gitpod Support.

This is pretty easily recreated by starting/restarting/stopping ~4 workspaces at once. I'll investigate more on Monday - to see if this is related or not.

[kylos101]

I am seeing some errors on workspace restart and stop:

Oh, no! Something went wrong!
Please try reloading the page. If the issue continues, please [get in touch](mailto:[email protected]?Subject=Gitpod%20Dashboard%20Error&Body=%0A%0AError%3A%20%5Bresource_exhausted%5D%20too%20many%20requests%20If%20this%20error%20is%20unexpected%2C%20please%20quote%20the%20request%20ID%20%27%27%20when%20reaching%20out%20to%20Gitpod%20Support.%0ACode%3A%208).

Reload
Code: 8
[resource_exhausted] too many requests If this error is unexpected, please quote the request ID '' when reaching out to Gitpod Support.

This is pretty easily recreated by starting/restarting/stopping ~4 workspaces at once. I'll investigate more on Monday - to see if this is related or not.

I was able to recreate this problem in dev-internal this morning w/o this change. Checked the logs, and created https://linear.app/gitpod/issue/EXP-696/