Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Codebuild fails due to lack of resources #9088

Open
xmfcx opened this issue Oct 15, 2024 · 2 comments
Open

Codebuild fails due to lack of resources #9088

xmfcx opened this issue Oct 15, 2024 · 2 comments
Assignees
Labels
type:ci Continuous Integration (CI) processes and testing. (auto-assigned)

Comments

@xmfcx
Copy link
Contributor

xmfcx commented Oct 15, 2024

Also:
image

https://github.com/autowarefoundation/autoware.universe/actions/runs/11341739442/job/31550579344

@mitsudome-r these are still managing to fail. Let's try to find a solution in the Software WG meeting.

Probably failing because of lack of memory.
These are CodeBuild machines come pre-set for GitHub actions.
I couldn't find a way to assign swap to these machines.

Current options:

  • Limit the number of parallel jobs without using taskset?
  • Reduce the cpus assigned to taskset?
  • Current codebuild cost is about $500 per month, maybe just find another place to simply host about 10 dedicated machines and set up github actions runners on them?

Originally posted by @xmfcx in #8946 (comment)

@xmfcx xmfcx self-assigned this Oct 15, 2024
@xmfcx xmfcx added the type:ci Continuous Integration (CI) processes and testing. (auto-assigned) label Oct 15, 2024
@xmfcx xmfcx moved this to Todo in Software Working Group Oct 15, 2024
@xmfcx
Copy link
Contributor Author

xmfcx commented Oct 15, 2024

Related branch by @mitsudome-r

@xmfcx xmfcx changed the title Codebuild failing due to lack of resources Codebuild fails due to lack of resources Oct 15, 2024
@mitsudome-r
Copy link
Member

The only solution that worked so far was reducing the number of cpu cores used for build:
https://github.com/autowarefoundation/autoware.universe/actions/runs/11345123914

Some other options other that what Fatih mentioned could be:

  • find out a root cause why Autoware build requires so much memory
  • build the packages that require a lot of memory separately (if we can find out a way to manipulate build order)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:ci Continuous Integration (CI) processes and testing. (auto-assigned)
Projects
Status: Todo
Development

No branches or pull requests

2 participants