Skip to content

Commit

Permalink
Increase memory and timeout configuration for the transform lambda
Browse files Browse the repository at this point in the history
We started seeing out of memory errors in the transform lambda
when onboarding the variant-nowcast-hub. This is the first time
we've onboarded a hub to S3 that includes sample data, and it
looks like the default memory allocation of 128MB isn't
sufficient.

For example, a single UMass submission is 27MB in memory after
the parquet file is read by pyarrow. We can and should
investigate ways to make the transform process better about
memory, but let's increase the memory now to ensure we're
in a good place until then.
  • Loading branch information
bsweger committed Nov 15, 2024
1 parent 0047104 commit 51f6c50
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion src/hubverse_infrastructure/shared/hubverse_transforms.py
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,8 @@ def create_transform_lambda(
s3_bucket=s3_bucket,
s3_key=s3_key,
tags={"hub": "hubverse"},
timeout=210,
memory_size=500,
timeout=600,
opts=ResourceOptions(depends_on=[hubverse_asset_bucket]),
)

Expand Down

0 comments on commit 51f6c50

Please sign in to comment.