cached assets on s3 for all datasets

[AndreaFrancis]

No description provided.

[codecov-commenter]

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (3c6c79b) 87.65% compared to head (40bb174) 82.07%.
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1882      +/-   ##
==========================================
- Coverage   87.65%   82.07%   -5.59%     
==========================================
  Files          22       13       -9     
  Lines         875      926      +51     
==========================================
- Hits          767      760       -7     
- Misses        108      166      +58     

Flag	Coverage Δ
services_admin	?
services_rows	84.16% <100.00%> (?)
services_search	78.55% <33.33%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
services/rows/src/rows/routes/rows.py	57.47% <100.00%> (ø)
services/rows/tests/routes/test_rows.py	93.13% <100.00%> (ø)
services/search/src/search/routes/search.py	53.60% <33.33%> (ø)

... and 32 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[severo]

OK. I was wondering if it's OK to delete the alternative option (fielsystem). But I think it's easy to set it back if needed, right?

Can we also deprecate some environment variables at the same time, or are they all needed yet

[lhoestq]

let's go !

[AndreaFrancis]

OK. I was wondering if it's OK to delete the alternative option (fielsystem). But I think it's easy to set it back if needed, right?

I think so, but the problem is with DNS configuration, not sure if there is a way to redirect some datasets to EFS and others to S3.
Maybe @rtrompier could help us confirm.

[severo]

Oh, yes, I hadn't this in mind. OK to do that. Should we remove /cached-assets from the nginx reverse proxy configuration?

[severo]

(I'm wondering how we will handle the tests: end to end, unit)

[AndreaFrancis]

Oh, yes, I hadn't this in mind. OK to do that. Should we remove /cached-assets from the nginx reverse proxy configuration?

I think so, I created this issue for tracking #1885

[severo]

Currently the docker compose files don't use S3. I think it means we cannot use /rows and /search with docker for images and audio, right? We should be testing this in the e2e tests, I think.

Should we keep the switch between directory and S3 in the code, to support docker compose (and thus: the init tests and the e2e tests)? The alternatives are: a) to include S3 in docker compose (and thus: use or mock S3 for tests), b) to remove docker compose completely and only use k8s/helm (it still means that we need S3 for the tests)

[rtrompier]

I think so, but the problem is with DNS configuration, not sure if there is a way to redirect some datasets to EFS and others to S3.

Nop we can't. The switch is based on url path in cloudfront.
But if you can have a different path between datasets, it should be possible.

[lhoestq]

Another alternative: use a a storage agnostic tool like fsspec to store the images (currently we use boto if I'm not mistaken).
In prod it would point to s3://bucket-name and in docker-compose it would point to a volume shared with the reverse proxy

[severo]

Another alternative: use a a storage agnostic tool like fsspec to store the images (currently we use boto if I'm not mistaken). In prod it would point to s3://bucket-name and in docker-compose it would point to a volume shared with the reverse proxy

I like this option -> I created #1887