[Epic][AP] CMS Public S3 file asset storage

[timcosgrove]

Background

CMS-managed file assets - images and PDFs primarily, that are added by CMS editors as part of their content - are stored in a file store that is within the VA network, like the CMS itself is. This means that when CMS content data that references these files, the URLs are to locations that are not accessible by regular users.

Content Build solves this problem by downloading every file asset used in VA.gov pages with every single content release. This is time-consuming.

To solve this problem for Next Build, we have set up file sync from the CMS file stores to public S3 buckets that can operate as web servers. Files are synced automatically as they are added to the CMS; the sync typically takes a few minutes.

As pages are built by Next Build, we need to substitute the URL to a given file as the CMS understands it with a URL to the public S3 bucket location. This will need to be done on all text where file references could existn.

Assumptions

• In all cases here we are speaking of file assets that are managed by the CMS and delivered with a page - images and PDFs primarily.

Acceptance Criteria

• Files that are saved in the production or staging CMS are stored in a publicly-reachable S3 bucket acting as a web server.
• All existing CMS-managed files are migrated to these same S3 buckets
• Next Build is updated to make use of these new locations.
• Optionally: Content Build is also updated to make use of these locations and the old download functionality is removed.

Tasks

Preview Give feedback

1. Defect: BRD Staging environment does not contain files from Production #18437
   CMS Team +1
   

Completed Tasks

Preview Give feedback

1. Set up DataSync between remaining EFS and Public S3 #16929
   0 of 2
   Accelerated Publishing Needs refining +2
   
2. Rewrite Next Build file paths to use public S3 locations. #16930
   0 of 1
   Accelerated Publishing Icebox Needs refining +3
   
3. Next Build should use a custom request header when requesting API data #18071
   0 of 3
   Accelerated Publishing Needs refining +2
   
4. Allow CMS to use different public file base URL on request #18070
   0 of 3
   Accelerated Publishing Needs refining +2
   
5. Set up DataSync between Staging Test EFS and Staging Test Public S3 #16925
   2 of 2
   Accelerated Publishing Needs refining +2
   
6. Set up DataSync between Prod Test EFS and Prod Test Public S3 #16926
   0 of 2
   Accelerated Publishing Needs refining +2
   
7. Set up DataSync between Staging EFS and Staging Public S3 #16928
   0 of 2
   Accelerated Publishing Needs refining +2

[timcosgrove]

See #10894 for anything useful.

[timcosgrove]

Note that this epic was written with the assumption of a more managed solution to files. We have also proposed 'dumb sync' where anything written by the CMS to its asset file system automatically syncs to S3.

We need to investigate that proposal, write it up, put out an RFC