Reconfigure current VA sitemap to support multiple sitemap files

[timcosgrove]

Requirements

We should set up our primary sitemap file as an index pointing to other sitemap files, so that different systems can manage their own sitemaps without interfering with each other.

Acceptance criteria

Preview Give feedback

1. Next Build content release workflow #17022
   0 of 3
   Accelerated Publishing +1
   
2. Content Build outputs its primary sitemap to a new named file, for example sitemap-cb.xml
3. Next Build outputs its primary sitemap to a new named file, for example sitemap-nb.xml
4. Content Build and Next Build both output a sitemap index to sitemap.xml which sources the above two files
5. It is confirmed that a sitemap index that references a sitemap file that is not there does not cause issues for processing the sitemap.
6. When sitemap.xml is processed, it returns the same set of URLs as the current sitemap.xml
7. When the Content Build changes are on production at va.gov, it is confirmed that the sitemap is working and the site can be spidered as normal.

Background & implementation details

This will be a production change to Content Build and it should be tested and deployed with the care that entails.

Currently both Content Build and Next Build generate sitemaps for the content they each manage. However, they have no awareness of each other's content, so in order to generate a complete sitemap, we need to combine them.

Currently both Content Build and Next Build generate their sitemap to sitemap.xml, which contains a large number of URLs. This should be changed such that both systems generate their large sitemap file to a specifically named sitemap file:

• Content Build generates sitemap-cb.xml
• Next Build generates sitemap-nb.xml

Then, additionally, each system must generate a new sitemap.xml that points to each of these. This sitemap will be a 'sitemap index' that points to the two files above.

See documentation on Sitemap indexes: https://developers.google.com/search/docs/crawling-indexing/sitemaps/large-sitemaps

We should confirm that a sitemap index can contain a reference to a file that is not available and that this does not cause problems. Initially, va.gov will not have the Next Build version of the file available for loading.

Since there's not really a decision yet on which system will initially own the sitemap index at sitemap.xml, it is probably easiest for both systems to output the same index file.

Implementation tasks

Preview Give feedback

1. Use this if it helps you or feel free to delete.