Split big sitemaps

[stefandoorn]

According to https://www.sitemaps.org/faq.html:

Sitemaps should be no larger than 50MB (52,428,800 bytes) and can contain a maximum of 50,000 URLs. These limits help to ensure that your web server does not get bogged down serving very large files.

It would be good that sitemaps have a default of maximum 50.000 items (configurable) after which they'll be split up. The first file of a specific provider should have the default name (as now) and the consecutive files should have a follow up number appended. Also the sitemap index should point to the proper files.

E.g.: sitemap/products.xml & sitemap/products_2.xml.

Alternative:

sitemap_index.xml refers to sitemap/products.xml which itself is a sitemap index again which points to sitemap/products/1.xml & sitemap/products/2.xml. Not sure though that is allowed, I guess there is a maximum inheritance to index files.

Relates to #78.

[loevgaard]

Nested indices are not allowed (https://support.google.com/webmasters/answer/7451001).

[Gamesh]

@loevgaard while nested indices are not allowed you can reference multiple product sitemaps in the main sitemap

https://developers.google.com/search/docs/crawling-indexing/sitemaps/large-sitemaps

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/sitemap1.xml.gz</loc>
    <lastmod>2024-08-15</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemap2.xml.gz</loc>
    <lastmod>2022-06-05</lastmod>
  </sitemap>
</sitemapindex>