🐛 filter non-editorial content out of sitemap

[ikesau]

Fixes #2726

Uses the wbdb.getPosts() method (and fixes up / simplifies some incorrect typing) which correctly filters out reusable blocks. There's a default filter for posts that end in -country-profile in this method, but AFAICT that's to filter out the templates, not the pages themselves - those still get added:

As well as default country pages:

Also uses the Gdoc.getPublishedGdocs() method which filters out fragments and unpublished documents.

On my local environment:
sitemap.xml length before: 6056
sitemap.xml length after: 6043

The filtered pages:

http://localhost:3030/co2-country-profile
http://localhost:3030/coronavirus-country-profile
http://localhost:3030/energy-country-profile
http://localhost:3030/untitled-reusable-block-128
http://localhost:3030/untitled-reusable-block-129
http://localhost:3030/untitled-reusable-block-130
http://localhost:3030/untitled-reusable-block-131
http://localhost:3030/untitled-reusable-block-145
http://localhost:3030/untitled-reusable-block-171
http://localhost:3030/untitled-reusable-block-180
http://localhost:3030/untitled-reusable-block-266
http://localhost:3030/untitled-reusable-block-275
http://localhost:3030/untitled-reusable-block-277
http://localhost:3030/untitled-reusable-block-295
http://localhost:3030/untitled-reusable-block-7
http://localhost:3030/untitled-reusable-block-77

[mlbrgl]

did you mean this ?

Suggested change

	)
	alreadyPublishedViaGdocsSlugs[0].map((row: any) => row.slug)

I'm not too familiar with knex raw queries but it seems we're getting an array out of it, so
alreadyPublishedViaGdocsSlugs contains an array of rows and fields. Without this change, no slug make it into the set, and end up as duplicates in the sitemap, e.g. diet-affordability (which on my local has been ported over (isGdocPublished = 1))

[ikesau]

Damn! Good catch! I'd just copied the code (it was on the cusp of my DRY threshold) without checking it. My local didn't have any published Gdocs successors when I diffed the sitemaps so I didn't notice it wasn't working.

I've extracted the function and fixed it! Thanks 🙂

[mlbrgl]

Nice reuse of the current APIs!

[mlbrgl]

I noticed an existing bug related to this that you might want to address here: #2864