feat: grapher page related research from DB content graph #3176

mlbrgl · 2024-02-07T09:32:29Z

This PR replaces the fortunejs content graph with a DB-based alternative, using the links stored in the posts_links and posts_gdoc_links tables.

This PR also continues the deprecation work started in #3166.

fully rebake site on staging
fixed malformed prominent link in content (ourworldindata.org/ instead of https://ourworldindata.org/)
- https://owid.cloud/wp/wp-admin/post.php?post=34103&action=edit
- https://owid.cloud/wp/wp-admin/post.php?post=34066&action=edit

Inconsistency on how grapher chart links are stored in the DB: there are a handful of chart links stored with the https://ourworldindata.org/grapher prefix. These disappear upon saving the containing articles, which indicates that they were created with an older version of the codebase. I'm then opting for not supporting them in the content graph.

    SELECT pgl.*, pg.published from posts_gdocs_links pgl
    JOIN posts_gdocs pg on pg.id = pgl.sourceId
    WHERE pgl.target LIKE "https://ourworldindata.org/grapher%"

Testing links

Below are some testing links to grapher pages, backlinking to posts in different configurations.

mlbrgl · 2024-02-07T09:32:43Z

Current dependencies on/for this PR:

master
- PR feat(wp): add wp api snapshot migration #3165
  - PR Bake posts, pages and blocks from WP API snapshots #3166
    - PR feat: grapher page related research from DB content graph #3176 👈
      - PR Merge content graph and chart references #3179
        
        PR refactor: bake embeds from DB content graph #3182
        
        PR Add redirects table and migration script #3185
        
        PR refactor: bake redirects from db #3186
        
        PR feat: add about page template #3190
        
        PR refactor: pass knex as parameter #3194

This stack of pull requests is managed by Graphite.

coderabbitai · 2024-02-07T09:32:47Z

Important

Auto Review Skipped

Auto reviews are disabled on base/target branches other than the default branch. Please add the base/target branch pattern to the list of additional branches to be reviewed in the settings.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository.

To trigger a single review, invoke the @coderabbitai review command.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit-tests for this file.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit tests for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository from git and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit tests.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
The JSON schema for the configuration file is available here.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

CodeRabbit Discord Community

Join our Discord Community to get help, request features, and share feedback.

mlbrgl · 2024-02-07T11:25:42Z

Back to the drawing board, evaluating whether getReferencesByChartId could be a good fit.

mlbrgl · 2024-02-07T19:59:44Z

Turns out it is: #3179. Given the refactor in the next PR, this one then becomes more of a stepping stone than a final destination. Comments regarding the content graph backreferences query should rather be made in #3179.

danyx23

Looks good! Two minor comments but neither is very important

danyx23 · 2024-02-13T16:00:40Z

db/model/Chart.ts

+                    MAX(chart_tags.keyChartLevel) as keyChartLevel
+                FROM charts
+                INNER JOIN chart_tags ON charts.id=chart_tags.chartId
+                WHERE JSON_CONTAINS(config->'$.dimensions', '{"variableId":${variableId}}')


I think it's better to join on the chart_dimensions table rather than this json_contains (normal tables are much faster to filter than nested json in mysql)

thanks, that makes sense. This was only moved during a refactor so I made an issue for it: #3203.

danyx23 · 2024-02-13T18:23:47Z

db/model/Post.ts

+            -- note: we are not filtering by linkType to cast of wider net: if a post links to an
+            -- explorer having the same slug as the grapher chart, we want to surface it as
+            -- a "Related research" as it is most likely relevant.


I don't quite get this comment - can you try to explain to me in different words what you mean?

The idea here was to not restrict linkType to "grapher" only, so a post embedding an explorer sharing the same slug as the chart would also be returned (next to posts linking to the chart only). This was more of an edge case optimization which I ended up dropping when I merged references (in the grapher admin) with related posts (in the grapher page) in a58fea7.

mlbrgl · 2024-02-19T17:02:44Z

Merge activity

Feb 19, 12:02 PM EST: @mlbrgl started a stack merge that includes this pull request via Graphite.
Feb 19, 12:03 PM EST: Graphite rebased this pull request as part of a merge.
Feb 19, 12:04 PM EST: @mlbrgl merged this pull request with Graphite.

This PR merges the content graph queries used in grapher pages (see #3176) and the references queries used in the chart admin to indicate where a chart is being used. If fixes two issues on the tab page: - Overridden Wordpress posts now ignored - Remove duplicate posts Reusable blocks are no longer surfaced. This is ok since reusable blocks are now dereferenced in the embedding article, which means a chart within a reusable block would show in the parent article. One exception is explorer content blocks, which are still standalone. A quick manual check on all explorers reveals that only one explorer references a chart: https://ourworldindata.org/explorers/plastic-pollution. Given these blocks' scope and limited lifetime, this caveat is acceptable. _A future PR should use these references during the check performed before deleting a chart. Currently, [only gdocs are being checked](https://github.com/owid/owid-grapher/blob/e5acb212754bbba7d5d563fcd856adbb32ab8a36/adminSiteServer/apiRouter.ts#L603-L611)._ ### Before <img width="546" alt="Screenshot 2024-02-07 at 17 07 50" src="https://github.com/owid/owid-grapher/assets/13406362/1a4c25b8-3398-44b7-b14c-10c752d442d3"> ### After <img width="530" alt="Screenshot 2024-02-07 at 20 38 49" src="https://github.com/owid/owid-grapher/assets/13406362/1fad824e-8613-4825-9342-6d18a089d900"> ### Testing links Below are some testing links to grapher pages, backlinking to posts in different configurations. - gdoc: [https://ourworldindata.org/grapher/agricultural-export-subsidies](https://ourworldindata.org/grapher/agricultural-export-subsidies) - [x] [http://localhost:3030/grapher/agricultural-export-subsidies](http://localhost:3030/grapher/agricultural-export-subsidies) - [x] [http://staging-site-content-graph-references/grapher/agricultural-export-subsidies](http://staging-site-content-graph-references/grapher/agricultural-export-subsidies) - gdocs: [https://ourworldindata.org/grapher/pollution-deaths-from-fossil-fuels](https://ourworldindata.org/grapher/pollution-deaths-from-fossil-fuels) - [x] [http://localhost:3030/grapher/pollution-deaths-from-fossil-fuels](http://localhost:3030/grapher/pollution-deaths-from-fossil-fuels) - [x] [http://staging-site-content-graph-references/grapher/pollution-deaths-from-fossil-fuels](http://staging-site-content-graph-references/grapher/pollution-deaths-from-fossil-fuels) - wp: [https://ourworldindata.org/grapher/dalys-rate-from-all-causes](https://ourworldindata.org/grapher/dalys-rate-from-all-causes) - [x] [http://localhost:3030/grapher/dalys-rate-from-all-causes](http://localhost:3030/grapher/dalys-rate-from-all-causes) - [x] [http://staging-site-content-graph-references/grapher/dalys-rate-from-all-causes](http://staging-site-content-graph-references/grapher/dalys-rate-from-all-causes) - wp (with chart redirect): [https://ourworldindata.org/grapher/age-standardized-death-rate-from-pm25-pollution-per-100000-vs-gdp-per-capita-int-](https://ourworldindata.org/grapher/age-standardized-death-rate-from-pm25-pollution-per-100000-vs-gdp-per-capita-int-) - [x] [http://localhost:3030/grapher/age-standardized-death-rate-from-pm25-pollution-per-100000-vs-gdp-per-capita-int-](http://localhost:3030/grapher/age-standardized-death-rate-from-pm25-pollution-per-100000-vs-gdp-per-capita-int-) - [x] [http://staging-site-content-graph-references/grapher/age-standardized-death-rate-from-pm25-pollution-per-100000-vs-gdp-per-capita-int-](http://staging-site-content-graph-references/grapher/age-standardized-death-rate-from-pm25-pollution-per-100000-vs-gdp-per-capita-int-) - gdoc (with chart redirect): [https://ourworldindata.org/grapher/population-long-run-with-projections?time=earliest..2100&country=~OWID_WRL](https://ourworldindata.org/grapher/population-long-run-with-projections?time=earliest..2100&country=~OWID_WRL) - [x] [http://localhost:3030/grapher/population-long-run-with-projections?time=earliest..2100&country=~OWID_WRL](http://localhost:3030/grapher/population-long-run-with-projections?time=earliest..2100&country=~OWID_WRL) - [x] [http://staging-site-content-graph-references/grapher/population-long-run-with-projections?time=earliest..2100&country=~OWID_WRL](http://staging-site-content-graph-references/grapher/population-long-run-with-projections?time=earliest..2100&country=~OWID_WRL) - none: [https://ourworldindata.org/grapher/death-rates-alcohol-drug-overdoses-by-age-who](https://ourworldindata.org/grapher/death-rates-alcohol-drug-overdoses-by-age-who) - [x] [http://localhost:3030/grapher/death-rates-alcohol-drug-overdoses-by-age-who](http://localhost:3030/grapher/death-rates-alcohol-drug-overdoses-by-age-who) - [x] [http://staging-site-content-graph-references/grapher/death-rates-alcohol-drug-overdoses-by-age-who](http://staging-site-content-graph-references/grapher/death-rates-alcohol-drug-overdoses-by-age-who)

github-actions bot assigned mlbrgl Feb 7, 2024

This was referenced Feb 7, 2024

feat(wp): add wp api snapshot migration #3165

Merged

Bake posts, pages and blocks from WP API snapshots #3166

Merged

mlbrgl marked this pull request as ready for review February 7, 2024 10:14

mlbrgl requested a review from danyx23 February 7, 2024 10:14

mlbrgl marked this pull request as draft February 7, 2024 11:25

mlbrgl force-pushed the bake-from-snapshot branch from cb1d456 to 8265f67 Compare February 7, 2024 13:45

mlbrgl force-pushed the db-content-graph branch from e6ae373 to 346822f Compare February 7, 2024 13:45

mlbrgl mentioned this pull request Feb 7, 2024

Merge content graph and chart references #3179

Merged

12 tasks

mlbrgl marked this pull request as ready for review February 7, 2024 19:55

mlbrgl mentioned this pull request Feb 8, 2024

refactor: bake embeds from DB content graph #3182

Merged

mlbrgl changed the title ~~feat: content graph from db~~ feat: grapher page related research from DB content graph Feb 8, 2024

This was referenced Feb 8, 2024

Tracking issue: sunsetting Wordpress #3097

Closed

Add redirects table and migration script #3185

Merged

refactor: bake redirects from db #3186

Merged

mlbrgl force-pushed the bake-from-snapshot branch from 8265f67 to c2e9291 Compare February 13, 2024 08:41

mlbrgl force-pushed the db-content-graph branch from 346822f to 3b47356 Compare February 13, 2024 08:41

mlbrgl mentioned this pull request Feb 13, 2024

feat: add about page template #3190

Merged

danyx23 approved these changes Feb 13, 2024

View reviewed changes

mlbrgl force-pushed the db-content-graph branch from 3b47356 to 8568bca Compare February 13, 2024 20:37

mlbrgl force-pushed the bake-from-snapshot branch from 8c1eac2 to 6c8f189 Compare February 14, 2024 08:38

mlbrgl force-pushed the db-content-graph branch from 8568bca to 445f734 Compare February 14, 2024 08:38

This was referenced Feb 14, 2024

refactor: pass knex as parameter #3194

Merged

refactor: rewrite getRelatedChartsForVariable query #3203

Closed

mlbrgl force-pushed the bake-from-snapshot branch from 6c8f189 to 85cf4fe Compare February 18, 2024 08:59

mlbrgl force-pushed the db-content-graph branch from 445f734 to 05ecc94 Compare February 18, 2024 08:59

mlbrgl force-pushed the db-content-graph branch from 05ecc94 to bbcf834 Compare February 19, 2024 14:02

mlbrgl force-pushed the bake-from-snapshot branch from f15615f to e93d1ba Compare February 19, 2024 14:35

mlbrgl force-pushed the db-content-graph branch from bbcf834 to 3ba5145 Compare February 19, 2024 14:35

Base automatically changed from bake-from-snapshot to master February 19, 2024 17:03

mlbrgl added 10 commits February 19, 2024 17:03

feat: content graph from db

ccc9452

refactor: remove unused fortune content graph code

05f8128

refactor(wp): move getRelatedArticles to Post

7e23550

refactor(wp): move getPermalinks to Post

2d9cf46

refactor(wp): move getPostTags to Post

c39d345

refactor(wp): move getRelatedChartsForVariable to Chart

f3bd95c

refactor(wp): move getRelatedResearchAndWritingForVariable to Post

129794c

refactor(wp): deprecate ENTRIES_CATEGORY_ID

fa68aa8

refactor(wp): mark FOR_SYNC_ONLY

3a76907

refactor(wp): deprecate OWID_API_ENDPOINT

08fe279

mlbrgl force-pushed the db-content-graph branch from 3ba5145 to 08fe279 Compare February 19, 2024 17:03

mlbrgl merged commit ee5bd52 into master Feb 19, 2024
14 of 16 checks passed

mlbrgl deleted the db-content-graph branch February 19, 2024 17:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: grapher page related research from DB content graph #3176

feat: grapher page related research from DB content graph #3176

mlbrgl commented Feb 7, 2024 •

edited

Loading

mlbrgl commented Feb 7, 2024 •

edited

Loading

coderabbitai bot commented Feb 7, 2024 •

edited

Loading

Auto Review Skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

CodeRabbit Discord Community

mlbrgl commented Feb 7, 2024

mlbrgl commented Feb 7, 2024

danyx23 left a comment

danyx23 Feb 13, 2024

mlbrgl Feb 15, 2024

danyx23 Feb 13, 2024

mlbrgl Feb 15, 2024

mlbrgl commented Feb 19, 2024 •

edited

Loading

feat: grapher page related research from DB content graph #3176

feat: grapher page related research from DB content graph #3176

Conversation

mlbrgl commented Feb 7, 2024 • edited Loading

Testing links

mlbrgl commented Feb 7, 2024 • edited Loading

coderabbitai bot commented Feb 7, 2024 • edited Loading

Auto Review Skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

CodeRabbit Discord Community

mlbrgl commented Feb 7, 2024

mlbrgl commented Feb 7, 2024

danyx23 left a comment

Choose a reason for hiding this comment

danyx23 Feb 13, 2024

Choose a reason for hiding this comment

mlbrgl Feb 15, 2024

Choose a reason for hiding this comment

danyx23 Feb 13, 2024

Choose a reason for hiding this comment

mlbrgl Feb 15, 2024

Choose a reason for hiding this comment

mlbrgl commented Feb 19, 2024 • edited Loading

Merge activity

mlbrgl commented Feb 7, 2024 •

edited

Loading

mlbrgl commented Feb 7, 2024 •

edited

Loading

coderabbitai bot commented Feb 7, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)

mlbrgl commented Feb 19, 2024 •

edited

Loading