🔨 (db) migrate Dataset and Source to knex #3131

sophiamersmann · 2024-01-24T15:40:31Z

Makes the Dataset and Source TypeORM classes obsolete by switching db access to knex
This PR does not do any clean-up, e.g. SQL queries in the router files are not moved, although they ultimately should live in the db folder
I added two helper functions that can be used as drop-in replacements for queryMysql and mysqlFirst that are often used in our codebase, knexRaw and knexRawFirst (both live in db.ts)
Routes that I could easily test are tested, but not all of the code is!

@danyx23 let me know if this is how you imagined it, or if I should do anything differently.

coderabbitai · 2024-01-24T15:40:40Z

Important

Auto Review Skipped

Auto reviews are disabled on base/target branches other than the default branch. Please add the base/target branch pattern to the list of additional branches to be reviewed in the settings.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository.

To trigger a single review, invoke the @coderabbitai review command.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit-tests for this file.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit tests for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository from git and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit tests.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
The JSON schema for the configuration file is available here.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

CodeRabbit Discord Community

Join our Discord Community to get help, request features, and share feedback.

sophiamersmann · 2024-01-24T15:40:53Z

Current dependencies on/for this PR:

🔨 (db) migrate Pageview to knex #3139
🔨 (db) migrate Dataset and Source to knex #3131 👈
🔨 make name on datasets mandatory #3143
🔨 remove obsolete tables and country standardizer #3065
🔨 add types for database rows and prepare use of knex instead of TypeORM #3064 : 1 other dependent PR (#3145 )
Extract grapher and core table types #3056
⚒️ Move types out of utils and into their own types package #3054
master

This stack of pull requests is managed by Graphite.

adminSiteServer/apiRouter.ts

danyx23

Thanks a lot for test driving the knex approach! This looks very good overall. There are two things that I would change on the db-types PR learning from this and two things I'd ask you to change.

The two changs I'd like to see to this PR is:

not make the knex instance optional in the knexRaw and knexRawFirst helper functions
and to wrap all API functions we touch in a transaction scope even if we only read data as we go through them. The reasoning here is that we want reads to be consistent and it's probably better to just implement a standard pattern everywhere rather than having to evaluate if it is necessary each time.

The two things I'll change on the other PR are:

bump the knex version so that we can create readyOnly transactions which are a bit more performant than generic ones
add a migration to make name on datasets mandatory rather than nullable

db/model/Source.ts

danyx23 · 2024-01-26T09:05:54Z

db/db.ts

+    str: string,
+    params?: any[],
+    knex?: Knex<any, any[]>
+): Promise<TRow[]> => (await (knex ?? knexInstance()).raw(str, params ?? []))[0]


I understand that this is convenient but I really think we should only get the knexInstance at the top level entry point and not implicitly in a helper function. This stuff can lead to some nested code using the global knex instance where it actually should be using the knex instance that is handed down from the transaction scope.

Of course in theory you could also just call knexInstance() in a nested utility funciton but then this is an explicit call that is easy to find via grep.

So I'm strongly in favour of making the knex parameter not optional

adminSiteServer/apiRouter.ts

danyx23 · 2024-01-26T11:54:59Z

db/db.ts

+export const knexRawFirst = async <TRow = unknown>(
+    str: string,
+    params?: any[],
+    knex?: Knex<any, any[]>


Dito here of course

danyx23

Thanks a lot for test driving the knex approach! This looks very good overall. There are two things that I would change on the db-types PR learning from this and two things I'd ask you to change.

The two changs I'd like to see to this PR is:

not make the knex instance optional in the knexRaw and knexRawFirst helper functions
and to wrap all API functions we touch in a transaction scope even if we only read data as we go through them. The reasoning here is that we want reads to be consistent and it's probably better to just implement a standard pattern everywhere rather than having to evaluate if it is necessary each time.

The two things I'll change on the other PR are:

bump the knex version so that we can create readyOnly transactions which are a bit more performant than generic ones
add a migration to make name on datasets mandatory rather than nullable

github-actions · 2024-02-15T07:05:49Z

This PR has had no activity within the last two weeks. It is considered stale and will be closed in 3 days if no further activity is detected.

sophiamersmann

Good to go I think!

Before the name field was made non-nullable in the db, I added a check to make sure a dataset name is given. We could get rid of that now :)

sophiamersmann · 2024-02-27T14:39:10Z

db/model/Dataset.ts

+    return dataPackage
+}
+
+export function isDatasetWithName(


I think we can remove this check (and all its references) now since we've made the dataset name non-nullable in the db.

sophiamersmann · 2024-02-27T14:44:30Z

adminSiteServer/apiRouter.ts

        })

        try {
-            await removeDatasetFromGitRepo(dataset.name, dataset.namespace, {
+            await removeDatasetFromGitRepo(dataset.name!, dataset.namespace, {


Again, one of my changes... The '!' shouldn't be necessary anymore

danyx23 · 2024-02-27T15:59:16Z

Merge activity

Feb 27, 10:59 AM EST: @danyx23 started a stack merge that includes this pull request via Graphite.
Feb 27, 10:59 AM EST: @danyx23 merged this pull request with Graphite.

github-actions bot assigned sophiamersmann Jan 24, 2024

sophiamersmann commented Jan 24, 2024

View reviewed changes

adminSiteServer/apiRouter.ts Outdated Show resolved Hide resolved

sophiamersmann commented Jan 24, 2024

View reviewed changes

adminSiteServer/apiRouter.ts Outdated Show resolved Hide resolved

sophiamersmann requested a review from danyx23 January 24, 2024 16:00

sophiamersmann changed the title ~~🔨 (types) migrate Dataset and Source to knex~~ 🔨 (db) migrate Dataset and Source to knex Jan 26, 2024

sophiamersmann force-pushed the db-types branch from 51be3d9 to 92618af Compare January 26, 2024 09:12

sophiamersmann force-pushed the db-types-migrate-dataset-and-source branch from 605992e to ce15753 Compare January 26, 2024 09:12

sophiamersmann mentioned this pull request Jan 26, 2024

🔨 (db) migrate Pageview to knex #3139

Merged

danyx23 requested changes Jan 26, 2024

View reviewed changes

danyx23 force-pushed the db-types branch from 92618af to d4da1d0 Compare January 26, 2024 13:31

danyx23 force-pushed the db-types-migrate-dataset-and-source branch 3 times, most recently from 486942e to ce49d9f Compare January 27, 2024 00:19

danyx23 force-pushed the db-types branch from fc059df to 9831a73 Compare January 27, 2024 01:28

This was referenced Jan 27, 2024

🔨 remove obsolete tables and country standardizer #3065

Merged

🔨 make name on datasets mandatory #3143

Merged

danyx23 changed the base branch from db-types to datasets-table-name-field-nullable January 27, 2024 01:29

danyx23 force-pushed the db-types-migrate-dataset-and-source branch 2 times, most recently from a834c76 to 435d84a Compare January 29, 2024 08:38

danyx23 mentioned this pull request Jan 29, 2024

Convert users and variables types to knex #3145

Merged

danyx23 force-pushed the datasets-table-name-field-nullable branch from 67b9b99 to c28f073 Compare January 29, 2024 14:11

danyx23 force-pushed the db-types-migrate-dataset-and-source branch from 435d84a to a35b059 Compare January 29, 2024 14:11

danyx23 force-pushed the datasets-table-name-field-nullable branch from c28f073 to 253ae3b Compare January 29, 2024 14:15

danyx23 force-pushed the db-types-migrate-dataset-and-source branch from a35b059 to 009d6fb Compare January 29, 2024 14:15

Base automatically changed from datasets-table-name-field-nullable to master January 31, 2024 08:39

mlbrgl mentioned this pull request Feb 13, 2024

Bake posts, pages and blocks from WP API snapshots #3166

Merged

2 tasks

github-actions bot added the stale label Feb 15, 2024

sophiamersmann added the pinned label Feb 15, 2024

danyx23 force-pushed the db-types-migrate-dataset-and-source branch from 009d6fb to 8c71da0 Compare February 22, 2024 13:31

sophiamersmann and others added 2 commits February 27, 2024 10:40

🔨 (types) migrate Dataset and Source to knex

1c129c0

🔨 minor fixes

92444de

danyx23 force-pushed the db-types-migrate-dataset-and-source branch from c538a40 to 92444de Compare February 27, 2024 12:07

🐝 unify access method to knexRaw

1a3b3ea

sophiamersmann commented Feb 27, 2024

View reviewed changes

🔨 incorporate PR feedback

e70e177

danyx23 merged commit a2b1345 into master Feb 27, 2024
19 of 24 checks passed

danyx23 deleted the db-types-migrate-dataset-and-source branch February 27, 2024 15:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🔨 (db) migrate Dataset and Source to knex #3131

🔨 (db) migrate Dataset and Source to knex #3131

sophiamersmann commented Jan 24, 2024 •

edited

Loading

coderabbitai bot commented Jan 24, 2024

Auto Review Skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

CodeRabbit Discord Community

sophiamersmann commented Jan 24, 2024 •

edited

Loading

danyx23 left a comment

danyx23 Jan 26, 2024

danyx23 Jan 26, 2024

danyx23 Jan 26, 2024

danyx23 left a comment

github-actions bot commented Feb 15, 2024

sophiamersmann left a comment

sophiamersmann Feb 27, 2024

sophiamersmann Feb 27, 2024

danyx23 commented Feb 27, 2024 •

edited

Loading

🔨 (db) migrate Dataset and Source to knex #3131

🔨 (db) migrate Dataset and Source to knex #3131

Conversation

sophiamersmann commented Jan 24, 2024 • edited Loading

coderabbitai bot commented Jan 24, 2024

Auto Review Skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

CodeRabbit Discord Community

sophiamersmann commented Jan 24, 2024 • edited Loading

danyx23 left a comment

Choose a reason for hiding this comment

danyx23 Jan 26, 2024

Choose a reason for hiding this comment

danyx23 Jan 26, 2024

Choose a reason for hiding this comment

danyx23 Jan 26, 2024

Choose a reason for hiding this comment

danyx23 left a comment

Choose a reason for hiding this comment

github-actions bot commented Feb 15, 2024

sophiamersmann left a comment

Choose a reason for hiding this comment

sophiamersmann Feb 27, 2024

Choose a reason for hiding this comment

sophiamersmann Feb 27, 2024

Choose a reason for hiding this comment

danyx23 commented Feb 27, 2024 • edited Loading

Merge activity

sophiamersmann commented Jan 24, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)

sophiamersmann commented Jan 24, 2024 •

edited

Loading

danyx23 commented Feb 27, 2024 •

edited

Loading