Skip to content

Commit

Permalink
Add storing single charts to local R2 bucket to syncGraphersToR2 tool
Browse files Browse the repository at this point in the history
The syncGraphersToR2 tool let's you sync all needed entries from the chart_configs table to an actual R2 bucket. This is useful for production and staging servers but in local dev the R2 bindings currently can't connect to an actual R2 bucket and instead simulate one using a local folder.

This PR adds a tool to store single chart configs by slug from the database into the local R2 bucket by invoking the `wrangler` cli tool for storing R2 objects.

---

For more details, open the [Copilot Workspace session](https://copilot-workspace.githubnext.com/owid/owid-grapher?shareId=XXXX-XXXX-XXXX-XXXX).
  • Loading branch information
danyx23 committed Aug 13, 2024
1 parent ed7d910 commit 92d502e
Showing 1 changed file with 37 additions and 1 deletion.
38 changes: 37 additions & 1 deletion devTools/syncGraphersToR2/syncGraphersToR2.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ import {
import { string } from "ts-pattern/dist/patterns.js"
import { chunk, take } from "lodash"
import ProgressBar from "progress"
import { getEnrichedChartBySlug } from "../../db/model/Chart.js"
import { exec } from "child_process"

type HashAndId = Pick<DbRawChartConfig, "fullMd5" | "id">

Expand Down Expand Up @@ -277,7 +279,41 @@ async function sync(parsedArgs: parseArgs.ParsedArgs, dryRun: boolean) {
}

async function storeDevBySlug(parsedArgs: parseArgs.ParsedArgs, dryRun: boolean) {
console.log("Dummy implementation for store-dev-by-slug")
const slug = parsedArgs._[1]
if (!slug) {
console.error("No slug provided")
return
}

await knexReadonlyTransaction(async (trx) => {
const chart = await knexRawFirst<DbRawChartConfig>(

Check failure on line 289 in devTools/syncGraphersToR2/syncGraphersToR2.ts

View workflow job for this annotation

GitHub Actions / bundlewatch

Cannot find name 'knexRawFirst'.

Check failure on line 289 in devTools/syncGraphersToR2/syncGraphersToR2.ts

View workflow job for this annotation

GitHub Actions / testdbcheck

Cannot find name 'knexRawFirst'.

Check failure on line 289 in devTools/syncGraphersToR2/syncGraphersToR2.ts

View workflow job for this annotation

GitHub Actions / bundlewatch

Cannot find name 'knexRawFirst'.

Check failure on line 289 in devTools/syncGraphersToR2/syncGraphersToR2.ts

View workflow job for this annotation

GitHub Actions / testdbcheck

Cannot find name 'knexRawFirst'.
trx,
`SELECT full FROM chart_configs WHERE slug = ?`,
[slug]
)
if (!chart) {
console.error(`No chart found for slug ${slug}`)
return
}

const fullConfig = JSON.stringify(chart.full)
const command = `npx wrangler r2 object put --local $GRAPHER_CONFIG_R2_BUCKET/$GRAPHER_CONFIG_R2_BUCKET_PATH/grapher/by-slug/${slug}.json --pipe --content-type application/json --persist-to ./cfstorage`

const process = exec(command, (error, stdout, stderr) => {
if (error) {
console.error(`Error executing wrangler command: ${error.message}`)
return
}
if (stderr) {
console.error(`Wrangler stderr: ${stderr}`)
return
}
console.log(`Wrangler stdout: ${stdout}`)
})

process.stdin.write(fullConfig)

Check failure on line 314 in devTools/syncGraphersToR2/syncGraphersToR2.ts

View workflow job for this annotation

GitHub Actions / bundlewatch

'process.stdin' is possibly 'null'.

Check failure on line 314 in devTools/syncGraphersToR2/syncGraphersToR2.ts

View workflow job for this annotation

GitHub Actions / testdbcheck

'process.stdin' is possibly 'null'.

Check failure on line 314 in devTools/syncGraphersToR2/syncGraphersToR2.ts

View workflow job for this annotation

GitHub Actions / bundlewatch

'process.stdin' is possibly 'null'.

Check failure on line 314 in devTools/syncGraphersToR2/syncGraphersToR2.ts

View workflow job for this annotation

GitHub Actions / testdbcheck

'process.stdin' is possibly 'null'.
process.stdin.end()

Check failure on line 315 in devTools/syncGraphersToR2/syncGraphersToR2.ts

View workflow job for this annotation

GitHub Actions / bundlewatch

'process.stdin' is possibly 'null'.

Check failure on line 315 in devTools/syncGraphersToR2/syncGraphersToR2.ts

View workflow job for this annotation

GitHub Actions / testdbcheck

'process.stdin' is possibly 'null'.

Check failure on line 315 in devTools/syncGraphersToR2/syncGraphersToR2.ts

View workflow job for this annotation

GitHub Actions / bundlewatch

'process.stdin' is possibly 'null'.

Check failure on line 315 in devTools/syncGraphersToR2/syncGraphersToR2.ts

View workflow job for this annotation

GitHub Actions / testdbcheck

'process.stdin' is possibly 'null'.
})
}

async function main(parsedArgs: parseArgs.ParsedArgs) {
Expand Down

0 comments on commit 92d502e

Please sign in to comment.