feat(backup): add new command dataset backup for server-side backups #5571

j33ty · 2024-01-26T09:28:17Z

Description

This PR introduces backup command for server-side dataset backups: sanity backup -h. This command primarily intended to be a consumer of Backup APIs and has very little client side logic. If you are a Sanity colleague, kindly refer to #prj-dataset-backups for more context.

Usage

User experience of interaction with the CLI has been kept very similar to using sanity export. The downloaded tar file should be importable through sanity dataset import.

Interaction with server-side backups

; sanity backup -h
usage: sanity backup [--default] [-v|--version] [-d|--debug] [-h|--help] <command> [<args>]

Commands:
   disable   Disable backup for a dataset.
   download  Download a dataset backup to a local file.
   enable    Enable backup for a dataset.
   list      List available backups for a dataset.

See 'sanity help backup <command>' for specific information on a subcommand.

Enable backup for a dataset

Usage

; sanity backup enable -h
usage: sanity backup enable [DATASET_NAME]

   Enable backup for a dataset.

Examples
  sanity backup enable DATASET_NAME

Sample Output

; sanity backup enable production
Enabled daily backups for dataset production.
*Retention policies may apply depending on your plan and agreement.*

List backups

Usage

; sanity backup list -h
usage: sanity backup list [DATASET_NAME]

   List available backups for a dataset.

Options
  --limit <int>     Maximum number of backups returned. Default 30.
  --after <string>  Only return backups after this timestamp (inclusive)
  --before <string> Only return backups before this timestamp (exclusive). Cannot be younger than <after> if specified.

Examples
  sanity backup list DATASET_NAME
  sanity backup list DATASET_NAME --limit 50
  sanity backup list DATASET_NAME --after 2024-01-01 --limit 10
  sanity backup list DATASET_NAME --after 2024-01-01 --before 2024-01-10

Sample Output

; sanity backup list
? Select the dataset name: production
┌──────────┬────────────┬─────────────────────────────────────────────────┐
│ RESOURCE │ CREATED AT │ BACKUP ID                                       │
├──────────┼────────────┼─────────────────────────────────────────────────┤
│ Dataset  │ 2024-01-25 │ 2024-01-25-005f0043-b2c2-4b9b-95a4-4c65222bbdd2 │
│ Dataset  │ 2024-01-25 │ 2024-01-25-40a9c750-142a-473c-8584-0d49fc5b8e6b │
│ Dataset  │ 2024-01-26 │ 2024-01-26-006fd426-4cc6-435f-bbe3-c4ee50068e3b │
│ Dataset  │ 2024-01-26 │ 2024-01-26-58922f75-36cc-4161-8995-3124e9c0434a │
│ Dataset  │ 2024-01-26 │ 2024-01-26-8cb692e4-7ab7-46b9-9917-bf4342074477 │
│ Dataset  │ 2024-01-26 │ 2024-01-26-addfa998-ba2f-4531-95c2-e3633ec8c80b │
└──────────┴────────────┴─────────────────────────────────────────────────┘

Download a backup

Usage

; sanity backup download -h
usage: sanity backup download [DATASET_NAME]

   Download a dataset backup to a local file.

Options
  --backup-id <string> The backup ID to download. (required)
  --out <string>       The file or directory path the backup should download to.
  --overwrite          Allows overwriting of existing backup file.
  --concurrency <num>  Concurrent number of backup item downloads. (max: 24)

Examples
  sanity backup download DATASET_NAME --backup-id 2024-01-01-backup-1
  sanity backup download DATASET_NAME --backup-id 2024-01-01-backup-2 --out /path/to/file
  sanity backup download DATASET_NAME --backup-id 2024-01-01-backup-3 --out /path/to/file --overwrite

Sample Output

; sanity backup download production --backup-id 2024-01-25-005f0043-b2c2-4b9b-95a4-4c65222bbdd2 --out backup/ --overwrite --concurrency 5
╭───────────────────────────────────────────────────────────╮
│                                                           │
│ Downloading backup for:                                   │
│ projectId: h5hc8cgs                                       │
│ dataset: production                                       │
│ backupId: 2024-01-25-005f0043-b2c2-4b9b-95a4-4c65222bbdd2 │
│                                                           │
╰───────────────────────────────────────────────────────────╯

Downloading backup to "/Users/ry/Documents/sanity/sanity/dev/test-studio/out/production-backup-2024-02-02-19083f95-1741-4e29-997a-76b56c773e24.tar.gz"
✔ [2.2s] Setting up backup environment...
✔ [5.4s] Reading backup files... (545/545)
✔ [2.3s] Downloading documents and assets... (545/545)
✔ [130ms] Archiving files into a tarball, 4.16 MB bytes written...
✔ [1ms] Cleaning up temporary files at /var/folders/1/foo/A/backup-1707275511960-78377
✔ Backup download complete [10.2s]

Disable backup for a dataset

Usage

; sanity backup disable -h
usage: sanity backup disable [DATASET_NAME]

   Disable backup for a dataset.

Examples
  sanity backup disable DATASET_NAME

Sample Output

; sanity backup disable production
Disabled daily backups for dataset

Potential Optimizations

For sanity backup download, we are first iterating over all the backup file and then downloading them. This behaviour may not be scalable when number of files is huge. To handle this, we can start downloading files as soon as we are reading them.
For sanity backup download, if it is possible, stream downloaded files directly into the tarball without writing them to disk first.

Notes for release

Since we are not marketing this feature at the moment, it is okay to skip these commits from public release notes.

vercel · 2024-01-26T09:28:24Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
performance-studio	✅ Ready (Inspect)	Visit Preview		Feb 12, 2024 5:14pm
test-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 12, 2024 5:14pm

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
studio-workshop	⬜️ Ignored (Inspect)	Visit Preview		Feb 12, 2024 5:14pm

socket-security · 2024-01-26T22:28:34Z

New dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
npm/@types/[email protected]	None	`+2`	88.1 kB	types
npm/[email protected]	Transitive: environment, filesystem	`+19`	279 kB	ctalkington
npm/[email protected]	filesystem Transitive: environment	`+36`	1.48 MB	ctalkington
npm/[email protected]	None	`0`	54.7 kB	dirtyhairy

View full report↗︎

bjoerge

Looks great! Left a few comments, mostly nice-to-haves.

bjoerge · 2024-02-09T16:32:24Z

packages/sanity/src/_internal/cli/commands/backup/backupGroup.ts

+import {CliCommandGroupDefinition} from '@sanity/cli'
+
+// defaultApiVersion is the backend API version used for dataset backup.
+export const defaultApiVersion = 'vX'


Should this not be a proper API version instead of vX? If it's intentional to keep it at vX, can you add a comment explaining why and what it would take to move it to an official version?

This should be released with vX Content Lake API version. Added the reason inline:

// First version of the backup API is vX since this feature is not yet released // and formal API documentation is pending.

bjoerge · 2024-02-09T16:36:55Z

packages/sanity/src/_internal/cli/commands/backup/downloadBackupCommand.ts

+import cleanupTmpDir from '../../actions/backup/cleanupTmpDir'
+import {defaultApiVersion} from './backupGroup'
+
+const debug = require('debug')('sanity:backup')


Not a blocker, but ideally this should be

Suggested change

const debug = require('debug')('sanity:backup')

import createDebug from 'debug'

const debug = createDebug('sanity:backup')

bjoerge · 2024-02-09T16:37:48Z

packages/sanity/src/_internal/cli/commands/backup/downloadBackupCommand.ts

+const DEFAULT_DOWNLOAD_CONCURRENCY = 10
+const MAX_DOWNLOAD_CONCURRENCY = 24
+
+type DownloadBackupOptions = {


(Nit, so feel free to ignore: we tend to use interface when we can)

Changed type to interface.

bjoerge · 2024-02-09T16:43:06Z

packages/sanity/src/_internal/cli/commands/backup/downloadBackupCommand.ts

+    // Create a unique temporary directory to store files before bundling them into the archive at outputPath.
+    // Temporary directories are normally deleted at the end of backup process, any unexpected exit may leave them
+    // behind, hence it is important to create a unique directory for each attempt.
+    const tmpOutDir = await mkdtemp(path.join(tmpdir(), `backup-`))


would be great if we could add sanity- to the tempfile here so we can leave a hint about who made it.

Goood point. Added it.

binoy14

most of them are nit-picky comments or questions. Not a blocker

packages/sanity/src/_internal/cli/actions/backup/archiveDir.ts

packages/sanity/src/_internal/cli/actions/backup/chooseBackupIdPrompt.ts

packages/sanity/src/_internal/cli/actions/backup/downloadAsset.ts

packages/sanity/src/_internal/cli/actions/backup/archiveDir.ts

packages/sanity/src/_internal/cli/actions/backup/fetchNextBackupPage.ts

packages/sanity/src/_internal/cli/actions/backup/resolveApiClient.ts

packages/sanity/src/_internal/cli/commands/backup/downloadBackupCommand.ts

binoy14 · 2024-02-09T16:41:38Z

packages/sanity/src/_internal/cli/commands/backup/downloadBackupCommand.ts

+  context: CliCommandContext,
+  args: CliCommandArguments,
+): Promise<[SanityClient, DownloadBackupOptions]> {
+  const flags = args.extOptions


Take a look at this PR I believe we want to use yargs to parse the flags, it enforces types and in future we want to move to making sure all flags are defined properly. It will also help remove some of the type checking code below

Good point. Updated code to follow this pattern.

binoy14 · 2024-02-09T16:43:08Z

packages/sanity/src/_internal/cli/commands/backup/listBackupCommand.ts

+  helpText,
+  action: async (args, context) => {
+    const {output, chalk} = context
+    const flags = args.extOptions


same comment here, it would be good to re-parse the arguments using yargs

This is fixed as well.

binoy14 · 2024-02-09T17:14:13Z

packages/sanity/src/_internal/cli/commands/backup/listBackupCommand.ts

+    } catch (error) {
+      const msg = error.statusCode
+        ? error.response.body.message
+        : error.message || error.statusMessage


was able to get this message, maybe a default message if none of the messages are populated?

Fixed in latest commit to print the proper error.

…odule namespace in imports, use interface in place of type, handle unhandled rejection

…et backup command, refactor enable/disable commands, fix backup list CLI to use correct response type and error handling

…selection for backup ID and dataset names, add progress bar

…le names that contain a path segment to prevent archiving failure

… spinner, void using long names for temporary dir to prevent hitting max length limit, handle archive warning, enable compression of archived file by default

…command

…rency safe download of documents, refactor code into modules that can be easily tested, improve progress tracking for dataset backup, install progress-stream correctly, address review feedback, fix API usage for list backups

…odule namespace in imports, use interface in place of type, handle unhandled rejection

…response, use yargs to parse CLI flags

j33ty · 2024-02-12T17:09:31Z

I have rebased from latest next branch with new linter config and reformatted files in this PR.

binoy14

LGTM, thanks!

ricokahler force-pushed the next branch from 57fd3e0 to ab0dbe1 Compare January 26, 2024 14:41

j33ty force-pushed the feat/dataset-backups-cli branch 2 times, most recently from 69c224d to e260037 Compare January 26, 2024 22:27

vercel bot had a problem deploying to Preview – performance-studio January 26, 2024 22:32 Failure

vercel bot had a problem deploying to Preview – test-studio January 26, 2024 22:32 Failure

j33ty force-pushed the feat/dataset-backups-cli branch from e260037 to b5031c9 Compare January 29, 2024 15:46

vercel bot had a problem deploying to Preview – test-studio January 29, 2024 15:50 Failure

vercel bot had a problem deploying to Preview – performance-studio January 29, 2024 15:50 Failure

bjoerge force-pushed the next branch from 958869d to 6fa1750 Compare February 1, 2024 10:53

vercel bot had a problem deploying to Preview – test-studio February 5, 2024 11:59 Failure

vercel bot had a problem deploying to Preview – performance-studio February 5, 2024 11:59 Failure

vercel bot had a problem deploying to Preview – test-studio February 5, 2024 12:24 Failure

vercel bot had a problem deploying to Preview – performance-studio February 5, 2024 12:25 Failure

j33ty force-pushed the feat/dataset-backups-cli branch from 5c6cd3f to e7bd85a Compare February 5, 2024 13:10

vercel bot deployed to Preview – performance-studio February 5, 2024 13:15 View deployment

vercel bot deployed to Preview – test-studio February 5, 2024 13:16 View deployment

vercel bot deployed to Preview – performance-studio February 6, 2024 17:37 View deployment

vercel bot deployed to Preview – test-studio February 6, 2024 17:37 View deployment

vercel bot deployed to Preview – performance-studio February 7, 2024 03:19 View deployment

vercel bot deployed to Preview – test-studio February 7, 2024 03:20 View deployment

j33ty force-pushed the feat/dataset-backups-cli branch from a6e4371 to b9d56e0 Compare February 7, 2024 03:24

vercel bot deployed to Preview – performance-studio February 7, 2024 03:29 View deployment

vercel bot deployed to Preview – test-studio February 7, 2024 03:30 View deployment

j33ty marked this pull request as ready for review February 7, 2024 10:04

j33ty requested a review from a team as a code owner February 7, 2024 10:04

j33ty requested review from binoy14 and removed request for a team February 7, 2024 10:04

vercel bot deployed to Preview – performance-studio February 7, 2024 11:22 View deployment

j33ty self-assigned this Feb 9, 2024

j33ty requested review from bjoerge and binoy14 February 9, 2024 12:20

bjoerge approved these changes Feb 9, 2024

View reviewed changes

binoy14 approved these changes Feb 9, 2024

View reviewed changes

binoy14 reviewed Feb 9, 2024

View reviewed changes

j33ty added a commit that referenced this pull request Feb 12, 2024

fix(cli): address PR #5571 comments in dataset backup CLI, add node m…

7e0cf9b

…odule namespace in imports, use interface in place of type, handle unhandled rejection

vercel bot deployed to Preview – performance-studio February 12, 2024 11:53 View deployment

vercel bot deployed to Preview – test-studio February 12, 2024 11:54 View deployment

vercel bot deployed to Preview – performance-studio February 12, 2024 16:59 View deployment

vercel bot deployed to Preview – test-studio February 12, 2024 17:00 View deployment

codebymatt and others added 8 commits February 12, 2024 18:02

feat(cli): new dataset backup enable/disable commands, add list and g…

c53ebc0

…et backup command, refactor enable/disable commands, fix backup list CLI to use correct response type and error handling

feat(cli): group commands into dataset-backup subcommand, add prompt …

66db7b7

…selection for backup ID and dataset names, add progress bar

feat(cli): Rename dataset backup subcommand to just backup, handle fi…

a875ec2

…le names that contain a path segment to prevent archiving failure

fix(cli): refactor input into its own function; fix progress tracking…

c46c038

… spinner, void using long names for temporary dir to prevent hitting max length limit, handle archive warning, enable compression of archived file by default

fix(CLI): fix dataset backups to be importable through sanity import …

27ccb46

…command

fix(cli): address PR #5571 comments in dataset backup CLI, add node m…

fcd7c74

…odule namespace in imports, use interface in place of type, handle unhandled rejection

fix(cli): in backup CLI, add common action to extract error from API …

b0622b9

…response, use yargs to parse CLI flags

j33ty force-pushed the feat/dataset-backups-cli branch from 6a27918 to b0622b9 Compare February 12, 2024 17:08

j33ty requested review from binoy14 and bjoerge February 12, 2024 17:12

vercel bot deployed to Preview – performance-studio February 12, 2024 17:13 View deployment

vercel bot deployed to Preview – test-studio February 12, 2024 17:14 View deployment

binoy14 approved these changes Feb 12, 2024

View reviewed changes

j33ty added this pull request to the merge queue Feb 12, 2024

Merged via the queue into next with commit f04c76e Feb 12, 2024
40 checks passed

j33ty deleted the feat/dataset-backups-cli branch February 12, 2024 20:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(backup): add new command dataset backup for server-side backups #5571

feat(backup): add new command dataset backup for server-side backups #5571

j33ty commented Jan 26, 2024 •

edited

Loading

vercel bot commented Jan 26, 2024 •

edited

Loading

socket-security bot commented Jan 26, 2024 •

edited

Loading

bjoerge left a comment

bjoerge Feb 9, 2024

j33ty Feb 12, 2024

bjoerge Feb 9, 2024

j33ty Feb 12, 2024

bjoerge Feb 9, 2024

j33ty Feb 12, 2024

bjoerge Feb 9, 2024

j33ty Feb 12, 2024

binoy14 left a comment

binoy14 Feb 9, 2024

j33ty Feb 12, 2024

binoy14 Feb 9, 2024

j33ty Feb 12, 2024

binoy14 Feb 9, 2024

j33ty Feb 12, 2024

j33ty commented Feb 12, 2024

binoy14 left a comment

-const debug = require('debug')('sanity:backup')
+import createDebug from 'debug'
+const debug = createDebug('sanity:backup')

feat(backup): add new command dataset backup for server-side backups #5571

feat(backup): add new command dataset backup for server-side backups #5571

Conversation

j33ty commented Jan 26, 2024 • edited Loading

Description

Usage

Interaction with server-side backups

Enable backup for a dataset

Usage

Sample Output

List backups

Usage

Sample Output

Download a backup

Usage

Sample Output

Disable backup for a dataset

Usage

Sample Output

Potential Optimizations

Notes for release

vercel bot commented Jan 26, 2024 • edited Loading

socket-security bot commented Jan 26, 2024 • edited Loading

bjoerge left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

binoy14 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

j33ty commented Feb 12, 2024

binoy14 left a comment

Choose a reason for hiding this comment

j33ty commented Jan 26, 2024 •

edited

Loading

vercel bot commented Jan 26, 2024 •

edited

Loading

socket-security bot commented Jan 26, 2024 •

edited

Loading