sysops: Add backup and restore docs, reorganize sysops docs

docs/index.md

@@ -19,10 +19,12 @@ To develop using Skytable and maintain your deployment you will want to learn ab
       - [**DCL**](blueql/dcl): Data control with BlueQL
     - [**Querying**](querying): Introduces different query modes and when to choose a specific query mode
     - [**System administration**](system):
-      - [**Configuration**](system/configuration): Information to help you configure Skytable with custom settings such as custom ports, hosts, TLS, and etc.
-      - [**User management**](system/user-management): Information on access control, user and other administration features
-      - [**Global management**](system/global-management): Global settings management
-      - [**Operations**](system/operations): Learn about administration operations
+      - [**Configuration**](system/configuration): Configuration modes (CLI, environment variables, configuration files) and options
+      - [**User management**](system/user-management): Account types, permissions, creating and managing multiple users
+      - [**Global management**](system/global-management): Learn how to check system health and manage the global state of your database instances
+      - [**Disk usage**](system/disk-usage): Understand disk usage and compaction
+      - [**Backup and restore**](system/backup-and-restore): Backing up data and restoring data from backups
+      - [**Data recovery**](system/recovery): Understanding data loss, mitigation and recovery options
     - **Resources**:
       - [**Useful links**](resources/useful-links): Links to helpful resources
       - [**Migration**](resources/migration): For old our returning Skytable users who are coming from older versions

docs/system/1.configuration.md → docs/system/a.configuration.md

@@ -44,7 +44,7 @@ To start the server with a configuration file, simply run `skyd --config <path t
 Here's an explanation of all the keys:
 - `system`:
   - `mode`: set to either `dev` / `prod` mode. `prod` mode will generally make some things stricters (such as background services)
-  - `rs_window`: **This is a very important setting!** It is set to `300` by default and is called the "reliability service window" which ensures that if any changes are observed in `300` (or whatever value you set) seconds, then they reach the disk as soon as that time elapses. For example, in the default configuration the system checks for changes every 5 minutes and if there are any dataset changes, they are immediately synced. [Read more here](operations#understanding-data-loss)
+  - `rs_window`: **This is a very important setting!** It is set to `300` by default and is called the "reliability service window" which ensures that if any changes are observed in `300` (or whatever value you set) seconds, then they reach the disk as soon as that time elapses. For example, in the default configuration the system checks for changes every 5 minutes and if there are any dataset changes, they are immediately synced. [Read more here](recovery#understanding-data-loss)
 - `auth`:
   - `plugin`: this is the authentication plugin. we currently only have `pwd` that is a simple password based authentication system where the password is stored as an [`rcrypt` hash](https://github.com/ohsayan/rcrypt) on disk. More `plugin` options are set to be implemented for more advanced authentication, especially in enterprise settings
   - `root_pass`: this is the root account password. **It must have atleast 16 characters**

docs/system/3.global-management.md → docs/system/c.global-management.md

@@ -11,7 +11,7 @@ The following query returns an `Empty` response or an error code depending on th
 SYSCTL REPORT STATUS
 ```
 
-If you receive an error code, we recommend you to connect to the host and check logs. If the server has crashed, you may need to [recover the database](operations#data-recovery).
+If you receive an error code, we recommend you to connect to the host and check logs. If the server has crashed, you may need to [recover the database](recovery).
 
 ## Inspecting all spaces
 

docs/system/d.disk-usage.md

@@ -0,0 +1,31 @@
+---
+id: disk-usage
+title: Disk usage
+---
+
+## Directory structure
+
+This is the general directory structure (subdirectories omitted):
+```
+├── data
+├── gns.db-tlog
+└── .sky_pid
+```
+
+- `gns.db-tlog` (file): This is a very important file that stores system tables and other data
+- `data` (directory): This directory contains subdirectories with all the spaces (which in turn contain all the data for each space)
+- `.sky_pid` (file): This is a temporary PID file that is created whenever the database is started. If the database crashes, then you may have to remove
+  it manually
+
+
+## Managing disk usage
+
+Over time, as you continue to use your database your database files will grow in size, as you would expect. However, sometimes database files may grow beyond an efficient size resulting in high memory usage or slowdowns. To counter this, Skytable uses internal heuristics to determine when a database file is "larger than needed" and automatically compacts them at startup.
+
+However, in some cases you may wish to perform a compaction regardless in order to reduce the file size. In order to do this you will have to run:
+
+```sh
+skyd compact
+```
+
+The server will then compact all files (even if a compaction wasn't triggered by internal heuristics) to their optimum size.

docs/system/e.backup-and-restore.md

@@ -0,0 +1,62 @@
+---
+id: backup-and-restore
+title: Backup and restore
+---
+
+## Backing up data
+
+To back data up, you can use the subcommand `skyd backup` as follows:
+
+```sh
+skyd backup --type=direct --to=<path to backup> [--from <directory>]
+```
+
+- `--type=direct`: This specifies the kind of backup created. The `direct` type indicates that it's a simple copy of the data files and directories
+- `--to=<path to backup>`: This specifies where this backup is to be created
+- `--from <path to installation>` *(optional)*: When this is not provided, the `backup` subcommand assumes that the current working directory is the installation directory. If you're running it from a different directory then set this option.
+
+**Example**:
+
+```sh
+skyd backup \
+    --type=direct \
+    --from=/var/lib/skytable \
+    --to=/mnt/backupnfsdrive/quick-backup-before-migration
+```
+
+:::info Backup types
+Note that in the future we may add more backup types including compressed archives or other modes. The only type of backup (specified using `--type`) is `direct` which clones the data files and directories. But you do not need to worry about this as the restore subcommand will take care of determining what kind of backup is being pointed to.
+:::
+
+### Backup protections
+
+The `backup` subcommand includes some protections to create consistent and valid backups. These include not allowing backups if the database is currently using the data files and some other parameters. If you need to override any of these parameters, then please check the help menu with `skyd backup --help`.
+
+## Restoring data
+
+To restore data from a backup, you can use the subcommand `skyd restore` as follows:
+
+```sh
+skyd restore --from=<path to backup> [--to <installation directory>]
+```
+
+- `--from=<path to backup>`: Specifies the path to the backup
+- `--to <installation directory>` *(optional)*: By default, it is assumed that the current directory is the installation directory. If not, set this option.
+
+**Example**:
+
+```sh
+skyd restore \
+    --from=/mnt/backupnfsdrive/quick-backup-before-migration \
+    --to=/var/lib/skytable
+```
+
+### Data restore protections
+
+The `restore` subcommand also has some safeguards in place that prevent you from accidentally restoring incorrect data. Some of these safeguards include:
+
+- **Backup has correct time signatures**
+- **Backup is compatible**
+- **Was created by the same host:** you will obviously need to override this when recovering from a crash and this should be okay to do. The reason this protection exists is in a situation where you're running a cluster and have multiple backups and accidentally restore from the wrong backup.
+
+If you need to override any of these conditions in special cases, then please check the help menu with `skyd restore --help`.

docs/system/operations.md → docs/system/f.recovery.md

@@ -1,20 +1,8 @@
 ---
-title: Operations
+id: recovery
+title: Data recovery
 ---
 
-## Managing disk usage
-
-Over time, as you continue to use your database your database files will grow in size, as you would expect. However, sometimes database files may grow beyond an efficient size resulting in high memory usage or slowdowns. To counter this, Skytable uses internal heuristics to determine when a database file is "larger than needed" and automatically compacts them at startup.
-
-However, in some cases you may wish to perform a compaction regardless in order to reduce the file size. In order to do this you will have to run:
-
-```sh
-skyd compact
-```
-
-The server will then compact all files (even if a compaction wasn't triggered by internal heuristics) to their optimum size.
-
-## Data recovery
 
 In the unforeseen event that a power failure or other catastrophic system failure causes the database to crash, the Skytable server will fail to start normally. Usually it will exit with a nonzero code and an error message such as "journal-corrupted." In such cases, you will need to recover the journal(s) and/or any other corrupted file(s).
 

docs/system/index.md

@@ -8,7 +8,9 @@ In the following sections, we explore general system administration options with
 
 Here's an overview of the different administration guides:
 
-- [**Configuration**](configuration): Understand how Skytable can be configured using command-line arguments, environment variables or a configuration file and what all configuration options are available
-- [**User management**](user-management): Learn about account types, permissions and how you can manage multiple users
+- [**Configuration**](configuration): Configuration modes (CLI, environment variables, configuration files) and options
+- [**User management**](user-management): Account types, permissions, creating and managing multiple users
 - [**Global management**](global-management): Learn how to check system health and manage the global state of your database instances
-- [**Operations**](operations): Understand administrator operations tasks such as backups, recovery and more
+- [**Disk usage**](disk-usage): Understand disk usage and compaction
+- [**Backup and restore**](backup-and-restore): Backing up data and restoring data from backups
+- [**Data recovery**](recovery): Understanding data loss, mitigation and recovery options

docusaurus.config.js

@@ -163,8 +163,8 @@ module.exports = {
             to: '/protocol/specification'
           },
           {
-            from: '/system/recovery',
-            to: '/system/operations#data-recovery'
+            from: '/system/operations',
+            to: '/system',
           }
         ]
       }]

sidebars.ts

@@ -27,7 +27,9 @@ module.exports = {
         "system/configuration",
         "system/user-management",
         "system/global-management",
-        "system/operations",
+        "system/disk-usage",
+        "system/backup-and-restore",
+        "system/recovery",
       ],
       link: {
         type: 'doc',