Added backup pruning (#19)

* Added backup pruning (#18)

* Added settings for pruning

* Separated the date parsing logic

* Refactored to accomodate pruning

* Refactored to accomodate pruning

* Tided up the prune functions

* Updated readme

* Updated the readme

* Added emojis

* Resized logos

* Updated readme

* Updated the readme

* Fixed some bugs

* Updated recursive functions

* Removed logging and updated gitignore

* Updated the changelog

.gitignore

@@ -22,6 +22,7 @@ composer.lock
 *.esproj
 *.sublime-workspace
 *.sublime-project
+craft-test.code-workspace
 *.tmproj
 *.tmproject
 .vscode/*

CHANGELOG.md

@@ -89,3 +89,10 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/) and this p
 
 - Add missing console command
 - Updated docs
+
+## 1.1.0 - 2020-02-08
+
+### Changed
+
+- Added backup pruning
+- Bugfixes

README.md

@@ -9,7 +9,7 @@ Backup and restore your database and volume assets across environments from the
 - Craft CMS 3 or later on Linux or MacOS (untested on Windows as of yet)
 - A private AWS S3 bucket for backups
 
-## Installation
+## 💾 Installation
 
 To install the plugin, follow these instructions.
 
@@ -22,12 +22,12 @@ To install the plugin, follow these instructions.
 2. Then tell Composer to load the plugin:
 
    ```sh
-   $ composer require weareferal/env-sync
+   composer require weareferal/env-sync
    ```
 
 3. In the Control Panel, go to Settings → Plugins and click the *Install* button for Craft Env Sync.
 
-## Overview
+## 🏔 Overview
 
 Craft Env Sync is a plugin that makes it possible to sync your database and volume assets across numerous environments from the comfort of the Craft Control Panel. This makes it much easier to move your site from local development to staging and onto production.
 
@@ -42,13 +42,17 @@ This plugin uses AWS S3 (more providers to come soon) as the "single source of t
 
 For more information read our blog post on ["Syncing your DB and assets across environments in Craft 3"](https://weareferal.com/tldr/syncing-your-db-and-assets-across-environments-in-craft-3/) or get in touch at [[email protected]](mailto:[email protected])
 
-This plugin is inspired by ![Andrew Welsch's `craft-scripts` library](https://github.com/nystudio107/craft-scripts) who also [has a great blog post on syncing you DB and assets in Craft](https://nystudio107.com/blog/database-asset-syncing-between-environments-in-craft-cms).
+This plugin is inspired by [Andrew Welsch's `craft-scripts` library](https://github.com/nystudio107/craft-scripts) who also [has a great blog post on syncing you DB and assets in Craft](https://nystudio107.com/blog/database-asset-syncing-between-environments-in-craft-cms).
 
-## Configuration
+## 🔧 Configuration
+
+Configuration is done through the dedicated "Sync" settings panel. 
+
+### Provider
 
 ![Craft Env Sync Setting Screenshot](resources/img/settings-screenshot.png)
 
-Configuration is done through the dedicated "Sync" settings panel. The details entered here correspond to your AWS S3 account and bucket that you want to use for backups. It's recommended to set up a new IAM user that has programmatic access (meaning via a acces/secret key) to a private S3 bucket.
+The details entered here correspond to your AWS S3 account and bucket that you want to use for backups. It's recommended to set up a new IAM user that has programmatic access (meaning via a acces/secret key) to a private S3 bucket.
 
 Once you have set this bucket up, you can either enter your AWS S3 details directly into the setting page, or you can use environment variables via your `.env` file (this is the recommended approach as seen in the screenshot above). This latter approach is more portable and secure as it prevents any private access/secret key values being included in files that you might commit to Github. Furthermore is means these variables can be reused in other plugins etc.
 
@@ -57,20 +61,20 @@ Here is an example portion of a  `.env` file:
 ```sh
 ...
 
-AWS_ACCESS_KEY = 
-AWS_SECRET_KEY = 
+AWS_ACCESS_KEY =
+AWS_SECRET_KEY =
 AWS_REGION = "us-west-2"
 AWS_BUCKET_NAME = "feral-backups"
 AWS_BUCKET_PREFIX = "craft-backups/my-site"
 ```
 
-## Usage
+## 💻 Usage
 
 ### Control Panel
 
 ![Craft Env Sync Utilities Screenshot](resources/img/utilities-screenshot.png)
 
-Once you have entered your settings variables you should be able to use the "sync" tab on the "utilities" section of the control panel.
+Once you have entered your settings variables you should be able to use the *Sync* tab on the *Utilities* section of the control panel.
 
 There are two broad sections: one for the database and one for volume assets. Each section has four options to create a local backup, push that local backup to S3, pull all remote backups _from_ S3 and finally to restore a particular backup.
 
@@ -80,34 +84,122 @@ There are also console commands available for creating, pushing and pulling back
 
 ```sh
 - env-sync/database                         Sync database backups
-    env-sync/database/create-backup         Create a local database backup
+    env-sync/database/create                Create a local database backup
+    env-sync/database/prune                 Prune database backups
     env-sync/database/pull                  Pull remote database backups from cloud
     env-sync/database/push                  Push local database backups to cloud
 
-- env-sync/volumes                          Sync volumes backup
-    env-sync/volumes/create-backup          Create a local volumes backup
-    env-sync/volumes/pull                   Pull remote volume backups from cloud
-    env-sync/volumes/push                   Push local volume backups to cloud
+- env-sync/volume                          Sync volumes backup
+    env-sync/volume/create                 Create a local volumes backup
+    env-sync/volume/prune                  Prune volume backups
+    env-sync/volume/pull                   Pull remote volume backups from cloud
+    env-sync/volume/push                   Push local volume backups to cloud
 ```
 
 For example:
 
 ```sh
-./craft env-sync/database/create-backup
+./craft env-sync/database/create
 ```
 
 These commands can be used alongside cron or your deployment scripts to automatically/periodically create backups.
 
-## Functionality
+## 📝 Functionality
+
+![Image of backups](resources/img/backup-screenshot.png)
 
 All local backups are stored in the existing `storage/backups` folder that Craft uses for its own database backup script.
 
-For database backups and restorations we use the existing Craft scripts - they are just included in our dashboard as an easier consolodated interface.
+- Database backups are created in a similar manner to the native Craft backup utility. In fact, the plugin uses this script behind-the-scenes, it just uses a slightly different naming scheme.
+- For volume assets backups, we simply create a versioned zip file containing the handles of all volume assets currently saved in the system.
+
+All backups have the following filename structure:
+
+```sh
+my_site_dev_200202_200020_yjrnz62yj4_v3.3.20.1.sql
+```
+
+Which includes:
+
+- Your site name
+- Your current environment
+- Date & time of backup
+- Random 10 character string
+- Craft version
+
+It's important not to manually rename these files as the plugin relies on this structure.
+
+To create new backups and push/pull backups to the cloud you can use the "Sync" utility
+
+### Queue
+
+You can choose to use the Craft queue to perform create, push and pull operations from the CP utilities section. To enable use of the queue, toggle the "Use Queue" setting.
+
+#### Control Panel errors
+
+When not using the queue, if there is an issue pulling/pushing a backup you will get feedback (an alert box). You won't get the same feedback when using the queue. Instead it will look like the operation has been successful. To see if the operation was actually successul you'll need to check the queue manually.
+
+#### CLI commands and the queue
+
+The CLI commands ignore the queue setting. In other words, they will always run synchrously. This is by design as it's likely you will want to see the results of these operations if they are part of your crontab or deployment script.
+
+### Deleting/Pruning old backups
+
+![Pruning settings](resources/img/pruning-screenshot.png)
+
+Env Sync supports pruning/deleting of old backups. To enable this feature toggle the "Prune Backup" setting. When you toggle this setting you will see a number of inputs for controlling the number of backups to be retained for a number of backup periods: daily, weekly, monthly, yearly. By default Env Sync will keep:
+
+- The 14 most recent daily backups
+- The earliest backups of the 4 most recent weeks
+- The earliest backups of the 6 most recent months
+- The earliest backups of the 3 most recent years
+
+Bear in mind that depending on how many backups you have, some of these might overlay. For example, a backup that is retained as part of the 4 most recent weekly backups might also be one of the 6 most recent monthly backups too.
+
+When enabled, backups will be pruned whenever a new backup is created via the Control Panel. Backups will be pruned independently. In other words, if you create a database backup, only the old database backups will be deleted, not the volume backups. You can also prune database or volume backups independently on the command line:
+
+```sh
+./craft env-sync/database/prune
+./craft env-sync/volume/prune
+```
+
+Bear in mind these commands _will_ respect the settings. In other words, you won't be able to prune backups via the command line if the setting in the control panel is disabled.
+
+Also note that the `./craft env-sync/[database|volume]/create` command does not automatically run the `prune` command (unlike the control panel).
+
+### Automating backups
+
+There is no built-in way to automate backups (periodic queue jobs are't something supported by the Craft queue). That said, it's very easy to automate backups either via cron or your own deployment script (if using Forge for example).
+
+#### Cron
+
+Here is an example daily cron entry to backup and prune daily at 01:00:
+
+```cron
+00 01 * * * /path/to/project/craft env-sync/database/prune
+05 01 * * * /path/to/project/craft env-sync/database/create
+10 01 * * * /path/to/project/craft env-sync/volume/prune
+15 01 * * * /path/to/project/craft env-sync/volume/create
+```
+
+### Providers
+
+The plugin has been built with the ability to add new providers relatively easily using a backend system. Currently the only provider available is AWS S3.
+
+If you require another provider, please leave an issue on Github.
+
+## 🚨 Troubleshooting
+
+If you are getting errors while pushing/pulling/creating/restoring or pruning, the first thing to check is the Craft logs at `storage/logs/web.log`.
+
+### Credentials
+
+For pushing and pulling, the most likely issue is with your credentials, so double check that those are OK.
 
-For volume assets backups, we simply create a versioned zip file containing the handles of all volume assets currently saved in the system. Bear in mind if you have a large number of assets this process may take some time and take up a significant amount of storage.
+### Memory limit creating volumes
 
-## Troubleshooting
+When you create a new volume backup, it's possible that your PHP memory limit will cause the process to crash. Make sure your memory limit is > than the volume folder you are trying to backup.
 
-If you are getting errors when you try to pull/push databases or assets, the first thing to check is the Craft logs at `storage/logs/web.log`. All errors should be logged here. The most likely issue is with your credentials, so double check that those are OK.
+## 📞 Credits and support
 
-Brought to you by [Feral](https://weareferal.com). Any issues email [[email protected]](mailto:[email protected]?subject=Craft%20Env%20Sync%20Question)
+Brought to you by [Feral](https://weareferal.com). Any problems email [[email protected]](mailto:[email protected]?subject=Craft%20Env%20Sync%20Question) or leave an issue on Github.

src/Sync.php

@@ -13,12 +13,8 @@
 
 use Craft;
 use craft\base\Plugin;
-use craft\web\UrlManager;
 use craft\services\Utilities;
-use craft\services\Plugins;
-use craft\events\PluginEvent;
 use craft\events\RegisterComponentTypesEvent;
-use craft\events\RegisterUrlRulesEvent;
 use craft\events\RegisterUserPermissionsEvent;
 use craft\services\UserPermissions;
 
@@ -27,6 +23,7 @@
 use weareferal\sync\utilities\SyncUtility;
 use weareferal\sync\models\Settings;
 use weareferal\sync\services\SyncService;
+use weareferal\sync\assets\SyncSettingAsset;
 
 
 class Sync extends Plugin
@@ -82,7 +79,10 @@ protected function createSettingsModel(): Settings
 
     protected function settingsHtml(): string
     {
-        return Craft::$app->getView()->renderTemplate(
+        $view = Craft::$app->getView();
+        $view->registerAssetBundle(SyncSettingAsset::class);
+        $view->registerJs("new Craft.SyncSettings('main-form');");
+        return $view->renderTemplate(
             'env-sync/settings',
             [
                 'settings' => $this->getSettings()

src/assets/SyncSettingAsset.php

@@ -0,0 +1,21 @@
+<?php
+namespace weareferal\sync\assets;
+
+use craft\web\AssetBundle;
+use craft\web\assets\cp\CpAsset;
+
+
+class SyncSettingAsset extends AssetBundle
+{
+    public function init()
+    {
+        $this->sourcePath = __DIR__ . '/dist';
+        $this->depends = [
+            CpAsset::class,
+        ];
+        $this->js = [
+            'SyncSetting.js'
+        ];
+        parent::init();
+    }
+}

src/assets/SyncAsset.php → src/assets/SyncUtilityAsset.php

@@ -5,7 +5,7 @@
 use craft\web\assets\cp\CpAsset;
 
 
-class SyncAsset extends AssetBundle
+class SyncUtilityAsset extends AssetBundle
 {
     public function init()
     {

src/assets/dist/SyncSetting.js

@@ -0,0 +1,21 @@
+(function($) {
+    Craft.SyncSettings = Garnish.Base.extend(
+        {
+            init: function(formId) {
+                this.$form = $('#' + formId);
+                this.$pruneLightswitch = $("#settings-prune", this.form);
+
+                this.addListener(this.$pruneLightswitch, 'change', 'updatePruneSettings');
+            },
+
+            updatePruneSettings: function(ev) {
+                var $lightswitch = this.$pruneLightswitch.data('lightswitch');
+                var $settings = $("#settings-env-sync-prune-settings", this.form);
+                if ($lightswitch.on) {                    
+                    $settings.show();
+                } else {
+                    $settings.hide();
+                }
+            }
+        });
+})(jQuery);