-
Notifications
You must be signed in to change notification settings - Fork 76
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'catalyst:MOODLE_402_STABLE' into MOODLE_402_STABLE
- Loading branch information
Showing
122 changed files
with
5,403 additions
and
484 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| # Migration guides | ||
|
|
||
| ## Migrating from local_azure_storage to local_azureblobstorage | ||
|
|
||
| Since March 2024, Microsoft officially discontinued support for the PHP SDK for Azure storage. This means the `local_azure_storage` plugin which is a wrapper of the SDK will no longer be updated, and is already out of date with newer php versions. | ||
|
|
||
| The plugin `local_azureblobstorage` was created to replace this, with a simpler and cleaner API for interacting with the Azure blob storage service via REST APIs. Objectfs has been updated with a new client handler class to enable you to cut over to the new storage system as easily as possible. | ||
|
|
||
| This new library is only supported in higher PHP versions. | ||
|
|
||
| ### Steps | ||
| 1. If you are on Moodle 4.2, ensure you have updated the previous `local_azure_storage` to the `MOODLE_42_STABLE` branch. This fixes some fatal errors caused by Guzzle namespace conflicts. | ||
| 2. Install `local_azureblobstorage` https://github.com/catalyst/moodle-local_azureblobstorage | ||
| 3. In the objectfs settings, change the `filesystem` config variable to `\tool_objectfs\azure_blob_storage_file_system` and save. ObjectFS will now be using the new API to communicate with Azure. You do not need to enter new credentials, the credentials are shared with the old client. | ||
| 4. Test and ensure the site works as expected. | ||
| 5. If you encounter any issues and wish to revert back, simply change the `filesystem` configuration back to the old client. This will immediately begin to use the old libraries again. | ||
| 6. Once you are happy, simply uninstall the `local_azure_storage` plugin. The migration is now complete. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| # Tagging | ||
| Tagging allows extra metadata about your files to be send to the external object store. These sources are defined in code, and currently cannot be configured on/off from the UI. | ||
|
|
||
| Currently, this is only implemented for the S3 file system client. | ||
| **Tagging vs metadata** | ||
|
|
||
| Note object tags are different from object metadata. | ||
|
|
||
| Object metadata is immutable, and attached to the object on upload. With metadata, if you wish to update it (for example during a migration, or the sources changed), you have to copy the object with the new metadata, and delete the old object. This is not ideal, since deletion is optional in objectfs. | ||
|
|
||
| Object tags are more suitable, since their permissions can be managed separately (e.g. a client can be allowed to modify tags, but not delete objects). | ||
|
|
||
| ## File system setup | ||
| ### S3 | ||
| [See the S3 docs for more information about tagging](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html). | ||
|
|
||
| You must allow `s3:GetObjectTagging` and `s3:PutObjectTagging` permission to the objectfs client. | ||
|
|
||
| ## Sources | ||
| The following sources are implemented currently: | ||
| ### Environment | ||
| What environment the file was uploaded in. Configure the environment using `taggingenvironment` in the objectfs plugin settings. | ||
|
|
||
| This tag is also used by objectfs to determine if tags can be overwritten. See [Multiple environments setup](#multiple-environments-setup) for more information. | ||
|
|
||
| ### Location | ||
| Either `orphan` if the file no longer exists in the `files` table in Moodle, otherwise `active`. | ||
|
|
||
| ## Multiple environments setup | ||
| This feature is designed to work in situations where multiple environments (e.g. prod, staging) points to the same bucket, however, some setup is needed: | ||
|
|
||
| 1. Turn off `overwriteobjecttags` in every environment except the production environment. | ||
| 2. Configure `taggingenvironment` to be unique for all environments. | ||
|
|
||
| By doing the above two steps, it will allow the production environment to always set its own tags, even if a file was first uploaded to staging and then to production. | ||
|
|
||
| Lower environments can still update tags, but only if the `environment` matches theirs. This allows staging to manage object tags on objects only it knows about, but as soon as the file is uploaded from production (and therefore have it's environment tag replaced with `prod`), staging will no longer touch it. | ||
|
|
||
| ## Migration | ||
| Only new objects uploaded after enabling this feature will have tags added. To backfill tags for previously uploaded objects, you must do the following: | ||
|
|
||
| - Manually run `trigger_update_object_tags` scheduled task from the UI, which queues a `update_object_tags` adhoc task that will process all objects marked as needing sync. | ||
| or | ||
| - Call the CLI to execute a `update_object_tags` adhoc task manually. | ||
|
|
||
| You may need to update the DB to mark objects tag sync status as needing sync if the object has previously been synced before. | ||
| ## Reporting | ||
| There is an additional graph added to the object summary report showing the tag value combinations and counts of each. | ||
|
|
||
| Note, this is only for files that have been uploaded from the respective environment, and may not be consistent for environments where `overwriteobjecttags` is disabled (because the site does not know if a file was overwritten in the external store by another client). | ||
|
|
||
| ## For developers | ||
|
|
||
| ### Adding a new source | ||
| Note the rules about sources: | ||
| - Identifier must be < 32 chars long. | ||
| - Value must be < 128 chars long. | ||
|
|
||
| While external providers allow longer key/values, we intentionally limit it to reserve space for future use. These limits may change in the future as the feature matures. | ||
|
|
||
| To add a new source: | ||
| - Implement `tag_source` | ||
| - Add to the `tag_manager` class | ||
| - As part of an upgrade step, mark all objects `tagsyncstatus` to needing sync (using `tag_manager` class, or manually in the DB) | ||
| - As part of an upgrade step, queue a `update_object_tags` adhoc task to process the tag migration. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| <?php | ||
| // This file is part of Moodle - http://moodle.org/ | ||
| // | ||
| // Moodle is free software: you can redistribute it and/or modify | ||
| // it under the terms of the GNU General Public License as published by | ||
| // the Free Software Foundation, either version 3 of the License, or | ||
| // (at your option) any later version. | ||
| // | ||
| // Moodle is distributed in the hope that it will be useful, | ||
| // but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| // GNU General Public License for more details. | ||
| // | ||
| // You should have received a copy of the GNU General Public License | ||
| // along with Moodle. If not, see <http://www.gnu.org/licenses/>. | ||
|
|
||
| namespace tool_objectfs; | ||
|
|
||
| use tool_objectfs\local\store\azure_blob_storage\file_system; | ||
|
|
||
| /** | ||
| * File system for Azure Blob Storage. | ||
| * | ||
| * This file tells objectfs that this storage system is available for use. | ||
| * E.g. via $CFG->alternative_file_system_class | ||
| * | ||
| * @package tool_objectfs | ||
| * @author Matthew Hilton <[email protected]> | ||
| * @copyright Catalyst IT | ||
| * @license http://www.gnu.org/copyleft/gpl.html GNU GPL v3 or later | ||
| */ | ||
| class azure_blob_storage_file_system extends file_system { | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| <?php | ||
| // This file is part of Moodle - http://moodle.org/ | ||
| // | ||
| // Moodle is free software: you can redistribute it and/or modify | ||
| // it under the terms of the GNU General Public License as published by | ||
| // the Free Software Foundation, either version 3 of the License, or | ||
| // (at your option) any later version. | ||
| // | ||
| // Moodle is distributed in the hope that it will be useful, | ||
| // but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| // GNU General Public License for more details. | ||
| // | ||
| // You should have received a copy of the GNU General Public License | ||
| // along with Moodle. If not, see <http://www.gnu.org/licenses/>. | ||
|
|
||
| namespace tool_objectfs\check; | ||
|
|
||
| use core\check\check; | ||
| use core\check\result; | ||
| use core\task\manager; | ||
| use html_table; | ||
| use html_writer; | ||
| use tool_objectfs\task\update_object_tags; | ||
|
|
||
| /** | ||
| * Tagging migration status check | ||
| * | ||
| * @package tool_objectfs | ||
| * @author Matthew Hilton <[email protected]> | ||
| * @copyright Catalyst IT | ||
| * @license http://www.gnu.org/copyleft/gpl.html GNU GPL v3 or later | ||
| */ | ||
| class tagging_migration_status extends check { | ||
| /** | ||
| * Link to ObjectFS settings page. | ||
| * | ||
| * @return \action_link|null | ||
| */ | ||
| public function get_action_link(): ?\action_link { | ||
| $url = new \moodle_url('/admin/category.php', ['category' => 'tool_objectfs']); | ||
| return new \action_link($url, get_string('pluginname', 'tool_objectfs')); | ||
| } | ||
|
|
||
| /** | ||
| * Get result | ||
| * @return result | ||
| */ | ||
| public function get_result(): result { | ||
| // We want to check this regardless if enabled or supported and not exit early. | ||
| // Because it may have been turned off accidentally thus causing the migration to fail. | ||
| $tasks = manager::get_adhoc_tasks(update_object_tags::class); | ||
|
|
||
| if (empty($tasks)) { | ||
| return new result(result::NA, get_string('tagging:migration:nothingrunning', 'tool_objectfs')); | ||
| } | ||
|
|
||
| $table = new html_table(); | ||
| $table->head = [ | ||
| get_string('table:taskid', 'tool_objectfs'), | ||
| get_string('table:iteration', 'tool_objectfs'), | ||
| get_string('table:status', 'tool_objectfs'), | ||
| ]; | ||
|
|
||
| foreach ($tasks as $task) { | ||
| $table->data[$task->get_id()] = [$task->get_id(), $task->get_iteration(), $task->get_status_badge()]; | ||
| } | ||
| $html = html_writer::table($table); | ||
|
|
||
| $ataskisfailing = !empty(array_filter($tasks, function($task) { | ||
| return $task->get_fail_delay() > 0; | ||
| })); | ||
|
|
||
| if ($ataskisfailing) { | ||
| return new result(result::WARNING, get_string('check:tagging:migrationerror', 'tool_objectfs'), $html); | ||
| } | ||
|
|
||
| return new result(result::OK, get_string('check:tagging:migrationok', 'tool_objectfs'), $html); | ||
| } | ||
| } |
Oops, something went wrong.