diff --git a/00-prerequisites/README.md b/00-prerequisites/README.md index c07f200..9994459 100644 --- a/00-prerequisites/README.md +++ b/00-prerequisites/README.md @@ -2,16 +2,16 @@ To be able to participate in the workshop exercises that will take place, you will have to have a basic setup of development tools on your machine. We have -prepared a variety of solutions for you to achieve this setup. +prepared a variety of solutions for you to achieve this setup. Which solution should I use? It depends :-) -- **VirtualBox:** Good for in-person workshops. Ensures coherent setup for all +- **VirtualBox:** Good for in-person workshops. Ensures coherent setup for all participants, and makes life easier for trainers. -- **Local setup:** Good for new developers. Ensures your environment is ready +- **Local setup:** Good for new developers. Ensures your environment is ready doing development after having completed the tutorials. -- **Vagrant:** Similar to VirtualBox, but requires to also have Vagrant - installed. +- **Vagrant:** Similar to VirtualBox, but requires to also have Vagrant + installed. ## VirtualBox (for workshops) diff --git a/01-getting-started/README.md b/01-getting-started/README.md index 751ef03..dc71a01 100644 --- a/01-getting-started/README.md +++ b/01-getting-started/README.md @@ -13,8 +13,8 @@ First ensure you have prepared your environment according to First, open a terminal and checkout the trainings' source code: ```bash -$ cd ~/src -$ git clone https://github.com/inveniosoftware/training.git +cd ~/src +git clone https://github.com/inveniosoftware/training.git ``` **Tip:** To copy/paste into the terminal inside the Ubuntu virtual machine @@ -25,7 +25,7 @@ use: Ctrl+Shift+V (paste), Ctrl+Shift+C (copy), Ctrl+Shift+X (cut). Scaffold the skeleton for your first Invenio instance: ```bash -$ cookiecutter gh:inveniosoftware/cookiecutter-invenio-instance -c v3.4 --no-input +cookiecutter gh:inveniosoftware/cookiecutter-invenio-instance -c v3.4 --no-input ``` ## Step 4: Install @@ -33,14 +33,14 @@ $ cookiecutter gh:inveniosoftware/cookiecutter-invenio-instance -c v3.4 --no-inp Navigate to the scaffolded code, and start the Docker services (database, Elasticsearch, RabbitMQ and Redis cache): ```bash -$ cd my-site -$ docker-compose up -d +cd my-site +docker-compose up -d ``` Install and build the Python and NPM dependencies: ```bash -$ ./scripts/bootstrap +./scripts/bootstrap ``` ## Step 5: Run @@ -48,19 +48,19 @@ $ ./scripts/bootstrap Setup the database tables, search indexes, queues and caches: ```bash -$ ./scripts/setup +./scripts/setup ``` Start a development server and background job worker: ```bash -$ ./scripts/server +./scripts/server ``` Last, open [https://127.0.0.1:5000/](https://127.0.0.1:5000/) in your browser: ```bash -$ firefox https://127.0.0.1:5000/ +firefox https://127.0.0.1:5000/ ``` Firefox and other browsers will display a security warning because we are trying diff --git a/03-infrastructure-tour/README.md b/03-infrastructure-tour/README.md index 9a83ea7..374226b 100644 --- a/03-infrastructure-tour/README.md +++ b/03-infrastructure-tour/README.md @@ -28,11 +28,11 @@ commands: ```bash # Build our Invenio application images first -$ ./docker/build-images.sh -$ docker-compose -f docker-compose.full.yml up -d +./docker/build-images.sh +docker-compose -f docker-compose.full.yml up -d ``` -To make sure our instance is running properly, open +To make sure our instance is running properly, open Here's a full diagram of what the `docker-compose.full.yml` infrastructure looks like: @@ -232,11 +232,11 @@ The load balancer, being at the edge of our infrastructure, besides serving the web application at , is also exposing a statistics panel at : -![](./images/haproxy.png) +![HAProxy](./images/haproxy.png) ## What did we learn -![](./images/diagram-labels.png) +![Diagram Labels](./images/diagram-labels.png) - The different services composing an Invenio instance - How to interface with them on a basic level diff --git a/04-running-invenio/README.md b/04-running-invenio/README.md index 5d3ddcb..2acb5dd 100644 --- a/04-running-invenio/README.md +++ b/04-running-invenio/README.md @@ -22,8 +22,8 @@ First, let's bring back the basic development container setup: ```bash # Bring down the full setup -$ docker-compose -f docker-compose.full.yml stop -$ docker-compose up -d +docker-compose -f docker-compose.full.yml stop +docker-compose up -d ``` We are now running only the database, Elasticsearch, Redis and RabbitMQ diff --git a/05-customizing-invenio/README.md b/05-customizing-invenio/README.md index 4740c59..d78ed43 100644 --- a/05-customizing-invenio/README.md +++ b/05-customizing-invenio/README.md @@ -155,7 +155,7 @@ If we want to be more precise and change a concrete CSS rule we can add it direc After changing the file, we have to rebuild our assets using the `invenio webpack` command: -```bash +```console (my-site) $ invenio webpack buildall ...webpack ``` @@ -166,7 +166,7 @@ If we reload our page now we should see our brand new design: You can watch for changes and automatically rebuild the assets by running: -```bash +```console (my-site) $ invenio webpack run start ``` @@ -242,7 +242,7 @@ export const MysiteResultsGridItem = ({ result, index }) => { Again, we'll have to run the `invenio webpack buildall` command: -```bash +```console (my-site) $ invenio webpack buildall ...webpack ``` diff --git a/06-developing-with-invenio/README.md b/06-developing-with-invenio/README.md index 676dbeb..820edc3 100644 --- a/06-developing-with-invenio/README.md +++ b/06-developing-with-invenio/README.md @@ -48,7 +48,7 @@ The documentation has already written basic information for you. However you can Build documentation: -```bash +```console (my-site)$ python setup.py build_sphinx ``` @@ -80,7 +80,7 @@ To run the tests you can use the test script provided in the repository (the scr To run the test functions one by one you should activate the virtualenv of your project and use pytest command, like on the example below: -```bash +```console (my-site)$ pytest tests/api/test_api_record_files.py::test_record_creation ``` @@ -114,7 +114,7 @@ Run tests again: In order to test a new package, simply install it in the virtualenv using `pip` tool: -```bash +```console (my-site)$ pip install Pillow ``` diff --git a/07-data-models-new-field/README.md b/07-data-models-new-field/README.md index 256ce09..f313710 100644 --- a/07-data-models-new-field/README.md +++ b/07-data-models-new-field/README.md @@ -109,7 +109,7 @@ We have created and started a new DB and ES along with the updated schemas and m **Note**: Make sure you have up and running our development server by running: ```bash -$ ./scripts/server +./scripts/server ``` Run the below command to create our new record: @@ -189,7 +189,7 @@ Our new record was successfully created! **Note**: Make sure you have up and running our development server by running: ```bash -$ ./scripts/server +./scripts/server ``` Let's search now for our newly created record. Replace the `` with the actual `id` of the diff --git a/08-data-models-from-scratch/README.md b/08-data-models-from-scratch/README.md index 8e5e913..5492da7 100644 --- a/08-data-models-from-scratch/README.md +++ b/08-data-models-from-scratch/README.md @@ -24,14 +24,18 @@ such as storing and searching. - [What did we learn](#what-did-we-learn) ## Step 1: Bootstrap exercise + ### 1.1 + If you completed the previous tutorial, you can skip this step. If instead you would like to start from a clean state run the following commands: ```bash cd ~/src/training/ ./start-from.sh 07-data-models-new-field ``` + ### 1.2 + **Note**: In order to reduce the amount of code that we need to write we have prepared beforehand the module structure in `/08-data-models-from-scratch/author_module` folder in which will go through and **uncomment** the needed code snippets to enable different functionalities and eventually build our module! Run the below command to copy the module over: diff --git a/12-managing-access/README.md b/12-managing-access/README.md index a3619c0..169b69b 100644 --- a/12-managing-access/README.md +++ b/12-managing-access/README.md @@ -1,6 +1,7 @@ # Tutorial 12 - Record access management ## Table of contents + - [Step 1 - Allow for access only from the owner](#step-1---allow-for-access-only-from-the-owner) - [Step 2 - search filter](#step-2---search-filter) - [Step 3 - Create permissions](#step-3---create-permissions) @@ -13,126 +14,125 @@ Prerequisites: 1. previous steps with owner field 2. at least two different users -```commandline -(my-site) $ my-site users create admin@test.ch -a --password=123456 # create admin user ID 1 -(my-site) $ my-site users create manager@test.ch -a --password=123456 # create admin user ID 2 -(my-site) $ my-site users create visitor@test.ch -a --password=123456 # create visitor user ID 3 -``` + ```console + (my-site) $ my-site users create admin@test.ch -a --password=123456 # create admin user ID 1 + (my-site) $ my-site users create manager@test.ch -a --password=123456 # create admin user ID 2 + (my-site) $ my-site users create visitor@test.ch -a --password=123456 # create visitor user ID 3 + ``` 3. at least two records -```commandline -curl -k --header "Content-Type: application/json" --request POST --data '{"title":"My test record", "contributors": [{"name": "Doe, John"}], "owner": 1}' https://localhost:5000/api/records/?prettyprint=1 - -curl -k --header "Content-Type: application/json" --request POST --data '{"title":"Second test record", "contributors": [{"name": "Copernicus, Mikolaj"}], "owner": 2}' https://localhost:5000/api/records/?prettyprint=1 -``` + ```bash + curl -k --header "Content-Type: application/json" --request POST --data '{"title":"My test record", "contributors": [{"name": "Doe, John"}], "owner": 1}' https://localhost:5000/api/records/?prettyprint=1 + curl -k --header "Content-Type: application/json" --request POST --data '{"title":"Second test record", "contributors": [{"name": "Copernicus, Mikolaj"}], "owner": 2}' https://localhost:5000/api/records/?prettyprint=1 + ``` ## Step 1 - Allow for access only from the owner -### Use case: +### Use case Restrict the access to read, edit and delete action for the record only to its owner. 1. We implement the permission factory. The permission requires a need to be fulfilled by a user for a record. In this case we remember that: -```json - "owner": { - "type": "integer" - }, -``` + ```json + "owner": { + "type": "integer" + }, + ``` -so the permission factory requires users to provide their ID as stored in the the `"owner"` field of the record. -Add the following `my_site/records/permissions.py` file: + so the permission factory requires users to provide their ID as stored in the the `"owner"` field of the record. + Add the following `my_site/records/permissions.py` file: -```diff - from invenio_access import Permission, any_user -+from flask_principal import UserNeed + ```diff + from invenio_access import Permission, any_user + +from flask_principal import UserNeed - def files_permission_factory(obj, action=None): - """Permissions factory for buckets.""" - return Permission(any_user) -+ -+def owner_permission_factory(record=None): -+ """Permission factory with owner access to the record.""" -+ return Permission(UserNeed(record["owner"])) -``` + def files_permission_factory(obj, action=None): + """Permissions factory for buckets.""" + return Permission(any_user) + + + +def owner_permission_factory(record=None): + + """Permission factory with owner access to the record.""" + + return Permission(UserNeed(record["owner"])) + ``` 2. We use the permission factory in the configuration file to let the application know that this endpoint has a permission requirement (RUD). Edit `my_site/records/config.py`: -```diff -+from my_site.records.permissions import owner_permission_factory - -RECORDS_REST_ENDPOINTS = { - 'recid': dict( - pid_type='recid', - pid_minter='recid', - pid_fetcher='recid', - default_endpoint_prefix=True, - search_class=RecordsSearch, - indexer_class=RecordIndexer, - search_index='records', - search_type=None, - record_serializers={ - 'application/json': ('my_site.records.serializers' - ':json_v1_response'), - }, - search_serializers={ - 'application/json': ('my_site.records.serializers' - ':json_v1_search'), - }, - record_loaders={ - 'application/json': ('my_site.records.loaders' - ':json_v1'), - }, - list_route='/records/', - item_route='/records/', - default_media_type='application/json', - max_result_window=10000, - error_handlers=dict(), - create_permission_factory_imp=allow_all, -- read_permission_factory_imp=check_elasticsearch, -- update_permission_factory_imp=allow_all, -- delete_permission_factory_imp=allow_all, -+ read_permission_factory_imp=owner_permission_factory, -+ update_permission_factory_imp=owner_permission_factory, -+ delete_permission_factory_imp=owner_permission_factory, - list_permission_factory_imp=allow_all - ), -} -"""REST API for my-site.""" -``` + ```diff + +from my_site.records.permissions import owner_permission_factory + + RECORDS_REST_ENDPOINTS = { + 'recid': dict( + pid_type='recid', + pid_minter='recid', + pid_fetcher='recid', + default_endpoint_prefix=True, + search_class=RecordsSearch, + indexer_class=RecordIndexer, + search_index='records', + search_type=None, + record_serializers={ + 'application/json': ('my_site.records.serializers' + ':json_v1_response'), + }, + search_serializers={ + 'application/json': ('my_site.records.serializers' + ':json_v1_search'), + }, + record_loaders={ + 'application/json': ('my_site.records.loaders' + ':json_v1'), + }, + list_route='/records/', + item_route='/records/', + default_media_type='application/json', + max_result_window=10000, + error_handlers=dict(), + create_permission_factory_imp=allow_all, + - read_permission_factory_imp=check_elasticsearch, + - update_permission_factory_imp=allow_all, + - delete_permission_factory_imp=allow_all, + + read_permission_factory_imp=owner_permission_factory, + + update_permission_factory_imp=owner_permission_factory, + + delete_permission_factory_imp=owner_permission_factory, + list_permission_factory_imp=allow_all + ), + } + """REST API for my-site.""" + ``` 3. log in as manager user 4. visit `/api/records/` (first record) -```json -{ -"message": "You don't have the permission to access the requested resource. It is either read-protected or not readable by the server.", -"status": 403 -} -``` + ```json + { + "message": "You don't have the permission to access the requested resource. It is either read-protected or not readable by the server.", + "status": 403 + } + ``` -4. visit `/records/` +5. visit `/records/` -Record still not protected! + Record still not protected! -5. Set permission factory also for UI endpoints in `my_site/records/config.py`: -```diff -RECORDS_UI_ENDPOINTS = dict( - recid=dict( - pid_type='recid', - route='/records/', - template='records/record.html', - record_class='invenio_records_files.api:Record', -+ permission_factory_imp='my_site.records.permissions:owner_permission_factory', - ), -``` +6. Set permission factory also for UI endpoints in `my_site/records/config.py`: -6. visit `/records/` + ```diff + RECORDS_UI_ENDPOINTS = dict( + recid=dict( + pid_type='recid', + route='/records/', + template='records/record.html', + record_class='invenio_records_files.api:Record', + + permission_factory_imp='my_site.records.permissions:owner_permission_factory', + ), + ``` +7. visit `/records/` ## Step 2 - search filter @@ -141,82 +141,82 @@ The details pages of records are now protected. But if we visit `/search?page=1& 1. We implement a search filter that will display records in the search results only to their owner. Let' s create `my_site/records/search.py`: -```python -from elasticsearch_dsl import Q -from flask_security import current_user + ```python + from elasticsearch_dsl import Q + from flask_security import current_user -def owner_permission_filter(): - """Search filter with permission.""" - return [Q('match', owner=current_user.get_id())] + def owner_permission_filter(): + """Search filter with permission.""" + return [Q('match', owner=current_user.get_id())] -``` + ``` 2. We implement a search class that uses the implemented filter (also in `search.py`). -```diff -from elasticsearch_dsl import Q -from flask_security import current_user -+from invenio_search.api import DefaultFilter, RecordsSearch + ```diff + from elasticsearch_dsl import Q + from flask_security import current_user + +from invenio_search.api import DefaultFilter, RecordsSearch -def owner_permission_filter(): - """Search filter with permission.""" - return [Q('match', owner=current_user.get_id())] + def owner_permission_filter(): + """Search filter with permission.""" + return [Q('match', owner=current_user.get_id())] -+class OwnerRecordsSearch(RecordsSearch): -+ """Class providing permission search filter.""" -+ -+ class Meta: -+ index = 'records' -+ default_filter = DefaultFilter(owner_permission_filter) -+ doc_types = None + +class OwnerRecordsSearch(RecordsSearch): + + """Class providing permission search filter.""" + + + + class Meta: + + index = 'records' + + default_filter = DefaultFilter(owner_permission_filter) + + doc_types = None -``` + ``` 3. We add the search class to the configuration in `my_site/records/config.py`: -```diff - -+from my_site.records.search import OwnerRecordsSearch - -RECORDS_REST_ENDPOINTS = { - 'recid': dict( - pid_type='recid', - pid_minter='recid', - pid_fetcher='recid', - default_endpoint_prefix=True, -+ search_class=OwnerRecordsSearch, - indexer_class=RecordIndexer, - search_index='records', - search_type=None, - record_serializers={ - 'application/json': ('my_site.records.serializers' - ':json_v1_response'), - }, - search_serializers={ - 'application/json': ('my_site.records.serializers' - ':json_v1_search'), - }, - record_loaders={ - 'application/json': ('my_site.records.loaders' - ':json_v1'), - }, - list_route='/records/', - item_route='/records/', - default_media_type='application/json', - max_result_window=10000, - error_handlers=dict(), - create_permission_factory_imp=allow_all, - read_permission_factory_imp=owner_permission_factory, - update_permission_factory_imp=owner_permission_factory, - delete_permission_factory_imp=owner_permission_factory, - list_permission_factory_imp=allow_all - ), -} -"""REST API for my-site.""" -``` + ```diff + + +from my_site.records.search import OwnerRecordsSearch + + RECORDS_REST_ENDPOINTS = { + 'recid': dict( + pid_type='recid', + pid_minter='recid', + pid_fetcher='recid', + default_endpoint_prefix=True, + + search_class=OwnerRecordsSearch, + indexer_class=RecordIndexer, + search_index='records', + search_type=None, + record_serializers={ + 'application/json': ('my_site.records.serializers' + ':json_v1_response'), + }, + search_serializers={ + 'application/json': ('my_site.records.serializers' + ':json_v1_search'), + }, + record_loaders={ + 'application/json': ('my_site.records.loaders' + ':json_v1'), + }, + list_route='/records/', + item_route='/records/', + default_media_type='application/json', + max_result_window=10000, + error_handlers=dict(), + create_permission_factory_imp=allow_all, + read_permission_factory_imp=owner_permission_factory, + update_permission_factory_imp=owner_permission_factory, + delete_permission_factory_imp=owner_permission_factory, + list_permission_factory_imp=allow_all + ), + } + """REST API for my-site.""" + ``` 4. Go to the API search page `https://127.0.0.1:5000/api/records/?prettyprint=1` and check that it displays only the records owned by the current user @@ -228,60 +228,60 @@ RECORDS_REST_ENDPOINTS = { 1. Implement the permission factory in `my_site/records/permissions.py` -```python -from invenio_access import Permission, authenticated_user + ```python + from invenio_access import Permission, authenticated_user -def authenticated_user_permission(record=None): - """Return an object that evaluates if the current user is authenticated.""" - return Permission(authenticated_user) + def authenticated_user_permission(record=None): + """Return an object that evaluates if the current user is authenticated.""" + return Permission(authenticated_user) -``` + ``` 2. Add the permission factory to the configuration of the records REST endpoints in `my_site/records/config.py` -```diff --from my_site.records.permissions import owner_permission_factory -+from my_site.records.permissions import owner_permission_factory, \ -+ authenticated_user_permission - -RECORDS_REST_ENDPOINTS = { - 'recid': dict( - pid_type='recid', - pid_minter='recid', - pid_fetcher='recid', - default_endpoint_prefix=True, - search_class=OwnerRecordsSearch, - indexer_class=RecordIndexer, - search_index='records', - search_type=None, - record_serializers={ - 'application/json': ('my_site.records.serializers' - ':json_v1_response'), - }, - search_serializers={ - 'application/json': ('my_site.records.serializers' - ':json_v1_search'), - }, - record_loaders={ - 'application/json': ('my_site.records.loaders' - ':json_v1'), - }, - list_route='/records/', - item_route='/records/', - default_media_type='application/json', - max_result_window=10000, - error_handlers=dict(), -- create_permission_factory_imp=allow_all -+ create_permission_factory_imp=authenticated_user_permission, - -``` + ```diff + -from my_site.records.permissions import owner_permission_factory + +from my_site.records.permissions import owner_permission_factory, \ + + authenticated_user_permission + + RECORDS_REST_ENDPOINTS = { + 'recid': dict( + pid_type='recid', + pid_minter='recid', + pid_fetcher='recid', + default_endpoint_prefix=True, + search_class=OwnerRecordsSearch, + indexer_class=RecordIndexer, + search_index='records', + search_type=None, + record_serializers={ + 'application/json': ('my_site.records.serializers' + ':json_v1_response'), + }, + search_serializers={ + 'application/json': ('my_site.records.serializers' + ':json_v1_search'), + }, + record_loaders={ + 'application/json': ('my_site.records.loaders' + ':json_v1'), + }, + list_route='/records/', + item_route='/records/', + default_media_type='application/json', + max_result_window=10000, + error_handlers=dict(), + - create_permission_factory_imp=allow_all + + create_permission_factory_imp=authenticated_user_permission, + + ``` 3. Perform a POST request by using curl to test permission to create records as an unauthenticated user (should fail) -```commandline -curl -k --header "Content-Type: application/json" --request POST --data '{"title":"Second test record", "contributors": [{"name": "Copernicus, Mikolaj"}], "owner": 2}' https://localhost:5000/api/records/?prettyprint=1 -``` + ```bash + curl -k --header "Content-Type: application/json" --request POST --data '{"title":"Second test record", "contributors": [{"name": "Copernicus, Mikolaj"}], "owner": 2}' https://localhost:5000/api/records/?prettyprint=1 + ``` ## Extras @@ -289,122 +289,121 @@ curl -k --header "Content-Type: application/json" --request POST --data '{"title Use case: We would like to allow our site's managers to edit and delete records -``` +```text NOTE: we have existing records already, we would not like to add the group access one by one to each record. - ``` +``` 1. Create a managers role (group) -```commandline -(my-site)$ my-site roles create managers -``` + ```console + (my-site)$ my-site roles create managers + ``` 2. Connect manager user with created role -```commandline -(my-site)$ my-site roles add manager@test.ch managers -``` + ```console + (my-site)$ my-site roles add manager@test.ch managers + ``` 3. Create the permission factory for role and owner -```python -from invenio_access import Permission -from flask_principal import UserNeed, RoleNeed + ```python + from invenio_access import Permission + from flask_principal import UserNeed, RoleNeed -def owner_manager_permission_factory(record=None): - """Returns permission for managers group.""" - return Permission(UserNeed(record["owner"]), RoleNeed('managers')) + def owner_manager_permission_factory(record=None): + """Returns permission for managers group.""" + return Permission(UserNeed(record["owner"]), RoleNeed('managers')) + + ``` -``` 4. Implement search filter for role and owner -```python -from elasticsearch_dsl import Q -from flask_security import current_user -from invenio_search.api import DefaultFilter, RecordsSearch + ```python + from elasticsearch_dsl import Q + from flask_security import current_user + from invenio_search.api import DefaultFilter, RecordsSearch -def owner_manager_permission_filter(): - """Search filter with permission.""" - if current_user.has_role('managers'): - return [Q(name_or_query='match_all')] - else: - return [Q('match', owner=current_user.get_id())] + def owner_manager_permission_filter(): + """Search filter with permission.""" + if current_user.has_role('managers'): + return [Q(name_or_query='match_all')] + else: + return [Q('match', owner=current_user.get_id())] -class OwnerManagerRecordsSearch(RecordsSearch): - """Class providing permission search filter.""" - class Meta: - index = 'records' - default_filter = DefaultFilter(owner_manager_permission_filter) - doc_types = None + class OwnerManagerRecordsSearch(RecordsSearch): + """Class providing permission search filter.""" -``` + class Meta: + index = 'records' + default_filter = DefaultFilter(owner_manager_permission_filter) + doc_types = None + + ``` 5. Update the configuration file with your new filter and factory -```python -from my_site.records.permissions import owner_permission_factory, \ - authenticated_user_permission, owner_manager_permission_factory -from my_site.records.search import OwnerManagerRecordsSearch - -RECORDS_REST_ENDPOINTS = { - 'recid': dict( - pid_type='recid', - pid_minter='recid', - pid_fetcher='recid', - default_endpoint_prefix=True, - search_class=OwnerManagerRecordsSearch, - indexer_class=RecordIndexer, - search_index='records', - search_type=None, - record_serializers={ - 'application/json': ('my_site.records.serializers' - ':json_v1_response'), - }, - search_serializers={ - 'application/json': ('my_site.records.serializers' - ':json_v1_search'), - }, - record_loaders={ - 'application/json': ('my_site.records.loaders' - ':json_v1'), - }, - list_route='/records/', - item_route='/records/', - default_media_type='application/json', - max_result_window=10000, - error_handlers=dict(), - create_permission_factory_imp=authenticated_user_permission, - read_permission_factory_imp=owner_manager_permission_factory, - update_permission_factory_imp=owner_manager_permission_factory, - delete_permission_factory_imp=owner_manager_permission_factory, - list_permission_factory_imp=allow_all - ), -} -"""REST API for my-site.""" -``` + ```python + from my_site.records.permissions import owner_permission_factory, \ + authenticated_user_permission, owner_manager_permission_factory + from my_site.records.search import OwnerManagerRecordsSearch + + RECORDS_REST_ENDPOINTS = { + 'recid': dict( + pid_type='recid', + pid_minter='recid', + pid_fetcher='recid', + default_endpoint_prefix=True, + search_class=OwnerManagerRecordsSearch, + indexer_class=RecordIndexer, + search_index='records', + search_type=None, + record_serializers={ + 'application/json': ('my_site.records.serializers' + ':json_v1_response'), + }, + search_serializers={ + 'application/json': ('my_site.records.serializers' + ':json_v1_search'), + }, + record_loaders={ + 'application/json': ('my_site.records.loaders' + ':json_v1'), + }, + list_route='/records/', + item_route='/records/', + default_media_type='application/json', + max_result_window=10000, + error_handlers=dict(), + create_permission_factory_imp=authenticated_user_permission, + read_permission_factory_imp=owner_manager_permission_factory, + update_permission_factory_imp=owner_manager_permission_factory, + delete_permission_factory_imp=owner_manager_permission_factory, + list_permission_factory_imp=allow_all + ), + } + """REST API for my-site.""" + ``` 6. Visit `https://127.0.0.1:5000/search?page=1&size=20&q=` and `https://127.0.0.1:5000/api/records/?prettyprint=1` as manager user and check if all the records are listed. - ### Explicit access per action type - additional exercise 1. Implement access management for the record having in mind the structure below -```json -{ - "_access": { - "read": { - "systemroles": ["campus_user"] - }, - "update": { - "users": [1], - "roles": ["curators"] + ```json + { + "_access": { + "read": { + "systemroles": ["campus_user"] + }, + "update": { + "users": [1], + "roles": ["curators"] + } } } -} -``` - - + ``` diff --git a/13-securing-your-instance/README.md b/13-securing-your-instance/README.md index 8b55af5..eecaa8c 100644 --- a/13-securing-your-instance/README.md +++ b/13-securing-your-instance/README.md @@ -2,7 +2,7 @@ In this session, you will discover the key points which will ensure that your Invenio instances are secure. You will learn how to protect the web application with configuration, package management and authentication. -## Table of contents: +## Table of contents - [Step 1: Bootstrap exercise](#step-1-bootstrap-exercise) - [Step 2: Lets create some demo data](#step-2-lets-create-some-demo-data) @@ -23,29 +23,27 @@ In this session, you will discover the key points which will ensure that your In If you completed the previous tutorial, you can skip this step. If instead you would like to start from a clean state run the following commands: ```bash -$ cd ~/src/training/ -$ ./start-from.sh 12-managing-access +cd ~/src/training/ +./start-from.sh 12-managing-access ``` ## Step 2: Lets create some demo data We will clean all the data created before and create some users and records for this tutorial. -```console -$ ~/src/my-site -$ ./scripts/setup -$ # with an instance of ./scripts/server running we create users and records -$ . ~/src/training/13-securing-your-instance/demo-data.sh +```bash +~/src/my-site +./scripts/setup +# with an instance of ./scripts/server running we create users and records +. ~/src/training/13-securing-your-instance/demo-data.sh ``` - - ## Step 3: Configuration allowed hosts You should update our `APP_ALLOWED_HOSTS` to the correct value in your production instances. If you try to make a request with different host header than this one you will be blocked. -```console -$ curl -ki -H "Host: evil.io" https://127.0.0.1:5000/api/records/ -H "Authorization: Bearer $BOOTCAMP_ACCESS_TOKEN" +```bash +curl -ki -H "Host: evil.io" https://127.0.0.1:5000/api/records/ -H "Authorization: Bearer $BOOTCAMP_ACCESS_TOKEN" HTTP/1.0 400 BAD REQUEST Content-Type: application/json Content-Length: 69 @@ -80,8 +78,8 @@ Lets say now that you allow now any host in `my_site/config.py`: Now potential attackers could inject a host header and make all your self links point to their evil site: -```console -$ curl -kI -H "Host: evil.io" "https://127.0.0.1:5000/api/records/?prettyprint=1" -H "Authorization: Bearer $BOOTCAMP_ACCESS_TOKEN" +```bash +curl -kI -H "Host: evil.io" "https://127.0.0.1:5000/api/records/?prettyprint=1" -H "Authorization: Bearer $BOOTCAMP_ACCESS_TOKEN" HTTP/1.0 200 OK Content-Type: application/json Content-Length: 1351 @@ -117,10 +115,11 @@ Change in `my_site/config.py` your `SECRET_KEY` and store it safely with only on ``` For example, we could use the Python 3 `secrets` library: -```console -$ export OLD_SECRET_KEY=CHANGE_ME -$ export NEW_SECRET_KEY=`python -c 'import secrets; print(secrets.token_hex(32))'` -$ sed -i "s/$OLD_SECRET_KEY/$NEW_SECRET_KEY/g" my_site/config.py + +```bash +export OLD_SECRET_KEY=CHANGE_ME +export NEW_SECRET_KEY=`python -c 'import secrets; print(secrets.token_hex(32))'` +sed -i "s/$OLD_SECRET_KEY/$NEW_SECRET_KEY/g" my_site/config.py ``` ## Step 5: Configuration SSL certificates @@ -136,8 +135,8 @@ two servers, the HAProxy load balancer and the Nginx reserve proxy, before arriv ## Step 7: Invenio HTTP headers walk-through -```console -$ curl -kI https://127.0.0.1:5000/api/records/ -H "Authorization: Bearer $BOOTCAMP_ACCESS_TOKEN" +```bash +curl -kI https://127.0.0.1:5000/api/records/ -H "Authorization: Bearer $BOOTCAMP_ACCESS_TOKEN" HTTP/1.0 200 OK Content-Type: application/json Content-Length: 822 @@ -195,15 +194,16 @@ Note: It is possible to run into problems regarding CSP rules when dealing with ## Step 9: Keeping packages up to date It is really important that you keep your packages up to date. Since we are using `pipenv` to manage our application we should follow [`pipenv`'s upgrade workflow](https://pipenv.readthedocs.io/en/latest/basics/#example-pipenv-upgrade-workflow) -```console -$ pipenv update --outdated -$ pipenv update [all|specific-outdated-package] + +```bash +pipenv update --outdated +pipenv update [all|specific-outdated-package] ``` `pipenv` also offers a way of checking all your dependencies and spot package versions which have been publicly discovered as vulnerable. -```console -$ pipenv check +```bash +pipenv check ``` ## Step 10: Secure file uploads @@ -249,16 +249,16 @@ We have been using access tokens during the exercises, but if you want to create Or through the command line interface: -```console -$ my-site tokens create -n tokenname -u +```bash +my-site tokens create -n tokenname -u newsupersecrettoken ``` Once you have your token you can start doing authenticated requests by adding the token in the HTTP header: -```console -$ export MY_SITE_ACCESS_TOKEN=newsupersecrettoken -$ curl -k "https://127.0.0.1:5000/api/records/2?prettyprint=1" -H "Authorization: Bearer $MY_SITE_ACCESS_TOKEN" +```bash +export MY_SITE_ACCESS_TOKEN=newsupersecrettoken +curl -k "https://127.0.0.1:5000/api/records/2?prettyprint=1" -H "Authorization: Bearer $MY_SITE_ACCESS_TOKEN" { "created": "2019-03-17T08:32:29.935720+00:00", "id": "2", @@ -284,23 +284,25 @@ $ curl -k "https://127.0.0.1:5000/api/records/2?prettyprint=1" -H "Authorization All user tokens are encrypted when stored in the database. Therefore, if the application `SECRET_KEY` is changed, these tokens need to be migrated: -```console -$ export NEW_SECRET_KEY=myoldsecretkey -$ export EVEN_NEWER_SECRET_KEY=`python -c 'import secrets; print(secrets.token_hex(32))'` -$ sed -i "s/$NEW_SECRET_KEY/$EVEN_NEWER_SECRET_KEY/g" my_site/config.py +```bash +export NEW_SECRET_KEY=myoldsecretkey +export EVEN_NEWER_SECRET_KEY=`python -c 'import secrets; print(secrets.token_hex(32))'` +sed -i "s/$NEW_SECRET_KEY/$EVEN_NEWER_SECRET_KEY/g" my_site/config.py ``` If we just change the secret key, our users will not be able to use their credentials: -```console -$ curl -k "https://127.0.0.1:5000/api/records/2?prettyprint=1" -H "Authorization: Bearer $MY_SITE_ACCESS_TOKEN" + +```bash +curl -k "https://127.0.0.1:5000/api/records/2?prettyprint=1" -H "Authorization: Bearer $MY_SITE_ACCESS_TOKEN" {"message":"The server could not verify that you are authorized to access the URL requested. You either supplied the wrong credentials (e.g. a bad password), or your browser doesn't understand how to supply the credentials required.","status":401} ``` We need to migrate all tokens: -```console -$ my-site instance migrate-secret-key --old-key $NEW_SECRET_KEY + +```bash +my-site instance migrate-secret-key --old-key $NEW_SECRET_KEY Successfully changed secret key. -$ curl -k "https://127.0.0.1:5000/api/records/2?prettyprint=1" -H "Authorization: Bearer $MY_SITE_ACCESS_TOKEN" +curl -k "https://127.0.0.1:5000/api/records/2?prettyprint=1" -H "Authorization: Bearer $MY_SITE_ACCESS_TOKEN" { "created": "2019-03-17T08:32:29.935720+00:00", "id": "2", diff --git a/14-deployment-monitoring/README.md b/14-deployment-monitoring/README.md index db77899..002203b 100644 --- a/14-deployment-monitoring/README.md +++ b/14-deployment-monitoring/README.md @@ -16,26 +16,26 @@ It does not make much sense to run this stress test in the development environme Ensure that docker-compose **full** is running. ```bash -$ cd ~/src/my-site -$ docker-compose stop -$ docker-compose -f docker-compose.full.yml up +cd ~/src/my-site +docker-compose stop +docker-compose -f docker-compose.full.yml up ``` Let's install `locust` in our virtualenv and run the server. ```bash -$ cd ~/src/my-site -$ pipenv run pip install locust -$ ./scripts/server +cd ~/src/my-site +pipenv run pip install locust +./scripts/server ``` In another terminal, now copy the file `locustfile.py` in your `my-site` folder (to be in the virtualenv) and then run locust in the same folder: -``` -$ cp ~/src/training/14-deployement-monitoring/locustfile.py ~/src/my-site/ -$ cd ~/src/my-site -$ pipenv run locust --host=https://127.0.0.1:5000/ -$ firefox http://127.0.0.1:8089 +```bash +cp ~/src/training/14-deployement-monitoring/locustfile.py ~/src/my-site/ +cd ~/src/my-site +pipenv run locust --host=https://127.0.0.1:5000/ +firefox http://127.0.0.1:8089 ``` ## Step 2: Example of Sentry configuration