Merge pull request #85 from UCL-ARC/updating-docs

Updating docs

README.md

@@ -4,13 +4,13 @@ The production website is hosted at http://museumofdreamworlds.eu-west-2.elastic
 
 The staging one is http://museumofdreams.eu-west-2.elasticbeanstalk.com/ and is built from the `development` branch
 
-The more public versions are https://museumofdreamworlds.org and https://staging.museumofdreamworlds.org
+The offical versions are https://museumofdreamworlds.org and https://staging.museumofdreamworlds.org
 
-Development should be done locally and staged to the website. At present this is not a production version.
+Development should be done locally and pushed to the staging website where researchers can test features and sign off on them before putting the code on production.
 
 ## Getting Started
 
-To work on this project as is, clone the repo into an appropriate folder (eg. `museum_of_dreams_project`). Create a venv at the top level and start it. Then install the requirements and launch the app
+To work on this project as is, clone the repo into an appropriate folder (eg. `museum_of_dreams_project`). Create a venv at the top level and start it. Then install the requirements and launch the app. We use `requirements-base.txt` as AWS looks for `requirements.txt` and we don't need to install MySQL locally (we use a local sqlite db).
 
 ```
 python3 -m venv modvenv
@@ -26,6 +26,10 @@ If it's your first time initialising the app on your machine, you may need to ru
 python manage.py migrate
 python manage.py createsuperuser
 ```
+### AWS
+
+This project is hosted on AWS, if you do not have access to the account, let [Amanda Ho-Lyn](mailto:[email protected]) know and she will arrange this.
+You may find it beneficial to read through the [current AWS setup docs](docs/baseAWSSetup.md) to gain an understanding of the architecture of the project. Also have a look at the [development SOP](docs/developmentSOP.md) for an idea of the general development flow.
 
 ### Running tests
 
@@ -36,12 +40,28 @@ To run the tests locally, run
 python manage.py test mod_app/tests
 ```
 
+### Technologies used
+
+This project uses a number of technologies, including:
+- [Django 4.2](https://docs.djangoproject.com/en/4.2/)
+- [OpenProps](https://open-props.style/#colors) (CSS variable package)
+- [CK Editor 4](https://ckeditor.com/docs/ckeditor4/latest/index.html)
+- [Fuse.js](https://www.fusejs.io/)
+- [AWS](aws.com)
+
+Others which have tangentially helped with development:
+- Hypothesis
+- Figma
 ---
 
-### See these other files for recreating the setup etc.
+### See these other files for recreating the AWS setup and other features.
+
+#### [Setting up from scratch](docs/howtoSetupBase.md) This covers how you'd go about recreating the AWS setup from scratch
+
+#### [Current AWS Settings](docs/baseAWSSetup.md) This describes our current AWS settings without the processes of setting up, with a bit more explanation of why they are what they are
 
-#### [Setting up from scratch](docs/fullSetup.md)
+#### [How to set up the AWS chatbot for notifications](docs/AWSchatbot.md)
 
-#### [Current AWS Settings](docs/AWScurrentSetup.md)
+#### [Using S3 for staticfiles](docs/s3ForStatic.md)
 
-#### [AWS chatbot](docs/AWSchatbot.md)
+#### [Setting up email notifications](docs/AWSses.md)

docs/AWSses.md

@@ -0,0 +1,11 @@
+# Setting up email notification system for changes on the website
+
+These instructions assume you have purchased a domain name.
+
+I followed the instructions in [this article](https://medium.com/hackernoon/the-easiest-way-to-send-emails-with-django-using-ses-from-aws-62f3d3d33efd). You don't need the seciton on sending from personal emails, but do need the section sending from a domain (that you've purchased).
+
+When you have set up a domain in SES, you need to ensure you have selected "publish to Route 53" so that the details (DKIM) get added to the relevant domain name. You don't need to set up the `Custom MAIL FROM domain` unless you want to (allows you to send from abc.yourdomain), if you do, ensure you publish those credentials to Route 53.
+
+To include it in the admin for select models, I created a mixin which expands the `save_model` function and added the mixin to the admin models. This way, whenever the model is saved in the admin, the researchers should be notified. I also opted to use a template for the html message rather than plain text.
+
+You can view the [code for the mixin](../mod_app/utils/mixins.py) and the [email template](../mod_app/templates/email_template.html)

docs/awsSSL.md

@@ -18,4 +18,4 @@ Go to the EB environment, go to the `Configuration` tab and go to the section ab
 
 Once the changes have been applied, go to your `EC2 Security Groups` and here you should see a new load balancer type for your environment. I recommend changing the name to include LB or something similar so you can tell them apart easily.
 
-On your environment's security group, the inbound rules should include http traffic from the load balancer, add it if it doesn't. Update the inbound rules in the load balancer's security group to allow HTTPS traffic from: itself, 0.0.0.0 (anywhere), the database and the environment. Also allow HTTP from anywhere.
+On your environment's security group, the inbound rules should include http traffic from the load balancer, add it if it doesn't. Update the inbound rules in the load balancer's security group to allow HTTPS traffic from 0.0.0.0 (anywhere), the database and the environment. Also allow HTTP from anywhere.

docs/baseAWSSetup.md

@@ -0,0 +1,85 @@
+# Current Setup
+
+Currently the AWS system is set up such that:
+
+## VPC
+
+There is one VPC called `MuseumofDreams`.
+This has subnets across 3 availability zones (eu-west-2 a, b & c)
+Below is a diagram of how the subnets are laid out.
+
+![VPC resource map](imgs/VPCresource_map.png)
+
+Internet gateways allow the assigned routetables and their subnets to be exposed to the internet. Both `mod_routetable_subnet`s are exposed to the internet and are used for the EB apps and their load balancers, whilst the `RDS-Pvt-rt` is for the databases, shielding them from public internet access.
+
+They have been distributed between dev and prod as shown because load balancers are required to be in at least 2 different availability zones and require subnets with internet gateways. The instances of the EB apps can have subnets with and without an internet gateway. The configuration for these will be covered in the ElasticBeanstalk section.
+
+
+## IAM
+
+The IAM configuration is that laid out in the [IAM Roles section](howtoSetupBase.md) of setting up. There is a role called `MuseumofDreams_EB_EC2`.
+There is also a user called `mod_site`.
+
+## RDS
+
+There are two RDS instances, one for staging (`mod-mysql-dev-db`) and one for prod (`mod-mysql-ebdb-prod`).
+
+They are almost identical in their setup aside from their availability zones and secondary security groups to connect with their respective environments - `rds-ec2-1` for production and `ec2-rds-1` for staging.
+
+
+## ElasticBeanstalk
+
+There is one EBS application - Museum of Dreams site and it has 2 environments - ...`env` which is production and ...`env-dev` for staging.
+
+The workflow is that development is done on a feature branch and this gets merged into `development` and the CodePipeline for the staging site deploys these updates. When these have been reviewed and approved, they can be merged into `main` and the production CodePipeline will deploy a new version of production.
+
+There are saved configurations for both dev and prod environments with load balancers (`mod prod config` and `dev scaling config`), which have most of the necessary selections, the exception being regarding security groups and subnets that will be covered below. You can also set these up with scaling single instances instead of load balancers and then change to load balancers later on (necessary for [SSL](awsSSL.md))
+
+They should also both use the `moddbkey` for EC2 access and `MuseumofDreams_EB_EC2` for the EC2 instance profile.
+
+### Configuration
+**Production**
+
+| Parameter name | Value/s |
+|----|---|
+|instance subnets | `mod-subnet-a1`, `mod-subnet-c1`, `RDS-Pvt-subnet-3`|
+|additional ec2 security groups | `rds-ec2-1`|
+|load balancer subnets | `mod-subnet-a1`, `mod-subnet-c1`|
+---
+
+
+**Staging**
+
+| Parameter name | Value/s |
+|----|---|
+|instance subnets | `mod-subnet-a2`, `mod-subnet-b1`, `RDS-Pvt-subnet-2`|
+|additional ec2 security groups | `ec2-rds-1`|
+|load balancer subnets | `mod-subnet-a2`, `mod-subnet-b1`|
+---
+
+Use the saved configurations to launch instances if an environment needs to be terminated and spun up again. Make sure you detach the security groups before doing this (can be done from the security group console).
+Ensure the private subnets for the instances are in a different availability zone from the public subnets (which overlap with the load balancer subnets) to prevent instances being launched in a private subnet by mistake.
+
+
+## Security Groups
+
+Security groups allow traffic to pass between the instances attached to the SG and any others that have been defined in the incoming/outgoing rules. If traffic is allowed in via one rule it is also allowed out by that same rule, which means you only need to define it once.
+
+Each EC2 instance has a default SG and a load balancer SG and these are automatically populated when an instance is launched. The additional groups are attached to the EC2 instances so that they can communicate with the MySQL databases.
+
+
+## Restoring DB instances
+
+If ever you need to spin up a new verison of the db, this should be done from *backups*, **not** snapshots as snapshots do not allow you to set an initial db name, which is necessary for EBS to connect to the RDS.
+
+# Accessing the db through an EC2 shell
+
+The following commands are important to run anytime you have to use the shell to manually run commands (this shouldn't be often!)
+
+```
+cd /var/app
+source venv/staging-LQM1lest/bin/activate
+cd current
+
+export $(/opt/elasticbeanstalk/bin/get-config --output YAML environment |  sed -r 's/: /=/' | xargs)
+```

docs/developmentSOP.md

@@ -0,0 +1,15 @@
+# Standard development procedure
+
+This document outlines the recommended way we develop and publish features for this project.
+
+#### 1. Begin to develop code locally on new branch
+
+#### 2. If you are working on a feature that needs to be tested on staging whilst in development, use the `dev-feature-test-pipeline` pipeline and edit it to point towards your branch. You will have to manually release the change. Be sure to notify your colleagues _before_ doing this to ensure there is no conflict in the versions.
+
+#### 3. Once you've finished developing the feature, open a PR to `development`. Once merged, double check the behaviour is as expected on staging.
+
+#### 4. Once confirmed, open a PR to `main` and once merged, check the behaviour is as expected on production. You may wish to inform the researchers via email.
+
+#### 5. On the the next! 😁
+
+If you have any questions, contact [Amanda Ho-Lyn](mailto:[email protected])