DevOps

Current Deployment

We are currently hosting the backend on AWS.

Re-deployment Steps

1. Connect to the server (get the connection details from En Rong)
2. Pull the latest backend build on the dist-dev (dev build) or dist (main build) branch after CI has completed
   • E.g. git fetch origin dist-dev followed by git checkout origin/dist-dev
   • Use git log to check that you are on the correct branch head
3. Run yarn reload:stg or yarn reload:prod to redeploy the main backend
   • WARNING: new migration files are automatically run
4. Run reload:stg:cronjob or reload:prod:cronjob to redeploy the cronjob microservice

Monitoring Deployment Status

• pm2 monit to check the status of each deployed instance
• pm2 list for a quick list overview of all deployed instances
• pm2 logs to see the latest logs

Deploying a new instance

1. Connect to the server
2. Setup the repo
   1. Clone the repo: E.g. git clone <repo-url>
   2. Fetch the latest dev build: E.g. git fetch origin dist-dev (or dist branch for the main build)
   3. Checkout the build branch: E.g. git checkout origin/dist-dev
3. Setup the environment variables
   1. Create a new env file in the root of the repo directory
      • .env.staging for staging
      • .env.production for production
4. Run yarn deploy:stg <num-instances> or yarn deploy:prod <num-instances> to deploy the main backend with the specified number of instances behind a round robin load balancer
   • This will create the database if it does not yet exist
5. Run yarn seed:stg or yarn seed:prod to run the database seeds in the respective environments
6. Run yarn deploy:stg:cronjob 1 or yarn deploy:prod:cronjob 1 to deploy a single instance of the cronjob microservice

Future Plans

1. Write a CI/CD pipeline to automatically build and deploy the backend without having to ssh manually

   • We can dockerize the application and follow the GitOps model with IaC
   • Use image tagging in ECR to deploy to prod instances
   • Shift to K8s

2. Logging pipeline

   • Shift to a proper logging pipeline such as the ELK stack

3. Metrics and Analytics

   • Track usage patterns and behaviours
   • Aggregate them in dashboards for analysis
   • Explore SaaS options like Mixpanel

4. Trace Data (low priority)

   • Instrument application telemetry to track and trace control flow throughout our backend when an API endpoint is hit
   • Trace spans are useful for analysing bugs in prod, and for optimizing the backend in the future
   • E.g. OpenTelemetry
   • Overkill at the moment as we are still small, but a possible area we can look into as the application grows in complexity

5. Application Monitoring (low priority)

   • Alerts and monitoring on our infra
   • E.g. Prometheus
   • Overkill at the moment as we are still small, but a possible area we can look into as the application grows in complexity and user load grows