Skip to content

DevOps: Diffusion Marketplace Server Setup from Scratch (for VAEC)

Jump to bottom
Emmanuel Nwakire edited this page Jul 12, 2024 · 15 revisions

Diffusion Marketplace Server Setup from Scratch

Note: the struck out instructions are for historical and debugging purposes. You do not have to run those commands to set up the server.

Note: original instructions here: https://docs.google.com/document/d/1DhaWb1eTRSPMYWpaOJj2sRnN1DmEpjpvYLgkA0fXq0w/edit?usp=sharing

Get into VAEC

https://prod.adfs.federation.va.gov/adfs/ls/idpinitiatedsignon.aspx

Get contents of .env file

  • Ask dm-devops team for contents of .env file, or figure it out from the README on Github

Create Redhat instance.

  • Choose vaec-ami-dmva ami-0b06502d141079259
  • Choose DMA VAEC RHEL - ami-06470d67 this has the RHEL license and updates installed already
    • If you do not have the above, choose RH 7.6 ITOPS-RHEL HVM Encrypted 2019-03-19 - ami-7b76341a
      • this is the VA flavored AMI. You need to request and install a RHEL license to be able to install anything with yum.
      • AWS Marketplace should be fine to use if you do not have a RHEL license, though it may not pass nessus scans (RHEL-7.6_HVM*)
  • Save the key as a filename that is memorable and remember where you put it. It is important to have it.
  • Add the server to the default security group. Otherwise, it cannot communicate with the database

## Register the server instance https://vaww.vashare.oit.va.gov/sites/euo/EUOWiki/Wiki/Enterprise%20Satellite%20Server%20Registration.aspx

To perform the registration, download the basic bootstrap script from the Enterprise Satellite server. You will then edit the file to include the activation keys that you were given and execute the script.

Example:

cd ~
wget http://vasat.aac.va.gov/pub/bootstrap/bootstrap.sh
vim bootstrap.sh
### see below for edits
bash bootstrap.sh

Edit line 75 to include your activation keys. For example if you were given the keys 3-example-app, 3-loc-wdc, and 3-v2s: Change this:

ACTIVATION_KEYS=

To this:

ACTIVATION_KEYS=3-example-app,3-loc-wdc,3-v2s

Run the bootstrap.sh script

chmod +x bootstrap.sh
sudo bash bootstrap.sh

Create S3 bucket

  • Remember the name of the bucket
  • Create a policy and role to use the bucket via CloudFormation
    • Sample of role to upload:
---
AWSTemplateFormatVersion: 2010-09-09
Description: Example role, policy and instance profile for project admin using IAM boundary
  # MUST HAVE for mult-region - Stack name starts with "project-"

Resources:
  rS3Policy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      # MUST HAVE - Policy name begins with "project-"
      # MUST HAVE for mult-region - Stack name starts with "project-"
      # Auto-assigned name = <stack-name>-<resourcename>-<random>
      # Example Auto assigned name = project-s3-role-example-rS3Policy-1ST3R7IYC9JJ5
      Description: Project Administrator created S3 Policy
      # MUST HAVE - Policy document Path: /project/
      Path: /project/
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Sid: AllowS3ReadAccess
            Effect: Allow
            Action:
              - s3:ListObject
              - s3:ListBucket
              - s3:GetObject
              - s3:GetObjectVersion
              - s3:PutObjectAcl
              - s3:PutObject
              - s3:DeleteObject

            Resource:
              - !Sub arn:${AWS::Partition}:s3:::dev-dm
              - !Sub arn:${AWS::Partition}:s3:::dev-dm/*

  rEc2Role:
    Type: AWS::IAM::Role
    Properties:
      # MUST HAVE - IAM role name begins with "project-"
      # MUST HAVE for mult-region - Stack name starts with "project-"
      # Auto-assigned name = <stack-name>-<resourcename>-<random>
      # Example Auto assigned name = project-s3-role-example-rEc2Role-1ST3R7IYC9JJ5
      # MUST HAVE - Role's Path: /project/
      Path: /project/
      # MUST HAVE - PermissionsBoundary set exactly as shown below, otherwise role creation will fail
      PermissionsBoundary: !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:policy/vaec/project-admin
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
        - Effect: Allow
          Principal:
            Service: ec2.amazonaws.com
          Action:
          - sts:AssumeRole
      ManagedPolicyArns:
      - !Ref rS3Policy

  rInstProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      # MUST HAVE - IAM role name begins with "project-"
      # MUST HAVE for mult-region - Stack name starts with "project-"
      # Auto-assigned name = <stack-name>-<resourcename>-<random>
      # Example Auto assigned name = project-s3-role-example-rInstProfile-1ST3R7IYC9JJ5
      # MUST HAVE - Instance Profile Path: /project/
      Path: /project/
      Roles:
      - !Ref rEc2Role
  • Attach role to the server created in step 3

Create Postgres instance in RDS

  • Postgres 9.6
  • Be sure to enable encryption for the data
  • Be sure to securely save the username, password, database name, and the database host (endpoint)

Create a Redis instance

  • Be sure to securely save the endpoint
  • Also add it to the security group and/or VPC that your server is on
  • Expose port 6379 of the security group

Generate certs

  • sudo openssl req -new -newkey rsa:2048 -keyout /root/cert/<common.name>.key -out /root/cert/<common.name>.csr -config /root/cert/<common.name>.cnf
  • request ssl certs https://vaww.pki.va.gov/ssltls/

Install SSL Certs

  • import SSL cert to AWS' Certificate Manager
  • create an ELB (ElasticLoadBalancer)
  • add your instance to the ELB
  • add your certificate to the ELB
  • open up port 443, close port 80
  • health check: make sure it is tcp to port 80

Request a CNAME record to resolve load balancer dns to whatever host name we want, such as staging.marketplace.va.gov

Affected User
Blake Henderson

Assignment group
IO.SS.FF.ACTIVEDIRECTORY.TEAM8

Assigned to
Denver Griffith (vaaitcgriffd) 

Preview record for field: Assigned to
Impact
1 - Critical

Urgency
3- Medium

Priority
3 - Moderate

Short Description
Create/Modify Existing DNS Entry - Diffusion Marketplace -VA Enterprise Cloud AWS GovCloud

Description
Create dev.marketplace.va.gov
and point to dev Diffusion Marketplace load balancer
internal-dev-dm-2048994434.us-gov-west-1.elb.amazonaws.com

I would like https://internal-dev-dm-2048994434.us-gov-west-1.elb.amazonaws.com/ to resolve to https://dev.marketplace.va.gov/ if at all possible, please. Thank you :)

Get access to jumpbox via Reflections

  • That’s a whole process in itself to get into the group cldunixp_userprofiles using the ePas process
  • vac10acsecs201.va.gov
  • vac10acsecs202.va.gov

ssh into created server in step 3

  • Get key onto this jumpbox
    • vi <environment name>-<application name>.key
    • Open up the key from step 3b in a text editor and copy the contents
    • Paste the contents in the VI opened file
    • Save the file
  • ssh -i <path to key here> ec2-user@<private ip address of server here>
  • You’ll usually find this in the ec2 dashboard when you click on the instance

Install all dependencies:

  • sudo yum update -y
  • sudo amazon-linux-extras install docker -y
    • for amazon linux 2 instances
  • sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
    • sometimes this works and sometimes it doesn't
  • sudo yum install -y nano wget openssl-devel gcc-c++ expat-devel autoconf yum-utils device-mapper-persistent-data lvm2
    • curl-devel wasn't working this time
  • sudo yum install -y gettext-devel openssl-devel perl-CPAN perl-devel zlib-devel
  • sudo yum install -y http://mirror.centos.org/centos/7/extras/x86_64/Packages/container-selinux-2.107-1.el7_6.noarch.rpm
  • yum-config-manager --enable rhel-7-server-rhui-optional-rpms
  • yum-config-manager --enable rhel-7-server-rhui-supplementary-rpms
  • yum-config-manager --enable rhel-7-server-rhui-extras-rpms

Install git

  • yum install -y wget openssl-devel curl-devel gcc-c++ expat-devel autoconf
  • sudo wget https://www.kernel.org/pub/software/scm/git/git-2.28.0.tar.gz
  • sudo tar -zxf git-2.28.0.tar.gz
  • ```cd git-2.28.0`````````````
  • sudo make configure
  • sudo ./configure --prefix=/usr
  • sudo make all
  • sudo make install
  • git --version
  • cd ~

Install docker

  • sudo yum install -y http://mirror.centos.org/centos/7/extras/x86_64/Packages/container-selinux-2.107-1.el7_6.noarch.rpm
  • sudo yum install -y yum-utils device-mapper-persistent-data lvm2
  • sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
  • wget https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-18.09.9-3.el7.x86_64.rpm
  • sudo yum install -y docker-ce-18.09.9-3.el7.x86_64.rpm
    • If this doesn't work, do sudo yum install -y docker-ce with the preface that the yum-config-manager command worked
  • sudo usermod -aG docker $USER
  • sudo systemctl start docker
    • Note: if there is a way to start docker without sudo, that would be the way to go
  • docker --version

Install docker-compose

  • sudo curl -L https://github.com/docker/compose/releases/download/1.25.0-rc2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose -k
  • sudo chmod +x /usr/local/bin/docker-compose
  • sudo docker-compose --version
  • Note, you have to start docker as root, but compose is installed as the user, so, we need to link the executable to root, so run this:
    • sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
  • Jut ran into this error: docker-compose: error while loading shared libraries: libz.so.1: failed to map segment from shared object: Operation not permitted
    • resolved with this: sudo mount /tmp -o remount,exec

Clone the repo

  • git config --global http.sslVerify false
  • git clone https://github.com/department-of-veterans-affairs/diffusion-marketplace.git
  • cd diffusion-marketplace
    • (optional) git fetch origin <branch> && git checkout <branch>
      • This is if there are different configs or features on a certain branch that only work in vaec or a certain build we want to deploy

Install certificates on Application Image:

  • sudo cp /etc/pki/ca-trust/source/anchors/VA-Internal-S2-RCA1-v1.cer ~/diffusion-marketplace

Set environment variables

  • sudo yum install -y nano
    • I like using nano as a text editor
    • Nothing against vi or vim, I just suck at using them
  • nano .env
  • Ask dm-devops team for contents of .env file, or figure it out from the README on Github
  • Example .env:
# AWS_ACCESS_KEY_ID=S3BUCKETUSER # may not need this
AWS_REGION=us-gov-west-1
# AWS_SECRET_ACCESS_KEY=1234 # may not need this
FORCE_SSL=true
[email protected]
GA_TRACKING_ID=fix-me
HOSTNAME=https://dev.marketplace.va.gov
LDAP_USERNAME=fix-me
LDAP_PASSWORD=fix-me
[email protected]
POSTGRES_DB=vaecdiffusionmarketplacedev
POSTGRES_HOST=vaecdiffusionmarketplacedev.cychy2qycdiv.us-gov-west-1.rds.amazonaws.com
POSTGRES_PASSWORD=fix-me
POSTGRES_PORT=5432
POSTGRES_USER=vaecdiffusionmarketplacedev
RAILS_ENV=production
REDIS_URL=redis://dev-dm.sybd24.0001.usgw1.cache.amazonaws.com:6379/0
S3_BUCKET_NAME=dev-dm
SES_SMTP_PASSWORD=fix-me
SES_SMTP_USERNAME=fix-me
SURVEY_MONKEY_PASSWORD=fix-me
SURVEY_MONKEY_TOKEN=fix-me
SURVEY_MONKEY_USERNAME=fix-me
SURVEY_MONKEY_EP201=fix-me
SURVEY_MONKEY_EP202=fix-me
SURVEY_MONKEY_EP203=fix-me
USE_NTLM=true
GOOGLE_API_KEY=fix-me
  • Place the contents into the .env file you opened in step b
  • Replace values with any real values you have available
  • Crucial ones are the db connection variables
POSTGRES_HOST
POSTGRES_DB
POSTGRES_USER
POSTGRES_PORT
POSTGRES_PASSWORD
  • Set environment variables via the .env file
    • while read LINE; do export "$LINE"; done < ./.env
  • printenv
  • Check if all of the variables were added

Build and run the docker containers

  • docker-compose build
  • docker-compose up -d
  • note: you may need sudo in front of these two commands

Browse to the private IP address provided for the server made in step 3

  • You’ll usually find this in the ec2 dashboard when you click on the instance

Enable ipv4 Forwarding

  • sudo nano /etc/sysctl.conf
  • add net.ipv4.ip_forward = 1 to the bottom of the file
  • Save the file
  • Confirm it was changed by running sysctl net.ipv4.ip_forward

Create Restart Server Service

  • vi /etc/systemd/system/appstartup.service
#!/bin/bash

################################################################################
# appstartup.service
#
# This service unit is for testing my systemd startup service
# By David Both
# Licensed under GPL V2
#
################################################################################
# This program should be placed in /usr/local/lib/systemd/system/.
# Create a symlink to it from the /etc/systemd/system directory.
################################################################################

[Unit]

Description=Runs /usr/local/bin/appstartup.sh


[Service]

ExecStart=/usr/local/bin/appstartup.sh


[Install]

WantedBy=multi-user.target
  • Create a symlink to systemd directory ln -s /usr/local/lib/systemd/system/appstartup.service /etc/systemd/system/appstartup.service
  • Create shell script to start application vi /usr/local/bin/appstartup.sh 
#!/bin/bash

################################################################################
# mystartup.sh
#
# This shell program is for testing a startup like rc.local using systemd.
# By DMVA
# Licensed under GPL V2
#
################################################################################

# This program should be placed in /usr/local/bin

################################################################################
# This is a test entry

echo `date +%F" "%T` "Startup worked" >> /root/mystartup.log

cd /home/ec2-user/diffusion-marketplace
./scripts/start_appcontainer.sh
  • Make service executable sudo chmod +x /usr/local/bin/appstartup.sh
  • Start service sudo systemctl start appstartup
  • Check status of service sudo systemctl status appstartup
  • Enable service sudo systemctl enable appstartup

SETTING UP NEW EC2 SERVERS FROM PRE-EXISTING INSTANCES FOR VADM.

_Note: This is to address setting up new servers from pre-existing servers that are currently running. You might want to ask why there is a need to set up new servers. This has to be a decision made by the team. Reasons might include but are not limited to newly upgraded VAEC Golden AMIs (Amazon Machine images) or the need to set up more than one production server.

get into the VAEC AWS environment

  • The link is provided above.

Get the contents of the .env file

  • Ask the dm-DevOps team for the contents of the .env file, kindly note that each server/instances have different .env credentials attached to it.
  • You can also get content of the .env file by Ssh into the server ( Prod, Dev, or Staging) cd into Diffusion marketplace and pass the command ll -a,
  • This will bring out all files and directories under the DM project, you then cat or vi into .env, copy the necessary details, and paste them on your notepad

Choosing an ami

  • Look at the existing AMI tag on the server(Prod, Dev, Staging) you wish to spin up, for example as of 6/20/2023 the new Dev Server has a VAEC AMZLINUX golden image 2023 attached to it. If the team chooses to use a newer AMI, You can get that from the My AMIs section of the instance setup.

Attaching IAM Role to the new server you have just spun up.

  • In doing this, kindly see the IAM role that all the servers have inherited and attach that same role to the new server, this will enable SSH, S3, and other AWS resources needed for the EC2 instance to have access to.

_Note: Since you are replicating servers all former server parameters remain the same by virtue of the credentials in .ENV file, You would not need to create new s3 Buckets, Redis instance, or POSTGRESQL Database.

Install all Dependencies, Install git, Install Docker, Install Docker-compose, and Clone the repo. kindly see the instructions above.

Install certificates on Application Image:

  • `sudo cp /etc/pki/ca-trust/source/anchors/VA-Internal-S2-RCA1-v1.cer, you need to be ~ directory to install this certificate.

Set environment variable

  • Use the detailed information above in doing this. Pls, note that the Postgres information required should be correct and specific to each server. If you wish to get more information kindly locate the RDS dashboard on the AWS console and locate each server instance to get the right information.

Build and run the docker containers

Restart the Application

  • ./scripts/start_appcontainer.sh

Migrating web Traffic from the Old Server to the new server.

  • To update the Load Balancer, you will need access to the VADM AWS account. Navigate to AWS DASHBOARD -> EC2 -> instance(Running)-> Scroll down to Load Balancer -> Choose the specific resource you wish to reconfigure -> Select the instance tab -> Click edit instance and add the new server IP address.
  • Then check the server on the web(via CAG or GFE and externally) the server should be up and running.

Jenkins authentication for new server

  • Access http://internal-jenkinsdevops-837096002.us-gov-west-1.elb.amazonaws.com/ on any web browser internally.
  • Login with your assigned username and password.
  • Locate and click on Manage Jenkins -> Configure Systems -> Environment Variables -> You can now reconfigure the new server IP address, then apply and save.
  • access the new server where Jenkins has been installed using EC2 instance Connect. Then Vi into the /etc/passwd file locate the Jenkins user, configure the user to bin/bash then save and exit.
  • Switch to Jenkins user and cd to ~/.ssh, if you don't see any public(id_rsa.pub) and private keys(id_rsa) kindly generate one.
  • cat .ssh/id_rsa on the Jenkins .ssh directory copy the private(id_rsa)key
  • On the new server(Dev, staging, Production) vi or nano into the .ssh/authorized_keys file, and paste the Private key copied from the Jenkins server.
  • return to your Jenkins dashboard and locate "deploy to dev, staging or Prod" and click on build now this should return a successful build.

INSTANCE PASSING HEALTH CHECKS ON LOAD BALANCER

  • Kindly follow these guidelines if your instances are not passing health checks on the load balancer level
  • Install Firewalld on the respective instance sudo yum install firewalld
  • Start, enable, and check status to be sure Firewalld is running sudo systemctl start firewalld
  • Open the port that the load balancer listens on to communicate to the application, sudo firewall-cmd --zone=public --add-port=8080/tcp --permanent sudo firewall-cmd --reload change 8080 to the necessary port. If the health check continues by failing, check the necessary AWS resources and configuration for load balancers.
Clone this wiki locally