Update Docs for ClamAV in Container (#16058)

* use clamav container for local development and review instances

* rubocop formatting & CODEOWNERS

* fix Dockerfile

* update CODEOWNERS and docker-compose version for review instances

* change docker-compose verions for review

* fix review ports

* update github workflows to match k8s

* update github workflows

* update code_checks for docker compose for tests

* add bundle exec to setup db step in audit service tags

* reset db in docker compose review & update audit service tag docker services

* remove clamd and freshclam from procfile

* add sidekiq enterprise license to test env var

* set sidekiq license to docker build args

* update makefile to work with new docker services

* update hybrid docs

* add clamav host & port to settings

* add restart services to docker-compose review

* update hybrid and native docs for clamav

* remove clamav setup for ubuntu

* add clamav configration info for docker setup

* Added virus scan to uploads spec (#16393)

* Added virus scan to uploads spec

* Fixing Virus scan spec

* resolve merge conflict

* change directory from /app to /srv/vets-api/src

* use previous master version for RI + clam containers

* add USER_ID to RI docker-compose

* set working directory back to app

* minor clean up with new line EOF & remove k8s deploy related code

* update makefile up command with foreman

* fix docker-compose clamav new line

* more cleanup and rubocop formatting

* remove redis ports from docker-compose test

* update redis port

* revert docker-compose test to original master

* update api service name to web

* rubocop formatting

* add Procfile to CODEOWNERS

* update CODEOWNERS

* add secret and pki volumes back to RI

* add some settings in review docker-compose

* fix merge conflict mistake in Dockerfile

* fix merge conflict

* add rvm install ruby version troubleshooting tip

* update dockerfile ruby version

* resolve merge conflict

* rephrase heading

* update clamav daemon socket docs and comment socket env var

* correct phrasing

* update from GH comments

---------

Co-authored-by: Rachal Cassity <[email protected]>

Makefile

@@ -7,6 +7,12 @@ else
     ENV_ARG	 := dev
 endif
 
+ifdef clam
+	FOREMAN_ARG := all=1
+else
+	FOREMAN_ARG := all=1,clamd=0,freshclam=0
+endif
+
 COMPOSE_DEV  := docker-compose
 COMPOSE_TEST := docker-compose -f docker-compose.test.yml
 BASH         := run --rm --service-ports web bash

config/initializers/clamav.rb

@@ -7,6 +7,7 @@
   ENV['CLAMD_TCP_PORT'] = Settings.clamav.port
 
   # If running ClamAV natively (via daemon)
-  # Update host and port on settings.local.yml to override the tcp connection
-  ENV['CLAMD_UNIX_SOCKET'] = '/usr/local/etc/clamav/clamd.sock'
+  # Uncomment this line if running with daemon
+  # Remove clamav host and port on settings.local.yml to override the tcp connection
+  # ENV['CLAMD_UNIX_SOCKET'] = '/usr/local/etc/clamav/clamd.sock'
 end

docs/setup/docker.md

@@ -9,3 +9,18 @@
 
 - [Docker Engine](https://docs.docker.com/engine/install/#server)
 - [Docker Compose](https://docs.docker.com/compose/install/#install-compose-on-linux-systems)
+
+### Configuring ClamAV antivirus
+
+Prior to EKS, ClamAV (the virus scanner) was deployed in the same process as Vets API. With EKS, ClamAV has been extracted out into it’s own service. Locally you can see the docker-compose.yml config for clamav.
+
+1. In settings.local.yml add the following:
+
+```
+clamav:
+  mock: false
+  host: 'clamav'
+  port: '3310'
+```
+
+These setting at the default, so they can removed as well

docs/setup/hybrid.md

@@ -2,13 +2,11 @@
 
 In hybrid mode, you will run vets-api natively, but run Postgres and Redis in Docker. By doing so you avoid any challenges of installing these two software packages and keeping them upgraded to the appropriate version.
 
-
-
 ## Base Setup
 
 1. Install Docker as referenced in the [Docker setup instructions](docker.md).
 
-1. Follow the [Native setup instructions](native.md), but skip any steps related to installing Postgres, Postgis, or Redis. You *will* need to install the other dependencies such as pdftk and clamav.
+1. Follow the [Native setup instructions](native.md), but skip any steps related to installing Postgres, Postgis, Redis or ClamAV. You *will* need to install the other dependencies such as pdftk.
 
 1. Configure vets-api to point to the Docker-ized dependencies. Add the following to `config/settings.local.yml`:
 
@@ -27,9 +25,18 @@ redis:
 
 *Note: If you have local instances of Postgres or Redis that were only for use by vets-api, you can stop them to save system resources.*
 
-## Running
+## Running Deps
 
 1. To start Postgres and Redis: run `docker-compose -f docker-compose-deps.yml up` in one terminal window.
-2. In another terminal window, start `vets-api` as per the [native running instructions](running_natively.md).
+2. In another terminal window, start `vets-api` as per the [native running instructions](running_natively.md). 
   * Run `bin/setup` first to create the needed database tables.
 3. Confirm the API is successfully running by seeing if you can visit [the local Flipper page.](http://localhost:3000/flipper/features)
+
+### Mock ClamAV
+
+If you wish to mock ClamAV, please set the clamav mock setting to true in settings.local.yml. This will mock the clamav response in the [virus_scan code](https://github.com/department-of-veterans-affairs/vets-api/blob/k8s/lib/common/virus_scan.rb#L14-L23). 
+
+```
+clamav:
+  mock: true
+```

docs/setup/native.md

@@ -6,18 +6,26 @@ Vets API requires:
 - PostgreSQL 15.x (including PostGIS 3)
 - Redis 6.2.x
 
-  The most up-to-date versions of each key dependency will be specified in the `docker-compose.yml` [file](https://github.com/department-of-veterans-affairs/vets-api/blob/master/docker-compose.yml) and the `Dockerfile`.
+The most up-to-date versions of each key dependency will be specified in the `docker-compose.yml` [file](https://github.com/department-of-veterans-affairs/vets-api/blob/master/docker-compose.yml) and the `Dockerfile`.
 
-  We suggest using a Ruby version manager such as [`rbenv`](https://github.com/rbenv/rbenv#installation), `asdf`, `rvm`, or `chruby` to install and maintain your version of Ruby.
+## Installing a Ruby Version Manager
 
-## Installing RVM
+We suggest using a Ruby version manager such as `rbenv`, `asdf`, `rvm`, or `chruby` to install and maintain your version of Ruby.
 
-1. Install `rvm` with `brew install rvm`. This could take a while.
-2. Check the ruby version number in `.ruby-version`. Use this number to install the needed Ruby version in the command `rvm install <version_number>`. This could also take a while.
-3. Run `rvm use` within the repo to confirm that the correct version is being used.
-4. After installing a new version of Ruby, run `gem install bundler` and `bundle install` to ensure all gems are installed for the current version.
+- [rbenv](https://github.com/rbenv/rbenv)
+- [rvm](https://rvm.io/)
+- [asdf](https://asdf-vm.com/)
+- [chruby](https://github.com/postmodern/chruby)
 
-Steps 2-4 must be repeated if the repo's Ruby version is updated later.
+If the repo's Ruby version is updated later, you will need to install the newer ruby (i.e., `rvm install <version_number>`) which is located in `.ruby-version`
+
+### RVM Troubleshooting
+
+If you see an error like `Error running '__rvm_make -j10'` while installing a ruby version, this usually occurs because of a mismatch with the openssl package. 
+
+Many of these types of errors occur because either the openssl path needs to be specified or there's a compatibility issue with the ruby version and the install openssl version. They may get resolved by explicitly adding the directory or trying newer openssl version.    
+
+For example: `rvm install 3.2.4 -C --with-openssl-dir=/$(brew --prefix openssl@3)`
 
 ## Base Setup
 
@@ -88,36 +96,45 @@ This file has the necessary configuration settings for local development as well
 
 ### Configuring ClamAV antivirus
 
-In many cases, there in no need to run ClamAV for local development, even if you are working with uploaded files since the scanning functionality is already built into our CarrierWave and Shrine file upload base classes.
+**NOTE:** In many cases, there in no need to run ClamAV for local development, even if you are working with uploaded files since the scanning functionality is already built into our CarrierWave and Shrine file upload base classes.
 
-If you would like to run a fake ClamAV "scanner" that will quickly produce a virus-free scan, you can configure the application to use the executable bash script `bin/fake_clamd`. This configuration is commented out in `config/settings.local.yml`
+Prior to EKS, ClamAV (the virus scanner) was deployed in the same process as Vets API. With EKS, ClamAV has been extracted out into it’s own service. Locally you can see the docker-compose.yml config for clamav.
 
-```yaml
-binaries:
-  # For NATIVE and DOCKER installation
-  # A "virus scanner" that always returns success for development purposes
-  # NOTE: You may need to specify a full path instead of a relative path
-  clamdscan: ./bin/fake_clamdscan
+1. In settings.local.yml add the following:
+
+```
+clamav:
+  mock: false
+  host: '0.0.0.0'
+  port: '33100'
 ```
 
-If you wish to run ClamAV, you'll need to check the platform specific notes.
+#### Mock ClamAV
+
+If you wish to mock ClamAV, please set the clamav mock setting to true in settings.local.yml. This will mock the clamav response in the [virus_scan code](https://github.com/department-of-veterans-affairs/vets-api/blob/master/lib/common/virus_scan.rb#L14-L23). 
+
+```
+clamav:
+  mock: true
+```
 
 ## Platform Specific Notes
 
 Specific notes for our most common native installation platforms are in this section. Note that most Windows users tend to use Docker instead of a native installation.
 
 ### OSX
 
+
 All of the OSX instructions assume `homebrew` is your [package manager](https://brew.sh/)
 
 1. Install Postgresql & PostGIS
 
-   1. It is MUCH easier to use the [Postgres.app](https://postgresapp.com/downloads.html) which installs the correct combination of Postgresql and PostGIS versions.
+   1. It is *_MUCH_* easier to use the [Postgres.app](https://postgresapp.com/downloads.html) which installs the correct combination of Postgresql and PostGIS versions.
 
    - Download the Postgres.app with PostgreSQL 15
    - Install Instructions here: https://postgresapp.com/
    - `sudo mkdir -p /etc/paths.d && echo /Applications/Postgres.app/Contents/Versions/latest/bin | sudo tee /etc/paths.d/postgresapp`
-   - `ARCHFLAGS="-arch x86_64" gem install pg -v 1.2.3`
+   - `ARCHFLAGS="-arch x86_64" gem install pg -v 1.5.6`
    2. Alternatively Postgresql 15 & PostGIS 3 can be installed with homebrew
       - `brew install postgresql@15`
       - `brew services start postgresql@15`
@@ -134,13 +151,13 @@ All of the OSX instructions assume `homebrew` is your [package manager](https://
     brew services start redis
     ```
 
-
 3. Install binary dependencies:
     ```bash
     brew bundle
-    ```
+    ```	 
+
 
-4. Among other things, the above `brew bundle` command installs ClamAV, but does not enable it. To enable ClamAV:
+4. (Optional see Running Natively for more info) Enable ClamAV daemon:
 
    ```bash
    brew info clamav
@@ -203,13 +220,11 @@ All of the OSX instructions assume `homebrew` is your [package manager](https://
    - `sudo apt install -y imagemagick`
 5. Install Poppler
    - `sudo apt install -y poppler-utils`
-6. Install ClamAV
-   - `sudo apt install -y clamav`
-7. Install pdftk
+6. Install pdftk
    - `sudo apt install -y pdftk`
-8. continue with [Base setup](native.md#base-setup)
+7. continue with [Base setup](native.md#base-setup)
 
-9. Updating Postgres and PostGIS if you already have them installed
+8. Updating Postgres and PostGIS if you already have them installed
 
    Backup your existing database
    ```bash
@@ -316,4 +331,4 @@ All of the OSX instructions assume `homebrew` is your [package manager](https://
    sudo pg_dropcluster ## main
 
    Done!!!
-  ```
+  ```

docs/setup/running_natively.md

@@ -13,7 +13,6 @@ browser and start interacting with the API. Changes to the source in your local
 directory will be reflected automatically via a docker volume mount, just as
 they would be when running rails directly.
 
-
 ### Running tests
 
 - `bundle exec rake spec` - Run the entire test suite  ( for `rspec spec`). Test coverage statistics are in `coverage/index.html`.
@@ -33,3 +32,48 @@ they would be when running rails directly.
 ### Running a rails interactive console
 
 - `rails console` -  runs an IRB like REPL in which all of the API's classes and environmental variables have been loaded.
+
+### Running with ClamAV
+
+#### Run with ClamAV containers (recommended)
+
+1. In `settings.local.yml` turn mocking off:
+```
+clamav:
+  mock: false
+  host: '0.0.0.0'
+  port: '33100'
+```
+
+1. In another terminal window, navigate to the project directory and run 
+```
+docker-compose -f docker-compose-clamav.yml up
+```
+
+1. In the original terminal run the following command
+```
+foreman start -m all=1,clamd=0,freshclam=0
+```
+
+This overrides any configurations that utilize the daemon socket
+
+#### Run with ClamAV daemon
+
+1. In `settings.local.yml` turn mocking off and make sure the host and port are removed:
+```
+clamav:
+  mock: false
+```
+
+1. Uncomment socket env var in `config/initializers/clamav.rb`
+
+```
+ENV['CLAMD_UNIX_SOCKET'] = '/usr/local/etc/clamav/clamd.sock'
+```
+
+*Note you will need to comment this line out before pushing to GitHub*
+
+1. In terminal run the following command
+```
+foreman start -m all=1
+```