Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ClamAV in Containers #15965

Merged
merged 45 commits into from
Apr 29, 2024
Merged

ClamAV in Containers #15965

merged 45 commits into from
Apr 29, 2024

Conversation

stevenjcumming
Copy link
Contributor

@stevenjcumming stevenjcumming commented Mar 15, 2024

Summary

This is an early step to replace review instances with preview environments. This is also an early step in merging k8s into master.

  • Review Instances will use the same Dockerfile as the named environment.
  • replace clam_scan with clamav-client
  • remove clamav binaries & packages
  • allow developers to mock clamav via settings.local.yml
  • most files are 1:1 copies of k8s branch
  • review instances should use clamav container
  • updated Dockerfile to match k8s
  • minor updates to docker-compose files for test and review
  • updated Makefile to work with new docker-compose files

Related issue(s)

Testing done

  • Manual local testing for mocking ClamAV
  • Manual local testing for not-mocking ClamAV
  • Manual testing for ClamAV on Review Instances

Acceptance criteria

  • Developers can use ClamAV with native, docker, & hybrid setup
  • Developers can mock ClamAV
  • ClamAV (and file virus scan) works in Review Instances
  • Can log in on review instances
  • Successfully filled out a 10-10EZR form on Review Instances

@stevenjcumming stevenjcumming self-assigned this Mar 15, 2024
Copy link

github-actions bot commented Mar 15, 2024

1 Error
🚫 This PR changes 681 LoC (not counting whitespace/newlines).

In order to ensure each PR receives the proper attention it deserves, those exceeding
500 will not be reviewed, nor will they be allowed to merge. Please break this PR up into
smaller ones.

If you have reason to believe that this PR should be granted an exception, please see the
Submitting pull requests for approval - FAQ.

File Summary

Files

  • .github/CODEOWNERS (+12/-3)

  • .github/workflows/audit_service_tags.yml (+5/-6)

  • .github/workflows/code_checks.yml (+5/-6)

  • .gitignore (+4/-0)

  • Dockerfile (+51/-88)

  • Gemfile (+1/-1)

  • Makefile (+6/-13)

  • Procfile (+0/-2)

  • app/uploaders/uploader_virus_scan.rb (+4/-3)

  • bin/fake_clamdscan (+0/-5)

  • config/clamd.conf (+4/-2)

  • config/freshclam.conf (+1/-1)

  • config/initializers/clamav.rb (+6/-0)

  • config/initializers/clamscan.rb (+0/-5)

  • config/settings.local.yml.example (+4/-4)

  • config/settings.yml (+7/-0)

  • docker-compose-clamav.yml (+10/-0)

  • docker-compose-deps.yml (+8/-1)

  • docker-compose.review.yml (+54/-39)

  • docker-compose.test.yml (+1/-2)

  • docker-compose.yml (+46/-34)

  • lib/clamav/commands/patch_scan_command.rb (+43/-0)

  • lib/clamav/patch_client.rb (+76/-0)

  • lib/common/file_helpers.rb (+17/-0)

  • lib/common/virus_scan.rb (+12/-4)

  • lib/shrine/plugins/validate_virus_free.rb (+5/-3)

  • spec/lib/shrine/plugins/validate_virus_free_spec.rb (+9/-23)

  • spec/models/persistent_attachments/dependency_claim_spec.rb (+5/-2)

  • spec/models/persistent_attachments/lgy_claim_spec.rb (+5/-2)

  • spec/models/persistent_attachments/pension_burial_spec.rb (+5/-2)

  • spec/requests/claim_documents_spec.rb (+2/-2)

  • spec/simplecov_helper.rb (+0/-1)

  • spec/spec_helper.rb (+0/-1)

  • spec/support/uploader_helpers.rb (+1/-7)

  • spec/uploaders/uploader_virus_scan_spec.rb (+3/-7)

    Note: We exclude files matching the following when considering PR size:

    *.csv, *.json, *.tsv, *.txt, Gemfile.lock, app/swagger, modules/mobile/docs, spec/fixtures/, spec/support/vcr_cassettes/, modules/mobile/spec/support/vcr_cassettes/, db/seeds, modules/vaos/app/docs, modules/meb_api/app/docs, modules/appeals_api/app/swagger/, *.bru
    

Big PRs are difficult to review, often become stale, and cause delays.

Generated by 🚫 Danger

@va-vfs-bot va-vfs-bot temporarily deployed to sjc-clamav-container/main/main March 18, 2024 17:05 Inactive
@va-vfs-bot va-vfs-bot temporarily deployed to sjc-clamav-container/main/main March 19, 2024 16:25 Inactive
@va-vfs-bot va-vfs-bot temporarily deployed to sjc-clamav-container/main/main March 20, 2024 00:30 Inactive
@va-vfs-bot va-vfs-bot temporarily deployed to sjc-clamav-container/main/main March 20, 2024 11:48 Inactive
@va-vfs-bot va-vfs-bot temporarily deployed to sjc-clamav-container/main/main March 20, 2024 12:21 Inactive
@va-vfs-bot va-vfs-bot temporarily deployed to sjc-clamav-container/main/main March 20, 2024 14:45 Inactive
@va-vfs-bot va-vfs-bot temporarily deployed to sjc-clamav-container/main/main March 20, 2024 14:51 Inactive
@va-vfs-bot va-vfs-bot temporarily deployed to sjc-clamav-container/main/main March 21, 2024 18:34 Inactive
@stevenjcumming stevenjcumming marked this pull request as ready for review April 24, 2024 17:56
Comment on lines +4 to +5
ENV['CLAMD_TCP_HOST'] = Settings.clamav.host
ENV['CLAMD_TCP_PORT'] = Settings.clamav.port
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have these saved in the RI settings.yml? I think that's somewhere over in the devops repo.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah nvm, I guess if not defined, It'll use settings.yml

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's just in the settings.yml file

@LindseySaari
Copy link
Contributor

LindseySaari commented Apr 24, 2024

I would recommend moving over some of the README info around using clam locally from the k8s branch to master also.

Specifically these two places

  1. https://github.com/department-of-veterans-affairs/vets-api/blob/k8s/docs/setup/native.md#configuring-clamav-antivirus
  2. https://github.com/department-of-veterans-affairs/vets-api/blob/k8s/docs/setup/hybrid.md#running

@va-vfs-bot va-vfs-bot temporarily deployed to sjc-clamav-container/main/main April 24, 2024 18:50 Inactive
@stevenjcumming
Copy link
Contributor Author

I would recommend moving over some of the README info around using clam locally from the k8s branch to master also.

Specifically these two places

  1. https://github.com/department-of-veterans-affairs/vets-api/blob/k8s/docs/setup/native.md#configuring-clamav-antivirus
  2. https://github.com/department-of-veterans-affairs/vets-api/blob/k8s/docs/setup/hybrid.md#running

I have a separate PR for the doc changes (although it might need some more updating)

#16058

@va-vfs-bot va-vfs-bot temporarily deployed to sjc-clamav-container/main/main April 29, 2024 15:37 Inactive
@stevenjcumming
Copy link
Contributor Author

For some reason the coverage report isn't being uploaded to /coverage

Screenshot 2024-04-29 at 11 52 05 AM

.github/CODEOWNERS Show resolved Hide resolved
Dockerfile Show resolved Hide resolved
Dockerfile Show resolved Hide resolved
docker-compose-clamav.yml Show resolved Hide resolved
@stevenjcumming stevenjcumming merged commit 0857270 into master Apr 29, 2024
17 of 19 checks passed
@stevenjcumming stevenjcumming deleted the sjc-clamav-container branch April 29, 2024 19:06
stevenjcumming added a commit that referenced this pull request Apr 29, 2024
* ClamAV in Containers (#15965)

* use clamav container for local development and review instances

* rubocop formatting & CODEOWNERS

* fix Dockerfile

* update CODEOWNERS and docker-compose version for review instances

* change docker-compose verions for review

* fix review ports

* update github workflows to match k8s

* update github workflows

* update code_checks for docker compose for tests

* add bundle exec to setup db step in audit service tags

* reset db in docker compose review & update audit service tag docker services

* remove clamd and freshclam from procfile

* add sidekiq enterprise license to test env var

* set sidekiq license to docker build args

* update makefile to work with new docker services

* add clamav host & port to settings

* add restart services to docker-compose review

* Added virus scan to uploads spec (#16393)

* Added virus scan to uploads spec

* Fixing Virus scan spec

* resolve merge conflict

* change directory from /app to /srv/vets-api/src

* use previous master version for RI + clam containers

* add USER_ID to RI docker-compose

* set working directory back to app

* minor clean up with new line EOF & remove k8s deploy related code

* update makefile up command with foreman

* fix docker-compose clamav new line

* more cleanup and rubocop formatting

* remove redis ports from docker-compose test

* update redis port

* revert docker-compose test to original master

* update api service name to web

* rubocop formatting

* add Procfile to CODEOWNERS

* update CODEOWNERS

* add secret and pki volumes back to RI

* add some settings in review docker-compose

* fix merge conflict mistake in Dockerfile

* update dockerfile ruby version

---------

Co-authored-by: Rachal Cassity <[email protected]>

* delete code checks yml

---------

Co-authored-by: Rachal Cassity <[email protected]>
RachalCassity added a commit that referenced this pull request Apr 29, 2024
RachalCassity added a commit that referenced this pull request Apr 29, 2024
RachalCassity added a commit that referenced this pull request Apr 29, 2024
stevenjcumming pushed a commit that referenced this pull request Apr 30, 2024
gabezurita added a commit that referenced this pull request Apr 30, 2024
* master: (33 commits)
  [DBX-74810] Add flipper with LH BI api for 4142 (#16483)
  upadte param dig call (#16570)
  98 add code to poll average days (#16471)
  8147 claim model (#16525)
  Toxic Exposure: Save TE "Add Conditions" fields for future transformation (#16548)
  Revert "ClamAV in Containers (#15965)" (#16557)
  Dangerfile updates for db migration (#16511)
  API-35829-error_handling (#16522)
  add loggers to see what dd api call is doing in staging (#16539)
  8440: fix appointments contract schema (#16528)
  ClamAV in Containers (#15965)
  Protect against calling `.any?` on a non-array (#16547)
  BE | Ask VA Api: Refactor classes and specs (#16542)
  Added '10-10EZR' to the list of freeze schemas (#16478)
  Add prefill for 10-7959c (#16448)
  Bump shrine from 3.5.0 to 3.6.0 (#16532)
  change limit on place and location. update question number (#16537)
  Add words_to_remove for simple forms 21-0966 (#16477)
  updgrade sidekiq and super_diff (#16536)
  BE | Ask Va Api:  Update `Inquiries::Creator` `Correspondences::Creator` Error Handling (#16529)
  ...
stevenjcumming added a commit that referenced this pull request May 1, 2024
* prefer pdftk-java with debian bookworm

* 8440: fix appointments contract schema (#16528)

* add loggers to see what dd api call is doing in staging (#16539)

* API-35829-error_handling (#16522)

* Adds error handling for new synchronous endpoint. Iterates through errors and adds them to the ecss_response on the claim. Adds to the retry? method to check the evss_response for the error key.

* Addressese PR suggestions

* REmoves error message, and uses evss_response on the claim instead.

* Adds a save rigt after updating the status on the claim

* Adds several tests

* Alters a test to more accurately test the scenario IRL

* Dangerfile updates for db migration (#16511)

* Revert "ClamAV in Containers (#15965)" (#16557)

This reverts commit 0857270.

* Toxic Exposure: Save TE "Add Conditions" fields for future transformation (#16548)

* add cause field to disabilities array

* another flipper fix

* remove debug comment

* add comment for future engs

* tests

* nil check for input_disability

* break from loop, not break from method

* 8147 claim model (#16525)

---------

Co-authored-by: Kris Pethtel <[email protected]>
Co-authored-by: s-caso <[email protected]>
Co-authored-by: Jennica Stiehl <[email protected]>
Co-authored-by: Rebecca Tolmach <[email protected]>
Co-authored-by: Rachal Cassity <[email protected]>
Co-authored-by: Seth Darr <[email protected]>
Co-authored-by: Andrew Herzberg <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants