LZ to DMZ Sync

[jayjaybunce]

🗣 Description

• Changed VS Sync scan to post org data to endpoint in chunks of 1000 orgs
• Added endpoint to ingest data from VS Sync scan
• Added a local copy of the MDL (the one in DMZ)
• Added syndnewmdl script to stand up the tables for the local MDL
• Implemented CSV parser middleware to allow CSV data ingests on the new endpoint

💭 Motivation and context

Resolves CRASM-714 + CRASM-785

🧪 Testing

Testing Steps

Prepare your local environment

1. Stop all containers
2. Add the below to your .env

MDL_USERNAME=mdl_local
MDL_PASSWORD=mini_data_lake
MDL_HOST=xfd-db-1
MDL_DATABASE=mini_data_lake_local
DMZ_API_KEY=d196030065c564bed15f8a8fc3ca90c7
OLD_MDL_USERNAME=mdl 
OLD_MDL_PASSWORD=password 
OLD_MDL_NAME=crossfeed_mini_datalake 

3. Remove the contents of /postgres-data => This will reset your local app database
4. /backend => npm run mdl-datagen

• See below if you have an M1 or M2 chip

5. Start all containers (ensure they are rebuilt)

6. /backend => npm run syncdb -- -d populate

7. /backend => npm run syncnewmdl

8. /backend => npm run syncmdl

9. docker cp /path/to/your/ mdl_backup_data.sql xfd-db-1:/tmp/mdl_backup_data.sql

10. Connect to the database docker => Run: docker exec -it xfd-db-1 bash
    10b. psql -U mdl -d crossfeed_mini_datalake -f tmp/mdl_backup_data.sql

11. Login to the application

12. Open developer tools and go to network => Copy the Auth header from any request

13. Add this to .env

DMZ_API_KEY=<yourtoken>

14. Build worker => /backend => npm run build-worker
15. Restart all containers

• See below if you have an M1 or M2 MacBook

16. Go to Admin-Tools
17. Create a new scan => vulnScanningSync
18. Click the play button next to the scan to manually trigger it
19. Connect to the local MDL with the below connection details

host: 127.0.0.1
username: mdl_local
password:: mini_data_lake
database: mini_data_lake_local

20. Confirm there are now records in the organization, cidr and cidr_organizations tables

Build Worker for M1/M2 Apple chips

A. Add the following below line 4 in /backend/Dockerfile.worker

RUN apk add --no-cache python3 py3-pip make g++

# Ensure python command points to python3
RUN ln -sf python3 /usr/bin/python```

B. /backend => docker build -t crossfeed-worker -f Dockerfile.worker .

✅ Pre-approval checklist

• This PR has an informative and human-readable title.
• Changes are limited to a single goal - eschew scope creep!
• All future TODOs are captured in issues, which are referenced
  in code comments.
• All relevant type-of-change labels have been added.
• I have read the CONTRIBUTING document.
• These code changes follow cisagov code standards.
• All relevant repo and/or project documentation has been updated
  to reflect the changes in this PR.
• Tests have been added and/or modified to cover the changes in this PR.
• All new and existing tests pass.

✅ Pre-merge checklist

• Revert dependencies to default branches.
• Finalize version.

✅ Post-merge checklist

• Create a release.