Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ASM Sync to the SQS process #712

Merged
merged 1 commit into from
Nov 7, 2024
+19 −3
Merged

Add ASM Sync to the SQS process #712

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion backend/Dockerfile.pe
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ RUN ./aws/install
# Sync the latest from cf-staging branch
RUN git clone -b crossfeed-SQS https://github.com/cisagov/ATC-Framework.git && \
cd ATC-Framework && \
git checkout 6b5a8753ad5bcdb4426b5ba24ceffd77f901c228 && \
git checkout 22c3de519337e577775f2580ba09c7ba51c08904 && \
pip install .

RUN python -m spacy download en_core_web_lg
Expand Down
7 changes: 7 additions & 0 deletions backend/serverless.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,13 @@ resources:
VisibilityTimeout: 18000 # 5 hours
MaximumMessageSize: 262144 # 256 KB
MessageRetentionPeriod: 604800 # 7 days
ASMQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: ${self:provider.stage}-asm-queue
VisibilityTimeout: 18000 # 5 hours
MaximumMessageSize: 262144 # 256 KB
MessageRetentionPeriod: 604800 # 7 days

functions:
- ${file(./src/tasks/functions.yml)}
Expand Down
11 changes: 9 additions & 2 deletions backend/src/tasks/scanExecution.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,14 @@ import { integer } from 'aws-sdk/clients/cloudfront';
const ecs = new AWS.ECS();
let docker: any;
const QUEUE_URL = process.env.QUEUE_URL!;
const SCAN_LIST = ['dnstwist', 'intelx', 'cybersixgill', 'shodan', 'xpanse'];
const SCAN_LIST = [
'dnstwist',
'intelx',
'cybersixgill',
'shodan',
'xpanse',
'asmSync'
];

if (process.env.IS_LOCAL) {
const Docker = require('dockerode');
Expand Down Expand Up @@ -215,7 +222,7 @@ export const handler: Handler = async (event) => {
await startDesiredTasks(scanType, desiredCount);
} else {
console.log(
'Shodan, DNSTwist, IntelX, Xpanse, and Cybersixgill are the only script types available right now. Must be all lowercase.'
'Shodan, ASMSync, DNSTwist, IntelX, Xpanse, and Cybersixgill are the only script types available right now. Must be all lowercase.'
);
}
} catch (error) {
Expand Down
2 changes: 2 additions & 0 deletions backend/worker/pe-worker-entry.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ while true; do
COMMAND="pe-source cybersixgill --org=$ORG --soc_med_included"
elif [[ "$SERVICE_TYPE" = *"xpanse"* ]]; then
COMMAND="pe-source xpanse --org='$ORG'"
elif [[ "$SERVICE_TYPE" = *"asmSync"* ]]; then
COMMAND="pe-asm-sync asm-sqs --org='$ORG'"
else
echo "Unsupported SERVICE_TYPE: $SERVICE_TYPE"
break
Expand Down
Loading