Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HOTFIX]: Add manual trigger for cache updater stack #1001

Merged
merged 10 commits into from
Feb 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/deploy-prd-enhanced-cache.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
on:
release:
types: [released]
workflow_dispatch:
inputs:
forceRun:
description: Force the cache to be rebuilt.
default: 'false'
concurrency:
group: environment-prd-enhanced-cacheUpdate-${{ github.ref }}
cancel-in-progress: true
Expand All @@ -24,3 +29,4 @@ jobs:
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
WORKSPACE: ${{ github.workspace }}
FORCE_RUN: ${{ inputs.forceRun }}
7 changes: 6 additions & 1 deletion cdk-infra/bin/cdk-infra.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,12 @@ async function main() {
env,
});

new CacheUpdaterStack(app, `${stackName}-cache`, { vpc, env, githubSecret: workerSecureStrings.GITHUB_SECRET });
new CacheUpdaterStack(app, `${stackName}-cache`, {
vpc,
env,
githubSecret: workerSecureStrings.GITHUB_SECRET,
githubBotPassword: workerSecureStrings.GITHUB_BOT_PASSWORD,
});
}

main();
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,21 @@ import { Bucket } from 'aws-cdk-lib/aws-s3';
import { Construct } from 'constructs';
import path from 'path';
import { getSnootyParserVersion } from '../../../utils/env';
import { StringParameter } from 'aws-cdk-lib/aws-ssm';

const SNOOTY_CACHE_BUCKET_NAME = 'snooty-parse-cache';

interface CacheUpdaterWorkerConstructProps {
vpc: IVpc;
githubBotPassword: string;
}

export class CacheUpdaterWorkerConstruct extends Construct {
readonly clusterName: string;
readonly taskDefinition: TaskDefinition;
readonly containerName: string;

constructor(scope: Construct, id: string, { vpc }: CacheUpdaterWorkerConstructProps) {
constructor(scope: Construct, id: string, { vpc, githubBotPassword }: CacheUpdaterWorkerConstructProps) {
super(scope, id);

const cluster = new Cluster(this, 'cacheUpdaterCluster', {
Expand All @@ -34,15 +36,16 @@ export class CacheUpdaterWorkerConstruct extends Construct {
snootyParseCacheBucket.grantWrite(taskRole);

const taskDefinition = new FargateTaskDefinition(this, 'cacheUpdaterWorker', {
cpu: 2048,
memoryLimitMiB: 4096,
cpu: 4096,
memoryLimitMiB: 8192,
taskRole,
});

const containerName = 'cacheUpdaterWorkerImage';
const taskDefLogGroup = new LogGroup(this, 'cacheUpdaterWorkerLogGroup');

const snootyParserVersion = getSnootyParserVersion();
const githubBotUsername = StringParameter.valueFromLookup(this, '/env/prd/docs/worker_pool/github/bot/username');

taskDefinition.addContainer('cacheUpdaterWorkerImage', {
image: ContainerImage.fromAsset(path.join(__dirname, '../../../../'), {
Expand All @@ -52,7 +55,10 @@ export class CacheUpdaterWorkerConstruct extends Construct {
}),
environment: {
SNOOTY_CACHE_BUCKET_NAME,
GITHUB_BOT_PASSWORD: githubBotPassword,
GITHUB_BOT_USERNAME: githubBotUsername,
},

logging: LogDrivers.awsLogs({
streamPrefix: 'cacheupdater',
logGroup: taskDefLogGroup,
Expand Down
9 changes: 7 additions & 2 deletions cdk-infra/lib/stacks/cache-updater-stack.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,20 @@ import { CacheUpdaterApiConstruct } from '../constructs/cache-updater/cache-upda
interface CacheUpdaterStackProps extends StackProps {
vpc: Vpc;
githubSecret: string;
githubBotPassword: string;
}
export class CacheUpdaterStack extends Stack {
constructor(scope: Construct, id: string, { vpc, githubSecret, ...props }: CacheUpdaterStackProps) {
constructor(
scope: Construct,
id: string,
{ vpc, githubSecret, githubBotPassword, ...props }: CacheUpdaterStackProps
) {
super(scope, id, props);

const { clusterName, taskDefinition, containerName } = new CacheUpdaterWorkerConstruct(
this,
'cache-updater-resources',
{ vpc }
{ vpc, githubBotPassword }
);

new CacheUpdaterApiConstruct(this, 'cache-updater-api', {
Expand Down
2 changes: 2 additions & 0 deletions src/cache-updater/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ A cache invalidation event occurs under two circumstances:

2. An updated version of the Snooty Parser is used by the Autobuilder; all doc sites' Snooty caches will be rebuilt in this event

The cache update process can be manually triggered by going to the GitHub Actions for the `docs-worker-pool` repository, going to the workflow for the cache updater, and then clicking run workflow. Make sure to set force run to be `true`.

### Architecture

#### Worker
Expand Down
16 changes: 14 additions & 2 deletions src/cache-updater/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,27 @@ import { S3Client } from '@aws-sdk/client-s3';
import { executeCliCommand } from '../commands/src/helpers';

const readdirAsync = promisify(fs.readdir);
const SNOOTY_CACHE_BUCKET_NAME = process.env.SNOOTY_CACHE_BUCKET_NAME;
const { SNOOTY_CACHE_BUCKET_NAME, GITHUB_BOT_USERNAME, GITHUB_BOT_PASSWORD } = process.env;

if (!SNOOTY_CACHE_BUCKET_NAME) throw new Error('ERROR! SNOOTY_CACHE_BUCKET_NAME is not defined');

async function cloneDocsRepo(repoName: string, repoOwner: string) {
if (!GITHUB_BOT_USERNAME) {
const errorMessage = `ERROR! GITHUB_BOT_USERNAME is not set. ${repoOwner}/${repoName} will not have their cache updated`;
console.error(errorMessage);
throw new Error(errorMessage);
}

if (!GITHUB_BOT_PASSWORD) {
const errorMessage = `ERROR! GITHUB_BOT_PASSWORD is not set. ${repoOwner}/${repoName} will not have their cache updated`;
console.error(errorMessage);
throw new Error(errorMessage);
}

try {
const cloneResults = await executeCliCommand({
command: 'git',
args: ['clone', `https://github.com/${repoOwner}/${repoName}`],
args: ['clone', `https://${GITHUB_BOT_USERNAME}:${GITHUB_BOT_PASSWORD}@github.com/${repoOwner}/${repoName}`],
});

console.log('clone: ', cloneResults);
Expand Down
Loading