worker_pubsub

Google Cloud Functions Recipes

Word Count Sample

Overview

This recipe demonstrates how to create a simple word count sample using a master-worker pattern using Cloud Pub/Sub.

Where applicable:

Replace [PROJECT-ID] with your Cloud Platform project ID

What's going on here?

1. Client calls the "master" function via HTTP
2. Master function pulls file from Google Cloud Storage
3. Master function segments the file and published mutliple messages to the "In Topic"
4. Cloud Functions allocates multiple workers (as necessary) to process segment messages and workers publish results to "Out Topic"
5. Master function subscribes to "Out Topic" and collects results from workers
6. Master function returns aggregate result to the client

Cooking the Recipe

1. Follow the Cloud Functions quickstart guide to setup Cloud Functions for your project

2. Clone this repository

   cd ~/
   git clone https://github.com/jasonpolites/gcf-recipes.git
   cd gcf-recipes/worker_pubsub
   

3. Create a Cloud Storage Bucket to stage our deployment

   gsutil mb gs://[PROJECT-ID]-gcf-recipes-bucket
   

4. Upload the sample file to the bucket

   gsutil cp sample.txt gs://[PROJECT-ID]-gcf-recipes-bucket
   

5. Create a Pub/Sub Topic for the workers to subscribe to

   gcloud alpha pubsub topics create gcf-mapr-in
   

6. Create a Pub/Sub Topic for the workers to publish to

   gcloud alpha pubsub topics create gcf-mapr-out		
   

7. Deploy the "master" function with an HTTP trigger

   gcloud alpha functions deploy mapr-pubsub-master --bucket [PROJECT-ID]-gcf-recipes-bucket --trigger-http --entry-point master
   

8. Deploy the "worker" function with a Pub/Sub trigger, using the 'gcf-mapr-in' topic as the source

   gcloud alpha functions deploy mapr-pubsub-worker --bucket [PROJECT-ID]-gcf-recipes-bucket --trigger-topic gcf-mapr-in --entry-point worker
   

9. Call the "master" function using the sample file, with both topics as function arguments

   gcloud alpha functions call mapr-pubsub-master --data '{"bucket": "[PROJECT-ID]-gcf-recipes-bucket", "file": "sample.txt", "in-topic": "gcf-mapr-in", "out-topic":"gcf-mapr-out"}'
   

You should see something like this in your console

The file sample.txt has 114 words

Running Tests

This recipe comes with a suite of unit tests. To run the tests locally, just use npm test

npm install
npm test

The tests will also produce code coverage reports, written to the /coverage directory. After running the tests, you can view coverage with

open coverage/lcov-report/index.html