README.md

AWSKinesisStreams Examples

This document describes the example applications provided with the AWSKinesisStreams library:

• DataProducer, which writes data records to an AWS Kinesis Stream.
• DataConsumer, which reads data records from an AWS Kinesis Stream.

To read data you need to run DataConsumer in parallel with DataProducer. We recommend that you run DataProducer as the agent of one imp and run DataConsumer as the agent of a second imp.

Each example is described below. If you wish to try one out, you will find setup instructions further down the page.

IMPORTANT Before proceeding, please review Amazon Kinesis Streams Pricing. Running these examples may not be free of charge.

DataProducer

This example writes data records to the specified preconfigured AWS Kinesis Stream:.

• Data records are written every ten seconds.
• Data records are written using the putRecord() and putRecords() methods of AWSKinesisStreams.Producer, switching the method every ten seconds.
• Every data record contains:
  • A "value" attribute. This is an integer value which starts at 1 and increases by 1 with every record written. It restarts from 1 every time the example is restarted.
  • An "origin" attribute. This contains the name of the method used to write the record, ie. "putRecord" or "putRecords".

DataConsumer

This example reads new data records from all shards of the specified preconfigured AWS Kinesis Stream.

• Only new records, appeared in the AWS Kinesis Streams after the example is started, are read.
• The first read occurs 15 seconds after the example is started.
• After that, data records are read every 15 seconds.
• Every received data record is printed to the log.

Using the Examples

1. In the Electric Imp IDE create two new models, one for imp A and the other for imp B.
2. Copy the DataProducer source code and paste it into the IDE as agent code for imp A.
3. Copy the DataConsumer source code and paste it into the IDE as agent code for imp B.
4. Perform AWS Kinesis Streams Setup as described below.
5. Perform Agent Constants Setup as described below.
6. Build and Run DataProducer.
7. Check from the logs in the IDE that data is being written successfully.
8. Build and Run DataConsumer.
9. Check from the logs in the IDE that data is being read successfully.

AWS Kinesis Streams Setup

This process assumes you have an AWS account and are signed in to AWS in your web browser.

IMPORTANT Before proceeding, please review Amazon Kinesis Streams Pricing. Running these examples may not be free of charge.

Create an AWS Kinesis Stream

1. Open the Amazon Kinesis console in your web browser.
2. Click Go to the Streams console:
3. Click Create Kinesis stream:
4. Enter testStream into the Kinesis stream name field.
5. Enter 1 into the Number of shards field.
6. Click Create Kinesis stream:
7. You will be redirected to the Kinesis streams list page.
8. Wait until the status of your Stream is changed to ACTIVE.
9. Click to your Stream name:
10. Copy the Stream ARN of your Stream and paste it into a text document or equivalent for use later:
11. The ARN format is arn:aws:kinesis:region:account:stream/name. Copy the region, eg. us-east-2 and paste it into a text document or equivalent. It will be used as the value of the AWS_KINESIS_REGION constant in the agent code.

Create an AWS IAM Policy and User

1. Open the AWS IAM console in your web browser.
2. Click Policies in the left-hand menu:
3. Click Create policy:
4. Under Policy Generator, click Select:
5. Choose Amazon Kinesis as the AWS service.
6. Choose All Actions in the Actions field:
7. Enter your Stream ARN, which you retrieved above, in the Amazon Resource Name (ARN) field.
8. Click Add Statement:
9. Click Next Step:
10. Change Policy Name to testStreamPolicy.
11. Click Create Policy:
    
12. Click Users in the left-hand menu.
13. Click Add user:
14. Enter testStreamUser in the User name field.
15. For Access type choose Programmatic access.
16. Click Next: Permissions:
17. Click Attach existing policies directly:
18. In the Search field enter testStreamPolicy.
19. Check the box to the left of your policy name.
20. Click Next: Review:
21. Click Create user:
22. Copy the Access key ID and paste it onto a text document or equivalent. It will be used as the value of the AWS_KINESIS_ACCESS_KEY_ID constant in the agent code.
23. Click Show under Secret access key. Copy the Secret access key and paste it onto a text document or equivalent. It will be used as the value of the AWS_KINESIS_SECRET_ACCESS_KEY constant in the agent code:

Agent Constants Setup

1. For the AWS_KINESIS_REGION, AWS_KINESIS_ACCESS_KEY_ID and AWS_KINESIS_SECRET_ACCESS_KEY constants, set the values you retrieved in the previous steps. Set the same values for both DataProducer and DataConsumer: