Skip to content

Commit

Permalink
Add Lambda support
Browse files Browse the repository at this point in the history
  • Loading branch information
jimzucker committed Sep 7, 2020
1 parent a4e6270 commit 3f61a7d
Show file tree
Hide file tree
Showing 9 changed files with 74 additions and 36 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
.DS_Store
.idea/
info.log
# Byte-compiled / optimized / DLL files
Expand Down
21 changes: 21 additions & 0 deletions LAMBDA_README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Setting up Lambda
There are 3 parts to configuring this to run as a Lambda
1. IAM Role for Lambda - The IAM Role for you Lambda will have to give permissions for Lambda and Cost Explorer.
2. Trigger - Event Bridge(Cloudwatch Alarms) setup with a cron expression to trigger a run daily.
3. Function Code - You can directly paste in the get_forecast.py file is all ready to go.

## IAM Role
When you create the Lambda it will create a role for you with required Lambda permissions, you must add the following
permissions to it:
![Lambda IAM Role](https://github.com/jimzucker/aws-forecast/blob/master/images/IAM_permissions.png)

## Configuring Trigger
Here is an example trigger, keep in mind the cron runs in UTC Timezone.
![Lambda Trigger](https://github.com/jimzucker/aws-forecast/blob/master/images/event_bridge.png)

## Function Code
You can directly copy and paste get_forecast.py into the Lambda definition without modifications.
![Lambda Function Code](https://github.com/jimzucker/aws-forecast/blob/master/images/lambda_function.png)

# AWS Architecture
![AWS Architecture](https://github.com/jimzucker/aws-forecast/blob/master/images/aws_architecture.png)
21 changes: 16 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,7 @@ So I set out to automate this as a slack post daily to save time. While doing t
2. Application must produce a cleanly formatted one line output.
3. Code must be written as python functions that we can re-use to integrate into a slack-bot.
4. Post to slack if url is defined as an AWS secret (see below)

# Technical Notes
5. Provide example Lambda function that posts to slack on a cron 1 time per day

### Command line
```python3 get_forecast.py --profile <aws profile> --type [FORECAST | ACTUALS]```
Expand All @@ -24,8 +23,21 @@ So I set out to automate this as a slack post daily to save time. While doing t
### Enabling Slack
To enable posting the message to slack instead of outputing to the command line you must define a secret in secrets manager called 'awsgenie_forecast_slack_url' with key=slack_url and value=<slack url>, if this secret is not found then the output is to the console.

![Sample Output of get_forecast](https://github.com/jimzucker/aws-forecast/blob/master/images/aws_secret.png)
![Enabling Slack](https://github.com/jimzucker/aws-forecast/blob/master/images/aws_secret.png)

### Setting up Lambda
There are 3 parts to configuring this to run as a Lambda
1. IAM Role for Lambda - The IAM Role for you Lambda will have to give permissions for Lambda and Cost Explorer.
2. Trigger - Event Bridge(Cloudwatch Alarms) setup with a cron expression to trigger a run daily.
3. Function Code - You can directly paste in the get_forecast.py file is is all ready to go.

[Click here for instructions for setting up Lambda](https://github.com/jimzucker/aws-forecast/blob/master/LAMBDA_README.md)

# AWS Architecture
![AWS Architecture](https://github.com/jimzucker/aws-forecast/blob/master/images/aws_architecture.png)


# Technical Notes

#### SSL Errors posting message to slack
If you get SSL Cert errors defining this environment varialbe may help you:
Expand All @@ -43,6 +55,5 @@ In testing I found several situations where the calls to get_cost_forecast would
2. Failure on new accounts or start of the month - on some days the calc fails due to insufficient data and we have to fall back to actuals

# Backlog
1. Add example Lambda function that posts to slack on a cron 1 time per day
2. Deploy the whole thing as infrastructure as code with Cloud Formation
1. Deploy the whole thing as infrastructure as code with Cloud Formation

67 changes: 36 additions & 31 deletions get_forecast.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,16 +36,12 @@
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError

logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(threadName)s] [%(levelname)s] %(message)s",
handlers=[
logging.FileHandler("info.log"),
logging.StreamHandler()
])
# Initialize you log configuration using the base class
logging.basicConfig(level = logging.INFO)

# Retrieve the logger instance
logger = logging.getLogger()


def arg_parser():
"""Extracts various arguments from command line
Expand Down Expand Up @@ -75,6 +71,7 @@ def formatter(prog):

def get_secret(client, secret_name):
try:
text_secret_data = ""
get_secret_value_response = client.get_secret_value(
SecretId=secret_name
)
Expand Down Expand Up @@ -208,8 +205,7 @@ def calc_forecast(costs_explorer_client):
def display_output(forecast_slack_url,message):
if forecast_slack_url != "":
send_slack(forecast_slack_url, message)
else:
logger.info(message)
print(message)

def send_slack(slack_hook_url, message):
slack_message = {
Expand All @@ -227,29 +223,38 @@ def send_slack(slack_hook_url, message):
except URLError as e:
logger.error("Server connection failed: %s", e.reason)

def get_forecast():
def get_forecast(boto3_session, type):
print(boto3_session)
forecast_slack_url=""
try:
secrets_manager_client = boto3_session.client('secretsmanager')
forecast_slack_url='https://' + get_secret(secrets_manager_client, "awsgenie_forecast_slack_url")
except Exception as e:
logger.warning("Disabling Slack URL not found")
print(e)

costs_explorer_client = boto3_session.client('ce')
if type in ['FORECAST']:
forecast = calc_forecast(costs_explorer_client)
display_output(forecast_slack_url, forecast)
elif type in ['ACTUALS']:
raise Exception("not implimented - ACTUALS")
actuals = calc_forecast(costs_explorer_client)
display_output(forecast_slack_url, actuals)
else:
raise Exception("Invalid run type: cmdline_params.type . Please choose from: FORECAST, ACTUALS")

return forecast

def lambda_handler(event, context):
logger.info("Event: " + str(event))
get_forecast(boto3, "FORECAST")

def main():
try:
cmdline_params = arg_parser()
boto3_session = boto3.Session(profile_name=cmdline_params.profile)
costs_explorer_client = boto3_session.client('ce')

forecast_slack_url=""
try:
secrets_manager_client = boto3_session.client('secretsmanager')
forecast_slack_url='https://' + get_secret(secrets_manager_client, "awsgenie_forecast_slack_url")
except Exception as e:
logger.warning("Disabling Slack URL not found")

if cmdline_params.type in ['FORECAST']:
forecast = calc_forecast(costs_explorer_client)
display_output(forecast_slack_url, forecast)
elif cmdline_params.type in ['ACTUALS']:
raise Exception("not implimented - ACTUALS")
actuals = calc_forecast(costs_explorer_client)
display_output(forecast_slack_url, actuals)
else:
print("Invalid run type: cmdline_params.type . Please choose from: FORECAST, ACTUALS")
sys.exit(1)
get_forecast(boto3_session, cmdline_params.type)

except Exception as e:
error_message = f"Error: {e}\n {traceback.format_exc()}"
Expand All @@ -259,4 +264,4 @@ def get_forecast():
sys.exit(0)

if __name__ == '__main__':
get_forecast()
main()
Binary file added images/IAM_permissions.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/aws_architecture.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/aws_architecture.pptx
Binary file not shown.
Binary file added images/event_bridge.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/lambda_function.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 3f61a7d

Please sign in to comment.