Does not report failed deployment correctly

[edweip3]

If a deployment failed, EB will return status green with a previous build deployed. This causes the plugin to think deployment is successful, but it is not. It should instead check to make sure the version labels match after the deployment turns green. If they don't match, then the deployment has failed.

[myoung34]

Here's some extra info: ```
Started by user Marc Young
[EnvInject] - Loading node environment variables.
Building remotely on Jenkins Slave (OPS) (i-0cc1339e477a2aa63) (jenkins_ops_slave) in workspace /home/ubuntu/workspace/Deploy Beanstalk
Cloning the remote Git repository
Cloning repository [email protected]:redact/redact.git

git init /home/ubuntu/workspace/Deploy Beanstalk # timeout=10
Fetching upstream changes from [email protected]:redact/redact.git
git --version # timeout=10
git fetch --tags --progress [email protected]:redact/redact.git +refs/heads/:refs/remotes/origin/
git config remote.origin.url [email protected]:redact/redact.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url [email protected]:redact/redact.git # timeout=10
Fetching upstream changes from [email protected]:redact/redact.git
git fetch --tags --progress [email protected]:redact/redact.git +refs/heads/:refs/remotes/origin/
Seen branch in repository origin/master
Seen 1 remote branches
Checking out Revision 8eb5aa5a7620f0ec70cdacb646cf1e21d35ebc4c (origin/master)
git config core.sparsecheckout # timeout=10
git checkout -f 8eb5aa5a7620f0ec70cdacb646cf1e21d35ebc4c
git rev-list 8eb5aa5a7620f0ec70cdacb646cf1e21d35ebc4c # timeout=10
git tag -a -f -m Jenkins Build #28 jenkins-Deploy_Beanstalk-28 # timeout=10
[Deploy Beanstalk] $ /bin/bash -x /tmp/hudson737544988919118642.sh
++ git rev-parse --short HEAD

• echo GIT_SHORT_HASH=8eb5aa5
  ++ date -u +%Y-%m-%dT%H:%M:%SZ
• echo DEPLOY_TIME=2016-11-14T19:27:31Z
  [EnvInject] - Injecting environment variables from a build step.
  [EnvInject] - Injecting as environment variables the properties file path 'env.props'
  [EnvInject] - Variables injected successfully.
  AWSEB Deployment Plugin Version 0.3.10
  Zipping contents of Root File Object (Deploy Beanstalk) into tmp file awseb-4080026641504629791.zip (includes=, excludes=)
  Uploading file awseb-4080026641504629791.zip as s3://elasticbeanstalk-us-east-1-redact/redact/redact-8eb5aa5-2016-11-14T19:27:31Z.zip
  Creating application version 8eb5aa5-2016-11-14T19:27:31Z for application redact for path s3://elasticbeanstalk-us-east-1-redact/redact/redact-8eb5aa5-2016-11-14T19:27:31Z.zip
  Created version: 8eb5aa5-2016-11-14T19:27:31Z
  Using environmentId 'e-rm2ptmfham'
  No pending Environment Updates. Proceeding.
  Checking health/status of environmentId e-rm2ptmfham attempt 1/30
  Environment Status is 'Ready'. Moving on.
  Updating environmentId 'e-rm2ptmfham' with Version Label set to '8eb5aa5-2016-11-14T19:27:31Z'
  Checking health/status of environmentId e-rm2ptmfham attempt 1/30
  Versions reported: (current=8eb5aa5-2016-11-14T16:15:08Z, underDeployment: 8eb5aa5-2016-11-14T19:27:31Z). Should I move on? false
  Checking health/status of environmentId e-rm2ptmfham attempt 2/30
  Versions reported: (current=8eb5aa5-2016-11-14T16:15:08Z, underDeployment: 8eb5aa5-2016-11-14T19:27:31Z). Should I move on? false
  Checking health/status of environmentId e-rm2ptmfham attempt 3/30
  Versions reported: (current=8eb5aa5-2016-11-14T16:15:08Z, underDeployment: 8eb5aa5-2016-11-14T19:27:31Z). Should I move on? false
  Environment Status is 'Ready' and Health is 'Green'. Moving on.
  Deployment marked as 'successful'. Starting post-deployment cleanup.
  Cleaning up temporary file /tmp/awseb-4080026641504629791.zip
  Finished: SUCCESS

and the corresponding EB log:

2016-11-14 13:33:38 UTC-0600    ERROR   Failed to deploy application.
2016-11-14 13:33:38 UTC-0600    ERROR   Service:AmazonCloudFormation, Message:TemplateURL must reference a valid S3 object to which you have access.
2016-11-14 13:29:48 UTC-0600    INFO    Environment update is starting.

[aldrinleal]

Hm... interesting. Ok, I'll keep that in mind and push a change for it shortly ok?

[artemmeron]

another case with the same problem:

Versions reported: (current=[Jenkins]-[Integration]-[08-03-2017 1924], underDeployment: [BK-API-17]-[Integration]-[20-03-2017 1242]). Should I move on? false
Mon Mar 20 10:45:08 UTC 2017 [INFO] Started Application Update
Mon Mar 20 10:45:08 UTC 2017 [ERROR] Deployment Failed: Unexpected Exception
Mon Mar 20 10:44:59 UTC 2017 [INFO] Deploying new version to instance(s).
Checking health/status of environmentId e-kbhp2qmi3b attempt 2/30
Versions reported: (current=[Jenkins]-[Integration]-[08-03-2017 1924], underDeployment: [BK-API-17]-[Integration]-[20-03-2017 1242]). Should I move on? false
Environment Status is 'Ready' and Health is 'Green'. Moving on.
Deployment marked as 'successful'. Starting post-deployment cleanup.
Cleaning up temporary file C:\Users\ADMINI~1\AppData\Local\Temp\2\awseb-5418100666623237368.zip
Finished: SUCCESS`

[130n]

We currently use the following script after deploy-step to verify the correct version was deployed.

where ${VERSION}-timestamp-${BUILD_TIMESTAMP} is our Version Label Format

ELB_VERSION=`aws elasticbeanstalk describe-environments \
    --profile our-profile --region our-region \
    --environment-names $ENVIRONMENT --query 'Environments[0].VersionLabel' | sed 's/"//g'`
if [ "$ELB_VERSION" = "${VERSION}-timestamp-${BUILD_TIMESTAMP}" ]; then
    echo "correct version is deployed on ELB"
else
    echo "ELB version: $ELB_VERSION do not match deployment version: ${VERSION}-timestamp-${BUILD_TIMESTAMP}"
    exit 1
fi

Posting as it might be useful for someone.

[myoung34]

In our bot (python) I do something similar:

    events = client.describe_events(ApplicationName=eb_app, StartTime=deploy_start_time_utc, EnvironmentName=eb_env, Severity='ERROR')['Events']
    if len([event for event in events if event['Message'] == 'Failed to deploy application.']) == 0:
        register_deployment(eb_env, commit_hash)
    else:
        exit(1)