DD Alerts - Create alert for failed 0781 document uploads

[tblackwe]

When an 0781 upload is marked as failed in Document Polling, generate an alert so the team knows to go remediate it.

Tasks

Preview Give feedback

1. create Datadog monitor to watch metric and trigger alert
2. set notification channel

[freeheeling]

@tblackwe Should an alert be created only in the event a 0781 document upload fails, or would it be better to alert regardless of document type: veteran upload, 0781, 0781a, or BDD instructions?

Also, included in @NB28VT's original implementation is tracking to indicate a document has been in a pending state for more than 24 hours. I'm not sure a monitor is warranted for alerting on such a state, or if logging of a potential processing timeout status is sufficient. Thoughts?

[freeheeling]

Alert created – https://vagov.ddog-gov.com/monitors/244896?view=spans – but could probably benefit from some refinement, if you're able to help me finalize, @tblackwe. Thanks!

[tblackwe]

I had originally said 0781 because they are the only ones we would go fix... but Emily asked us to go a different path yesterday.

If we arent going to do anything about it, I am not sure that an alert is truly needed anymore. I am going to think on this.

In general, the alert looks correct and covers what was requested

[freeheeling]

Alert created and code developed. However, currently blocked by the following when I attempt to push up my local branch:

[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

indicating Git client is unable to authenticate with GitHub using SSH keys.

Will troubleshoot tomorrow.

Update: Resolved by regenerating and configuring my SSH key.

[freeheeling]

PR to 'log when Form 0781 document upload fails' ready for review.