MHV-66203: CCD generate polling throttled with backoff

[mattwrightva]

Summary

• Generate CCD polling throttled with a backoff, up to 30 seconds.

Related issue(s)

https://jira.devops.va.gov/browse/MHV-66203

Download CCD occasionally generates a 404 error

Description
Our monitor for Download endpoints occasionally trips because the Download CCD endpoint generates a 404. Here is a list of such errors.

Understand why these are happening and fix if necessary.

Linked GitHub issue.

Testing done

No new tests needed

Screenshots

No UI changes

What areas of the site does it impact?

MHV medical records - CCD download

Acceptance criteria

AC1 Research and understand why error is happening

AC2 Fix any issues found

AC3 Add a backoff throttle for the generate polling, start w/ 1 second and back off 5-10% every call up to some max

Quality Assurance & Testing

• I fixed|updated|added unit tests and integration tests for each feature (if applicable).
• No sensitive information (i.e. PII/credentials/internal URLs/etc.) is captured in logging, hardcoded, or specs
• Linting warnings have been addressed
• Documentation has been updated (link to documentation *if necessary)
• Screenshot of the developed feature is added
• Accessibility testing has been performed

Error Handling

• Browser console contains no warnings or errors.
• Events are being sent to the appropriate logging solution
• Feature/bug has a monitor built into Datadog or Grafana (if applicable)

Authentication

• Did you login to a local build and verify all authenticated routes work as expected with a test user