aft-account-request-processor Lambda throws InternalFailure

[yermulnik]

Terraform Version & Prov:

Terraform 1.9.5

AFT Version:

1.13.1

Bug Description
Every so often the aft-account-request-processor Lambda throws Internal Failure:

{
    "time_stamp": "2024-12-18 06:14:04,484",
    "module": "aft_account_request_processor",
    "log_level": "ERROR",
    "log_message": {
        "FILE": "aft_account_request_processor.py",
        "METHOD": "lambda_handler",
        "EXCEPTION": "An error occurred (InternalFailure) when calling the DescribeProductAsAdmin operation (reached max retries: 4): Internal Failure"
    }
}

Traceback (most recent call last):
  File "/var/task/aft_account_request_processor.py", line 43, in lambda_handler
    account_request = AccountRequest(auth=auth)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.11/site-packages/aft_common/account_request_framework.py", line 354, in __init__
    self.account_factory_product_id = aft_common.service_catalog.get_ct_product_id(
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.11/site-packages/aft_common/service_catalog.py", line 44, in get_ct_product_id
    response = client.describe_product_as_admin(Name=sc_product_name)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.11/site-packages/botocore/client.py", line 535, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/lib/python3.11/site-packages/botocore/client.py", line 980, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InternalFailure) when calling the DescribeProductAsAdmin operation (reached max retries: 4): Internal Failure

To Reproduce
Steps to reproduce the behavior:

1. Let aft-account-request-processor Lambda run and wait for when it fails.

Expected behavior
aft-account-request-processor Lambda doesn't throw ambiguous errot and iether handles it properly, or is fixed to not encounter going forward.

[yermulnik]

I also searched through this Lambda function's CloudWatch logs for the past month and it was all fine up until today when it threw this exception.
Previously we observed the same on Nov 6, Oct 17, Aug 6 (before this we, I guess, had not logging) — so it's kind of irregularly regular.