AWS SDK exception - Too many open files and Unknown host

[anAmateurDeveloper]

Hello, may I know what's the problem with my code snippet inside an AWS Lambda?

         final AmazonS3 s3Client = AmazonS3ClientBuilder.defaultClient();

        S3Object s3Object = s3Client.getObject(bucketName, key);
        InputStream inputStream = new ByteArrayInputStream(toByteArray(s3Object.getObjectContent()));
        s3Object.close();

        return inputStream;

This code throws SDK exception quite often (not always) if Lambda receives more intensive requests. "too many open files" or "Unknown host" issues are always shown as the root cause. We doubt that's caused by Lambda has run out of FDs. But how could it happen? We closed connection every time after got resources. Any idea on improvement?

[debora-ito]

"Too many open files" sounds like the connections are not being properly closed after use. Can you provide the full stacktrace?

[anAmateurDeveloper]

(1). Too many open files
"error":{"Error":"com.amazonaws.SdkClientException","Cause":"{"errorMessage":"Unable to execute HTTP request: Too many open files","errorType":"com.amazonaws.SdkClientException","stackTrace":["com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)","com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)","com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)","com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)","com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5219)","com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5165)","com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1502)","com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1356)","com.clarivate.client.aws.s3.S3Client.getFileInputStream(S3Client.java:43)","com.clarivate.util.S3PayloadUtil.getPayload(S3PayloadUtil.java:28)","com.clarivate.content.ingestion.request.AbstractRequestHandler.handleRequest(AbstractRequestHandler.java:66)","sun.reflect.GeneratedMethodAccessor339.invoke(Unknown Source)","sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)","java.lang.reflect.Method.invoke(Method.java:498)"],"cause":{"errorMessage":"Too many open files","errorType":"java.net.SocketException","stackTrace":["java.net.Socket.createImpl(Socket.java:460)","java.net.Socket.getImpl(Socket.java:520)","java.net.Socket.setSoTimeout(Socket.java:1141)","org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:120)","org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)","sun.reflect.GeneratedMethodAccessor186.invoke(Unknown Source)","sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)","java.lang.reflect.Method.invoke(Method.java:498)","com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)","com.amazonaws.http.conn.$Proxy22.connect(Unknown Source)","org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)","org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)","org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)","org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)","org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)","org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)","com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1333)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)","com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)","com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)","com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)","com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5219)","com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5165)","com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1502)","com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1356)","com.clarivate.client.aws.s3.S3Client.getFileInputStream(S3Client.java:43)","com.clarivate.util.S3PayloadUtil.getPayload(S3PayloadUtil.java:28)","com.clarivate.content.ingestion.request.AbstractRequestHandler.handleRequest(AbstractRequestHandler.java:66)","sun.reflect.GeneratedMethodAccessor339.invoke(Unknown Source)","sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)","java.lang.reflect.Method.invoke(Method.java:498)"]}}"}}

(2). Unknown host
"error":{"Error":"com.amazonaws.SdkClientException","Cause":"{"errorMessage":"Unable to execute HTTP request: cl-us-west-2-cc-prod-content-ingestion.s3.us-west-2.amazonaws.com","errorType":"com.amazonaws.SdkClientException","stackTrace":["com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)","com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)","com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)","com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)","com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5219)","com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5165)","com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1502)","com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1356)","com.clarivate.client.aws.s3.S3Client.getFileInputStream(S3Client.java:43)","com.clarivate.util.S3PayloadUtil.getPayload(S3PayloadUtil.java:28)","com.clarivate.content.ingestion.request.AbstractRequestHandler.handleRequest(AbstractRequestHandler.java:66)","sun.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)","sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)","java.lang.reflect.Method.invoke(Method.java:498)"],"cause":{"errorMessage":"cl-us-west-2-cc-prod-content-ingestion.s3.us-west-2.amazonaws.com","errorType":"java.net.UnknownHostException","stackTrace":["java.net.InetAddress.getAllByName0(InetAddress.java:1281)","java.net.InetAddress.getAllByName(InetAddress.java:1193)","java.net.InetAddress.getAllByName(InetAddress.java:1127)","com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)","com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)","org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)","org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)","sun.reflect.GeneratedMethodAccessor246.invoke(Unknown Source)","sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)","java.lang.reflect.Method.invoke(Method.java:498)","com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)","com.amazonaws.http.conn.$Proxy22.connect(Unknown Source)","org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)","org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)","org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)","org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)","org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)","org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)","com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1333)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)","com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)","com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)","com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)","com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)","com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5219)","com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5165)","com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1502)","com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1356)","com.clarivate.client.aws.s3.S3Client.getFileInputStream(S3Client.java:43)","com.clarivate.util.S3PayloadUtil.getPayload(S3PayloadUtil.java:28)","com.clarivate.content.ingestion.request.AbstractRequestHandler.handleRequest(AbstractRequestHandler.java:66)","sun.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)","sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)","java.lang.reflect.Method.invoke(Method.java:498)"]}}"}}

[anAmateurDeveloper]

We doubt "Unknown host" was caused by "Too many open files". And "Too many open files" could also cause java.lang.NoClassDefFoundError

[anAmateurDeveloper]

@debora-ito Are you suggesting I close s3Object in finally block or try-with-resource block? Is that the correct way to close connections?

[debora-ito]

(1) "Too many open files" seems related to the AWS Lambda limit for file descriptors like you mentioned. Closing the S3 object is good, and also double-check if you are closing other resources used in your lambda function like the InputStream. Another thing you can experiment on is to share the same S3 client instance between lambda invocations instead of creating a new client with every invocation - to know more about how this would work check the Lambda runtime lifecycle docs: https://docs.aws.amazon.com/lambda/latest/dg/runtimes-context.html#runtimes-lifecycle

(2) The UnknownHostException can be caused by a DNS caching issue in the JVM, you can get more insights in this thread: #1503 - one of the comments says in their case the issue was caused by running out of file descriptors in ECS.

[anAmateurDeveloper]

Thanks @debora-ito .

1. We don't close other InputStream resources within Lambda function. We may need to close them (Including the inputStream got from S3) as well after using them.
2. We can share s3 client between invocations, but the problem happens at

S3Object s3Object = s3Client.getObject(bucketName, key);

which cannot be shared between invocations because of different bucketName and key. So we still need to close s3Object nicely
3. My last question is why my original s3 is not closed appropriately? Is it because we didn't wrap it within any try-catch-finally block?

Let me know what you think. Thanks

[debora-ito]

3. My last question is why my original s3 is not closed appropriately? Is it because we didn't wrap it within any try-catch-finally block?

Closing resources in a finally block is a good practice in general.

I'll go ahead and close this, as it seems this issue is caused by the Lambda hitting the file descriptors limit. Feel free to reach out if you have further questions.

[github-actions]

COMMENT VISIBILITY WARNING

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.