IAM auth for source connector doesn't work

[mikekamornikov]

I'm using IAM role with proper policy to authenticate S3 requests. Everything works fine for sink connector. While source connector fails with the following error:

[2020-03-23 12:57:46,588] WARN Retryable error while polling. Will sleep and try again. (com.spredfast.kafka.connect.s3.source.S3SourceTask)
com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 033D3E15AA0DEF9C; S3 Extended Request ID: lsgEMCZYu2ZY1vQhCQpqc3gZuZ4t35qD0N3W4F90f77STA2n2zq4OfHiB/bb7+ASNiMzh02JPfI=), S3 Extended Request ID: lsgEMCZYu2ZY1vQhCQpqc3gZuZ4t35qD0N3W4F90f77STA2n2zq4OfHiB/bb7+ASNiMzh02JPfI=
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1799)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1383)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1359)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5052)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4998)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4992)
	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:895)
	at com.spredfast.kafka.connect.s3.source.S3FilesReader$1.nextObject(S3FilesReader.java:141)
	at com.spredfast.kafka.connect.s3.source.S3FilesReader$1.hasNext(S3FilesReader.java:266)
	at com.spredfast.kafka.connect.s3.source.S3SourceTask.getSourceRecords(S3SourceTask.java:169)
	at com.spredfast.kafka.connect.s3.source.S3SourceTask.poll(S3SourceTask.java:153)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:265)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:232)
	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

TODO:

• bump AWS SDK minor version to the latest available
• rewrite S3 listObjects request using V2 API
• test in real cluster