New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add AWS S3 Support #15

Open

krayzpipes opened this issue Jun 4, 2019 · 1 comment

Assignees

Labels

Owner

krayzpipes commented Jun 4, 2019

A helpful feature would be to pass in an S3 Bucket Object and then scan that object.. even if GZipped.

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html?highlight=streamingbody#S3.Object.get

Getting an S3 object provides a body which is a 'StreamingBody()'... so it may be possible to stream S3 files and work magic as it comes in.

I haven't tested the standard lib 'gzip' for this, but could probably write a wrapper of sorts if needed.

krayzpipes added the enhancement label

krayzpipes self-assigned this

Owner Author

krayzpipes commented Aug 6, 2019

https://botocore.amazonaws.com/v1/documentation/api/latest/reference/response.html#botocore.response.StreamingBody.iter_lines

StreamingBody object has iter_lines capability.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment