Close google/azure/s3 client when closing GoogleTransfer object [DDB-1160] #187
Conversation
kmichel-aiven
changed the title
| @@ -82,6 +82,9 @@ def __init__( | |||
| self.notifier = notifier or NullNotifier() | |||
| self.stats = StatsClient(statsd_info) | |||
|
|
|||
| def close(self) -> None: | |||
There was a problem hiding this comment.
I do not understand who is calling this close call?
I could not find api in this repo which will enforce the close call.
Should the Transfer object "user" remember to close the object after every usage?
There was a problem hiding this comment.
Otherwise a client pool may be used. Or any other simple way which makes it easy to use and reliable
There was a problem hiding this comment.
Close is part of the API for this class. A docstring might help here.
There was a problem hiding this comment.
Files, sockets, etc... need to be closed to reliably not have leaks, the same is true for APIs that use them.
I'll add a docstring to make it clear that this is part of the API.
rohmu/object_storage/google.py
Outdated
| @@ -188,10 +188,18 @@ def __init__( | |||
| self.proxy_info = proxy_info | |||
| self.google_creds = get_credentials(credential_file=credential_file, credentials=credentials) | |||
| self.gs: Optional[Resource] = self._init_google_client() | |||
| self.gs_object_client = None | |||
| self.gs_object_client: Optional[Any] = None | |||
There was a problem hiding this comment.
nit:
| self.gs_object_client: Optional[Any] = None | |
| self.gs_object_client: Any = None |
[DDB-1160]
kmichel-aiven
force-pushed
the
kmichel-close
branch
from
5c64b5b to
7ef096d
Compare
kmichel-aiven
changed the title
About this change - What it does
GoogleTransfer objects create clients and never close the client, if we create many clients, we leak sockets.
Add a API to close a Transfer object and implement it in the GoogleTransfer subclass to close the google client.
Also implement it in Azure/S3 transfer objects for consistency (and hopefully one day we'll stop creating buckets during
__init__, which will make the objects creation cheaper and free of side-effects.)Why this way
Because if we leak socket, then we eventually crash.