Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add minio client helper #25

Merged
merged 1 commit into from
Jun 6, 2024
Merged

add minio client helper #25

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions src/minio_utils/minio_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
"""
This module contains utility functions to interact with Minio.
"""

import os

from minio import Minio


def get_minio_client() -> Minio:
"""
Helper function to get the Minio client.

:return: A Minio client object
"""
return Minio(

Check warning on line 16 in src/minio_utils/minio_utils.py

View check run for this annotation

Codecov / codecov/patch

src/minio_utils/minio_utils.py#L16 

Added line #L16 was not covered by tests
os.environ['MINIO_URL'].replace("http://", ""),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here I'm wondering why we're just not excluding http:// in the env var in rancher. Is there a reason for that?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tested removing http:// from docker-compose file. When I tried to read delta files, it took 10 minutes without returning anything. I am guessing it's needed for spark custom s3 endpoint config.

access_key=os.environ['MINIO_ACCESS_KEY'],
secret_key=os.environ['MINIO_SECRET_KEY'],
secure=False
)
1 change: 1 addition & 0 deletions src/notebook/startup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@

from spark.utils import get_spark_session
from db_ops.spark_db_utils import create_namespace_if_not_exists, table_exists, remove_table
from minio_utils.minio_utils import get_minio_client
1 change: 1 addition & 0 deletions test/src/minio_utils/minio_utils_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
from src.minio_utils.minio_utils import *
Loading