Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace MinIO S3 with HDFS on Kubernetes Cluster for Fink Database Compatibility #913

Open
fjammes opened this issue Nov 6, 2024 · 0 comments · May be fixed by #930
Open

Replace MinIO S3 with HDFS on Kubernetes Cluster for Fink Database Compatibility #913

fjammes opened this issue Nov 6, 2024 · 0 comments · May be fixed by #930
Assignees
Labels
k8s Kubernetes deployment storage

Comments

@fjammes
Copy link
Contributor

fjammes commented Nov 6, 2024

Goal of the Issue:
The goal of this issue is to replace the current MinIO S3 storage with Hadoop Distributed File System (HDFS) on the Kubernetes (K8s) cluster to ensure compatibility with the Fink database, which relies on HDFS for its data storage and access needs.


Describe the Issue
A short description of what the issue is:
While MinIO S3 is currently operational on the K8s cluster and meets general storage requirements, it needs to be replaced by HDFS to support the Fink database. The Fink database requires HDFS as its primary storage layer to enable seamless data operations and compatibility with its architecture.

Expected Behaviour
What you expected to happen:
HDFS should be deployed on the K8s cluster, providing a compatible and efficient storage layer for the Fink broker.

Actual Behaviour
What actually happened, error messages go here:
MinIO S3 is currently in use for object storage for hte Fink broker, but it does not fully support the HDFS-based architecture of the Fink database. No errors are currently present, but Fink database operations cannot be executed as expected on MinIO.

To Reproduce
Steps to replace MinIO S3 with HDFS:

  1. Deploy HDFS on the Kubernetes cluster, using an appropriate procedure.
  2. Configure access policies for HDFS to ensure smooth integration with the Fink broker.

Additional Context
The replacement of MinIO S3 with HDFS is necessary for Fink database operations, as the database relies on HDFS for optimal data handling and integration. HDFS is expected to offer the required compatibility, scalability, and high-throughput performance for Fink broker workloads on the Kubernetes cluster.

@fjammes fjammes self-assigned this Nov 6, 2024
@fjammes fjammes added filesystem storage k8s Kubernetes deployment and removed filesystem labels Nov 6, 2024
@fjammes fjammes linked a pull request Dec 27, 2024 that will close this issue
@fjammes fjammes linked a pull request Dec 27, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
k8s Kubernetes deployment storage
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant