Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better HDFS support in CI #17604

Open
wcy-fdu opened this issue Jul 8, 2024 · 1 comment
Open

Better HDFS support in CI #17604

wcy-fdu opened this issue Jul 8, 2024 · 1 comment

Comments

@wcy-fdu
Copy link
Contributor

wcy-fdu commented Jul 8, 2024

We have been spotted and fixed several bugs and misses in the codes, Dockerfile.hdfs and docker-hdfs.sh. I suspect there will be more. The root cause is because we rely on manual effort to maintain the HDFS related codes and it is hard for developer to remember to update the hdfs related parts during daily development. For example, I saw that the nvm installation is missing in `Dockerfile.hdfs.
Given that we saw more and more requests on HDFS build, I start thinking about the following things:

  1. Shall we have a CI, at least per release, for the hdfs feature?
  2. Based on 1, should we publish a HDFS build image as well for each release?
  3. Should we unify the dockerfile and the build script for HDFS and non-HDFS build given that they share a lot of things in common?

The following two solutions await more investigation:

  • uncomment following lines and find a way to make cargo clippy more friendly with hdfs dependence.
[features]
hdfs-backend = ["opendal/services-hdfs", "dep:risingwave_jni_core"]
  • Rewrite Dockerfile.hdfs incrementally, or package hdfs from an RisingWave image as base image.
  • set up automatic pipeline to package hdfs per release.
@wcy-fdu wcy-fdu self-assigned this Jul 8, 2024
@github-actions github-actions bot added this to the release-1.10 milestone Jul 8, 2024
@wcy-fdu wcy-fdu removed this from the release-1.10 milestone Jul 8, 2024
Copy link
Contributor

github-actions bot commented Sep 7, 2024

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean.
Don't worry if you think the issue is still valuable to continue in the future.
It's searchable and can be reopened when it's time. 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant