GSoC Ideas: ofs, OpenDAL File System via FUSE

[Xuanwo]

Background

OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way. ofs can expose OpenDAL power in fuse way that allow users to mount storage services locally.

Objectives

Implement ofs, allowing users to mount storage services locally for read and write.

Features

In Scope:

• Continuous reading
• Continuous writing
• Random reading
• List dir
• Stat file

Out Scope:

• Random Write
• Xattrs
• Permissions

Tasks

• Implement features that in scope
• Implement tests suite

Recommended Skills

• Familiar with Rust
• Familiar with basic ideas of file system and fuse
• Familiar with OpenDAL Rust Core

Mentor

Mailing List: [email protected]
Mentor: junouyang, Apache OpenDAL PMC Member, [email protected]
Please leave comments if you want to be a mentor

---

Cross posted at https://issues.apache.org/jira/browse/GSOC-255

[oowl]

Can I mentor this GSOC project? And we have done some work for it before, but I am not sure if this project for GSOC is reasonable.

[bokket]

I interested in.Whether we discuss it before, we do the linux fuse or fuse3 first, and then consider the S3 support? @oowl

[oowl]

we do the linux fuse or fuse3 first, and then consider the S3 support? @oowl

we do not need to implement all these fuse libraries, just choose one of them.

I interested in

Welcome, but we need to wait for the nomination process of GSOC.

[bokket]

Welcome, but we need to wait for the nomination process of Gsoc.

The website shows that the organizations are announced on February 21st.

[simonsan]

Will ofs (opendal-fs?) be a library or a CLI tool?

[Xuanwo]

Will ofs (opendal-fs?) be a library or a CLI tool?

I expect to have both lib and cli, but that depends on the code owner's decision.

[simonsan]

Will ofs (opendal-fs?) be a library or a CLI tool?

I expect to have both lib and cli, but that depends on the code owner's decision.

Ok, I think it is a good way to separate these concerns for modularity and would be also helpful to link against it, when other projects, want to pull in that code. Not affiliated with it in any way, but I find it good, how abscissa applies that when initializing a new project, it's a good way to quickly start. So just leaving it here.

[Zheaoli]

I'm familiar with the fuse before. I would like to be one of the mentor for this idea

[ho-229]

I think we should update the goals of this idea, the goals listed above seems be implemented in #4179

[Xuanwo]

I think we should update the goals of this idea, the goals listed above seems be implemented in #4179

Agreed, I will update the goals.

[xiaguan]

It seems to be a local mapping of cloud storage. It looks like switching out Datenlord's metadata engine, etcd, for a local SQLite should do the trick.

Rather than the data layer stuff, I'm more intersted about some decisions regarding metadata

[xiaguan]

Based on the existing ofs code, it seems. This idea would require implementing a reader and writer to handle prefetch and concurrent asynchronous write-backs, while also managing a certain amount of memory usage. There are also some things to consider on the metadata level.
It looks like my previous page_cache repo might be of some help.

[bokket]

we discussed in discord.but not clear.

[ho-229]

I want to support Windows by implementing CloudFilter for ofs, I will write proposal later