chore: add more documents

ldclabs · Jul 14, 2024 · 7e7439a · 7e7439a
1 parent d227071
commit 7e7439a
Show file tree

Hide file tree

Showing 25 changed files with 410 additions and 186 deletions.
diff --git a/README.md b/README.md
@@ -1,4 +1,5 @@
 # `ic-oss`
+
 🗂 A decentralized Object Storage Service on the Internet Computer.
 
 💝 This project received a **$25k Developer Grant** from the [DFINITY Foundation](https://dfinity.org/grants).
@@ -9,10 +10,12 @@
 
 In decentralized enterprise applications, `ic-oss` will be an essential infrastructure.
 
-`ic-oss` is a file infrastructure service, not a user-facing file product, but it will provide a simple management interface.
+`ic-oss` is a file infrastructure service, not a user-facing product, but it will provide a simple management interface.
 
 > [!NOTE]
-> `ic-oss` is in development and is not suitable for production use yet.
+> The main functions of `ic-oss` have been developed, and the cluster management function is still under development (which will be completed soon). It can be used in the production environment.
+
+![IC-OSS](./ic-oss.webp)
 
 ## Features
 
@@ -25,107 +28,27 @@ In decentralized enterprise applications, `ic-oss` will be an essential infrastr
 - [ ] Implements file encryption storage using ICP's vetKeys mechanism.
 - [ ] Integrates with external storage, supporting file storage in decentralized file services like IPFS and Arweave, with `ic-oss` managing file metadata.
 
-## Running the project locally
-
-If you want to test your project locally, you can use the following commands:
-
-Deploy the bucket canister:
-```bash
-dfx canister create --specified-id mmrxu-fqaaa-aaaap-ahhna-cai ic_oss_bucket
-
-dfx deploy ic_oss_bucket --argument "(opt variant {Init =
-  record {
-    name = \"LDC Labs\";
-    file_id = 0;
-    max_file_size = 0;
-    max_folder_depth = 10;
-    max_children = 1000;
-    visibility = 0;
-    max_custom_data_size = 4096;
-    enable_hash_index = true;
-  }
-})"
-# Output:
-# ...
-# Installing code for canister ic_oss_bucket, with canister ID mmrxu-fqaaa-aaaap-ahhna-cai
-# Deployed canisters.
-# URLs:
-#   Backend canister via Candid interface:
-#     ic_oss_bucket: http://127.0.0.1:4943/?canisterId=bd3sg-teaaa-aaaaa-qaaba-cai&id=mmrxu-fqaaa-aaaap-ahhna-cai
-```
-
-Build cli tool:
-```bash
-cargo build -p ic-oss-cli
-
-# Run the cli tool
-./target/debug/ic-oss-cli --help
-./target/debug/ic-oss-cli identity --help
-./target/debug/ic-oss-cli upload --help
-
-# Generate a new identity
-./target/debug/ic-oss-cli identity --new --file myid.pem
-# Output:
-# principal: lxph3-nvpsv-yrevd-im4ug-qywcl-5ir34-rpsbs-6olvf-qtugo-iy5ai-jqe
-# new identity: myid.pem
-```
-
-Add a manager to the bucket canister:
-```bash
-dfx canister call mmrxu-fqaaa-aaaap-ahhna-cai admin_set_managers '(vec {principal "lxph3-nvpsv-yrevd-im4ug-qywcl-5ir34-rpsbs-6olvf-qtugo-iy5ai-jqe"})'
-```
-
-Upload a file to the bucket canister:
-```bash
-./target/debug/ic-oss-cli -i myid.pem upload -b mmrxu-fqaaa-aaaap-ahhna-cai --file test.tar.gz
-# Output:
-# ...
-# 2024-05-18 18:42:38 uploaded: 99.48%
-# 2024-05-18 18:42:38 uploaded: 99.66%
-# 2024-05-18 18:42:38 uploaded: 99.82%
-# 2024-05-18 18:42:38 uploaded: 100.00%
-# upload success, file id: 1, size: 147832281, chunks: 564, retry: 0, time elapsed: PT69.149941S
-```
-
-List files:
-```bash
-dfx canister call ic_oss_bucket list_files '(0, null, null, null)'
-```
-
-Get file info:
-```bash
-dfx canister call ic_oss_bucket get_file_info '(1, null)'
-# Output:
-# (
-#   variant {
-#     Ok = record {
-#       id = 1 : nat32;
-#       parent = 0 : nat32;
-#       status = 0 : int8;
-#       updated_at = 1_716_028_957_265 : nat;
-#       hash = opt blob "\b7\bb\90\40\d6\44\79\a7\ca\56\c8\e0\3a\e2\da\dd\c8\19\85\9f\7b\85\84\88\c0\b9\98\ee\de\d6\de\de";
-#       name = "test.tar.gz";
-#       size = 147_832_281 : nat;
-#       content_type = "application/gzip";
-#       created_at = 1_716_028_890_649 : nat;
-#       filled = 147_832_281 : nat;
-#       chunks = 564 : nat32;
-#       ert = null;
-#     }
-#   },
-# )
-
-dfx canister call ic_oss_bucket get_file_info_by_hash '(blob "\b7\bb\90\40\d6\44\79\a7\ca\56\c8\e0\3a\e2\da\dd\c8\19\85\9f\7b\85\84\88\c0\b9\98\ee\de\d6\de\de", null)'
-```
-
-Delete file:
-```bash
-dfx canister call ic_oss_bucket delete_file '(1, null)'
-```
-
-Download the file in browser:
-- by file id: `http://mmrxu-fqaaa-aaaap-ahhna-cai.localhost:4943/f/1`
-- by file hash:  `http://mmrxu-fqaaa-aaaap-ahhna-cai.localhost:4943/h/b7bb9040d64479a7ca56c8e03ae2daddc819859f7b858488c0b998eeded6dede`
+## Libraries
+
+| Library                                                                                  | Description                                                                                                              |
+| :--------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------- |
+| [ic_oss_bucket](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_bucket)           | An ICP smart contract and a storage bucket in the ic-oss cluster for storing files and folders.                          |
+| [ic_oss_cluster](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_cluster)         | An ICP smart contract and the manager of the ic-oss cluster.                                                             |
+| [ic-oss-can](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_can)                 | A Rust library for implementing large file storage in ICP canisters.                                                     |
+| [ic-oss-types](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_types)             | A Rust types library used for integrating with ic-oss cluster.                                                           |
+| [ic-oss-cose](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_cose)               | A Rust library based on COSE (RFC9052) and CWT (RFC8392) for issuing and verifying access tokens for the ic-oss cluster. |
+| [ic-oss](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss)                         | The Rust version of the client SDK for the ic-oss cluster.                                                               |
+| [ic-oss-cli](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_cli)                 | A command-line tool implemented in Rust for the ic-oss cluster.                                                          |
+| [examples/ai_canister](https://github.com/ldclabs/ic-oss/tree/main/examples/ai_canister) | A demonstration project used to show how to implement large file storage in the ICP canister by using `ic-oss-can`.      |
+
+## Integration Workflow
+
+![IC-OSS Sequence](./ic-oss-sequence.webp)
+
+How to integrate `ic-oss`:
+1. The backend of the Dapp calls the API of `ic_oss_cluster` to add access control policies for the user.
+2. The frontend of the Dapp uses the `ic-oss-ts` SDK to obtain the `access_token` of the target `ic_oss_bucket` from `ic_oss_cluster`.
+3. The frontend of the Dapp uses the `access_token` to call the API of the target `ic_oss_bucket` to operate the authorized files and folders.
 
 ## License
 Copyright © 2024 [LDC Labs](https://github.com/ldclabs).

diff --git a/examples/ai_canister/README.md b/examples/ai_canister/README.md
@@ -1,5 +1,11 @@
 # Example: `ai_canister`
 
+[ic-oss](https://github.com/ldclabs/ic-oss) is a decentralized Object Storage Service on the Internet Computer.
+
+`ai_canister` is a demonstration project used to show how to implement large file storage in the ICP canister. By using `ic-oss-can` to include the `ic_oss_fs!` macro in your canister, an `fs` module and a set of Candid file system APIs will be automatically generated. You can use the `ic-oss-cli` tool to upload files to the ICP canister.
+
+For more information about `ic-oss-can`, please refer to [ic-oss-can](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_can).
+
 ## Running the project locally
 
 If you want to test your project locally, you can use the following commands:
@@ -39,4 +45,4 @@ dfx canister call ai_canister list_files '(0, null, null, null)'
 ## License
 Copyright © 2024 [LDC Labs](https://github.com/ldclabs).
 
-`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](../../LICENSE-MIT) for the full license text.
+`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](../../LICENSE-MIT) for the full license text.
diff --git a/ic-oss-sequence.webp b/ic-oss-sequence.webp
diff --git a/ic-oss.webp b/ic-oss.webp
diff --git a/src/ic_oss/Cargo.toml b/src/ic_oss/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "ic-oss"
-description = "A rust client of ic-oss."
+description = "The Rust version of the client SDK for the ic-oss cluster."
 publish = true
 repository = "https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss"
 version.workspace = true

diff --git a/src/ic_oss/README.md b/src/ic_oss/README.md
@@ -1,15 +1,15 @@
 # `ic-oss`
-
 ![License](https://img.shields.io/crates/l/ic-oss.svg)
 [![Crates.io](https://img.shields.io/crates/d/ic-oss.svg)](https://crates.io/crates/ic-oss)
-[![CI](https://github.com/ldclabs/ic-oss/actions/workflows/ci.yml/badge.svg)](https://github.com/ldclabs/ic-oss/actions/workflows/ci.yml)
+[![Test](https://github.com/ldclabs/ic-oss/actions/workflows/test.yml/badge.svg)](https://github.com/ldclabs/ic-oss/actions/workflows/test.yml)
 [![Docs.rs](https://img.shields.io/docsrs/ic-oss?label=docs.rs)](https://docs.rs/ic-oss)
 [![Latest Version](https://img.shields.io/crates/v/ic-oss.svg)](https://crates.io/crates/ic-oss)
 
-A rust client of [ic-oss](https://github.com/ldclabs/ic-oss).
-`ic-oss` is a fully open-source decentralized object storage service running on the Internet Computer.
+[ic-oss](https://github.com/ldclabs/ic-oss) is a decentralized Object Storage Service on the Internet Computer.
+
+`ic-oss` is the Rust version of the client SDK for the ic-oss cluster.
 
 ## License
 Copyright © 2024 [LDC Labs](https://github.com/ldclabs).
 
-`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](LICENSE) for the full license text.
+`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](../../LICENSE-MIT) for the full license text.
diff --git a/src/ic_oss_bucket/Cargo.toml b/src/ic_oss_bucket/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "ic_oss_bucket"
-description = "A bucket canister of ic-oss"
+description = "An ICP smart contract and a storage bucket in the ic-oss cluster for storing files and folders."
 publish = false
 repository = "https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_bucket"
 version.workspace = true

diff --git a/src/ic_oss_bucket/README.md b/src/ic_oss_bucket/README.md
@@ -1,9 +1,48 @@
 # `ic_oss_bucket`
 
-[![CI](https://github.com/ldclabs/ic-oss/actions/workflows/ci.yml/badge.svg)](https://github.com/ldclabs/ic-oss/actions/workflows/ci.yml)
+[ic-oss](https://github.com/ldclabs/ic-oss) is a decentralized Object Storage Service on the Internet Computer.
+
+`ic_oss_bucket` is an ICP smart contract and a storage bucket in the `ic-oss` cluster for storing files and folders. The `ic-oss` cluster can deploy one or more buckets for horizontal file storage scaling, managed by `ic_oss_cluster`.
+
+## Features
+
+- [x] Supports large file uploads and downloads through file sharding, concurrent high-speed uploads, resumable uploads, and segmented downloads.
+- [x] Provides data verification based on ICP's verification mechanisms to ensure file integrity during reading.
+- [x] Supports file directory tree.
+- [x] Access control with permissions for public, private, read-only, and write-only for files, folders, and buckets.
+
+## Candid API
+
+```shell
+admin_set_auditors : (vec principal) -> (Result);
+admin_set_managers : (vec principal) -> (Result);
+admin_update_bucket : (UpdateBucketInput) -> (Result);
+batch_delete_subfiles : (nat32, vec nat32, opt blob) -> (Result_1);
+create_file : (CreateFileInput, opt blob) -> (Result_2);
+create_folder : (CreateFolderInput, opt blob) -> (Result_2);
+delete_file : (nat32, opt blob) -> (Result_3);
+delete_folder : (nat32, opt blob) -> (Result_3);
+get_bucket_info : (opt blob) -> (Result_4) query;
+get_file_ancestors : (nat32, opt blob) -> (Result_5) query;
+get_file_chunks : (nat32, nat32, opt nat32, opt blob) -> (Result_6) query;
+get_file_info : (nat32, opt blob) -> (Result_7) query;
+get_file_info_by_hash : (blob, opt blob) -> (Result_7) query;
+get_folder_ancestors : (nat32, opt blob) -> (Result_5) query;
+get_folder_info : (nat32, opt blob) -> (Result_8) query;
+http_request : (HttpRequest) -> (HttpStreamingResponse) query;
+list_files : (nat32, opt nat32, opt nat32, opt blob) -> (Result_9) query;
+list_folders : (nat32, opt blob) -> (Result_10) query;
+move_file : (MoveInput, opt blob) -> (Result_11);
+move_folder : (MoveInput, opt blob) -> (Result_11);
+update_file_chunk : (UpdateFileChunkInput, opt blob) -> (Result_12);
+update_file_info : (UpdateFileInput, opt blob) -> (Result_11);
+update_folder_info : (UpdateFolderInput, opt blob) -> (Result_11);
+validate_admin_set_auditors : (vec principal) -> (Result) query;
+validate_admin_set_managers : (vec principal) -> (Result) query;
+validate_admin_update_bucket : (UpdateBucketInput) -> (Result) query;
+```
 
-A bucket canister of [ic-oss](https://github.com/ldclabs/ic-oss).
-`ic-oss` is a fully open-source decentralized object storage service running on the Internet Computer.
+The complete Candid API definition can be found in the [ic_oss_bucket.did](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_bucket/ic_oss_bucket.did) file.
 
 ## Running locally
 
@@ -28,8 +67,10 @@ MYID=$(dfx identity get-principal)
 ic-oss-cli -i debug/uploader.pem identity
 # principal: nprym-ylvyz-ig3fr-lgcmn-zzzt4-tyuix-3v6bm-fsel7-6lq6x-zh2w7-zqe
 
+# add managers
 dfx canister call ic_oss_bucket admin_set_managers "(vec {principal \"$MYID\"; principal \"nprym-ylvyz-ig3fr-lgcmn-zzzt4-tyuix-3v6bm-fsel7-6lq6x-zh2w7-zqe\"})"
 
+# add public keys to verify the access tokens
 dfx canister call ic_oss_bucket admin_update_bucket '(record {
   name = null;
   max_file_size = null;
@@ -43,18 +84,25 @@ dfx canister call ic_oss_bucket admin_update_bucket '(record {
   trusted_eddsa_pub_keys = opt vec {vec {19; 152; 246; 44; 109; 26; 69; 124; 81; 186; 106; 75; 95; 61; 189; 47; 105; 252; 169; 50; 22; 33; 141; 200; 153; 126; 65; 107; 209; 125; 147; 202}};
 }, null)'
 
+# list files in the folder 2, with a access token
 dfx canister call ic_oss_bucket list_files '(2, null, null, opt blob "\84\44\a1\01\38\2e\a0\58\ac\a7\01\78\1b\61\6a\75\71\34\2d\72\75\61\61\61\2d\61\61\61\61\61\2d\71\61\61\67\61\2d\63\61\69\02\78\3f\7a\37\77\6a\70\2d\76\36\66\65\33\2d\6b\6b\73\75\35\2d\32\36\66\36\34\2d\64\65\64\74\77\2d\6a\37\6e\64\6a\2d\35\37\6f\6e\78\2d\71\67\61\36\63\2d\65\74\35\65\33\2d\6e\6a\78\35\33\2d\74\61\65\03\78\1b\6d\6d\72\78\75\2d\66\71\61\61\61\2d\61\61\61\61\70\2d\61\68\68\6e\61\2d\63\61\69\04\1a\66\8f\ce\68\05\1a\66\8f\c0\58\06\1a\66\8f\c0\58\09\78\18\46\6f\6c\64\65\72\2e\2a\3a\31\20\42\75\63\6b\65\74\2e\52\65\61\64\2e\2a\58\40\52\66\3e\e7\55\7e\99\2c\66\6d\65\56\54\9f\30\a1\2e\aa\56\69\66\b6\c6\e9\75\d7\c9\02\4c\24\1d\5d\7e\83\7d\c1\13\c6\00\91\56\d9\6a\ae\34\c3\a5\c9\b4\99\b3\47\b7\68\54\8d\dd\9c\9a\9b\a0\f9\1a\f5")'
+
+# list folders in the root folder 0
 dfx canister call ic_oss_bucket list_folders '(0, null)'
 
+# upload a file to the bucket
 ic-oss-cli -i debug/uploader.pem upload -b mmrxu-fqaaa-aaaap-ahhna-cai --file README.md
 
+# read the file info
 dfx canister call ic_oss_bucket get_file_info '(1, null)'
 
+# update the file 1's status to 0
 dfx canister call ic_oss_bucket update_file_info "(record {
   id = 1;
   status = opt 0;
 }, null)"
 
+# create a folder in the root folder
 dfx canister call ic_oss_bucket create_folder "(record {
   parent = 0;
   name = \"home\";
@@ -66,6 +114,7 @@ dfx canister call ic_oss_bucket create_folder "(record {
   name = \"jarvis\";
 }, null)"
 
+# move the file 1 from the root folder 0 to the folder 2
 dfx canister call ic_oss_bucket move_file "(record {
   id = 1;
   from = 0;
@@ -74,6 +123,7 @@ dfx canister call ic_oss_bucket move_file "(record {
 
 dfx canister call ic_oss_bucket list_files '(2, null, null, null)'
 
+# delete the file 1
 dfx canister call ic_oss_bucket delete_file '(1, null)'
 ```
 
@@ -85,4 +135,4 @@ Download the file in browser:
 ## License
 Copyright © 2024 [LDC Labs](https://github.com/ldclabs).
 
-`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](LICENSE) for the full license text.
+`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](../../LICENSE-MIT) for the full license text.
diff --git a/src/ic_oss_bucket/src/api_init.rs b/src/ic_oss_bucket/src/api_init.rs
@@ -12,14 +12,14 @@ pub enum CanisterArgs {
 
 #[derive(Clone, Debug, CandidType, Deserialize)]
 pub struct InitArgs {
-    name: String,
-    file_id: u32,
-    max_file_size: u64,
-    max_folder_depth: u8,
-    max_children: u16,
-    max_custom_data_size: u16,
-    enable_hash_index: bool,
-    visibility: u8, // 0: private; 1: public
+    name: String,              // bucket name
+    file_id: u32,              // the first file id, default is 0
+    max_file_size: u64,        // in bytes, default is 384GB
+    max_folder_depth: u8,      // default is 10
+    max_children: u16, //  maximum number of subfolders and subfiles in a folder., default is 1000
+    max_custom_data_size: u16, // in bytes, default is 4KB
+    enable_hash_index: bool, // if enabled, indexing will be built using file hash, allowing files to be read by their hash and preventing duplicate hash for files. default is false
+    visibility: u8,          // 0: private; 1: public, can be accessed by anyone, default is 0
 }
 
 #[derive(Clone, Debug, CandidType, Deserialize)]
@@ -97,6 +97,10 @@ fn init(args: Option<CanisterArgs>) {
                     args.max_custom_data_size
                 };
                 b.enable_hash_index = args.enable_hash_index;
+
+                // The root folder 0 is created by default
+                b.folder_id = 1;
+                b.folder_count = 1;
             });
         }
         CanisterArgs::Upgrade(_) => {

diff --git a/src/ic_oss_bucket/src/api_query.rs b/src/ic_oss_bucket/src/api_query.rs
@@ -160,7 +160,7 @@ fn list_files(
     take: Option<u32>,
     access_token: Option<ByteBuf>,
 ) -> Result<Vec<FileInfo>, String> {
-    let max_prev = store::state::with(|s| s.file_id).saturating_add(1);
+    let max_prev = store::state::with(|s| s.file_id);
     let prev = prev.unwrap_or(max_prev).min(max_prev);
     let take = take.unwrap_or(10).min(100);
     let canister = ic_cdk::id();

diff --git a/src/ic_oss_bucket/src/store.rs b/src/ic_oss_bucket/src/store.rs
@@ -815,7 +815,7 @@ pub mod fs {
     pub fn add_folder(metadata: FolderMetadata) -> Result<u32, String> {
         state::with_mut(|s| {
             FOLDERS.with(|r| {
-                let id = s.folder_id.saturating_add(1);
+                let id = s.folder_id;
                 if id == u32::MAX {
                     Err("folder id overflow".to_string())?;
                 }
@@ -828,7 +828,7 @@ pub mod fs {
                     s.max_children as usize,
                 )?;
 
-                s.folder_id = id;
+                s.folder_id = s.folder_id.saturating_add(1);
                 s.folder_count += 1;
                 Ok(id)
             })
@@ -838,7 +838,7 @@ pub mod fs {
     pub fn add_file(metadata: FileMetadata) -> Result<u32, String> {
         state::with_mut(|s| {
             FOLDERS.with(|r| {
-                let id = s.file_id.saturating_add(1);
+                let id = s.file_id;
                 if id == u32::MAX {
                     Err("file id overflow".to_string())?;
                 }
@@ -860,7 +860,7 @@ pub mod fs {
                     }
                 }
 
-                s.file_id = id;
+                s.file_id = s.file_id.saturating_add(1);
                 s.file_count += 1;
                 parent.files.insert(id);
                 FS_METADATA_STORE.with(|r| r.borrow_mut().insert(id, metadata));