Implements basic local and remote sources support (#17)

* feat: support local directories

* feat(example): add support for local compressed file in vorpal example

- Updated `example/rust/vorpal.rs` to include a new example for building from a local compressed file.
- Changed the name of the existing example from "example-source" to "example-local-directory".
- Modified `src/service/proxy/package/mod.rs` to simplify the call to `store::unpack_source` by removing unnecessary conversion to `PathBuf`.

* refactor(example): rename package and remove local compressed file example

Renamed the package from "example-local-directory" to "example-rust" for better clarity. Removed the example of a local compressed file package to streamline the code and reduce redundancy.

* feat: add new dependencies and update package preparation and build process

- Added new dependencies in Cargo.toml and Cargo.lock including `futures-util`, `git2`, `reqwest`, `tokio-stream`, `url`, and others.
- Updated `PackageService` to handle streaming `PrepareRequest`.
- Modified `prepare` function to handle different source types (Git, HTTP, Local) and prepare the source accordingly.
- Implemented `prepare_source` function to sign and send source data in chunks.
- Updated `build` function to handle new source preparation and build process.
- Renamed `vorpal.rs` to `build.rs` and updated its content to reflect new build process.
- Updated `flake.nix` to include new dependencies and build inputs.
- Refactored store module functions to use consistent naming and paths.
- Added new functions in notary module for signing data.
- Updated service modules to use new store path functions and handle new build process.
- Improved error handling and logging throughout the codebase.

* refactor(service): simplify error handling in build preparation

Simplified the error handling in the build preparation module by using the `map_err` method for more concise and readable code. Removed redundant error logging statements. This change improves code maintainability and readability.

* feat: add rust-overlay and cargo-udeps to flake configuration

- Added `rust-overlay` input to `flake.nix` and `flake.lock`.
- Included `cargo-udeps` in the list of inherited packages.
- Updated `devShells` to include `cargo-udeps` in `nativeBuildInputs`.
- Configured `rust-overlay` to follow `nixpkgs`.
- Added `cargo-nightly` application to `process-compose` configuration.

* refactor: rename build.rs to vorpal.rs and update references

- Renamed `build.rs` to `vorpal.rs` in `example/rust` directory.
- Updated `Cargo.toml` to reflect the new binary name and path.
- Modified `flake.nix` to include `clippy` and `rustfmt` in `nativeBuildInputs`.
- Replaced `.clone()` with dereferencing in `src/command/mod.rs`.
- Simplified function calls by removing unnecessary references in `src/notary/mod.rs`, `src/service/build/build/mod.rs`, `src/service/build/prepare/mod.rs`, `src/service/proxy/package/mod.rs`, and `src/store/mod.rs`.
- Improved code readability by using shorthand syntax for increment operations and string concatenations.

* refactor: rename modules and update build script

- Renamed `build/mod.rs` to `run_build.rs`
- Renamed `prepare/mod.rs` to `run_prepare.rs`
- Renamed `build/sandbox_default.rs` to `sandbox_default.rs`
- Updated `service.rs` to use new module names
- Simplified build script generation in `run_build.rs`
- Removed redundant error logging in `run_build.rs`
- Changed function parameters from `String` to `&str` in `proxy/package/mod.rs` and `store/mod.rs`
- Added unit tests for store directory paths in `store/mod.rs`
- Updated `flake.nix` to include clippy checks and tests in the build phase and removed clippy from devShells.

* refactor: update flake.nix and run_prepare.rs for improved build process

- Updated `checkPhase` in `flake.nix` to include `cargo fmt --check --verbose` and modified `cargo test` command to use `--locked --all-features --all-targets`.
- Added `rustfmt` to `nativeBuildInputs` in `flake.nix`.
- Removed `rustfmt` from `devShells.default.nativeBuildInputs` in `flake.nix`.
- Removed `cargo-nightly` app configuration from `apps` in `flake.nix`.
- Simplified the creation of `PrepareResponse` in `run_prepare.rs`.

* chore: update dependencies

- Bump `addr2line` from 0.21.0 to 0.22.0
- Bump `anstyle-query` from 1.0.3 to 1.1.0
- Bump `hyper` from 0.14.28 to 0.14.29
- Bump `backtrace` from 0.3.71 to 0.3.72
- Bump `cc` from 1.0.98 to 1.0.99
- Bump `clap` from 4.5.4 to 4.5.6
- Bump `clap_builder` from 4.5.2 to 4.5.6
- Bump `clap_derive` from 4.5.4 to 4.5.5
- Bump `clap_lex` from 0.7.0 to 0.7.1
- Bump `gimli` from 0.28.1 to 0.29.0
- Bump `object` from 0.32.2 to 0.35.0
- Bump `proc-macro2` from 1.0.84 to 1.0.85
- Bump `tar` from 0.4.40 to 0.4.41
- Bump `tokio` from 1.37.0 to 1.38.0
- Bump `tokio-macros` from 2.2.0 to 2.3.0
- Bump `utf8parse` from 0.2.1 to 0.2.2

Remove `cargo-udeps` from `flake.nix` and update `cargoSha256`.

Cargo.toml

@@ -12,17 +12,23 @@ anyhow = "1.0"
 clap = { version = "4.5", features = ["derive"] }
 dirs = "5.0"
 flate2 = "1.0"
+futures-util = "0.3"
+git2 = "0.18"
 hex = "0.4"
+infer = "0.3"
 prost = "0.12"
 rand = "0.8"
 rsa = { version = "0.9", features = ["sha2"] }
 rusqlite = { version = "0.31", features = ["bundled"] }
 sha256 = "1.5"
+reqwest = { version = "0.12", features = ["json"] }
 tar = "0.4"
 tempfile = "3.10"
 tera = { version = "1", default-features = false }
-tokio = { version = "1.37", features = ["macros", "process", "rt-multi-thread"] }
+tokio = { version = "1.37", features = ["full"] }
+tokio-stream = "0.1"
 tonic = "0.11"
+url = "2.5"
 uuid = { version = "1.8", features = ["v7"] }
 walkdir = "2.5"
 

api/v0/build/build.proto

@@ -6,6 +6,20 @@ service BuildService {
     rpc Package (PackageRequest) returns (PackageResponse);
 }
 
+enum PackageSourceKind {
+    UNKNOWN = 0;
+    LOCAL = 1;
+    HTTP = 2;
+    GIT = 3;
+}
+
+message PackageSource {
+    PackageSourceKind kind = 1;
+    optional string hash = 2;
+    repeated string ignore_paths = 3;
+    string uri = 4;
+}
+
 message PackageResponse {
     string source_id = 1;
     string source_hash = 2;
@@ -14,9 +28,8 @@ message PackageResponse {
 message PackageRequest {
     repeated PackageResponse build_deps = 1;
     repeated PackageResponse install_deps = 2;
-    repeated string ignore_paths = 3;
     string build_phase = 4;
     string install_phase = 5;
     string name = 6;
-    string source = 7;
+    PackageSource source = 7;
 }

api/v0/package/package.proto

@@ -3,7 +3,7 @@ syntax = "proto3";
 package vorpal.package.v0;
 
 service PackageService {
-    rpc Prepare (PrepareRequest) returns (PrepareResponse);
+    rpc Prepare (stream PrepareRequest) returns (PrepareResponse);
     rpc Build (BuildRequest) returns (BuildResponse);
 }