Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Orchestrator, Executor and Executor Spawners #3

Merged
merged 32 commits into from
Dec 11, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
db2d841
feat: initial orchestrator struct with simple test code
sifnoc Nov 22, 2023
280a8fd
feat: orchestrator, executor and executor spawner
sifnoc Nov 23, 2023
c6ddf89
feat: added basic test cases
sifnoc Nov 24, 2023
5cdb550
chore: remove unused csv files
sifnoc Nov 24, 2023
bf04d99
fix: use default csv file on test
sifnoc Nov 24, 2023
ca23759
fix: test cases
sifnoc Nov 24, 2023
4d1fee7
feat: error handling with cancellation
sifnoc Nov 25, 2023
3e39b10
feat: created mock spawner and added more test cases
sifnoc Nov 27, 2023
089c744
chore: cancel when occur error
sifnoc Nov 27, 2023
b1ffced
chore: Update README
sifnoc Nov 28, 2023
f13aa9f
chore: renamed spanwers and small updates on README
sifnoc Nov 28, 2023
24e7c7e
fix: warnings; Update README
sifnoc Nov 29, 2023
74e0a50
chore: update README with diagram
sifnoc Nov 29, 2023
34e4f07
fix: refactor; updated README
sifnoc Nov 30, 2023
8634a91
fix: assign unused port on mock server
sifnoc Nov 30, 2023
fc17fc7
feat: finish cloud spawner
sifnoc Dec 1, 2023
a049680
feat: added test case for CloudSpawner; Update README
sifnoc Dec 1, 2023
1186202
feat: added aggregation example
sifnoc Dec 3, 2023
4a4d71b
fix: clippy; feature flag to use nightly in example
sifnoc Dec 4, 2023
f32fc8f
fix: applied latest v1 improvement
sifnoc Dec 4, 2023
27b5d4f
fix: example; README
sifnoc Dec 4, 2023
6b1a145
fix: retrieving preimage from sibling mini tree on AggregationMerkleS…
sifnoc Dec 5, 2023
d6dd01e
fix: example; updated README
sifnoc Dec 5, 2023
4b4e10e
fix: more error handling; added retry on executor
sifnoc Dec 6, 2023
f227c1f
chore: added description as in-line comments
sifnoc Dec 6, 2023
e61a1da
fix: update README for mini tree server; removed some part on main RE…
sifnoc Dec 7, 2023
c6ed945
chore: update README and added description as in-line
sifnoc Dec 7, 2023
726dbd7
fix: latest v1; minor changes on comments
sifnoc Dec 7, 2023
c8a2d7f
chore: update README; move csv folder to root
sifnoc Dec 7, 2023
679ed56
fix: return a ref of empty array on un-intended fn on AggregationMerk…
sifnoc Dec 7, 2023
b108d1b
fix: the container now using unused port on host
sifnoc Dec 8, 2023
be9b8f2
fix: applied latest v1-improvement; updated README; fixed minor
sifnoc Dec 8, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ csv = "1.3.0"
rand = "0.8.5"
futures = "0.3.29"
bollard = "0.15.0"
tokio-util = "0.7.10"

[[bin]]
name = "mini-tree-server"
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,15 +127,15 @@ Before starting the tests, you need to build the `mini-tree-generator` image and
Build the image with the following command:

```bash
$ docker build . -t summa-aggregation
docker build . -t summa-aggregation
```

Ensure that the `summa-aggregation:latest` image exists in your local registry.

Then, you can run the tests using this command:

```bash
$ cargo test
cargo test
enricobottazzi marked this conversation as resolved.
Show resolved Hide resolved
```

## Mini Tree Generator
enricobottazzi marked this conversation as resolved.
Show resolved Hide resolved
Expand Down
4 changes: 1 addition & 3 deletions src/executor/container_spawner.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
use bollard::{
container::{
Config, CreateContainerOptions, RemoveContainerOptions, StartContainerOptions,
},
container::{Config, CreateContainerOptions, RemoveContainerOptions, StartContainerOptions},
service::ContainerInspectResponse,
Docker,
};
Expand Down
27 changes: 14 additions & 13 deletions src/executor/executor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -51,16 +51,20 @@ impl Executor {
pub async fn generate_tree<const N_ASSETS: usize, const N_BYTES: usize>(
&self,
json_entries: Vec<JsonEntry>,
) -> Result<MerkleSumTree<N_ASSETS, N_BYTES>, Box<dyn Error>> {
) -> Result<MerkleSumTree<N_ASSETS, N_BYTES>, Box<dyn Error + Send>> {
// Parse the response body into a MerkleSumTree
let json_tree = self
.client
.post(&self.url)
.json(&json_entries)
.send()
.await?
.await
.map_err(|err| Box::new(err) as Box<dyn Error + Send>)
.unwrap()
.json::<JsonMerkleSumTree>()
.await?;
.await
.map_err(|err| Box::new(err) as Box<dyn Error + Send>)
.unwrap();

let entries = json_entries
.iter()
Expand Down Expand Up @@ -100,25 +104,22 @@ impl Executor {
#[cfg(test)]
mod test {
use futures::future;
use std::{error::Error, sync::atomic::AtomicUsize};
use std::error::Error;

use bollard::Docker;
use super::Executor;
use crate::orchestrator::entry_parser;
use crate::executor::ContainerSpawner;
use crate::executor::spawner::ExecutorSpawner;
use crate::executor::ContainerSpawner;
use crate::orchestrator::entry_parser;
use bollard::Docker;
enricobottazzi marked this conversation as resolved.
Show resolved Hide resolved

#[tokio::test]
async fn test_executor() -> Result<(), Box<dyn Error>> {
let spawner = ContainerSpawner::new(
"summa-aggregation".to_string(),
"excutor_test".to_string(),
);
let spawner =
ContainerSpawner::new("summa-aggregation".to_string(), "excutor_test".to_string());

let executor = spawner.spawn_executor().await;

let entries = entry_parser::<_, 2, 14>("./src/orchestrator/csv/entry_16.csv").unwrap();
let merkle_tree = executor.generate_tree::<2, 14>(entries).await?;
let merkle_tree = executor.generate_tree::<2, 14>(entries).await.unwrap();

spawner.terminate_executors().await;

Expand Down
14 changes: 11 additions & 3 deletions src/orchestrator/entry_csv_parser.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,23 @@ struct CSVEntry {
/// Parses a CSV file stored at path into a vector of Entries
pub fn entry_parser<P: AsRef<Path>, const N_ASSETS: usize, const N_BYTES: usize>(
path: P,
) -> Result<Vec<JsonEntry>, Box<dyn Error>> {
) -> Result<Vec<JsonEntry>, Box<dyn Error + Send>> {
let mut json_entries = Vec::<JsonEntry>::new();
let file = File::open(path)?;
// let file = File::open(path)?;
let file = match File::open(path) {
Ok(f) => f,
Err(e) => return Err(Box::new(e) as Box<dyn Error + Send>),
};

let mut rdr = csv::ReaderBuilder::new()
.delimiter(b';') // The fields are separated by a semicolon
.from_reader(file);

for result in rdr.deserialize() {
let record: CSVEntry = result?;
let record: CSVEntry = match result {
Ok(r) => r,
Err(e) => return Err(Box::new(e) as Box<dyn Error + Send>),
};

let entry = JsonEntry {
username: record.username,
Expand Down
Loading