refactor: add arg run-mode to prover & cli (#1305)

* add forester, forester-test, rpc, full modes

refactor: update RetryConfig defaults

The RetryConfig struct now has updated default values for the max_retries and retry_delay fields. The max_retries value has been increased from 10 to 20, and the retry_delay value has been changed from 100 milliseconds to 1 second. This change ensures that the RetryConfig provides more robust and reliable retry behavior.

chore: update LoadKeys function signature

The LoadKeys function in the proving_keys_utils.go file now accepts an additional parameter called circuits. This parameter is a list of circuit names that will be used in the GetKeys function. This change allows for more flexibility in loading keys based on specific circuits.

feat: update TestIndexer initialization

The TestIndexer initialization in the test_compressed_pda.rs file has been updated to include a ProverConfig parameter. This parameter allows for specifying the run_mode and circuits for the TestIndexer. This change enables more fine-grained control over the TestIndexer behavior.

feat: update TestIndexer initialization

The TestIndexer initialization in the test.rs file has been updated to include a ProverConfig parameter. This parameter allows for specifying the run_mode and circuits for the TestIndexer. This change enables more fine-grained control over the TestIndexer behavior.

feat: update TestIndexer initialization

The TestIndexer initialization in the test_program_owned_trees.rs file has been updated to include a ProverConfig parameter. This parameter allows for specifying the run_mode and circuits for the TestIndexer. This change enables more fine-grained control over the TestIndexer behavior.

chore: update initTestEnv function signature

The initTestEnv function in the initTestEnv.ts file now accepts two additional parameters: proverRunMode and circuits. The proverRunMode parameter allows for specifying the run mode of the prover, and the circuits parameter is a list of circuit names. These changes provide more flexibility in initializing the test environment.

chore: update spawn_prover function signature

The spawn_prover function in the gnark.rs file now accepts a ProverConfig parameter instead of the ProverMode parameter. The ProverConfig allows for specifying the run_mode and circuits for the prover. This change provides more flexibility in spawning the prover.

chore: update spawn_validator function signature

The spawn_validator function in the test_utils.rs file now accepts a ProverConfig parameter instead of the LightValidatorConfig parameter. The ProverConfig allows for specifying the run_mode and circuits for the validator. This change provides more flexibility in spawning the validator.

chore: update killProcess function behavior

The killProcess function in the process.ts file has been updated to handle killing processes based on their name or command. It now uses the process name or command to identify the target processes to kill. This change improves the accuracy of process termination.

chore: update waitForServers function behavior

The waitForServers function in the process.ts file has been updated to increase the timeout value to 60 seconds and remove the simultaneous option. These changes ensure that the function waits for the servers to start up properly and improves the reliability of the server startup process.

chore: update init_program_test_env function signature

The init_program_test_env function in the e2e_test_env.rs file now accepts a ProverConfig parameter instead of the ProverMode parameter. The ProverConfig allows for specifying the run_mode and circuits for the TestIndexer. This change provides more flexibility in initializing the program test environment.

chore: update KeypairActionConfig behavior

The KeypairActionConfig struct in the e2e_test_env.rs file no longer has a prover_mode method. Instead, the inclusion and non_inclusion methods are used to determine the run mode of the prover. This change simplifies the logic for determining the prover run mode.

chore: update start-prover command behavior

The start-prover command in the index.ts file now accepts the --circuit option to specify individual circuits to enable. This change allows for more granular control over which circuits are enabled in the prover.

import ProverConfig in test.rs

refactor: update test functions for batch append and update

Refactor the test functions for batch append and update to use a larger tree depth of 26 instead of 10.

refactor: token-escrow: update import statements for ProverConfig and ProverMode

refactor: update ProverConfig defaults in test.rs

refactor: update ProverConfig defaults in test.rs

refactor: Remove unused prover.sh script

refactor: Update circuit builder

refactor: Update lightweight and full files in download_keys.sh

wip

wip

format

import ProverConfig

build CLI in Rust workflow

wip

wip

refactor: fix formatting in test_program_owned_trees.rs

fix

refactor: update prover configuration in test programs

* Refactor prover spawning logic and dependencies

* Add 'devenv' feature to light-prover-client

* Add step to build CLI in GitHub workflow

* Refactor circuit_utils.go and proving_keys_utils.go

.github/workflows/light-system-programs-tests.yml

@@ -71,6 +71,11 @@ jobs:
       - name: Setup and build
         uses: ./.github/actions/setup-and-build
 
+      - name: Build CLI
+        run: |
+          source ./scripts/devenv.sh
+          npx nx build @lightprotocol/zk-compression-cli
+
       - name: build-programs
         run: |
           source ./scripts/devenv.sh

.github/workflows/rust.yml

@@ -40,6 +40,11 @@ jobs:
       - name: Setup and build
         uses: ./.github/actions/setup-and-build
 
+      - name: Build CLI
+        run: |
+          source ./scripts/devenv.sh
+          npx nx build @lightprotocol/zk-compression-cli
+
       # concurrent merkle tree crate must run in isolation because otherwise it
       # tries to use the idl-build feature (which fails).
       - name: Test workspace (excluding light-concurrent-merkle-tree)

circuit-lib/light-prover-client/Cargo.toml

@@ -8,7 +8,8 @@ edition = "2021"
 
 [features]
 gnark = ["tokio", "reqwest"]
-default = ["gnark"]
+default = ["gnark", "devenv"]
+devenv = []
 
 [dependencies]
 

circuit-lib/light-prover-client/src/gnark/helpers.rs

@@ -1,5 +1,6 @@
 use log::info;
 use std::{
+    ffi::OsStr,
     fmt::{Display, Formatter},
     process::Command,
     sync::atomic::{AtomicBool, Ordering},
@@ -14,12 +15,40 @@ use serde_json::json;
 use sysinfo::{Signal, System};
 
 static IS_LOADING: AtomicBool = AtomicBool::new(false);
-#[derive(Debug, Clone, PartialEq)]
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub enum ProverMode {
+    Rpc,
+    Forester,
+    ForesterTest,
+    Full,
+    FullTest,
+}
+
+impl Display for ProverMode {
+    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+        write!(
+            f,
+            "{}",
+            match self {
+                ProverMode::Rpc => "rpc",
+                ProverMode::Forester => "forester",
+                ProverMode::ForesterTest => "forester-test",
+                ProverMode::Full => "full",
+                ProverMode::FullTest => "full-test",
+            }
+        )
+    }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq)]
 pub enum ProofType {
     Inclusion,
     NonInclusion,
+    Combined,
     BatchAppend,
     BatchUpdate,
+    BatchAppendTest,
+    BatchUpdateTest,
 }
 
 impl Display for ProofType {
@@ -30,31 +59,58 @@ impl Display for ProofType {
             match self {
                 ProofType::Inclusion => "inclusion",
                 ProofType::NonInclusion => "non-inclusion",
+                ProofType::Combined => "combined",
                 ProofType::BatchAppend => "append",
                 ProofType::BatchUpdate => "update",
+                ProofType::BatchAppendTest => "append-test",
+                ProofType::BatchUpdateTest => "update-test",
             }
         )
     }
 }
 
-pub async fn spawn_prover(restart: bool, proof_types: &[ProofType]) {
-    if let Some(project_root) = get_project_root() {
-        let path = "circuit-lib/light-prover-client/scripts/prover.sh";
-        let absolute_path = format!("{}/{}", project_root.trim(), path);
+#[derive(Debug, Clone)]
+pub struct ProverConfig {
+    pub run_mode: Option<ProverMode>,
+    pub circuits: Vec<ProofType>,
+}
+
+pub async fn spawn_prover(restart: bool, config: ProverConfig) {
+    if let Some(_project_root) = get_project_root() {
+        let prover_path: &str = {
+            #[cfg(feature = "devenv")]
+            {
+                &format!("{}/{}", _project_root.trim(), "cli/test_bin/run")
+            }
+            #[cfg(not(feature = "devenv"))]
+            {
+                "light"
+            }
+        };
+
         if restart {
+            println!("Killing prover...");
             kill_prover();
         }
 
         if !health_check(1, 3).await && !IS_LOADING.load(Ordering::Relaxed) {
             IS_LOADING.store(true, Ordering::Relaxed);
 
-            let proof_type_args: Vec<String> = proof_types.iter().map(|p| p.to_string()).collect();
-            let proof_type_str = proof_type_args.join(" ");
-            Command::new("sh")
-                .arg("-c")
-                .arg(format!("{} {}", absolute_path, proof_type_str))
-                .spawn()
-                .expect("Failed to start server process");
+            let mut command = Command::new(prover_path);
+            command.arg("start-prover");
+
+            if let Some(ref mode) = config.run_mode {
+                command.arg("--run-mode").arg(mode.to_string());
+            }
+
+            for circuit in config.circuits.clone() {
+                command.arg("--circuit").arg(circuit.to_string());
+            }
+
+            println!("Starting prover with command: {:?}", command);
+
+            let _ = command.spawn().expect("Failed to start prover process");
+
             let health_result = health_check(20, 5).await;
             if health_result {
                 info!("Prover started successfully");
@@ -68,15 +124,60 @@ pub async fn spawn_prover(restart: bool, proof_types: &[ProofType]) {
     }
 }
 
-pub fn kill_prover() {
+pub fn kill_process(process_name: &str) {
     let mut system = System::new_all();
     system.refresh_all();
 
     for process in system.processes().values() {
-        if process.name() == "light-prover" {
-            process.kill_with(Signal::Term);
+        let process_name_str = process.name().to_string_lossy();
+        let process_cmd = process.cmd().join(OsStr::new(" "));
+        let process_cmd_str = process_cmd.to_string_lossy();
+
+        // Match the exact process name
+        if process_name_str.contains(process_name) {
+            println!(
+                "Attempting to kill process: PID={}, Name={}, Cmd={}",
+                process.pid(),
+                process_name_str,
+                process_cmd_str
+            );
+            if process.kill_with(Signal::Kill).is_some() {
+                println!("Successfully killed process: PID={}", process.pid());
+            } else {
+                eprintln!("Failed to kill process: PID={}", process.pid());
+            }
         }
     }
+
+    // Double-check if processes are still running
+    system.refresh_all();
+    let remaining_processes: Vec<_> = system
+        .processes()
+        .values()
+        .filter(|process| {
+            let process_name_str = process.name().to_string_lossy();
+            process_name_str == process_name
+        })
+        .collect();
+
+    if !remaining_processes.is_empty() {
+        eprintln!(
+            "Warning: {} processes still running after kill attempt",
+            remaining_processes.len()
+        );
+        for process in remaining_processes {
+            eprintln!(
+                "Remaining process: PID={}, Name={}, Cmd={}",
+                process.pid(),
+                process.name().to_string_lossy(),
+                process.cmd().join(OsStr::new(" ")).to_string_lossy()
+            );
+        }
+    }
+}
+
+pub fn kill_prover() {
+    kill_process("prover");
 }
 
 pub async fn health_check(retries: usize, timeout: usize) -> bool {
@@ -146,3 +247,52 @@ where
         Err(_) => panic!("Merkle tree data invalid"),
     }
 }
+
+#[derive(Debug)]
+pub struct LightValidatorConfig {
+    pub enable_indexer: bool,
+    pub prover_config: Option<ProverConfig>,
+    pub wait_time: u64,
+}
+
+impl Default for LightValidatorConfig {
+    fn default() -> Self {
+        Self {
+            enable_indexer: false,
+            prover_config: None,
+            wait_time: 35,
+        }
+    }
+}
+
+pub async fn spawn_validator(config: LightValidatorConfig) {
+    if let Some(project_root) = get_project_root() {
+        let path = "cli/test_bin/run test-validator";
+        let mut path = format!("{}/{}", project_root.trim(), path);
+        println!("Starting validator...");
+        println!("Config: {:?}", config);
+        if !config.enable_indexer {
+            path.push_str(" --skip-indexer");
+        }
+        if let Some(prover_config) = config.prover_config {
+            prover_config.circuits.iter().for_each(|circuit| {
+                path.push_str(&format!(" --circuit {}", circuit));
+            });
+            if let Some(prover_mode) = prover_config.run_mode {
+                path.push_str(&format!(" --prover-run-mode {}", prover_mode));
+            }
+        } else {
+            path.push_str(" --skip-prover");
+        }
+
+        println!("Starting validator with command: {}", path);
+
+        Command::new("sh")
+            .arg("-c")
+            .arg(path)
+            .spawn()
+            .expect("Failed to start server process");
+        tokio::time::sleep(tokio::time::Duration::from_secs(config.wait_time)).await;
+        println!("Validator started successfully");
+    }
+}