update docs

ecgo/docs/apis.md → docs/apis.md

@@ -7,16 +7,16 @@ This document offers a comprehensive guide to the core APIs provided by the comp
 To integrate the compiler into your Go project, add the following import statement to your code:
 
 ```go
-import "github.com/PolyhedraZK/ExpanderCompilerCollection"
+import "github.com/PolyhedraZK/ExpanderCompilerCollection/ecgo"
 ```
 
 ### Compile Function
 
-The `ExpanderCompilerCollection.Compile` method is the primary interface to the compiler. It takes a `frontend.Circuit` as input and returns a `CompileResult`.
+The `ecgo.Compile` method is the primary interface to the compiler. It takes a `frontend.Circuit` as input and returns a `CompileResult`.
 
 ### CompileResult Structure
 
-The `ExpanderCompilerCollection.CompileResult` structure encapsulates the results of the compilation process, which includes both the layered circuit and the intermediate representation (IR).
+The `ecgo.CompileResult` structure encapsulates the results of the compilation process, which includes both the layered circuit and the intermediate representation (IR).
 
 The `CompileResult` provides three methods for accessing the data:
 
@@ -26,7 +26,7 @@ The `CompileResult` provides three methods for accessing the data:
 
 ### Builder API
 
-The `ExpanderCompilerCollection.API` serves as the interface for the builder API. It extends the `frontend.API` interface from gnark, offering additional features such as support for sub-circuits and utility functions.
+The `ecgo.API` serves as the interface for the builder API. It extends the `frontend.API` interface from gnark, offering additional features such as support for sub-circuits and utility functions.
 
 ## Sub-Circuit API
 
@@ -52,19 +52,17 @@ One key requirement for `SubCircuitFunc` is determinism. This means that for any
 
 ```go
 type API interface {
-	ToSingleVariable(frontend.Variable) frontend.Variable
 	Output(frontend.Variable)
-	LayerOf(frontend.Variable) int
-	ToFirstLayer(frontend.Variable) frontend.Variable
 	GetRandomValue() frontend.Variable
+	CustomGate(gateType uint64, inputs ...frontend.Variable) frontend.Variable
 }
 ```
 
-- `ToSingleVariable`: This method transforms an expression into a single base variable. If the input expression is already a single variable, it is returned as is. Otherwise, an internal variable or gate is created to encapsulate the expression.
 - `Output`: This method appends a variable to the circuit's output. It is typically used to designate certain variables as public outputs of the circuit.
-- `LayerOf`: This method estimates the layer in which a variable will appear in the compiled layered circuit. The term "estimate" is used because subsequent compilation optimizations may modify the exact layer placement initially determined during the Builder phase.
-- `ToFirstLayer`: This method employs a hint to pull a variable back to the first layer.
 - `GetRandomValue`: This method retrieves a random value directly, a more efficient approach than generating pseudo-random numbers using a hash function. This direct access to random numbers is facilitated by the Libra proving process.
+- `CustomGate`: This method is similar to Gnark's `NewHint` in that it essentially calls a hint function to compute a result. In the resulting layered circuit, it will be compiled into a custom gate of the specified gate type. Unlike `NewHint`, it requires pre-registering the hint function and other parameters. For specific details, see [the example](../ecgo/examples/custom_gate/main.go).
+
+Several other APIs exist for old pure Golang Expander Compiler compability, but they are no-op now.
 
 ## Builder Extensions
 

ecgo/docs/artifact_and_serialization.md → docs/artifact_and_serialization.md

@@ -29,37 +29,24 @@ A gate is considered random if its `Coef` is equivalent to `RootCircuit.Field`.
 
 ### Serialization Process
 
-At its core, `uint64` types are serialized in little-endian format, while `big.Int` types are serialized as 32-byte little-endian sequences.
+At its core, `uint64` types are serialized in little-endian format, while `big.Int` types are serialized as 1/4/32-byte little-endian sequences depending on the particular field.
 
 The serialization of an array begins with a `uint64` that denotes its length, followed by the serialized representation of its elements.
 
-In the serialized form, `Coef` values are constrained to be less than `RootCircuit.Field`. Random gates are represented using additional arrays.
+A unique identifier, the magic number 3770719418566461763 (b'CIRCUIT4'), is prefixed to the serialized RootCircuit data stream to ensure data integrity.
 
-The `Circuit` structure, when serialized, is represented as follows:
-
-```go
-type Circuit struct {
-    InputLen      uint64
-    OutputLen     uint64
-    SubCircuits   []SubCircuit
-    Mul           []GateMul
-    Add           []GateAdd
-    Cst           []GateCst
-    RandomCoefIdx []uint64
-}
-```
-In this structure, RandomCoefIdx denotes the indices of random gates within the combined arrays of Mul, Add, and Cst.
-
-A unique identifier, the magic number 3626604230490605891 (b'CIRCUIT2'), is prefixed to the serialized RootCircuit data stream to ensure data integrity.
+Refer [Go implementation](../ecgo/layered/serialize.go) and [Rust implementation](../expander_compiler/src/circuit/layered/serde.rs) for details.
 
 ## Input Solver
 
-The input solver is an intermediary form of the circuit that facilitates witness generation. It is defined in `ir/input_solver.go`.
+The input solver (`irwg.RootCircuit`) is an intermediary form of the circuit that facilitates witness generation.
 
-For Go-specific implementations, serialization is performed using the gob package.
+Refer [Go implementation](../ecgo/irwg/serialize.go), [Rust implementation for circuit](../expander_compiler/src/circuit/ir/common/serde.rs) and [Rust implementation for instruction](../expander_compiler/src/circuit/ir/hint_normalized/serde.rs) for details.
 
 ## Witness Serialization
 
 The witness, an array of `big.Int`, serves as the input for the layered circuit. It is also defined in `ir/input_solver.go`.
 
-The serialization process for the witness is straightforward. Given the known array length, the `big.Int` array is serialized as a sequence of 32-byte little-endian values.
+One witness file contains one or multiple witnesses, stored in a compact form.
+
+Refer [Go implementation](../ecgo/irwg/witness_gen.go) for details.

ecgo/docs/compilation_process.md → docs/compilation_process.md

@@ -2,14 +2,16 @@
 
 The compilation process, encapsulated within `api.go`, comprises several distinct phases:
 
-1. **Circuit Intermediate Representation (IR) Construction:** The initial phase involves processing the source code with the builder, which constructs the circuit's Intermediate Representation (IR). This builder, similar to gnark's internal builder, extends the `LinearExpression` to support quadratic terms. It maintains a tree structure that mirrors the computational graph of the circuit, serving as the foundation for IR generation.
+1. **Circuit Intermediate Representation (IR) Construction:** The initial phase involves processing the source code with the builder, which constructs the circuit's Intermediate Representation (IR). It maintains a tree structure that mirrors the computational graph of the circuit, serving as the foundation for IR generation.
 
 2. **IR Optimization:** The Intermediate Representation (IR) is a flexible representation of the circuit that allows for various optimizations. These can include merging redundant variables, splitting large variables to enhance efficiency, and performing other transformations that streamline the computational graph without altering its functionality.
 
 3. **Circuit Layering:** The IR undergoes a layering process that transforms it into a layered circuit. This process involves topologically sorting the directed acyclic graph (DAG) of sub-circuit call relationships, determining the necessary layers for each variable, and allocating the variables across these layers. The layering strategy is recursive, giving priority to sub-circuits and ensuring alignment with the Libra protocol's requirements.
 
 4. **Layered Circuit Optimization:** After the circuit is layered, additional optimizations can be applied. These may include expanding sub-circuits based on their frequency of occurrence or the number of variables they contain.
 
+In the old version of compiler, all steps are done in Go, but now only the step 1 is done in Go.
+
 ## Detailed Steps in Compilation
 
 ### Builder Implementation

ecgo/docs/example.md → docs/example.md

@@ -48,8 +48,8 @@ import (
 	"github.com/consensys/gnark-crypto/ecc"
 	"github.com/consensys/gnark/frontend"
 
-	"github.com/PolyhedraZK/ExpanderCompilerCollection"
-	"github.com/PolyhedraZK/ExpanderCompilerCollection/test"
+	"github.com/PolyhedraZK/ExpanderCompilerCollection/ecgo"
+	"github.com/PolyhedraZK/ExpanderCompilerCollection/ecgo/test"
 )
 
 type Circuit struct {
@@ -65,7 +65,7 @@ func (circuit *Circuit) Define(api frontend.API) error {
 func main() {
 	assignment := &Circuit{X: 1, Y: 1}
 
-	circuit, _ := ExpanderCompilerCollection.Compile(ecc.BN254.ScalarField(), &Circuit{})
+	circuit, _ := ecgo.Compile(ecc.BN254.ScalarField(), &Circuit{})
 	c := circuit.GetLayeredCircuit()
 	os.WriteFile("circuit.txt", c.Serialize(), 0o644)
 	inputSolver := circuit.GetInputSolver()
@@ -79,8 +79,8 @@ func main() {
 
 Please note that the Prover's implementation resides in [Expander](https://github.com/PolyhedraZK/Expander). As such, the `CheckCircuit` function in this context merely evaluates the circuit once to confirm its correctness. To generate and verify the actual proof, supply `circuit.txt` and `witness.txt` to Expander.
 
-In contrast to gnark, where the prover handles a portion of the witness generation, our compiler employs an `InputSolver` to perform this task. The `InputSolver` also supports serialization and deserialization. For more information, please consult the examples in [MIMC](../examples/mimc) and [MIMC Deserialize](../examples/mimc_deserialize).
+In contrast to gnark, where the prover handles a portion of the witness generation, our compiler employs an `InputSolver` to perform this task. The `InputSolver` also supports serialization and deserialization. For more information, please consult the examples in [MIMC](../ecgo/examples/mimc) and [MIMC Deserialize](../ecgo/examples/mimc_deserialize).
 
 For additional features, please refer to [APIs](./apis.md).
 
-For more examples, please visit [examples](../examples).
+For more examples, please visit [examples](../ecgo/examples).

readme.md

@@ -18,9 +18,9 @@ The APIs for this library are detailed in [APIs](./docs/apis.md).
 Refer to [this example](./docs/example.md) for a practical demonstration of our compiler. In this example, we illustrate how a gnark circuit can be compiled using `ExpanderCompilerCollection`. The output of this example includes a circuit description file `"circuit.txt"` and a corresponding witnesses file `"witness.txt"`. Our prover, [Expander](https://github.com/PolyhedraZK/Expander), utilizes these IRs to generate the actual proof.
 
 Additional examples include:
-- Hash functions like [sha2](./examples/gnark_std_sha2/main.go), [keccak](./examples/keccak/main.go), and [MIMC](./examples/mimc/main.go)
-- A [recursive circuit](./examples/gnark_recursive_proof/main.go)
-- A [mersenne field](./examples/m31_field/main.go)
+- Hash functions like [sha2](./ecgo/examples/gnark_std_sha2/main.go), [keccak](./ecgo/examples/keccak/main.go), and [MIMC](./ecgo/examples/mimc/main.go)
+- A [recursive circuit](./ecgo/examples/gnark_recursive_proof/main.go)
+- A [mersenne field](./ecgo/examples/m31_field/main.go)
 
 ## Deeper Dive in to the tech