feat(wasm-builder): New approach to build demos #3602

ark0f · 2023-12-17T20:20:43Z

This is a new approach to build Gear programs which can replace gear-wasm-builder.

Schematic architecture:

Pipeline:

Purpose of program build script is just write features to lock file and don't use any cargo:rerun-if instructions so default cargo heuristics are enabled.

Binaries crate build script list its build dependencies and corresponding reads lock files to obtain program features then it runs cargo build -p program1 -p program2 so cargo builds WASM binaries effectively and concurrently. Then it writes to lock files to skip cargo invocation in next or in another binaries crate to avoid few seconds cache verification

Pros:

Advanced feature tracking.
Features are tracked automatically so we don't need to re-run build script each time like in gear-wasm-builder (can be mitigated by cargo-metadata and Cargo.toml tracking)
Faster build.
gear-wasm-builder builds program via new cargo command in the same target directory so each cargo process blocks other. wasm-dep-builder collects list of programs so only one cargo process runs.

Cons:

Create and import binaries crate.
It is important to create binaries create that will list and build programs

Current problem:

When crate includes itself as dev dependency (any integration tests) with different feature set then cargo build and runs 2 different versions of build script - for workspace and for tests.

gear-wasm-builder case is that cargo builds and runs script just once with merged feature set.

So, how to resolve these features? Should we deliver different WASM binaries? Or just merge feature sets?
I vote for merging but we need some mechanism to obtain all of feature sets cargo resolved for each build script invocation of program before build script invocation of binaries crate if we don't want to waste time on WASM build with incomplete feature set

clearloop · 2023-12-19T13:48:58Z

plz check this #3588 (comment) btw

gshep

Good job! 🥇

gshep · 2024-02-02T06:46:18Z

gclient/binaries/build.rs

@@ -0,0 +1,21 @@
+// This file is part of Gear.
+//
+// Copyright (C) 2023 Gear Technologies Inc.


I suggest removing the year number from the copyright since it is not consistent across the source code and we will not update it later

I see most of files have year when just a few don't

utils/wasm-dep-builder/src/utils.rs

StackOverflowExcept1on · 2024-02-05T19:34:18Z

core/src/pages.rs

@@ -18,6 +18,8 @@

 //! Module for memory pages.

+#![allow(clippy::assertions_on_constants)]


merge master and remove this

StackOverflowExcept1on · 2024-02-05T20:08:23Z

examples/signal-wait/src/lib.rs

@@ -0,0 +1 @@
+


why examples/signal-wait/** is empty?

DennisInSky · 2024-02-06T10:41:47Z

Cargo.toml

@@ -213,6 +213,7 @@ gear-service = { path = "node/service", default-features = false }
 gear-stack-buffer = { path = "stack-buffer" }
 gear-utils = { path = "utils/utils" }
 gear-wasm-builder = { path = "utils/wasm-builder", default-features = false }
+wasm-dep-builder = { path = "utils/wasm-dep-builder" }


Do we want to prefix it with gear as for the others?

DennisInSky · 2024-02-06T10:43:48Z

examples/constructor/io/Cargo.toml

+[dependencies]
+gcore.workspace = true
+parity-scale-codec.workspace = true
+gstd = { workspace = true }


Suggested change

gstd = { workspace = true }

gstd.workspace = true

DennisInSky · 2024-02-06T11:02:51Z

examples/delayed-sender/io/Cargo.toml

+
+[dependencies]


DennisInSky · 2024-02-06T11:09:56Z

examples/reserve-gas/io/Cargo.toml

+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html


DennisInSky · 2024-02-06T11:24:17Z

examples/syscalls/io/Cargo.toml

+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html


DennisInSky · 2024-02-06T11:43:52Z

gclient/Cargo.toml

 gmeta = { workspace = true }
 gstd = { workspace = true, features = ["debug"] }
 demo-wat.workspace = true
+gclient-binaries = { path = "binaries" }


Would it be better to call it demo-binaries?

DennisInSky · 2024-02-06T11:54:07Z

gsdk/Cargo.toml

@@ -42,6 +42,7 @@ rand = { workspace = true, optional = true }

 [dev-dependencies]
 gsdk = { path = ".", features = ["testing"] }
+gsdk-binaries = { path = "binaries" }


-> demo-binaries?

DennisInSky · 2024-02-06T11:55:09Z

gtest/Cargo.toml

 demo-meta-io.workspace = true
+gtest-binaries = { path = "binaries" }


-> demo-binaries?

DennisInSky · 2024-02-06T11:56:21Z

node/authorship/Cargo.toml

@@ -41,7 +41,9 @@ sp-inherents = { workspace = true, features = ["std"] }
 prometheus-endpoint.workspace = true

 [dev-dependencies]
+gear-authorship-binaries = { path = "binaries" }


-> demo-binaries?

DennisInSky · 2024-02-06T12:05:27Z

utils/wasm-dep-builder/Cargo.toml

+repository.workspace = true
+
+[dependencies]
+cargo_metadata = "0.18.1"


Do you want to move them into the workspace deps?

ark0f · 2024-02-07T14:58:34Z

cargo check may be failed because of missing .lock. Investigating...

NikVolf · 2024-02-25T16:17:18Z

Can you describe the approach in the PR outline?

NikVolf · 2024-02-26T07:57:28Z

Can we maybe put "dependency builder" inside gear-wasm-builder itself?

breathx · 2024-02-26T16:12:51Z

Can we maybe put "dependency builder" inside gear-wasm-builder itself?

It will replace gear_wasm_builder some day, so is there any reason to merge them now?

Just for notice: as it was mentioned, there are a few problems with these approach, and as far as we discussed with @ark0f they're already solved, but not pushed

ark0f · 2024-03-12T18:14:09Z

Can you describe the approach in the PR outline?

Done

breathx · 2024-05-06T08:59:02Z

Temporary closing in favour of opened referencing issue as won't be implemented

ark0f added 7 commits December 13, 2023 03:16

Initial commit

576b56f

Add tracing based on cargo and lock file

b9e51b9

Use wasm-dep-builder in pallet-gear

45c2d40

Use wasm-dep-builder in all pallet-gear demos

5b4b12f

Build packages in workspace

53c2744

Fix cargo config is in wrong folder

1526bf1

Optimize binaries

df03809

ark0f added A1-inprogress Issue is in progress or PR draft is not ready to be reviewed D5-tooling Helper tools and utilities labels Dec 17, 2023

Merge origin/master

2384b97

ark0f added 19 commits December 25, 2023 00:34

Merge remote-tracking branch 'origin/master' into al/no-wasm-builder

c7e9636

Fix missing wasm_binary.rs file error appears sometime

84de8f3

Fix some demos are not rebuilt

c13b7f6

Make demo imports like before

ce5d39d

Make pallet-gear-binaries crate optional

fb310d6

Follow new approach in test-syscalls

2313d74

Make build command persistent

6dfbc83

Rename WASM binary consts

03ffa6e

Remove test-wasm-dep

6f3a50c

Introduce debug feature in binaries crate

fb5f2e9

Advanced feature tracking

0a2e664

Introduce LockFile

297513c

Introduce from_value constructors

96882c9

Remove obsolete environment variable

78f0d44

Make features tracking persistent

58f6b47

Refactor builder

4e98b52

Fix clippy warnings

0c850e6

Revert benchmarks.rs in fungible-token

3ee9e29

Use wasm-dep-builder in gtest

c3e0a2f

Merge remote-tracking branch 'origin/master' into al/no-wasm-builder

1c2dde2

gshep approved these changes Feb 2, 2024

View reviewed changes

ark0f added 7 commits February 5, 2024 18:57

Merge remote-tracking branch 'origin/master' into al/no-wasm-builder

42a7a33

Build in its crate's own directory

cda85fb

Rename read -> read_any for BinariesLockFileConfig

9f45efd

Remove debug warnings

54fc69a

Copy binaries to default directory

ae89a05

Fix unused imports warning

9e59065

Update copyright

943d9e4

ark0f requested review from DennisInSky, clearloop, breathx and StackOverflowExcept1on February 5, 2024 16:37

StackOverflowExcept1on reviewed Feb 5, 2024

View reviewed changes

examples/signal-wait/src/lib.rs

@@ -0,0 +1 @@

Copy link

Member

StackOverflowExcept1on Feb 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why examples/signal-wait/** is empty?

StackOverflowExcept1on approved these changes Feb 5, 2024

View reviewed changes

DennisInSky approved these changes Feb 6, 2024

View reviewed changes

ark0f added A5-dontmerge PR should not be merged yet A3-gotissues PR occurred to have issues after the review and removed A5-dontmerge PR should not be merged yet labels Feb 7, 2024

breathx added the A5-dontmerge PR should not be merged yet label Feb 26, 2024

ark0f mentioned this pull request Mar 25, 2024

feat(wasm-builder): Another approach to build demos #3837

Closed

breathx mentioned this pull request May 6, 2024

Consider options of redesigning building approach (deprecate gear-wasm-builder) #3945

Open

breathx closed this May 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(wasm-builder): New approach to build demos #3602

feat(wasm-builder): New approach to build demos #3602

ark0f commented Dec 17, 2023 •

edited

Loading

clearloop commented Dec 19, 2023

gshep left a comment

gshep Feb 2, 2024

ark0f Feb 5, 2024

StackOverflowExcept1on Feb 5, 2024

StackOverflowExcept1on Feb 5, 2024

DennisInSky Feb 6, 2024

DennisInSky Feb 6, 2024

DennisInSky Feb 6, 2024

DennisInSky Feb 6, 2024

DennisInSky Feb 6, 2024

DennisInSky Feb 6, 2024

DennisInSky Feb 6, 2024

DennisInSky Feb 6, 2024

DennisInSky Feb 6, 2024

DennisInSky Feb 6, 2024

ark0f commented Feb 7, 2024

NikVolf commented Feb 25, 2024

NikVolf commented Feb 26, 2024

breathx commented Feb 26, 2024

ark0f commented Mar 12, 2024

breathx commented May 6, 2024

		@@ -18,6 +18,8 @@

		//! Module for memory pages.

		#![allow(clippy::assertions_on_constants)]


		# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

		demo-meta-io.workspace = true
		gtest-binaries = { path = "binaries" }


		[dependencies]

feat(wasm-builder): New approach to build demos #3602

feat(wasm-builder): New approach to build demos #3602

Conversation

ark0f commented Dec 17, 2023 • edited Loading

clearloop commented Dec 19, 2023

gshep left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ark0f commented Feb 7, 2024

NikVolf commented Feb 25, 2024

NikVolf commented Feb 26, 2024

breathx commented Feb 26, 2024

ark0f commented Mar 12, 2024

breathx commented May 6, 2024

ark0f commented Dec 17, 2023 •

edited

Loading