feat(gnovm): Executable Markdown #2357

notJoon · 2024-06-14T05:41:35Z

Description

The doctest (tentative name) is a tool that allows code blocks in Markdown, especially those marked as "go" and "gno", to be executed using the gno VM.

It works as follows: First, it parses the code blocks (fenced code blocks) from the Markdown file. After that, the extracted code is written to a temporary file, and the code is executed at the VM level.

Also supports static analysis to automatically include imports or packages without explicitly specifying them before execution. However, in this case, it assumes the code always executes the main function, fixes the package as 'main', and only searches for missing library paths in the standard library.
Additionally, to prevent unnecessary executions, a feature to cache execution results has been added.

I wrote the behavior and functionality in the file gnovm/cmd/gno/testdata/doctest/1.md as a tutorial.

Moving forward, there is potential to further enhance this tool to support code documentation by allowing the execution of code embedded within comments, similar to Rust.

related #2245

CLI

The CLI command is used as follows:

gno doctest -path <markdown_file_path> [-run ] [-timeout ]

path literally means the path of the markdown file to be executed. The run flag allows you to specify a pattern to match against code block names or tags, similar to go test -run <name>. This will execute all code blocks that partially or completely match the given pattern. If run is not specified, all code blocks in the file will be executed.

The optional timeout flag allows you to set a maximum duration for the execution of each code block. If not specified, a default timeout will be used.

For example:

gno doctest -path ./README.md -run "example" -timeout 30s

This command will execute all code blocks in README.md that have "example" in their name or tag, with a timeout of 30 seconds for each block.

codecov · 2024-06-14T05:59:34Z

Codecov Report

All modified and coverable lines are covered by tests ✅

📢 Thoughts on this report? Let us know!

… into testable-markdown

deelawn

Nice work -- I found this very well written and and easy to follow, but there is a lot here, so I've left plenty of comments regarding style and questions about certain things. Thanks for this 😄

gnovm/cmd/gno/doctest.go

gnovm/cmd/gno/doctest_test.go

gnovm/pkg/doctest/parser.go

gnovm/pkg/doctest/analyzer.go

gnovm/pkg/doctest/exec.go

Co-authored-by: deelawn <[email protected]>

notJoon · 2024-08-05T11:29:48Z

Thank you for the detailed review @deelawn. I think I have addressed all the points you mentioned in your review. Could you please check and confirm? Thank you once again 👍

piux2 · 2024-08-05T16:12:53Z

gnovm/pkg/doctest/README.md

+
+Doctest also recognizes that a block of code is a gno. The code below outputs the same result as the example above.
+
+```go


This this a great idea, I have a few questions:

Do we intent to run both go code and gno code in the doctest?
How does Doctest detect this is a gno code?
What is the reason behind using "```go" instead of "````gno" for gno code block?

That's a really good question. Currrently, when rendering markdown, gno doesn't have universal syntax highlighting applied yet (except the custom syntax rule). So, even in official documents like "Effective gno" are using go instead of gno to specify the language.

Considering the purpose of markdown is more for documentation rather than testing, this was an inevitable purpose. So I made it recognize both languages.

However, it doesn't clearly distinguish whether it's completely go or gno. For now, I assume that it's mostly compatible with go and execute it at gno VM level. In other words, even if the language is atually written in go, everything runs as gno. This part may require deeper context analysis in the future.

To summarize in the order of your questions:

Even if the code block is specified as Go, it's actually recognized and executed as gno.

There's no feature to clearly distinguish between the two languages, and it relies on the language specified in the fenced code block.

Go is also set to be recognized due to syntax highlighting.

The main difference between Gno and Go is that Gno has a few native methods unique to blockchain usage. For example:

std.AssertOriginCall
std.GetHeight

Basically, a program can either be Go code or Gno code. We cannot mix them in one source file.

There are a few options I can think of

Users need to explicitly mark it as either Go code or Gno code, similar to indicating the source code type with a suffix in the filename: xxx.go vs xxx.gno.

Only support Gno.

By default, the MD is treated as Gno code. Use gnostd.XXX to indicate that it is Go code. (Not sure if this has much practical purpose; it is more complicated to implement and is only for the sake of supporting Go code with chain functionality in MD.)

piux2 · 2024-08-05T16:54:30Z

gnovm/pkg/doctest/README.md

+package main
+
+func main() {
+    println("Hello, World!")


the println does not print correct results compared with file tests.

package main type ints []int func main() { a := ints{1,2,3} println(a) }

doctest

// Output:
// (slice[(1 int),(2 int),(3 int)] gno.land/r/g14ch5q26mhx3jk5cxl88t278nper264ces4m8nt/run.ints)

file test result is correct. ints is a type defined in main package.
// Output:
// (slice[(1 int),(2 int),(3 int)] main.ints)

It seems this is happening because the code is executed through the VMKeeper.

func ExecuteCodeBlock(c codeBlock, stdlibDir string) (string, error) { // ... addr := crypto.AddressFromPreimage([]byte("addr1")) acc := acck.NewAccountWithAddress(ctx, addr) acck.SetAccount(ctx, acc) msg2 := vm.NewMsgRun(addr, std.Coins{}, files) res, err := vmk.Run(ctx, msg2) // ... }

I used this method to handle stdlibs imports. But honestly, I don't know how to maintain this functionality while also making it output main.

What about replacing it with the proper type when displaying the output in the Markdown?

gnovm/pkg/doctest/README.md

deelawn

Thanks for the work to resolve my comments. Everything looks good to me.

zivkovicmilos

Seems good, at least from my domain knowledge of the VM 💯

Please wait on @thehowl to give you a ✅ before moving forward 🙏

cc @mvertes for a look also

zivkovicmilos · 2024-11-20T03:35:30Z

gnovm/pkg/gnolang/nodes_test.go

@@ -1,10 +1,13 @@
-package gnolang_test


Is there a specific reason we had a different test package? @thehowl

zivkovicmilos · 2024-11-20T03:37:22Z

gnovm/cmd/gno/doctest.go

+
+func execDoctest(cfg *doctestCfg, _ []string, io commands.IO) error {
+	if cfg.markdownPath == "" {
+		return fmt.Errorf("markdown file path is required")


There is nothing formatted here

zivkovicmilos · 2024-11-20T03:38:25Z

gnovm/cmd/gno/doctest.go

+	resultChan := make(chan []string)
+	errChan := make(chan error)
+
+	go func() {


You can use error groups, and easily exit when the first routine encounters a problem

zivkovicmilos · 2024-11-20T03:38:55Z

gnovm/cmd/gno/doctest.go

+	case err := <-errChan:
+		return fmt.Errorf("failed to execute code block: %w", err)
+	case <-ctx.Done():
+		return fmt.Errorf("execution timed out after %v", cfg.timeout)


What do you think about formatting it to seconds?

zivkovicmilos · 2024-11-20T03:44:42Z

gnovm/pkg/doctest/exec.go

+	cache      = newCache(maxCacheSize)
+	regexCache = make(map[string]*regexp.Regexp)
+
+	addrRegex = regexp.MustCompile(`gno\.land/r/g[a-z0-9]+/[a-z.]+`)


Isn't this defined somewhere as standard?

zivkovicmilos · 2024-11-20T03:47:13Z

gnovm/pkg/doctest/exec.go

+
+	msg2 := vm.NewMsgRun(addr, std.Coins{}, files)
+
+	res, err := vmk.Run(stdlibCtx, msg2)


Maybe in the future we can support execution that doesn't require a local gno copy, but can interact with a remote chain. I can see this being useful when we have chain-version specific guides that require nuance in which VM version the source code is ran against

zivkovicmilos · 2024-11-20T05:13:15Z

gnovm/pkg/doctest/exec.go

+		return "", err
+	}
+
+	cache.set(hashKey, res)


How much does having this cache make a difference?
Since we're saving hashes of the code blocks here, how often do they repeat?

zivkovicmilos · 2024-11-20T05:14:23Z

gnovm/pkg/doctest/exec.go

+}
+
+// ref: gno.land/pkg/sdk/vm/common_test.go
+func setupEnv() (


setupEnv shouldn't be in the package like this, but in a _test.go file

zivkovicmilos · 2024-11-20T05:15:20Z

gnovm/pkg/doctest/exec.go

+
+// hashCodeBlock generates a SHA256 hash for the given code block.
+func hashCodeBlock(c codeBlock) string {
+	h := sha256.New()


Maybe this is overkill?

zivkovicmilos · 2024-11-20T05:42:34Z

gnovm/pkg/doctest/parser_test.go

+	}
+
+	for _, tt := range tests {
+		tt := tt


You don't need this in go 1.22

thehowl · 2024-12-12T17:22:12Z

@notJoon ping me once you've addressed the comments from milos, and i'll take a look :)

Gno2D2 · 2024-12-12T17:23:12Z

🛠 PR Checks Summary

🔴 The pull request head branch must be up-to-date with its base (more info)

Manual Checks (for Reviewers):

SKIP: Do not block the CI for this PR

✅ Automated Checks (for Contributors):

🟢 Maintainers must be able to edit this pull request (more info)
🔴 The pull request head branch must be up-to-date with its base (more info)

☑️ Contributor Actions:

Fix any issues flagged by automated checks.
Follow the Contributor Checklist to ensure your PR is ready for review.
- Add new tests, or document why they are unnecessary.
- Provide clear examples/screenshots, if necessary.
- Update documentation, if required.
- Ensure no breaking changes, or include BREAKING CHANGE notes.
- Link related issues/PRs, where applicable.

☑️ Reviewer Actions:

Complete manual checks for the PR, including the guidelines and additional checks if applicable.

📚 Resources:

Debug

Automated Checks
Maintainers must be able to edit this pull request (more info)
If
🟢 Condition met
└── 🟢 The pull request was created from a fork (head branch repo: notJoon/gno-core)
Then
🟢 Requirement satisfied
└── 🟢 Maintainer can modify this pull request
The pull request head branch must be up-to-date with its base (more info)
If
🟢 Condition met
└── 🟢 On every pull request
Then
🔴 Requirement not satisfied
└── 🔴 Head branch (notJoon:testable-markdown) is up to date with base (master): behind by 184 / ahead by 47
Manual Checks
**SKIP**: Do not block the CI for this PR
If
🟢 Condition met
└── 🟢 On every pull request
Can be checked by

Any user with comment edit permission

fenced code block parser

1e598f6

github-actions bot assigned notJoon Jun 14, 2024

github-actions bot added the 📦 🤖 gnovm Issues or PRs gnovm related label Jun 14, 2024

notJoon added 13 commits June 14, 2024 15:52

file handler

168259d

pseudo code for execute a parsed code

edb648e

generate hash key based on the parsed code

f60cb1c

use tree-sitter for parsing markdown content

a577afc

capturing output from buffer

b29530c

refactor

6430862

save

b7ef8ad

change to use MemDB and add execute options

b649727

save: need to handle import expr later

e53482b

basic CLI

f0df3b3

execute imported function

599bd6f

save

da567e3

fix: doctest CLI

2d363bc

notJoon changed the title ~~[WIP] Executable Markdown~~ feat(gnovm): Executable Markdown Jun 18, 2024

notJoon added 3 commits June 18, 2024 20:17

Merge branch 'master' into testable-markdown

83d7ca0

lint

f31a102

Merge branch 'testable-markdown' of https://github.com/notJoon/gno-core…

4f05a8e

… into testable-markdown

notJoon marked this pull request as ready for review June 18, 2024 11:23

notJoon requested review from a team, jaekwon, moul, piux2, deelawn and thehowl as code owners June 18, 2024 11:23

adr-sk mentioned this pull request Jun 19, 2024

Onbloc's Builder Journey gnolang/hackerspace#29

Open

static analysis for auto fix missing package and import statement

23052b9

thehowl mentioned this pull request Jul 23, 2024

CI: detect outdated docs' examples #2532

Open

deelawn reviewed Aug 2, 2024

View reviewed changes

notJoon and others added 8 commits August 2, 2024 14:01

Merge branch 'master' into testable-markdown

bf480c7

Update gnovm/pkg/doctest/analyzer.go

1f7f238

Co-authored-by: deelawn <[email protected]>

Apply suggestions from code review

1bb104a

Co-authored-by: deelawn <[email protected]>

resolve suggestions (doctest, exec, parser)

6bac3ad

fix: partial exec, parser

4978329

update: compareResults, ExecuteBlock

1577034

remove unnecessary lock

2d64669

fix: handle auto import for conflict names

317fd41

readme

ae25209

piux2 reviewed Aug 5, 2024

View reviewed changes

gnovm/pkg/doctest/README.md Outdated Show resolved Hide resolved

remove auto import

45265f2

deelawn approved these changes Aug 7, 2024

View reviewed changes

Kouteki added the review/triage-pending PRs opened by external contributors that are waiting for the 1st review label Oct 3, 2024

Merge branch 'master' into testable-markdown

b7a3d9d

Kouteki removed the review/triage-pending PRs opened by external contributors that are waiting for the 1st review label Oct 4, 2024

notJoon added 3 commits October 15, 2024 11:14

Merge branch 'master' into testable-markdown

1776969

fix execution errors

3142978

fix

d15d3f1

thehowl mentioned this pull request Oct 17, 2024

docs.gno.land revamp #2953

Open

Kouteki removed request for jaekwon and thehowl November 18, 2024 06:46

zivkovicmilos requested a review from thehowl November 20, 2024 03:43

zivkovicmilos approved these changes Nov 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(gnovm): Executable Markdown #2357

feat(gnovm): Executable Markdown #2357

notJoon commented Jun 14, 2024 •

edited

Loading

codecov bot commented Jun 14, 2024 •

edited

Loading

deelawn left a comment

notJoon commented Aug 5, 2024

piux2 Aug 5, 2024 •

edited

Loading

notJoon Aug 6, 2024

piux2 Oct 14, 2024

piux2 Aug 5, 2024 •

edited

Loading

notJoon Aug 6, 2024 •

edited

Loading

piux2 Oct 14, 2024 •

edited

Loading

deelawn left a comment

zivkovicmilos left a comment

zivkovicmilos Nov 20, 2024

thehowl Dec 12, 2024

zivkovicmilos Nov 20, 2024

zivkovicmilos Nov 20, 2024

zivkovicmilos Nov 20, 2024

zivkovicmilos Nov 20, 2024

zivkovicmilos Nov 20, 2024

zivkovicmilos Nov 20, 2024

zivkovicmilos Nov 20, 2024

zivkovicmilos Nov 20, 2024

zivkovicmilos Nov 20, 2024

thehowl commented Dec 12, 2024

Gno2D2 commented Dec 12, 2024 •

edited

Loading

✅ Automated Checks (for Contributors):

☑️ Contributor Actions:

☑️ Reviewer Actions:

📚 Resources:

If

Then

If

Then

If

Can be checked by


		Doctest also recognizes that a block of code is a gno. The code below outputs the same result as the example above.

		```go


		msg2 := vm.NewMsgRun(addr, std.Coins{}, files)

		res, err := vmk.Run(stdlibCtx, msg2)

feat(gnovm): Executable Markdown #2357

Are you sure you want to change the base?

feat(gnovm): Executable Markdown #2357

Conversation

notJoon commented Jun 14, 2024 • edited Loading

Description

CLI

codecov bot commented Jun 14, 2024 • edited Loading

Codecov Report

deelawn left a comment

Choose a reason for hiding this comment

notJoon commented Aug 5, 2024

piux2 Aug 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

piux2 Aug 5, 2024 • edited Loading

Choose a reason for hiding this comment

notJoon Aug 6, 2024 • edited Loading

Choose a reason for hiding this comment

piux2 Oct 14, 2024 • edited Loading

Choose a reason for hiding this comment

deelawn left a comment

Choose a reason for hiding this comment

zivkovicmilos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thehowl commented Dec 12, 2024

Gno2D2 commented Dec 12, 2024 • edited Loading

🛠 PR Checks Summary

Manual Checks (for Reviewers):

✅ Automated Checks (for Contributors):

☑️ Contributor Actions:

☑️ Reviewer Actions:

📚 Resources:

If

Then

If

Then

If

Can be checked by

notJoon commented Jun 14, 2024 •

edited

Loading

codecov bot commented Jun 14, 2024 •

edited

Loading

piux2 Aug 5, 2024 •

edited

Loading

piux2 Aug 5, 2024 •

edited

Loading

notJoon Aug 6, 2024 •

edited

Loading

piux2 Oct 14, 2024 •

edited

Loading

Gno2D2 commented Dec 12, 2024 •

edited

Loading