Provide a "Syntax Tree View" pane in VSCode

[ebousse]

The Langium Playground provides a fantastic "Syntax Tree View" in the right pane which instantly shows the AST obtained when parsing the program written in the middle pane. This is a small tool very useful for many purposes: debugging, teaching, writing a visitor using an example AST, etc.

My suggestion would be to make this "Syntax Tree View" part of the Langium VSCode extension (or part of each DSL extension generated with Langium), which could be opened in one click when editing a program of the DSL being made.

[spoenemann]

Maybe we could provide this using a tree view in VS Code. That would be considerably easier than creating a Webview. Including this as an optional part in the Yeoman generator might be a good approach.

[ebousse]

Indeed, great idea, a native tree view sounds even better!

[aabounegm]

Thanks to @spoenemann's suggestion, I took a dab at this today and came up with this: aabounegm/stella-experiment#1

It's still rough (didn't test it extensively), but still a good start. If anyone would like to review that PR, please feel free :)

As for where the implementation should live, I believe this is a feature mostly relevant only during development for debugging purposes, so it doesn't make sense to include it in the extension of the target language. I think it would be nice if the tree view is part of the Langium extension, and the server part (the command returning the AST) is also automatically injected by Langium (perhaps only in dev mode). If we do not want to pollute language servers that do not need it, then it can be opt-in by importing and registering a SyntaxTreeViewProvider for example.
Edit: On another thought, this might be too limiting for users who actually want this in their extensions. Ironically, the language in the PR above is for educational purposes in a university course, so it definitely makes sense to keep the tree view in the extension for the students.

Another option would be to have the implementation for both live outside Langium, perhaps in langium-tools by @borisbrodski, if he thinks it is suitable for the library.

Lastly, as suggested by @spoenemann, it could simply be part of the Yeoman template, but this is me least preferred approach because if a user created a template with it, and then an update to Langium itself caused some changes in the AST structure (or the serialization or anything), all users are stuck with having to fix it manually.

[spoenemann]

The problem with offering such an AST view in the Langium extension is that we can only do that in a playground style: use the default services everywhere, so no custom lexing, no custom scope provider etc. Including all the customizations requires to build and run the actual language code. But without the customizations, the result will be wrong for many languages.

One way to solve this is to check an environment variable (e.g. DEBUG_LANGIUM_AST) before the AST tree view is set up in the extension. That would make it available in dev mode by setting the variable in the launch configuration, but the view is hidden in production.

[aabounegm]

@spoenemann But why would it be so? The language server can override any services and just send the ready AST to the extension. For example, I implemented a custom scope provider (aabounegm/stella-experiment#2) and my tree view still works with it. The point is that the language server has to implement a custom request (I called it stella/syntaxTree in the PR linked above) which would simply send the serialized AST to the extension. This can also have a default implementation (seem my syntax-request.ts in Langium itself or simply be added to the Yeoman template.

[spoenemann]

Do you mean the view is contributed by the Langium extension, but the AST comes from your language server? Yes that could work, but it requires communication between the Langium extension and your own extension because the Langium extension doesn't have direct access to the language server.

Two approaches that come to my mind:

1. You expose a certain API from your extension and the Langium extension uses that to retrieve the AST. @dhuebner has used this approach for the devtools of vscode-messenger: https://github.com/TypeFox/vscode-messenger/blob/main/packages/vscode-messenger-devtools/src/devtool-ext.ts
2. The Langium extension opens a port and the language server connects to it via local socket. That way we don't need to go through custom JSON-RPC messages, but can communicate directly.

[spoenemann]

To clarify: the approach of creating the tree view in your own extension instead of the Langium extension is viable, too. But then we'd have to generate more code in the Yeoman template. Having a standard reusable view for it would be nicer, hence the two other approaches above.

[Lotes]

I started some similar discussion here. It also provides some ideas by @spoenemann . Maybe the same that are already here…

[aabounegm]

Yes, that's exactly what I mean, and the first approach (custom JSON-RPC request) is what I implemented in the PR I shared above (I actually created that project mainly just to test this feature). I preferred this approach over as it seems simpler and technically allows for reuse in any other LSP client.
It was also too simple I didn't think about trying vscode-messenger