[IDEA] Add JSDoc to AST

[linonetwo]

Is your feature request related to a problem? Please describe.

To build WYSIWYG editor, I need to maintain https://github.com/tiddly-gittly/wikiast , but it's dev experience is so bad that I don't want to touch it for a while.

Describe the solution you'd like

At least move AST part of typing from https://github.com/tiddly-gittly/TW5-Typed to the tw core, using JSDoc like how svelte does ( https://news.ycombinator.com/item?id=35932617 )

Example:

https://github.com/sveltejs/svelte/blob/1681b218cf3b42757f7d0f0a7c0dc45a79fea6f0/packages/svelte/src/store/shared/index.js#L7-L10

Describe alternatives you've considered

Directly use .d.ts may be better, but inline typing will make sure people update it.

Currently AST type in TW5-Typed is broken, the wikiast's code is in a mess. My time on opensource is liminted, so I need auto type checker to help GPT4 to help me. (GPT works better with type check output)

Additional context

I've heard that tw core maintainers don't like TS even JSDoc, but the fact is plugin developer need it, I need it, so please consider providing a better experience here. (Obsidian provides TS SDK and plugin template even it is close-sourced. Many people misunderstood Obsidian is opensourced because of its opensource plugin community. Consider how it attract modern web developers to contribute so much plugins)

[pmario]

I think the right article to link to - is: https://www.puruvj.dev/blog/get-to-know-typescript--using-typescript-without-typescript

It has been mentioned in: https://devclass.com/2023/05/11/typescript-is-not-worth-it-for-developing-libraries-says-svelte-author-as-team-switches-to-javascript-and-jsdoc/

I personally would love to get "autocomplete" and "type popups" if I hover a TW function in VSCode. IMO JSDOC is not too much different, to what we do have already. We will probably increase the source code size a bit, but I think the wins would be worth it.

What we have atm is suboptimal. eg: Just yesterday I had to lookup the GitHub-saver code for a thread at Talk

1. There are several function calls to this.wiki.* and $tw.utils.*
2. If we want to see how they work and if there are optional paramters we have to do a global search
   1. eg: $tw.utils.getPassword -- To have a chance to find the function definition we need to search for .getPassword
   2. In this case we are lucky, because there are only 10 matches in 7 files but for other search-terms there may be 100++ matches
   3. The pain point is, that VSCode searches the whole structure, so I constantly need to maintain "include" extension
   4. 
3. I need to open the file to see which "type" the "name" parameter is.
   1. 
   2. There is some documentation 3 lines, but no type in formation
4. The info for window.localStorage.getItem() is much nicer
   1. 
5. A bit of JSDoc woul look like this
   1. 
   2. and the popup would be like this
   3. 

That's nice, but how can we tell VSCode, where to find that info in $:/core/modules/savers/github.js

If we could improve the inline documentation in a way, that VSCode Intellisense automatically recognises it, in the whole project, I would support the effort.

VSCode does understand JSDoc -- and JSDoc understands commonjs - require(module-name) -- but JSDoc does not understand where to find the TW module: $:/core/modules/utils/dom.js in the project directory structure.

Because TW implements its own require function

So IMO $tw.modules.execute(moduleName,moduleRoot)
is responsible to actually find the relation between TW title: $:/core/modules/utils/dom.js and where to find it on disk.

We would need to let JSDoc know, how to use this info.

There is a second relation -- eg: $tw.utils.getPassword() actually comes from $:/core/modules/utils/dom.js because it is "merged" into the global namespace $tw.utils.* because it has modul-type: utils. -> Every exported function with that type is available as $tw.utils.*

JSDoc also has no idea about that relation, but it would need to know it to give proper info if you hover $tw.utils.getPassword

Just some thoughts.

@Jermolene -- Did you ever have a closer look at JSDoc in relation with TW source code?

[linonetwo]

@pmario

JSDoc does not understand where to find the TW module: $:/core/modules/utils/dom.js in the project directory structure.

Maybe sourcemap can handle this, no JSDoc/TS required.

Or use .d.ts file like https://github.com/tiddly-gittly/TW5-Typed/blob/master/src/modules/utils/parsetree.d.ts#L5

https://github.com/tiddly-gittly/TW5-Typed/blob/master/src/modules/utils/index.d.ts

https://github.com/tiddly-gittly/TW5-Typed/blob/master/src/core.d.ts#L164

I personally would love to get "autocomplete" and "type popups" if I hover a TW function in VSCode

And this will only require JSDoc, no sourcemap/TS reuqired.

Currently I only care about AST node, they change frequent than other methods. Other methods in https://github.com/tiddly-gittly/TW5-Typed haven't break since they are added.

I can add PR to add some type, but #8255 may cause conflict and rework. An example here #8533

[Jermolene]

I've heard that tw core maintainers don't like TS even JSDoc, but the fact is plugin developer need it, I need it, so please consider providing a better experience here

My views are:

• I am not personally enthusiastic for TypeScript for reasons too tedious and arcane to get into here, but I do recognise that others enjoy it, and I would absolutely like for TiddlyWiki to offer a great dev experience for people who choose to write plugins in TS
• I would very much like for TW to adopt JSDoc. Further, I'd like to include a JS parser in the core that understands JSDoc comments and formats them nicely

[CodaCodr]

• I am not personally enthusiastic for TypeScript for reasons too tedious and arcane to get into here

I'm with you. For those unaware of the negatives (sorry, there are many), and are unwilling to spend the hours necessary to do the full-on, deepdive analyses, grab a coffee (or six) and head here:

https://medium.com/javascript-scene/the-typescript-tax-132ff4cb175b

https://medium.com/javascript-scene/the-shocking-secret-about-static-types-514d39bf30a3

https://medium.com/javascript-scene/you-might-not-need-typescript-or-static-types-aa7cb670a77b

[rougsig]

Hi! I’m considering to add JSDoc comments to the TiddlyWiki codebase. Would this change be acceptable to the community?

[pmario]

@rougsig wrote

Hi! I’m considering to add JSDoc comments to the TiddlyWiki codebase. Would this change be acceptable to the community?

I think it would be great, if VSCode would be able to use IntellySense to show all TW function parameters as tooltips.

There are some points we do have to consider:

---

The main problem is, that the core code-size will be increased significantly, which is a problem. As you may know, the TW core code is intentionally not uglyfied.

The main target audience for TW are "standard users" not developers. That's why debugging the TW core code without the need for source maps is essential.

Even modifying parts of the core JS code, by interested non-developers, should be straight forward. That's not possible with uglified core code.

So we would need to implement JSDocs in a way, that does not increase core code size all too much or not at all. The core size is the main driver for the tiddlywiki.html single file size.

---

There is a proof of concept PR at: #8533

As you see in the discussion there. TW implements its own require() function, that makes it possible to use the same code in the server and client, without an additional build-step, that modifies the source code for the client.

IMO @linonetwo found out how to "teach" IntellySense to connect the JSDOC pieces using @typedef {import(' -- So the whole thing will be different to existing 3rd party tooling.

We do not want to add 3rd party dependencies if it can be avoided. TW already implements a "build-step" that parses our TW js-header and converts js-code into proper tiddlers.

/*\
title: $:/core/modules/parsers/wikiparser/rules/codeblock.js
type: application/javascript
module-type: wikirule

TW documentation

\*/

You can see the start of that discussion at: #8533 (review) and following posts.

There is a second discussion going on, which discusses lossless "core compression", which may reduce the core size by the factor of 8 to 10. As @Jermolene mentioned in the linked PR. If we can combine those 2 steps it should be doable.

The core would be compressed by the TW build-step and uncompressed at wiki startup.

[pmario]

Here is the discussion about core compression

• Please GZip $:/core, maybe? #4262

[rougsig]

@pmario

I’ve read through the discussion and think I understand the main points, but I’d like to clarify a few things:

1. Am I correct in understanding that JSDoc is not needed for "standard developers"? Could you please clarify what is meant by "standard developers"?
2. Is it acceptable to add dependencies to devDependencies, similar to how eslint is currently used?
3. Could you please specify where exactly the size of the core is particularly critical? Is it only when TiddlyWiki is packaged as a single HTML file, or is it a concern in other scenarios as well?
4. Would it make sense to start a dedicated discussion about JSDoc, so all related ideas and questions could be organized in one place?

Thank you for your time and insights!

[pmario]

1. Am I correct in understanding that JSDoc is not needed for "standard developers"?

You are right JSDoc is not needed for the TW development at the moment.

1.1 Could you please clarify what is meant by "standard developers"?

I would consider a user that is comfortable with using or figuring out what: git clone, node tiddlywiki.js ./editions/tw5.com --build index and node tiddlywiki.js ./editions/tw5.com-server --listen means is a "standard developer"

2. Is it acceptable to add dependencies to devDependencies, similar to how eslint is currently used?

Yes and no, with a tendency to: We try to avoid dependencies if possible.

Most 3rp party tools include nested dependencies that need constant maintenance. IMO The advantage that a new dependency brings will need to outweigh the disadvantages by at least a 100 to 1. Simply speaking "we can not afford it"

For me personally especially nested dependencies include a horrendously huge attack surface for "supply chain attacks", which will be the No.1 thread in the future.

3. Could you please specify where exactly the size of the core is particularly critical?

For the default usage of the tiddlywiki.html single page app. If you download empty.html you will see, that it is about 2.6MByte in file size.

While that is half the size than most images taken with a modern mobile phone, it still is a significant size. Especially if wikis have to be loaded, if bandwidth is expensive or limited.

In contrast TWclassic empty.html is about 400kByte. If we could be close to that, it would be a huge win. (BTW TWc is built using the TW5 build chain)

3.1 Is it only when TiddlyWiki is packaged as a single HTML file, or is it a concern in other scenarios as well?

Mainly for the single page HTML. A smaller default bandwidth footprint using node-js client-server configuration would be a welcome side effect.

4. Would it make sense to start a dedicated discussion about JSDoc, so all related ideas and questions could be organized in one place?

I think this issue already is about JSDoc. I think you can post your arguments here. Converting this issue into a discussion is still possible in the future.

[rougsig]

You are right JSDoc is not needed for the TW development at the moment.

Yes JSDoc is not needed, but it improve dev-experience, isn't it? Improve code readability, code understandability?

Mainly for the single page HTML. A smaller default bandwidth footprint using node-js client-server configuration would be a welcome side effect.

So, can we add a lot of JSDoc comments to the source code without any issues if we implement compression for a single-page HTML file?

[pmario]

So, can we add a lot of JSDoc comments to the source code without any issues if we implement compression for a single-page HTML file?

From my point of view -- yes, iff JSDoc is human created or human redacted and optimized.

As linonetwo mentioned at the POC PR, the JSDoc is created with an LLM. The quality is so bad, that "no jsdocs" would be better. See my comment: #8533 (comment) and #8533 (comment)

The concept that @linonetwo introduced with the POC PR, seems to be high quality.

[pmario]

Yes JSDoc is not needed, but it improve dev-experience, isn't it? Improve code readability, code understandability?

If done right. -- "dev-experience" with VSCode -- definitely yes.

Code readability -- not really. The syntax needed by the JSDoc parser is distracting and adds mental complexity.

Code understandability -- Once devs know how to "decode" the JSDoc syntax in their head -- yes, if it is placed outside the code itself. Mixing code and JSDocs is distracting.

ATM The main advantage of JSDoc comments is the possibility to use "automatic" processing of type definitions and make them usable.

Automatically create more advanced dev-docs in the future may be a second step. Exporting JSDoc developer documentation and importing it into https://tiddlywiki.com/dev would be great.

[linonetwo]

@rougsig For now you can try PR to my branch, like updating jsdoc to make them high quality. Current TODO is move typescript to build time, so there will be .d.ts file in npm tiddlywiki package, for plugin developers to use, and core developers can optionally not installing typescript.

But I'm writing my games, and we have https://github.com/tiddly-gittly/TW5-Typed as "temporary" workaround, so I don't feel this is urgent for myself.

[rougsig]

@linonetwo Okay got it. I will focus on better UX for editor and theme. Actually i can't understand why we can't do something like obsidian real time inline markup render. Working here https://github.com/rougsig/TiddlyWikiX

[linonetwo]

@rougsig You can search on CPL, you may find https://github.com/tiddly-gittly/slate-write . But it relys on #8258 which also needs attention.