Reduce JS bundle size

[v6ak]

I have a static site generated by Scala.js + sbt-web + HTML generators. It currently uses Play framework for development (not for production). I have tried to replace Play by Vite in order to speed up development and get a newer SASS compiler. The main issue is bundle size, which increased from ~1MiB to ~1.9MiB.

I know this is related to ES modules (required by Vite), which cause the Google Closure Compiler to be disabled. Related project-specific issue: v6ak/zbdb-stats#57

[Lukah0173]

There's some further info in the Scala.js repo:

scala-js/scala-js#4482
scala-js/scala-js#3893

It seems to be a few people considering / waiting on a native solution in the Scala.js toolchain. In the meantime, it's not ideal, but it's possible to use Terser to minimise assets (tested with Vite 4.4.X). Although, be cautious if dynamic imports are used as described in the above issue threads.

build: {
    ...
    minify: 'terser',
    terserOptions: {
        sourceMap: true,
        nameCache: {},
        format: {
            comments: false,
        },
        mangle: {
            properties: {
                debug: false,
                keep_quoted: true,
                reserved: ['$classData', 'main', 'toString', 'constructor', 'length', 'call', 'apply', 'NaN', 'Infinity', 'undefined'],
                regex: /^(Lweb|Lslinky|slinky|render__L|query__L|\$m_|.*__f_|Ljava|cats\$)/,
            }
        }
    },
},

[v6ak]

@Lukah0173 Thanks for linking the relevant issues.

Terser looks cool, I'll probably try it. Does it affect 3rd party JS (non-Scala.js) libraries? I think they deserve a more conservative approach (unless we are sure we can perform some more aggressive optimizations).

[Lukah0173]

Yep, it should also work for the third-party libraries - I'm not sure about external JS that haven't been transpiled by Scala.js. I've tested with the following and no issues, but I don't know enough about the concepts to be certain of safety / effectiveness.

We're using the following dependencies:

      Compile / npmDependencies ++=
        List(
          "@types/d3" -> "7.4.0",
          "@ui5/webcomponents" -> "1.17.0",
          "@ui5/webcomponents-fiori" -> "1.17.0",
          "@ui5/webcomponents-icons" -> "1.17.0",
          "d3" -> "7.4.0",
          "date-fns" -> "2.29.3",
          "date-fns-tz" -> "2.0.0",
          "keycloak-js" -> "19.0.2",
          "simple-statistics" -> "7.7.6",
          "tailwindcss" -> "3.3.0",
          "typescript" -> "4.6.2",
        ))

and, in addition to our source, here's the result of the build:

vite v4.4.11 building for production...
✓ 735 modules transformed.
../dist/index.html                     0.79 kB │ gzip:   0.47 kB
../dist/assets/index-51df973b.css     37.61 kB │ gzip:   6.90 kB
../dist/assets/app-a95d1b34.js     4,375.23 kB │ gzip: 749.36 kB

It only takes a few seconds or so to build, ymmv.

[v6ak]

I have similar list of JS dependencies (bootstrap, chart.js, moment +tz, chartjs-adapter-moment and comma-separated-values). The resulting size is much better, but still by ~330KiB larger than it is with closure compiler:

dist/assets/index-1a9a2c3d.js  1,389.92 kB │ gzip: 293.20 kB │ map: 5,549.54 kB

Also, it doesn't like the moment's style of calling functions:

client/target/scala-3.3.1/zbdb-stats-client-opt/client.js (8132:15) Cannot call a namespace ("$i_moment").

For now, I'll probably use Vite for dev only, keeping sbt-web for production.

[gzm0]

Also, it doesn't like the moment's style of calling functions:

That looks like there is a namespace import in a facade instead of a default import.

[v6ak]

I still believe there must be a way to proceed with Google Closure Compiler, even with few hacks. I believe I've almost reached the result, albeit with some hacks:

1. Produce ES modules with Scala.js
2. Run postprocessing and Closure Compiler: https://gist.github.com/v6ak/ccd0cadb43993afa854769519646ed46
3. Manually rewrite opt.js: replace require with imports. (I don't have a script for that.)
4. Build it with Vite

It seems to produce a sane result with the exception of calls to 3rd-party code. Calls to module's top-level functions seem to be OK, but calls to methods of their objects are mangled.

I've looked how Scala.js configures the Closure Compiler. It seems that the difference is primarily in the externs. It however seems that I cannot easily export them, as they seem to be kept in-memory. But maybe, when I adjust the code, I can get the externs.

[sjrd]

The externs in a tiny part of it. What really matters is that we emit external method/property references as foo["bar"] instead of foo.bar.

[v6ak]

Makes sense, but I wonder why it works with CommonJS modules. IIUC, there should be the same issue.

[sjrd]

For CommonJS, GCC leaves the require(...) calls alone, because they're normal, external function calls. It doesn't want to leave import statements alone, however.

[v6ak]

In my experiment, I've replaced all external JS imports (i.e., those that don't start with "./"*) by require function call before passing to GoCC, so the reason shouldn't be there. The difference might be in the type annotation of the require function. Maybe I need some recursive typedef like Dynamic = Object<String, Dynamic>. *) Well, my hacky regex checks just for the initial dot.

[v6ak]

Aha, I got it. Scala.js uses withOptimizeBracketSelect(false) when using GoCC, so GoCC doesn't mangle the references. With ES module, there is no way to configure optimizeBracketSelect, so GoCC mangles it.

I can play a bit more with that, and hopefully even send a PR. However, I probably can't get rid of the hack (rewriting imports to function calls before passing to GCC and rewriting it back when GCC is done). I just can write it in somewhat cleaner way (using AST transformations instead of regex, using a dedicated scalajs-specific name rather than require). Is this hack acceptable?

[sjrd]

Is this hack acceptable?

Probably not. But if it allows us to reliably use GCC with ES modules, we can at least consider it.

[v6ak]

I hope I can make GoCC reliably working with ES modules. It should work even for multi-module projects (internal modules are inlined). Inlining seems to be a suitable option for many scenarios (server/client/webworker code).

However, my willingness to work on it depends on your willingness to accept such patch. If you say that this hack would be probably unacceptable, I don't want to spend much time on that.

[sjrd]

Inlining seems to be a suitable option for many scenarios (server/client/webworker code).

Well ... it shouldn't contradict what the ModuleSplitStyle of the Scala.js guarantees. In particular in the presence of dynamic js.import() calls.

[v6ak]

Ad ModuleSplitStyle: OK, it sounds like some redesign would be needed for what I want. I'd like to use small modules for fast link and fat modules (=inlined internal modules) for production.

I have some good and bad news:

The bad: I struggle with AST rewriting.

The good: I've mostly succeeded by going some hacky way. It seems that I need to slightly adjust imports of moment.js and BS Modal, but everything else is probably fine. (EDIT: I've done few more adjustments and it works 100%! Also, the bundle size roughly matches the original non-Vite bundle size.) How I did it:

1. I adjusted the SBT plugin for Scala.js: added .withOptimizeBracketSelects(false) to BasicLinkerBackend. I can make it configurable and send a PR.
2. Compiled multiple modules with scalaJSLinkerConfig ~= {_.withModuleKind(ModuleKind.ESModule).withModuleSplitStyle(ModuleSplitStyle.FewestModules)}.
3. Replaced imports by requires (just 3rd party libraries; imports of internal modules were kept).
4. Ran google Closure Compiler.
5. Replaced requires by imports.
6. Ran Vite build

Updated script (it currently does both adjustments automatically): https://gist.github.com/v6ak/ccd0cadb43993afa854769519646ed46

[v6ak]

Thinking a bit more conceptually about it. For most of the parts, I can send a PR if you are interested:

1. Option for disabling optimizeBracketSelects

This allows running GoCC afterwards, outside of Scala.js SBT plugin.

What to do: I can prepare a PR.

2. Module inlining

Maybe it is not the best way to go, as it can produce more code than needed: Let's have independent modules A and B and a class X that is referred from both module A and B. However, method foo is referred just from module A. This might cause some extra code to be generated for module B, as their shared module will probably also contain method foo. Maybe GoCC will eliminate it, but IIRC GoCC isn't perfect (as we could have seen in early Scala.js versions without tree shaking).

What to do: use fat modules instead.

3. Fat modules / separate subprojects

They seem to be essentially supported, just not as ModuleSplitStyle, but one can achieve this by subprojects. However, this plugin doesn't support subprojects.

EDIT: It seems that subprojects are somehow supported as long as you apply the plugin multiple times. This probably increases startup time (as SBT is invoked multiple times), but it should basically work
EDIT: While I can apply the ScalaJS plugin multiple times, it tends to fail, as it runs SBT in parallel.
What to do: I can probably adjust this plugin for multiple subproject support and send a PR.
PR: #16

4. Different ModuleSplitStyle/scalaJSLinkerConfig for fastLink/fullLink

We can introduce some additional option(s) that allow different scalaJSLinkerConfig for fastLink/fullLink. I would have to think about backward compatibility a bit, though, because we are essentially splitting a single option to two separate options.

What to do: I can probably send a PR, hopefully having a backward-compatible solution.
EDIT: already done: https://github.com/scala-js/scala-js/blob/b8fb2f28aff3dac46e35c101550c732bf0dbe562/sbt-plugin/src/main/scala/org/scalajs/sbtplugin/ScalaJSPluginInternal.scala#L468-L474

5. Hacks for Google Closure compiler

I've struggled with doing this as AST transformations, as the GoCC's AST seems to have a very permissive type hierarchy. I believe I can proceed if I dedicate several hours to analysing the issue, but I don't want to do so if you think you most likely will not accept it.

There are however some things we could do instead:

a. If we have an option for disabling optimizeBracketSelects (see point no. 1), I can do this hack outside of the SBT plugin. Not the best solution, as this will not bring this optimisation out-of-box.
b. Configure project to produce CommonJS modules (at least in fullLinkJS, see point no. 4) and add support for CommonJS to Vite (there are some plugins, I haven't tried them).

[v6ak]

Prepared a PR for multiple subprojects: #16

[Lukah0173]

Interesting, I'm following your progress - Let me know if there's anything I can do to help. It's outside my area of focus but I'm eager to see a solution for this 👍

[gzm0]

Note that different configs for fastLink / fullLink have always been possible. In fact, it is how their difference is implemented:

https://github.com/scala-js/scala-js/blob/b8fb2f28aff3dac46e35c101550c732bf0dbe562/sbt-plugin/src/main/scala/org/scalajs/sbtplugin/ScalaJSPluginInternal.scala#L468-L474

[v6ak]

@Lukah0173 Thinking about it again:

1. Separate subprojects (no. 3) or fat modules aren't a basic thing, i.e., in some scenarios, you don't need it. They would be useful for SSG (my case), as I don't want to have the HTML generation / CSV downloading code in the client JS. I've prepared a PR, which fails on Windows, probably due to argument quoting/escaping. If you want to contribute (I currently don't have a Windows VM), you are welcome.
   In the meantime, I am using a sleep-based hack that allows multiple application of the plugin. The sleep prevents the race condition most of the time. If it doesn't, Vite just fails to start and you can try again. EDIT: I seem to sleep on a wrong place, which has no effect. Anyway, the race condition seems to just cause some startup fails.
2. Hacking it yourself: I originally thought we would need an option for disabling optimizeBracketSelects (which forces Scala.js to produce foo["bar"] rather than foo.bar for references of external bar). This is essential if you want to run GoCC on your own (after SBT build), as it prevents some unsound optimizations. However, maybe there is an easier way: we can let Scala.js to produce ES modules in dev and a single CommonJS module in prod. Then, we can configure Vite (probably via build.commonjsOptions.include) to translate CommonJS to ES modules. I haven't tried this way yet (you are welcome to try that) and I the documentation doesn't make it clear for me what to put in the build.commonjsOptions.include (actual path in the FS?). If you wish to try it and report results, you are welcome.
3. Implementing the AST transformations that allow GoCC to be run in Scala.js builds: I've tried and stalled. Since it isn't that likely to be accepted, I don't want to dedicate much time for that. But if you wish to do so, you are welcome (and I can share more about my current progress).

[Lukah0173]

@v6ak just fyi, there may be an official solution in-progress for this:

scala-js/scala-js#4930