OpenWrt (and other minimal embedded environment) support #3473
Comments
|
Hi David. Thanks for your issue, I will try to answer to all your questions.
Everything inside node_modules folder is needed. Apart from that I use this command after building ui and server to cleanup useless files: It essentially keeps only the folders/files that are listed and remove the rest. In-fact if you check what's inside docker you will see: In case of mqttjs (I'm also the maintainer of that library) I can easily use esbuild in order to create a bundle with all the files needed, that's not really easy to do with zwave-js-ui as it also have a webserver and many other things that makes that hard to bundle everything in a single file, paths to things will just be wrong everywhere or less predictable. About compiling it to bytecode, I already do that with pkg, you see the available builds in releases: https://github.com/zwave-js/zwave-js-ui/releases/tag/v9.5.1
Everything is loaded by default, UI, webserver etc... What could help here could be to use env vars to disable the webserver, dunno how much memory that could save BTW...
Not ideal but I let you decide
The
I think yes? @AlCalzone can tell you that
|
|
If I click on the controller in the node map, there's an orange button for 'Open' above the Background RSSI chart. If I click it, I get an empty browser window pointing at Is that because of our packaging? |
|
Oh that could be an issue, I changed the routing to hash based some months ago and I may have missed to fix the path for that specific window |
|
c618fd1 |
Thanks for the prompt response!
On top of what @nxhack already does in the package build, that basically just removing the LICENSE, README.md and public/ directory I think. There's still a lot of duplication under node_modules.
Is there something we should be doing to ensure that npm doesn't install a duplicate copy of mqttjs, and uses the one that's already supposed to be there? (Apologies, some of these are basic questions about node packaging).
That's 230MiB which is more than twice the size of existing package. Is that expected, or is that a bad comparison? I guess this x86_64 executable includes the node runtime itself, and that's probably quite large? Is it possible to package just the bytecode and not a separate copy of the runtime?
Probably depends on whether setting the environment variable causes the whole thing to drop out of visibility and not get loaded by the runtime at all? In an ideal world, we'd be able to make use of the existing webserver that's running on the system. That's not just about runtime footprint, but also about making the pages available from the same public-facing port as the existing user connection to the system. But that's probably a topic for another day.
Hm, so ZWave-JS-UI will download a new copy of the configuration db? And by default that would try to overwrite the original one in the pre-packaged I wonder if we could package the config-db separately in that case, and not ship a copy in the actual zwave-js-ui code package at all? Or even not ship it at all, and download only the information we need on demand? I think for now it's enough just to stop setting
I threw away everything except
Thanks. |
The only way I think is to try using tools like esbuild or else but as said that's not an easy route.
That could be improved a bit by enabling executable zip, anyway that means that on runtime it will unzip his bytecode content somewhere in the device to run it (and the size will be the same). The reason it is so heavy is that it also ships nodejs within it, so no need to install nodejs runtime on the device in that case. I use this tool to package the application and convert the files to bytecode: https://github.com/yao-pkg/pkg (I'm actually a maintainer of that too and there is no available valid alternative to it right now, the future of it will be nodejs SEA but it's definetly far from being usable in a complex application). Anyway pkg application are known to have a little more RAM impact as they store a reference on memory of each file packaged inside it (it creates a virtual filesystem to serve the files) |
|
Yeah, it ends up being a trade-off of "disk" (really flash) storage vs. RAM usage and in that case on this class of device it's often better to use more flash so that you use less RAM. If we can mmap files from the filesystem then the system can drop pages when they're not being accessed, and bring them back again on demand. But if they were decompressed into anonymous pages, there's nothing the system can do (we typically don't even have swap). We typically use a compressed file system too, which means that compressing the file internally doesn't even give much of a win; an uncompressed file doesn't actually take that much more space on the real flash anyway. Compression just makes demand-paging hard for little benefit. |
Aside from the ZUI settings, these are the most important files to preserve forever. Speaking generally, it would be a very poor user experience to require re-interviewing the network anytime ZUI restarts. |
|
It wouldn't be on a ZUI restart, not even a router reboot. Only on a full system upgrade to a new version of the OS (including packages such as ZUI). But yeah, let's start by preserving them even then. |
|
Changing most settings restarts ZUI, the process could crash, etc... EDIT: N/M, I get what you're saying, the files persist until router upgrade. |
|
I suspect the ideal option for a constrained environment would be to ship the code which runs locally as bytecode, but still use the system's node runtime. And for any "files" accessed by it (including those served by the web server) to be just present on the file system (as they are at the moment). Compiling to bytecode would presumably mean that we don't get to "share" certain libraries like mqttjs with other projects that also use them. But that sharing clearly isn't working right now anyway, and using bytecode will probably reduce the memory footprint and possible also the disk footprint too? Is that feasible? |
|
FWIW running with |
|
I think that you may be able to enable those sharing of modules using pnpm in place of npm? Not 100% sure but may be worth a try |
@nxhack I think that suggestion might be applicable to the whole of https://github.com/nxhack/openwrt-node-packages ? |
|
The issue is not the size of the package, but the memory size required by the node.js virtual machine. IMO, we would choose the hardware to run node.js to its full potential. |
|
Some thoughts on the above: @robertsLando this looks to me like
This is used for updating the configuration DB on an existing system without updating the entire stack. Usually this is only possible if you're using an up to date version and there have been only config |
Nope it doesn't it would be too complicated as there could be dinamically loaded files/libraries that would fail to load... Also I'm not sure using esbuild will make the application reliable but I can try... |
|
Thanks. I ran 'npm run bundle' on my x86 host and it seems to run OK for the But here it doesn't find the Should there have been a package.json somewhere in |
|
That's strange, doing the exact same on my end doesn't throw :( |
|
Oh wait I got it, let me try to fix |
|
Ok it has been more painful then expected but it's working now. Try it now @dwmw2 |
Will debug further... first suspicion is the serial port bindings, but it doesn't open the serial port until later, does it? Don't have gdb on this board (yet) but strace suggests it's doing MQTT things. |
|
(What I've done here is just run 'npm run bundle' on the x86 host and then rsync the build/ directory to /usr/lib/node/zwave-js-ui/ on the target). If I give it a different (and empty) STORE_DIR, it does start up OK. However, the HTTP service can't find its files. Even on th x86 host: |
|
Is it looking for its HTTP files within that massive index.js? If I strace on the x86 host, I see it doing this when it gets the request, before reporting the error: That's loading 10MiB of data into memory just to look for a snippet of HTML inside it? :) |
|
Thanks for working on this! I'm still seeing the same thing (load index.js and then give 404 error) with commit dd35bfb though. |
|
Try to rebuild now, I'm not getting that error, are you opening http://localhost:8081 ? |
No no that was just wrong, now it will load index.html from |
|
The UI is loading now, thanks! I see this on the target host which I don't think I've seen before: Still crashes with the 'real' config file. Separately, I tried adding 'minify: true' to esbuild options. Doesn't work but looks like it would be nice to have! |
|
On the x86 host with the same config file it just sits there and does nothing after printing the 'scenes.json not found' line. |
@dwmw2 If you google that error you will see some solutions, that's not something related to the bundle |
|
Ah, the logs are disabled in the config file, so this isn't a good comparison... |
|
It's working fine; just hadn't said anything about listening on port 8091. And couldn't open the serial port since the dongle isn't on the x86 host. Trying again with debug enabled on the target: |
|
LOL segmentation fault 😕 It's hard to debug that... |
|
Happens with MQTT disabled too. Still inclined to blame the prebuilt serial port bindings. I have a native copy of those built for this system. |
|
For sure that's from serialport as the segmentation fault only happens on native code. Did you tried to do a build from source for that? PS: I enabled minification and now the entire bundle is about 5.37MB :) |
|
Trying marking the serial port stuff as external so that it uses the native build. |
That works (/cc @nxhack). Next up... |
|
Hm, that one went away when I enabled logging (for which I had to reinstate the config override directory, which I'd set to "" to make it start up on the x86). I think it's working. Thanks so much for your help! |
|
The minified build fails here with: |
|
I get it on x86 too: |
|
With sourcemap: |
|
@dwmw2 Oh I know why... The solution is to don't do the minification from esbuild but do it after the npm run bundle. This is because I actually have to patch the output file here: Lines 85 to 97 in 330d085 And that find/replace is broken when I do minification. I suggest you to use terser or an online tool for now to minify it I have disabled minification from esbuild options now |
|
@robertsLando This is a problem, since some parts of Z-Wave JS match on the constructor names:
You'll have to enable the |
|
@AlCalzone That's only relevant when doing minification and I don't do that for the reason described above |
|
I have just pushed a change to the PR to also support minification with |
|
That works as of commit 72bdc62, thanks. The minified version is "only" 180MiB instead of 200MiB too, which I think is fairly repeatable. It's still a lot for something that used to be virtually nothing, but it's working. I can use it from Domoticz over MQTT and latency is minimal. |
If you are referencing to virtual memory I cannot do so much about that, I think that comes from all the things we keep in memory expecially zwave-js in its jsonl db that loads everything on RAM |
|
I think this can be closed? @dwmw2 |
|
Yes, I think we can close it; thanks. If we want to revisit ways of reducing memory usage in future that can be done separately. I'll work with @nxhack on OpenWrt packaging. |
|
Not sure if this is expected, but on a completely fresh setup if I just run 'npm run bundle' it says I ran |
Yeah that's expected, you must compile frontend before running npm run bundle command. |
We package Domoticz for the OpenWrt distribution, and have historically used OpenZWave with it. Domoticz is now switching to ZWave-JS-UI so it would be great to package ZWave-JS-UI to work out of the box on OpenWrt too: nxhack/openwrt-node-packages#1826
An always-on embedded router/AP is a perfect place to run home automation, but the environment can be a little constrained in CPU, memory and "disk" storage. For the last few years mine has been running on a MIPS24k with 128MiB of RAM.
We have a proof of concept basically working (not on that box, but on a quad Arm7 with 256MiB of RAM). I'd appreciate some help cleaning it up though, please. There are a few things to look at. Please let me know if I should file separate issues for each.
1. The package is huge — about 100MiB.
Are we doing it wrong? Are there any parts which can be left out? Here are some of the largest files....
Some of those seem like amalgamated or preprocessed versions of the original sources; in some cases we end up with what looks like three copies of the same code. Do we need to ship them all? In the case of MQTT there's a separate package for that anyway, which has the same three files that the zwave-js-ui package is shipping too.
Some help understanding how this stuff works would be much appreciated.
Also, is it possible to compile down to bytecode or even native code, and ship only that to the target system? That might help with...
2. Memory usage
Its virtual memory size is 180MiB. Is there anything which can be done to reduce that? Is the full UI loaded into the interpreter even when only the ZWave←→MQTT bridge is active? Is it possible to run a version with the UI completely removed? It would actually be OK even if the user has to stop the
zwave-js-uiprocess, and restart it with UI enabled.Of course it would be nice to have a separate minimal server, and run the UI on demand (and as much as possible on the browser side), but that's probably a lot more work. Anything we can do to reduce memory usage in a relatively non-intrusive fashion would be useful though.
3. File system usage in
/etc/zwave-js-ui.In OpenWrt we typically use a writable overlay flash file system on top of a read-only root, and have the following categories of storage:
• The initial root filesystem (including the 100MiB package, as discussed above).
• Files written to flash in the overlay, which are preserved across upgrade.
• Files written to flash in the overlay, which are lost across upgrade.
• Files in RAM, lost on every reboot.
We try to reduce the amount written to the flash file system — both in terms of size, and frequency of writes. We also try to minimize the amount that's preserved across system upgrade, as it's packaged into a smallish tarball which is then extracted into the new system after upgrade. Looking through the contents of /etc/zwave-js-ui, here's my idea of what might go where; please let me know if it's sane...
settings.json,users.json,nodes.json,scenes.json: Preserved over sysupgrade.logs/: In RAM. Or perhaps even sent only to syslog and not to files at all.backups/: Flash file system, not preserved.config-db/: Our packaging seems to point$ZWAVEJS_EXTERNAL_CONFIGto this, but I think that's wrong because it makes another copy of everything that's already shipped in the package? We should leave this to be used "sparingly, when custom files are absolutely necessary" as the documentation says. And then it should be in the flash file system and preserved over system upgrades.*.lock: In RAMxxxxxxxx.json,xxxxxxxx.metadata.json,xxxxxxxx.values.json: Flash file system, perhaps not preserved? They can all be recreated, I believe?config/,sessions/,snippets/: Not sure, these are empty for me.The text was updated successfully, but these errors were encountered: