"New UI" tabs not working on Linux?

[kengruven]

I have two installs of OLA "version 0.10.8". In the "New UI" web interface as served by macOS (Homebrew), the tabs work fine. In the "New UI" web interface as served by Linux (Debian), all of the tabs redirect back to the home page.

It doesn't matter which OS or browser I use to view the webpage. It only seems to matter which server I'm looking at.

I'd be happy to perform any tests, to try to diagnose this.

[peternewman]

I think this is a bug where Jquery is missing in the deb due to Debian politics.

Can we have your olad -l 4 logs:
https://www.openlighting.org/ola/get-help/ola-faq/#How_do_I_get_olad_-l_4_logs

But also just check with developer tools if you're getting a 404 downloading that library from our webserver?

[kengruven]

Hmm, doesn't look like it's related to jQuery. Resource "/new/libs/jquery/js/jquery.min.js" gives "HTTP/1.1 200 OK", and the contents look fine. jQuery is definitely included in the Debian OLA package.

[peternewman]

Okay, that's good they've fixed that bug then, I'd sort of lost track.

Can we see some olad -l 4 logs then please, and maybe check your browser console too?

[kengruven]

The level-4 logs are 500+ lines, mostly loading hardware plugins, and PosixInterfacePicker enumerating all my network interfaces multiple times. I tried diff'ing the Mac/Linux logs, and it was just a big mess. I'll see if I can turn off all the unused plugins to get a more minimal test case.

As for the browser side:

• On macOS (where it works), the Universe 1 URL is "http://localhost:9090/new/#/universe/1", and the "Faders" tab has URL "http://localhost:9090/new/#/universe/1/faders"
• On Linux (where it doesn't), the Universe 1 URL is "http://localhost:9090/new/#!/universe/1#%2F", and the "Faders" tab has the same relative link but clicking it takes me to "http://localhost:9090/new/#!/#%2Funiverse%2F1%2Ffaders"

In both cases, the web browser console has no errors or warnings.

[peternewman]

The level-4 logs are 500+ lines, mostly loading hardware plugins, and PosixInterfacePicker enumerating all my network interfaces multiple times. I tried diff'ing the Mac/Linux logs, and it was just a big mess. I'll see if I can turn off all the unused plugins to get a more minimal test case.

I'm quite happy to just trawl through them, I'm used to what to ignore, feel free to hide/delete your NICs if you're worried about that.

I think the log should show the requested URLs which probably would have clarified the below too.

As for the browser side:

• On macOS (where it works), the Universe 1 URL is "http://localhost:9090/new/#/universe/1", and the "Faders" tab has URL "http://localhost:9090/new/#/universe/1/faders"
• On Linux (where it doesn't), the Universe 1 URL is "http://localhost:9090/new/#!/universe/1#%2F", and the "Faders" tab has the same relative link but clicking it takes me to "http://localhost:9090/new/#!/#%2Funiverse%2F1%2Ffaders"

In both cases, the web browser console has no errors or warnings.

So the %2F is a URL encoded /, but there's the exclamation mark too!

Try copying the jquery file(s) from the Mac onto Linux and see if that fixes it (Linux will probably have some symlinks if you're using the deb package). It might be the new version is incompatible.

[kengruven]

It looks like the Mac jquery.min.js is "v2.1.3" (which is the same version that's included in the ola repo), and the Debian jquery.min.js is "v3.5.1" (!). I'll try swapping them later to see if this is in fact the issue.

[kengruven]

The jQuery version makes no difference, but reverting the angular and angular-route libraries which Debian substitutes (1.8.2-2) with the older version included in OLA (1.3.14) fixes it.

I'll file a bug report with Debian.

[kengruven]

Filed: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1016447

I have no idea if it's feasible or desirable to update OLA to use the latest version of Angular, so I'll leave it up to you whether to close this bug against OLA or not.

[kripton]

With https://blog.angular.io/discontinued-long-term-support-for-angularjs-cc066b82e65a in mind, I don't know if it's worth to put much effort into the "new web UI" as it is right now. However, if it's a point we need to fix in order to get our update into the Debian archives, I will work on updating it.
Alternatives:

• Drop the "new Web UI" from ola and simply keep on using the classic one
• Re-write the "new Web UI" using the latest "Angular", which also means converting it from JavaScript to TypeScript
• Re-write the "new Web UI" in another web framework that people are more familiar with (I'm personally quite skilled in React but it could also happen that it falls out of support at some time. The world of "web toolkits" is still moving quite fast)

[FloEdelmann]

Drop the "new Web UI" from ola and simply keep on using the classic one

I would be quite sad about that, as I regularly use the new Web UI instead of the old one. But having to maintain both in parallel seems like a waste of resources anyway.

Re-write the "new Web UI" using the latest "Angular"
Re-write the "new Web UI" in another web framework that people are more familiar with

Given it's a rewrite either way, I'd suggest using Nuxt.js (built on top of Vue.js), since we are already using it at https://github.com/OpenLightingProject/open-fixture-library. It makes writing a web server application quite easy, and requires a standardized folder structure (which is good for new contributors).

[kripton]

I would be quite sad about that, as I regularly use the new Web UI instead of the old one. But having to maintain both in parallel seems like a waste of resources anyway.

I definitely agree that simply dropping it would be quite a loss. Maintaining two in parallel is too much effort to justify it in the long run but the "new" one needs to be stable and future-proof before we decide to drop the old one.

Given it's a rewrite either way, I'd suggest using Nuxt.js (built on top of Vue.js), since we are already using it at https://github.com/OpenLightingProject/open-fixture-library. It makes writing a web server application quite easy, and requires a standardized folder structure (which is good for new contributors).

I certainly don't oppose switching to Nuxt.js for a rewrite (later). I personally don't have experince with it, neither with Vue.js, but it's not that I couldn't learn it and if we have people who are skilled with it it's even better.

For the moment I will try to make the new Web UI work with the "latest" AngularJS, which seems to be the one that Debian ships with (1.8.x). Maybe I can also merge #1739 while working on it but the focus is definitely 1.8 compatibility

[kripton]

Fixed by #1813 and will be solved with next release of 0.10.x. Shall we close this already or wait until the release has happened?

[peternewman]

Drop the "new Web UI" from ola and simply keep on using the classic one

I would be quite sad about that, as I regularly use the new Web UI instead of the old one. But having to maintain both in parallel seems like a waste of resources anyway.

I definitely agree that simply dropping it would be quite a loss. Maintaining two in parallel is too much effort to justify it in the long run but the "new" one needs to be stable and future-proof before we decide to drop the old one.

You seem ideally placed to work on the new one then @FloEdelmann 😉 We'd need the new one to be at feature parity with the old one (see some of #891) before we could drop the old one anyway.

Re-write the "new Web UI" using the latest "Angular"
Re-write the "new Web UI" in another web framework that people are more familiar with

Given it's a rewrite either way, I'd suggest using Nuxt.js (built on top of Vue.js), since we are already using it at https://github.com/OpenLightingProject/open-fixture-library. It makes writing a web server application quite easy, and requires a standardized folder structure (which is good for new contributors).

I'm certainly no expert on these things, but if we went for Nuxt.js, doesn't that double our exposure, as both that AND Vue.js have to survive for us to be able to keep using it?

Maybe I can also merge #1739 while working on it but the focus is definitely 1.8 compatibility

If you fancy getting that PR in too would be great @kripton

Fixed by #1813 and will be solved with next release of 0.10.x. Shall we close this already or wait until the release has happened?

I normally close them once the code has been merged, as there's less to keep track of that way, although it does occasionally lead to someone else opening a new bug if we've not got a release out yet...

Anyway I'll shut this, we've got #891 to track features needed/wanted in the new UI, #1741 to track switching it to Angular (and might be the right issue to hijack to consider switching it to something else.

[yoe]

Sorry for the late response -- I have been waaaay too busy with helping organize FOSDEM and then visiting family in Belgium (I emigrated from Belgium to South Africa a few years ago), and only now can I make some time here.

On Mon, Feb 06, 2023 at 11:54:44AM -0800, kripton wrote: * Re-write the "new Web UI" in another web framework that people are more familiar with (I'm personally quite skilled in React but it could also happen that it falls out of support at some time. The world of "web toolkits" is still moving quite fast)

I would suggest https://vuejs.org/ which works conceptually similar to "old" angular, but is still being maintained. It might not require as much work in porting?

…

-- ***@***.***{be,co.za} wouter@{grep.be,fosdem.org,debian.org} I will have a Tin-Actinium-Potassium mixture, thanks.

[kripton]

Sorry for the late response -- I have been waaaay too busy with helping organize FOSDEM and then visiting family in Belgium (I emigrated from Belgium to South Africa a few years ago), and only now can I make some time here.

No problem :) A "late" response is better than none and we all work asynchronously ;)

I would suggest https://vuejs.org/ which works conceptually similar to "old" angular, but is still being maintained. It might not require as much work in porting?

I'm currently working my way through the Nuxt.js tuorials (which is based on Vue 3). Hearing that it's conceptually similar to AngularJS is a positive thing, we'll see how well I cope with it ;)

[kengruven]

FYI, the OLA 0.10.9 package in Debian has fixed the Angular bug, but there's still a similar Bootstrap bug.

The lesson I get from this is: as long as there's a default UI and an other UI, distributions are going to break the latter.

[peternewman]

FYI, the OLA 0.10.9 package in Debian has fixed the Angular bug, but there's still a similar Bootstrap bug.

Thanks for letting us know @kengruven . I see this is being tracked downstream as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1032775 .

This one is over to @yoe as it's not a commit in our repo:
https://salsa.debian.org/wouter/ola/-/commit/175a46c5e0f8bdadc26a35597711159ebbfeff6d

I assume from your report on the Debian tracker that it works fine when you copy their (later?) Bootstrap over, so we don't necessarily need to even fix anything our side?

The lesson I get from this is: as long as there's a default UI and an other UI, distributions are going to break the latter.

I think it's more don't use UI libraries, as Debian seems to want to strip them out which unfortunately often causes unexpected issues.

[DaAwesomeP]

I think it's more don't use UI libraries, as Debian seems to want to strip them out which unfortunately often causes unexpected issues.

Is the issue that they are insisting on replacing bundled web dependencies with apt repo ones? Surely this can't be the only project with that issue; there is no way to completely replicate all of NPM in apt. This also doesn't make sense from a version-dependency resolving standpoint: If we upgraded to Bootstrap 5 (libjs-bootstrap5 is not available in Bullseye), would they choose to drop OLA in Debian Bullseye instead of using the bundled version? Web dependencies typically move a lot faster than binary dependencies.

I think switching to Vue or a newer web framework will probably require using Webpack or some other similar build system that would make it both really difficult and simply impractical to ever use distro dependencies. Webpack and the likes will compile everything into concatenated bundles for CSS and JS during the build process.

I wonder if there is a way for us to influence how the downstream Debian package is built with our upstream package. For example, if we add our own minification or concatenation using bundled dependencies during the build process, then we would effectively remove the ability to use distro runtime JS and CSS dependencies.

But maybe it's not worth doing that manually if we are already moving to Vue in later releases.

An unrelated workaround: @kengruven feel free to try out one of the Debian builds from my CI Debian build pull request #1812. I haven't actually tested them so they are a bit experimental, but if they work that would be awesome: https://github.com/OpenLightingProject/ola/actions/runs/4394335530

[kengruven]

I'm not a Debian developer, or an OLA developer, so I speak with zero authority here.

If we upgraded to Bootstrap 5 (libjs-bootstrap5 is not available in Bullseye), would they choose to drop OLA in Debian Bullseye instead of using the bundled version?

Bootstrap 5 is in "bookworm" (the next Debian release), which appears to be planned for release later this year, so I think the worst case scenario would be that Debian/stable would be stuck with the last pre-Bootstrap-5 version of OLA for half a year. That's the same situation as any other program (web or native) whose upstream has upgraded to newer versions of libraries than exist in a particular release of a stable Linux distro.

OLA is only at version 0.10.x, so I think it's generally understood that developers who need the latest code should pull it from the upstream repo themselves. I'm totally fine with the Debian "stable" package being slightly out-of-date, as long as their packaging process doesn't make it any worse than stock. :-)

Debian also has guidelines and tools (like npm2deb) for converting Node packages into Debian packages, or bundling them with the program that needs them. I don't think there's any fear that if OLA chooses to use JS library XYZ that Debian would drop OLA. So far, the OLA maintainer has been pretty responsive (e.g., pushed a fix less than 3 days after I reported this bug).

I think it's more don't use UI libraries, as Debian seems to want to strip them out which unfortunately often causes unexpected issues.

A thought: how difficult would it be to make a hard pass/fail test for web resources? If OLA had a test script which did "fetch localhost:9090/new/ and make sure none of the resources give a 404", then distributors could add that to their packaging process and make sure it passed, no matter what libraries OLA wanted to use.

[DaAwesomeP]

A thought: how difficult would it be to make a hard pass/fail test for web resources?

I like this a lot! I have the Debian build tests working 100% in #1812, so we can work on that once that gets merged.

We would probably need to either:

1. Use an actual frontend test framework capable of doing this:
   • Probably difficult for package maintainers since the test bits are not included in the package.
   • Frontend tests are easy for Vue but I am unsure about actually loading the site checking for resources, as they are usually just JS tests.
2. Self-checking JavaScript:
   • Write some quick JavaScript that simply places a comment <!-- resources ok --> just before </html>. Then we can use a terminal-based web browser that supports JS like edbrowse (in Debian apt repo) or browsh, save the resulting HTML to a file, then check for the comment.
3. Use a browser testing method available in distro package repos.
   • Use python3-selenium + chromium-driver in a simple Python script. I have used selenium before, and it should work great. I believe that we can interact with the web inspector, and we can definitely inject JS/run JS commands that can check for the assets. I can add this to my list of things to do.

[DaAwesomeP]

Also with Selenium we can write more thorough tests like:

• Dragging slider actually adjusts OLA DMX output value
• Checking box and hitting save actually patches universes
• Updating values are shown

And basically anything else that we want. Though many of these may be better served by a proper frontend test suite (which can run in CI but not necessarily by package maintainers) and aren't necessary to test during packaging.

[kripton]

I'm probably a bit late to the party and also pretty brief, but:

• Indeed, "current" web frameworks usually create bundles where all libraries and CSS that come in are merged into a small set of files. This usually makes sense for web pages that are "loaded over the internet" since it reduces the number of request/response ping-pong. It might not have big advantages when accessing the web page on the same machine. For such bundles, Debian would not be able to unbundle/replace anything and we could make sure that what we ship is always what gets installed/served
• I don't know the Debian policies, but do they require "bundled javascript frameworks" to be unbundled? I would understand the reasoning behind it but would they deny a package that insists on shipping its bundled libs?
• Shipping "pre-built web page bundles" really is a double-edged sword .... On the one hand, the "source code archive" that makes up an OLA release is expected to contain no "compiled" "binaries", just the source code. I would view the "pre-built web page bundle" as such. On the other hand, we do already ship compiled and minified JavaScript and the C/C++ toolchain + library dependencies are modelled and handled by distributions quite well. For the NPM universe, this is not the case and it would be unrealistic to package and request 200 NPM packages as dependencies for OLA. It's also unrealistic to just define them in the package.json and expect them to be downloaded & installed "at build time" when creating the "binary package" (.deb-file) since some distributions (such as Gentoo Linux) run the build process in a "network sandbox" (and do so for very good reasons). In such a case, "building OLA" would simply fails since it would be unable to download the new UI dependencies via NPM.
• Regarding a test while packaging: I would see this as some for of tests that should be run with make check. However, I don't know if we can/should that an instance of ola for that. As far as I know, the "regular" tests don't need to run an instance (please correct me if wrong) and for Java we do have a very small set of "offline" tests that run by default and some "online" tests that require an instance of ola running but they are not executed by default
• I agree that a some "real" "new UI" tests should be developed, I would propose Cypress for that. Yes, you can have a slider moved and the result verified with that. It can even record videos of the tests and one can watch them later to see why they failed. However, let me argue that implementing such tests doesn't make sense for the current "new UI" if the plan is to replace it with something else.

All that being said, my proposal would be to leave everything in the "new" folder of the Web UI as it is and not replace it with content from apt packages. It's prone to break stuff. If this violates a Debian policy, we can either be nice and talk or be a bit rude and make it a "bundled package" without any way to replace libraries at all. If they would exclude OLA in such a case, go back to the "be nice" step ;)

[DaAwesomeP]

I don't know the Debian policies, but do they require "bundled javascript frameworks" to be unbundled?

I think this is the main thing we should probably figure out before discussing further so as not to venture too far into hypothetical solutions.

It's also unrealistic to just define them in the package.json and expect them to be downloaded & installed "at build time" when creating the "binary package" (.deb-file) since some distributions (such as Gentoo Linux) run the build process in a "network sandbox" (and do so for very good reasons).

In my opinion this is an issue for the distro maintainers to solve. Building the web part of the package is a valid step for building the package. Retrieving the required dependencies is also a valid build step. I don't see it that far off as requiring a bunch of header files on a system before running a C/C++ compiler. They could provide the OLA node_modules as a single source package in its own directory.

Regarding a test while packaging: I would see this as some for of tests that should be run with make check.

make check is not running during the current Debian packaging tests. The current packaging tests really just check that everything is installed and accessible. These packaging tests effectively install the package first, so you don't necessarily have access to the full source tree, although maybe that is possible. But yes, there would be some crossover.

[kengruven]

My thoughts, as a user:

I don't know the Debian policies, but do they require "bundled javascript frameworks" to be unbundled?

Here are some guidelines I found. It looks like the answer could be anything from "sure, keep them bundled" to "don't include it in stable". You should loop in the Debian maintainer and/or the Debian JS team, if this is a concern, because at this point we're just guessing.

Debian would not be able to unbundle/replace anything

I think people are underestimating the lengths that Debian will go to patch software. :-) I can't imagine any way in which some common JS libraries would be used which couldn't be repacked, if it was decided that was the policy.

Another crazy thought: does the web interface(s) need to be the same project/package as the daemon? Going forward, if the plan is for the web interface to use "200 NPM packages" which constantly need updating, maybe that's not a great fit for stable Linux distros -- while OLA as a hardware interface still is. These seem like distinct types of software to me. Or maybe that would just make things worse. Or maybe I'm the only Debian user and the other 99.99% of OLA users build from source for their RPi. I don't know.

[yoe]

Sorry for being late to the party here, I missed this.

I think it's more don't use UI libraries, as Debian seems to want to strip them out which unfortunately often causes unexpected issues.

Is the issue that they are insisting on replacing bundled web dependencies with apt repo ones?

The specific problem is that Debian policies require that no "binaries" are shipped without source. In this context, minified Javascript counts as a binary (which makes sense, as nobody in their right mind would want to edit one of those). Since the "binaries" for the libraries used by the "new" ui are shipped but the sources aren't, I needed to remove them from the package. If that were remedied (together with the scripts and instructions to build the "binaries" from the source), then I could stop doing that.

Switching to JavaScript libraries that require transpilation is generally fine, although if it's a large framework with many dependencies that are not yet packaged for Debian, this could be problematic as my time for adding and maintaining such libraries properly in Debian is somewhat limited (but definitely I can look at adding one or two). Worst case, I think I can get away with adding instructions on how to build the JavaScript parts in a README.source, but I'd have to check.

At any rate, packaged Javascript libraries are installed on a Debian system in such a way that they can be used as a "local" repository by npm, and so building against them is feasible, but only against those packages that are in Debian at the version that is found in Debian.

[yoe]

make check is not running during the current Debian packaging tests. The current packaging tests really just check that everything is installed and accessible. These packaging tests effectively install the package first, so you don't necessarily have access to the full source tree, although maybe that is possible. But yes, there would be some crossover.

If by 'the Debian packaging tests' you talk about the autopkgtest, then yes you do have access to the source tree. The autopkgtest gets executed in an unpacked (and, by default, also not built, although requesting a build is possible) version of the source package. The test is indeed supposed to test the installed system, and can do whatever is necessary to test that the installed packages work fine. Doing browser tests there sounds like a great idea.

Another crazy thought: does the web interface(s) need to be the same project/package as the daemon? Going forward, if the plan is for the web interface to use "200 NPM packages" which constantly need updating, maybe that's not a great fit for stable Linux distros -- while OLA as a hardware interface still is. These seem like distinct types of software to me. Or maybe that would just make things worse. Or maybe I'm the only Debian user and the other 99.99% of OLA users build from source for their RPi. I don't know.

I don't think this is such a crazy idea. Many projects do this, and it does tend to sometimes simplify certain things. There are some downsides, mainly around API coupling (with the current setup, you know the version of the UI you are running and can make incompatible changes if need be, something which you lose if you ship the UI separately), but it would mean that the "new new" UI can be written and maintained separately, with multiple releases for people to test in between ola releases, etc.

[kengruven]

The "New UI" is now working in Debian (bookworm/sid), so I'm going to close this.

I applaud the goal wanting to make packaging easier for distros, but I fear there's a lot of second-guessing here about what Debian maintainers may do, or how they will interpret their own guidelines. IMHO, the priority should be to pick (one) way forward, and commit to it. As long as it's not totally crazy, distros who want to include OLA will figure out how to package it.

Thanks for all the work on this, everyone.