-
-
Notifications
You must be signed in to change notification settings - Fork 144
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Wayland Support #18
Comments
Refer also to VirtualGL/virtualgl#10 |
After experimenting with Weston under Fedora 25, it seems that my assertion regarding GPU access was incorrect. Wayland GL applications seem to work using a similar paradigm to GLX applications. They obtain a Wayland display handle by calling
When running on the "root" display (
(NOTE: Cannot use nVidia's proprietary drivers yet, because they don't yet have Wayland support) When launching Weston on the root display, which causes it to use the Wayland back end, the GL applications report the same OpenGL renderer and vendor. When launching Weston using the RDP or headless back end, then the GL applications report the following:
Having "VMWare, Inc." as the vendor string is a bit suspicious and immediately brought to mind the fact that VMWare uses Gallium to forward OpenGL calls from the guest O/S to the host O/S. I thought at first that Weston might be doing likewise, but the abysmal performance of weston-gears with the RDP and headless back ends (8-9 fps) would seem to indicate that it is truly using software rendering (or is it forwarding the OpenGL to the client?! It just occurred to me that that might explain why the RDP compositor didn't work with Microsoft's RDP client-- it required the FreeRDP client.) This would suggest that one of the following is likely going to be necessary in order to achieve a multi-user remote display solution with server-side hardware-accelerated 3D rendering under Wayland/Weston:
As far as implementing a TurboVNC server on top of this, it seems like much of the existing VNC server code, minus the parts that touch X.org, could be refactored into a Weston back end. It wouldn't be "easy" per se, but it would be at least straightforward. There are a lot of unanswered questions, though, such as how to start window managers in that environment, etc. But it seems as though the basic paradigm of a Wayland server, in terms of how it handles multiple displays, etc., is similar enough to an X server that the existing X proxy architecture could be adapted to it. |
The Wayland developers seem to think that Approach 2 is viable. The thread starts here: https://lists.freedesktop.org/archives/wayland-devel/2016-December/032275.html I need to do some legwork to figure out how difficult of a project it will be. |
After further discussions with the Wayland developers (https://lists.freedesktop.org/archives/wayland-devel/2017-February/033209.html) and poring over the code for the Weston backends (specifically the Wayland and RDP backends), it does indeed seem possible to implement Option 2 as described above, but it's not going to be an easy hack. Some of the challenges with this approach include:
At the moment, I don't think it makes much sense to pursue Option 2, because of the following:
There are definitely advantages to Option 2. It would eliminate the need for a vglrun-like wrapper script. It would potentially be much easier to integrate with NVENC. It would automatically run compositing window managers automatically. It wouldn't experience some of the esoteric issues that are sometimes encountered when using |
Hello. |
Not really. The Wayland EGL interposer has a project dependency within VirtualGL (VirtualGL/virtualgl#10, accessing the GPU through EGL rather than GLX and thus eliminating the need for a 3D X server), and I intend to implement that dependency this year. After that, implementing a Wayland EGL interposer for testing purposes would probably be a simple matter. However, I also saw where the Wayland developers just introduced a set of patches for adding experimental hardware-accelerated OpenGL support to the RDP back end, so Option 2 may bear fruit before the EGL interposer (Option 1) does. From a product point of view, none of this will become a major concern until Wayland makes its way into one of the enterprise Linux distributions (RHEL or Ubuntu LTS) and until nVidia's EGL implementation fully supports it. So anything I did at the moment would be purely for testing purposes. |
@dcommander Ubuntu artful (17.10) is now using Wayland and it's just a matter of time till it gets into bionic (18.04), which is a LTS version so time is coming |
Yes, at the moment I'm still waiting to see how nVidia and Red Hat support it, because that's going to be the driver for high-end 3D applications. |
Well, 18.04 has arrived already and Xorg still remains by default |
Well it at least seems that the GNOME/Wayland developers are working on this |
RHEL 8 will perhaps be a better litmus test, since they have already switched to Wayland by default in Fedora. To the best of my understanding, a big limiting factor here is still nVidia driver support. At least the last time I checked, you couldn't use Wayland with the nVidia proprietary drivers. You had to use nouveau. |
RHEL 8 Beta has arrived (three months ago): https://developers.redhat.com/blog/2018/11/15/red-hat-enterprise-linux-8-beta-is-here/ |
Yes, I know. I'm looking at it now. |
Hello. |
Right now, I am focused on intermediate steps, such as developing an EGL back end for VirtualGL and adding systemd support to TurboVNC, that will pave the way for Wayland support in the long term. Ultimately this feature will become high-priority when commercial 3D application vendors start adopting Wayland and those applications stop working in TurboVNC, but that shows no signs of happening anytime soon. Scientific/technical computing ISVs and users tend to be very slow adopters, and they will probably continue to use X11 as long as they are able to use X11. I predict that this will become an issue as soon as GUI frameworks like Qt and GTK, and the window managers built upon those frameworks, stop supporting X11, but I have no idea when that will happen. Large enterprises are only just now switching from RHEL 6 to RHEL 7, and they will have to switch to RHEL 8 before they can even run Wayland applications. From my point of view, Wayland is still a moving target. The underlying assumption was that I could base a TurboVNC Server implementation around a common Wayland compositor code base that would serve the same purpose that the X.org code base currently serves in the TurboVNC Server. However, that's not really how Wayland works. I would need to base a hypothetical Wayland TurboVNC Server implementation on Weston, and Weston is undergoing such rapid and bleeding-edge development right now that I can't even build that code base on RHEL 8 unless I check out version 6.0.1, which is nearly a year old. Thus, another factor that probably needs to converge here is Red Hat adopting Weston in some capacity, so an "enterprise stable" version of that code base could serve as the basis for a TurboVNC Server implementation. The needs of large-scale enterprises have primarily driven the development of TurboVNC and VirtualGL over the years. One of my enterprise customers has identified Wayland support as a long-term goal, but it's still low-priority at the moment. They have a lot of other long-term goals, and because funding is limited, we're focusing on projects that have the most "bang for the buck" in the short term. Wayland is, like other technologies we're looking at for long-term deployment (WebAssembly, for instance), still in the "wait and see" phase. Some people are using it, but most of the people who have most of the money aren't (yet). I'm also fighting a constant battle to maintain TurboVNC's niche, given that other products have been aggressively going after my customer base for some years, oblivious to the fact that-- if TurboVNC goes away-- so does VirtualGL (which those other products rely upon.) So I have enough to deal with just trying to remain afloat-- both strategically and financially. I can't spend much time right now looking at speculative stuff. |
It would be nice to have a wayland native vncviewer. We have a server now for wlroots based compositors: https://github.com/any1/wayvnc. I was going to see if I could add it myself, but all this Java stuff is a bit too much to handle. Remmina works, but it's pretty slow compared to TurboVNC which seems to be "the fastest VNC client in the west", even via xwayland. Running via XWayland tends to be glitchy and the GUI doesn't integrate well with sway. |
Weston supports now HW accelerated headless mode with: Wayvnc doesn't work with Weston:
(IMHO Weston is nicest compositor for Wayland development because it can run in a window on another Wayland or X11 display, which makes debugging things much easier.) |
I fail to see how any of this is relevant to this "issue". Please, excuse me for feeding into off-topic discussion, but I'd like to set a few things straight...
Sway and other wlroots based compositors have had this for a long time
Wayvnc is a VNC server for wlroots based compositors. It states so in the first sentence in the first paragraph of the README on the landing page of the github project, and probably other places as well. Weston is not wlroots based.
You can also do this with Sway. |
Great, thanks! (I hadn't found any mentions of Sway support for HW accelerated offscreen, or running in a window with quick Google search, they aren't mentioned in Sway Fedora manual page, nor in its README.md or Wiki: https://github.com/swaywm/sway/wiki) I see now that I should have checked docs for wlroots underlying it instead: https://github.com/swaywm/wlroots/blob/master/docs/env_vars.md Using options from there, Sway & WayVNC (in Fedora 33) indeed work fine in HW accelerated headless mode! |
The fact that there are multiple competing compositors just underscores my opinion that this is a rapidly-evolving field with insufficient technological convergence to be actionable from TurboVNC's point of view. VNC itself wasn't even invented until X Windows was 15 years old and had achieved broad acceptance and convergence. That tends to be the way of things-- you can't really build upon a piece of infrastructure that's a moving target. It is nice to know, however, that the concept has been proven. Perhaps it would be fruitful to look at accelerating NeatVNC, upon which WayVNC is based, using the TurboVNC encoding methods. |
Well, the interface for creating a windows and submitting buffers to them is pretty stable and Wayland support is implemented in Gtk, Qt, SDL, glfw and more. A regular window application that works on one Wayland compositor will work on the others too. Some protocols which are relevant to VNC servers that have not been agreed upon between different compositors are:
Protocols that are missing:
As you've already mentioned (5 years ago), h.264 encoding would lend itself pretty well to Wayland. Doing hardware encoding would be pretty ideal because it means that we can leave the buffers on the GPU until they've been encoded. This is not yet implemented in NeatVNC. We'll need to implement it for some clients as well. |
As an independent software developer, I have to go where the money is, and the industries that financially sponsor most of my work on this project are always at the tail end of the curve, not the front end. This will only become a high priority for them (and, by extension, me) when commercial applications (and the GUI frameworks on which they are built) stop supporting X11 altogether. As long as Qt, GTK, etc. have dual X11 and Wayland support, applications that use those frameworks should continue to work with TurboVNC, even if they use Wayland on the local display. Commercial ISVs cannot, at this point, choose to support only Wayland, because there are still enterprise/LTS Linux distributions in active support that don't have Wayland capabilities at all. That's why I feel that things will begin to move on this front once large enterprises start moving from RHEL 7 to RHEL 8, which won't happen for a few years. Referring to #19, there are potential ways that VirtualGL might be able to hand off a GPU buffer to the TurboVNC Server for encoding, but I agree that GPU-based encoding does lend itself more to Wayland than to a virtual X server. |
Intel added support for lossless render buffer compression (RBC) in GEN9 (Skylake etc), and support for having it compatible between media and 3D pipelines in later generations. Nvidia HW has supported RBC longer, but I'm not sure how well its drivers support Linux buffer modifiers [1], like Mesa does. And I don't whether there are any issues with the Mesa AMD and ARM GPU support, or how well their HW supports RBC. [1] In the latest X server release, user needs to enable (buffer) modifier support with X config ("dmabuf_capable") debug flag, whereas it's enabled by default in the X server Git version. [2] When there's no modifier support, GPU driver needs to resolve (uncompress) compressed render buffers when they're passed to another process (= do extra blit where write is uncompressed). Especially for content with large areas of uniform color, like is case with many desktop applications, GPUs' lossless compression could provide significant memory bandwidth improvement. At least in Weston case, there's been no need for that extra resolve, since support for modifiers was initially added to kernel+Mesa in late 2017 [2], but I guess it's the same also with other Wayland compositors? [2] X server modifiers support got to working state for Intel HW around May 2018, just when the last X release was done, but I guess support was still buggy for other GPUs, and that's why it wasn't enabled by default. :-/ |
@eero-t I don't understand how that's relevant to the topic at hand. Can you explain? |
I don't know enough of VirtGL and its integration with TurboVNC to comment on that, but AFAIK application window buffers can be accessed bandwidth-wise as optimally with X server as with Wayland, for potential video/JPEG compression [1] done on GPU. However, as I commented, one needs to use either X server git version, or enable dmabuf_capable debug option, whereas with Wayland things work already in released versions. What I don't know, is whether media drivers allow passing (3D HW compressed) buffers with modifiers to them yet. [1] Besides normal video formats, Intel media driver supports JPEG compression done on GPU (from GEN9 onwards), maybe other GPUs support that too. (Hm. Maybe this would have been more relevant for #19.) |
VirtualGL currently integrates with TurboVNC (and most other X proxies, for that matter) via the MIT-SHM X extension. Enabling GPU-based compression would require either:
In general, it seems like a less painful way to accomplish GPU-based compression would be to keep the TurboVNC framebuffer in GPU memory rather than CPU memory, but I don't have a sufficient understanding of how X.org interacts with physical hardware to be able to say whether that's even possible (in brief discussions with nVidia, they seemed to think that it wasn't possible with their driver, which is used in the vast majority of commercial VirtualGL+TurboVNC deployments.) If it were possible to get GPU acceleration in TurboVNC that way, then VirtualGL wouldn't even be necessary, and TurboVNC could choose whether to use the CPU or GPU for compression. Perhaps TurboVNC could be based on XWayland and use the Wayland EGL back end? The issue of how to implement GPU-based compression or VirtualGL-less 3D acceleration in TurboVNC isn't specific to Wayland, but it relates to Wayland in the sense that I can't envision how to make it happen without invoking Wayland somehow. But again, I confess to not having a thorough understanding of this stuff. |
I haven't used it myself, but I think DMA-Buf and interfaces built on top of that are what needs to be used for sharing that data:
(Nvidia proprietary blob probably needs separate support.) |
That is all Greek to me at the moment. I need to start with a more high-level understanding of how the various components interconnect. |
DMA-Buf buffers are represented by an FD, which can be passed around, see e.g: |
I'd be interested in sponsoring a developer to implement this feature... I'm working on a software project that will depend on remote execution of graphical applications for display with Wayland (streaming the UIs of individual applications, rather than full desktops). No accelerated 3D rendering would be necessary, at least initially. Please shoot me an e-mail (address in GitHub bio) if you might be able and available to work on this. |
Discussions in the KasmVNC (kasmtech/KasmVNC#193) and xrdp (neutrinolabs/xrdp#2637) communities suggest that this is way more complicated than I thought. In particular, the concept of a single Wayland VNC server that can accommodate multiple types of window managers (GNOME, KDE, wlroots, etc.) is likely a lost cause, since the window manager is tied to the compositor at the hip. A more appealing concept might be to design a Wayland compositor from the ground up that has remote display and remote window management (seamless windows) built in and uses an as-yet-to-be-defined legacy-free streaming protocol designed from the ground up to accommodate Wayland. However, that is obviously a huge lift. A more reasonable short-term approach might be to facilitate (via. a VNC server library) incorporating TurboVNC and TigerVNC's technology into various Wayland compositors. |
Referring again to kasmtech/KasmVNC#193, there would be licensing issues with incorporating a GPL-licensed VNC server library into some compositors, as well as incorporating TurboVNC's or TigerVNC's code into neatvnc. Also, a seamless window mode seems less feasible the more I dig into it. A more sane approach might be simply to extend LibVNCServer with the security and performance enhancements from TurboVNC, since the LibVNCServer and TurboVNC Server code bases are similar. That may give us a GNOME/Wayland VNC server for free, since GNOME's remote desktop feature uses LibVNCServer. However, in order to get the full complement of TurboVNC features, including session management, I anticipate the need to build a standalone Wayland TurboVNC server that uses the xdg-desktop-portal extension. tl;dr: Seamless windows don't seem feasible, but the idea of a Wayland TurboVNC Server does. It's going to be a very large effort, though. |
My (admittedly limited) understanding of Wayland is that it is essentially frame-based and image-based from the get-go. All of the drawing and rendering takes place within the application, and the application then tells the Wayland compositor, "Here's a frame. Display it in this window." This is a very natural fit for VNC, since a VNC server could simply be a Wayland compositor that processes each frame from an application and converts it into a single RFB framebuffer update. It would greatly simplify the VNC server, since it would no longer be necessary for it to hook into low-level drawing primitives or to convert the inherently single-buffered, fine-grained drawing approach used by X11 into a more "frame-like", coarse-grained, image-based approach suitable for remote interaction.
At first glance, it also seems like this could provide a straightforward way of supporting:
The text was updated successfully, but these errors were encountered: