GOG file hashing

[halgari]

This PR adds the following:

• it's now possible to log into GOG
• via a CLI command we can export product info about a game, including hashes
• we're able to get the build, depot and hash values
• also possible to get a seekable read-only stream to these files
• in order to facilitate this, we add WebView to the app, and setup a system for sending OAuth requests to this in-app web browser.

So far I'm pretty happy with how clean the CEF integration is here, there's a lot of benefit for having an embedded browser for manual file downloads and browsing Nexus Mods. But we'll see what the integration on Linux and macOS looks.

[Sewer56]

CEF integration is here, there's a lot of benefit for having an embedded browser for manual file downloads and browsing Nexus Mods.

[erri120]

Will review after the holiday. Got a lot to say about embedded browser.

[halgari]

I understand that CEF integration is something a lot of people have an opinion about, but having an in-app browser is something we will want sooner or later. Especially when it comes to manual downloads for collections.

As far as discussion of this point goes, file sizes are not a contributing factor to the discussion. Increasing the download size of the app by 100MB for something that manages 100GB of data is not relevant. I'm also considering "I don't want code by google on my machine" or other such political rationale to be irellevant to the discussion. Aside from those two, I'm happy to hear input on the subject.

[erri120]

Developer Issues

There is a risk when it comes to adding dependencies to a project. Most of the time, that risk is negligible, other times the risk is greater but the dependency is a necessity. Avalonia would fall into the latter, we absolutely need it to build a cross-platform C# graphical application, and most of codebase is interacting with it. Any changes on their side will affect us, whether those are features or bugs. However, we can easily look at the source code, debug issues in our IDE, and even fix issues and create PRs. We can do all of those things rather quickly because Avalonia is a another large C# codebase with some native interops. Obviously, if there are any issues with the native interops, the previous statements won't apply, as seen in RocksDB:

I recently tried to fix an issue where RocksDB had an issue opening a database file (#2394). We own our database code, so that's fine, but that code depends on a C# library that exposes and depends on native RocksDB code. Since the error came from native code, the C# stacktrace was essentially useless. I couldn't use my IDE to debug the issue, and it took me a long time to even find the source code that was creating the error. Debugging the native code was out of the question, I could only guess how the code interacted and try to create a stacktrace by following method usages.

Both Avalonia and RocksDB are dependencies with a higher risk than others, but both are essential for our application to work. An embedded browser combines the bad aspects of both Avalonia and RocksDB: it's a very large codebase of native code that has tons of platform-specific native code. Browsers are some of the largest and most complex codebases of userland programs in existence. There are many technologies used on the internet and the browser needs to support all of them. Everything from the networking, to the content parsing, rendering, and the interactivity is immensely complex.

The maintenance of an embedded browser can be a significant burden. Browsers frequently release updates for security and feature enhancements. If chromium updates, we'd have to wait for CEF to update, and then for the C# interop library to update. If we embed a browser the amount of development time spent on debugging issues related to it will be disproportionately high.

All of this is also assuming that the embedded browser even works on all of our supported platforms in the first place. We've had a ton of issues on Wabbajack regarding the WebView we're using. And that was a C# Windows-only project using a Microsoft made Windows-only OS-level supported WebView. Literally the best-case scenario for distributing a native application that also needs to render web content and it still causes issues for many users.

From a development standpoint, embedding a browser and taking it on as a non-essential dependency while valid alternatives exist is something I oppose greatly.

User Issues

Aside from the issues us developers will face if we embed a browser, the user experience is impacted as well. If there is one thing I can guarantee, it's that the user already has a browser installed. That should be undisputed, the browser is the most used application. It should be no surprise that many users also customize their browser. An embedded browser can't be customized. This means no accessibility features like changing the font type, font size, colors, or using screen reader, a dark reader, or any translation services. You can't install extensions into an embedded browser, so you can forget using a password manager or an adblocker. Browsing the internet without an adblocker is a huge security risk and a very miserable experience. Any other security, privacy, and networking settings like proxy, DNS, cookie, fingerprinting, HTTPS-Only settings also can't be changed by the user.

My favorite issues are infinite captcha locks combined with infinite redirects. Anti-bot technology is used everywhere and they are very trigger-happy with embedded and headless browsers. I've had multiple situations in embedded browsers where captchas would never resolve and where cloudflare redirects were stuck in a loop. Embedded browsers also sometimes can't interact with the clipboard because the parent application didn't set it up, so copy-pasting passwords also doesn't work in some cases. Newer websites using the latest web standards are also prone to not render correctly, websites requiring WASM or a WebGPU don't work in embedded browsers.

Embedding a browser means shipping with the compiled native code which can be anything between 100MB and 1GB in size, since the majority of applications nowadays are Electron trash, this is negligible and nothing new to users. Finally, there is a performance penalty. Going from a full browser to an embedded browser also has a performance impact. I can't comment on how noticeable that is to users, that is something you'd have to test on various platforms with different configurations.

Conclusion

There are real issues with embedded browsers on both the developer and user side, this list above is just a small excerpt of what I personally find the most egregious. Given all of that, I can't approve the usage of an embedded browser in the app in good faith when alternatives exist.

[halgari]

Thanks for the writeup @erri120. Initially I don't agree with several of these concerns, but I want to do some research before making a final decision. This writeup will help a lot.

[erri120]

Thanks for the writeup @erri120. Initially I don't agree with several of these concerns, but I want to do some research before making a final decision. This writeup will help a lot.

The biggest takeaway from the write-up that I probably should've emphasized more is the fact that alternatives exist and we should exhaust them first before looking at embedding a browser. We shouldn't have a conversation about why we want to use a browser or not, but rather a conversation comparing the different approaches and using the list above as a list of cons for browsers.

For this specific case, of getting GOG OAuth to work, one valid alternative is to just ask GOG for a new client, or an existing client, with a redirect URL to a protocol handler we have. Other alternatives probably exist as well, that would need some research, but what I want is for us to compare those alternatives to using an embedded browser.

[Sewer56]

I would rather prefer not to comment, but saying nothing would not be correct. Before I get started, I also would like to apologise for my dystopian attitude/take which follows; it may not be pleasant and very unlike me, but it would not be correct to stay silent. I too, am a user of the product after all. Presenting a dystopian reality story isetc.e best way I feel I can express my feelings about the subject.

---

I have long accepted defeat; months ago, this was inevitable.

There will be a web browser, like it or not; this is just the reality, the world we live in. The App will be 350MB bigger on Windows and 350MB/1GB larger on disk for macOS & Linux (depending on build flags used by 3rd party CEF distribution). If not careful, memory may not be freed, so after visiting a page, our app may use 1GiB while doing absolutely nothing in the background; you know, in the same way we already don't sometimes unload views in workspaces.

The download will be 100/300MB bigger, it will be a bit crappier; it's the reality, we just gotta face it, can't change it. This would have happened sooner or later anyway. Someone from product or above would eventually want to render some parts of the site in the App. Business would say 'just use a web browser, why waste time/money reimplementing it?'. I've long just accepted it.

Most users wouldn't know/care; critics will be few in number, even if it may include well known mod authors, not significant enough for it to be important. Even Steam can get away just fine with doing nothing in task tray while using 1GB of RAM while preventing access to games with it not running using the power of DRM.

Some hiccups will happen of course. For example, without proper controls in the UI tied to the browser, a user may get stuck in an infinite redirect loop by a dodgy link from a 'browse' entry in a collection; such as a captcha loop. A user, may get sent to a phishing site that could, for example pretend to be ours or start keylogging. It could even show different content if it can detect the client/user agent is or is not the App, to avoid the eyes of our moderators. There will be no safety mechanisms to stop the user from visiting known malicious sites and other safety mechanisms, like the ones found in a dedicated browser like Chrome. Visited pages may (auto)play audio with no way to quickly mute it. A user without browser history may find it harder to report a dodgy link. They may even run or interact with a dodgy ad with some interesting code; I remember the days of pop-up ads and later JS exploits.

A user wouldn't be able to customize their browsing experience. They wouldn't be able to easily bookmark a useful page related to a mod, for example, a guide attached to an external download. Disabled people may not be able to use their accessibility tools that they need to browse the web. People with impaired vision may not know they can (or even be able to) zoom the screen. User's light/dark preference will not be respected (unlike extensions which can enforce it). The user may need to login onto various sites again from the embedded browser without any password manager integration, for example, to begin an external file download. Embedded CEF sometimes can't interact with various OS features such as clipboard or attention/notifications which makes this worse (I believe it's usually a config/integration issue). Any forms a user may be asked to fill such as a mod description will be missing essential features such as a spell check. CEF (headless browser) may be mistaken as a bot by some websites. Native controller support likely wouldn't work well with it once we get around to it. A user (e.g. from [People's Republic of] China) may be unable to use a proxy if they need to get around censorship if they do it via browser. Corners may be taken, for example, why have an upload API if we can take the user to a web page with the fields pre-filled via GET/POST query strings? Why not embed entire mod pages? We will need to regularly update and test the CEF dependency on every single platform. The code will be native precompiled code, most likely from a third party. It will not be debuggable out the box, or be easy to edit, inspect, and fix issues in. CEF takes just barely over two hours to compile on my 12 core 5900X machine.

All in all. There will be unique problems, we will have to overcome them, just have to adapt.

I also understand there is some pride on the line to be held. Rest assured, despite being ready, I still had a heart attack when I woke up to the news at 3am; but we have to continue living. If it's any consolation, know that if everything is crap, then nothing is; and end users are used to crap being the modern standard.

---

In any case, a silly hypothetical aside.
I would much rather hear a problem statement, i.e. what we are trying to achieve. And to discuss how we're going to achieve them.

I know easier GOG Login (callback) and free user journey for downloads are two slices of the pie. What else is there, what problems do we exactly need to solve, what are our options, etc.

[Sewer56]

Oh, I thought everyone was off today, yet I see commits, hmm.

Well, in any case. Once the emotions have eventually died down, I guess, I'm okay with it in limited capacity.

However, I'm not necessarily comfortable letting an embedded browser access external domains (e.g. collections external files) outside of our control for the reasons above. Some simpler use cases like GOG login and free user download button access should be fine.

---

Misc Note: In my hotel room last week, I did make login work in around 40 minutes without an embedded browser (~100 lines of code). Sends you to regular browser, is cross platform, you can use password manager, etc. Completing login fires nxm:// link back to App.

But it required admin to quickly setup a MITM proxy, and part of that is a rather scary (in my opinion) dialog when temporarily installing a root certificate to intercept GOG traffic. Therefore I consider it a no-go personally.

[halgari]

@Sewer56 as always the rule is "do what I say, not what I do", I get the itch once an awhile to code up a few things, doesn't mean I want anyone else doing that :D. Don't worry I won't ninja commit this while people are off.

[Sewer56]

Ya, everything's good, I was just a little surprised if anything, ahaha. I'm not any better, always cooking some code.

[halgari]

Did a quick survey of some machines in my place to figure out how common CEF embedding is. Here's a list of what I found so far:

• Jetbrains
  • Rider
  • WebStorm
  • MPS
  • PyCharm
• GOG Galaxy
• Steam (I think Steam is a webapp, at least to some extent)
• Marvel Rivals
• Battle.NET
• Epic Launcher
• NVidia App

I'm going to survey some other machines in the house and add those results to this list.

[Sewer56]

Steam (I think Steam is a webapp, at least to some extent)

Steam used to be fully native, GPU accelerated with an optional embedded browser for visiting the store (it would only ever be loaded if you visit store). It used to use ~60MB of RAM total when not showing the browser. Its client GUI is now rendered with CEF however. There was a way to fallback, but they stopped supporting it after a while.

RAM usage during initial switchover was really bad, even by CEF standards but has improved a bit since. Steam is essentially rendering different parts of the UI in different headless webviews (if that makes sense). So various parts of the UI get different processes, each with a small webview.

The client (on my machine) uses 700MiB of RAM while doing nothing in the task tray; which is very unfortunate; especially since it's required to be in background due to DRM. (For me, this is completely unacceptable).
At the very least, it's not 1.2GiB, so they improved since the initial CEF switchover.

---

how common CEF embedding is

I think you could also include Electron applications in this list; it's just CEF with extensions if you think about it.
So Discord / Slack etc.

[Sewer56]

Wish Native WebView was an option, WebView2 on Windows, WKWebView on macOS and WebKitGTK on Linux, that's kind of a bummer. OS Webview would shave off download size, memory size and not require manual updating; but there doesn't seem to be a control/library for that I know of.

Edit: There was one, but repo got nuked for using the Avalonia name; probably as to not confuse with actual Avalonia package.