feat: auto ubuntu packages download for local browsers #1036

KuznetsovRoman · 2024-11-29T01:36:15Z

CI side (collect-ubuntu-browser-dependencies, is not being built):

Download browsers, if necessary
Resolve direct shared objects dependencies to apt package names per each ubuntu release
Save direct dependencies into "src/browser-installer/ubuntu-packages/autogenerated" (which is being packed)
Also saves its cache into "collect-ubuntu-browser-dependencies" in order to speed up next launches

Cache is stored in VCS and is splitted to 2 files:

Mutual cache. Ubuntu version independent (processed-browsers-linux.json). Removes the need to download browsers each time
Release specific cache. Could be different on different ubuntu versions (shared-objects-map-ubuntu-%.json). Removes the need to run "apt-file search" each time

Client side (src/browser-installer/ubuntu-packages, being packed):

Resolve direct dependencies to full list of apt packages
Download and extract them
Set LD_LIBRARY_PATH when launching webdriver so it could populate to browser, so browser would use downloaded packages

KuznetsovRoman · 2024-11-29T01:38:33Z

src/browser-installer/chromium/revisions/linux.ts

@@ -8,7 +8,6 @@ export default {
    79: 707231,
    80: 722374,
    81: 737198,
-    82: 750023,


Removed as API is not responding to request "which version of chromedriver should i use with this version of chrome": https://chromedriver.storage.googleapis.com/LATEST_RELEASE_82

KuznetsovRoman · 2024-11-29T01:38:48Z

.github/workflows/collect-deps.yml

+name: Collect ubuntu browser dependencies
+on:
+    schedule:
+        - cron: 0 0 1 * *


once per month

KuznetsovRoman · 2024-11-29T01:39:15Z

package.json

@@ -9,10 +9,11 @@
  ],
  "scripts": {
    "build": "tsc --build && npm run copy-static && npm run build-bundles",
-    "copy-static": "copyfiles 'src/browser/client-scripts/*' build",
+    "copy-static": "copyfiles 'src/browser/client-scripts/*' 'src/**/*.json' build",


Also copy these autogenerated json

looks dangerous because it can be any json file, for example, stub for unit-tests

Restricted to "src//[!cache]*/autogenerated//*.json"
So, jsons need to be in "autogenerated" directory

KuznetsovRoman · 2024-11-29T01:39:45Z

src/browser-installer/chrome/index.ts

+    const [chromeDriverPath] = await Promise.all([
+        installChromeDriver(chromeVersion),
+        installChrome(chromeVersion),
+        shouldInstallUbuntuPackageDependencies ? installUbuntuPackageDependencies() : null,


Ensure ubuntu packages are ready before spawning child process

KuznetsovRoman · 2024-11-29T01:40:52Z

src/browser-installer/install.ts

+        return await Promise.all([
+            installChrome(browserVersion, { force }),
+            shouldInstallWebDriver && installChromeDriver(browserVersion, { force }),
+            needToInstallUbuntuPackages && installUbuntuPackageDependencies(),


Ensure ubuntu package are ready (e.g. for devtools)

KuznetsovRoman · 2024-11-29T01:53:52Z

src/collect-ubuntu-browser-dependencies/cache/index.ts

+        await fs.outputJSON(this._sharedObjectsMapPath, sortObject(sharedObjectsMap), { spaces: 4 });
+        await fs.outputJSON(this._processedBrowsersCachePath, sortObject(processedBrowsers), { spaces: 4 });


Using 4 spaces because the linter wants it

KuznetsovRoman · 2024-11-29T01:55:38Z

src/collect-ubuntu-browser-dependencies/constants.ts

+// Those are couldn't be seen with readelf -d
+export const EXTRA_FIREFOX_SHARED_OBJECTS = ["libdbus-glib-1.so.2", "libXt.so.6"];


Usually there is a note in dynamic section of an elf file "which libraries does it load", but for a long time, starting from the very first versions, firefox needs these 2 shared objects, and they are not recorded in dynamic section of an elf file

KuznetsovRoman · 2024-11-29T01:56:45Z

src/collect-ubuntu-browser-dependencies/index.ts

+
+    logger.log(`Fetched ${browserVersions.length} browser milestones`);
+
+    const browsersToDownload = cache.filterProcessedBrowsers(browserVersions);


Browsers are only downloaded once
If we would like to add new ubuntu release, we would not need to download any browser

KuznetsovRoman · 2024-11-29T01:58:39Z

src/collect-ubuntu-browser-dependencies/shared-object.ts

+        return "libc6";
+    }
+
+    const relevantPackageName = _.minBy(packages, packageName => calcLevenshtein(sharedObject, packageName)) as string;


Using levenshtein in order to resolve shared objects to their main packages, and not just to some package, which is more popular (in example, if libnss3.so is used in firefox, it would return "libnss3" even if firefox is more popular)

KuznetsovRoman · 2024-11-29T01:59:21Z

src/collect-ubuntu-browser-dependencies/ubuntu/index.ts

Directory with ubuntu binaries call wrappers

KuznetsovRoman · 2024-11-29T19:18:04Z

.github/workflows/collect-deps.yml

@@ -0,0 +1,43 @@
+name: Collect ubuntu browser dependencies


Scheduled action:

Collects ubuntu dependencies for new browser versions

Creates a PR to add dependency if there is any new library

If there are no extra dependencies, do nothing.

If there is extra browser, but no new dependencies, creates PR to update cache, so it would not need to download the browser later

.github/workflows/collect-deps.yml

sipayRT · 2024-12-03T12:24:22Z

package.json

@@ -9,10 +9,11 @@
  ],
  "scripts": {
    "build": "tsc --build && npm run copy-static && npm run build-bundles",
-    "copy-static": "copyfiles 'src/browser/client-scripts/*' build",
+    "copy-static": "copyfiles 'src/browser/client-scripts/*' 'src/**/*.json' build",


looks dangerous because it can be any json file, for example, stub for unit-tests

package.json

sipayRT · 2024-12-03T16:33:10Z

src/collect-ubuntu-browser-dependencies/index.ts

+async function main(): Promise<void> {
+    const ubuntuMilestone = await getUbuntuMilestone();
+
+    logger.log(`Detected ubuntu release: "${ubuntuMilestone}"`);


can't it be undefined/null here?

It relies on VERSION_ID in /etc/os-release
If it is ubuntu, VERSION_ID exists and defined
getUbuntuMilestone implies it is ubuntu
So, no

sipayRT · 2024-12-04T11:55:28Z

src/collect-ubuntu-browser-dependencies/index.ts

+import { getUbuntuMilestone, writeUbuntuPackageDependencies } from "../browser-installer/ubuntu-packages";
+import logger from "../utils/logger";
+
+const createResolveSharedObjectToPackageName =


so create or resolve? mb, createSharedObjectToPackageNameResolver?

createSharedObjectToPackageNameResolver

Resolver is a noun. We could call what is returned as Resolver only if its object with some method, but its function

This function creates function "resolveSharedObjectToPackageName", so it is called "createResolve..."

sipayRT · 2024-12-04T12:00:30Z

src/collect-ubuntu-browser-dependencies/shared-object.ts

+};
+
+export const getBinarySharedObjectDependencies = async (binaryPath: string): Promise<string[]> => {
+    const sharedObjectRegExp = /^\s*\dx\d+\s\(NEEDED\)\s*Shared library: \[(.*)\]/gm;


please, add an example for readElf to understand what is going on here

In code section?
There is in test section

sipayRT · 2024-12-04T12:04:49Z

src/collect-ubuntu-browser-dependencies/shared-object.ts

+    while (regExpResult && regExpResult[1]) {
+        sharedObjectDependencies.push(regExpResult[1]);
+
+        regExpResult = sharedObjectRegExp.exec(readElfResult);


how does it work? why does the second call return a different value?

Works just like it does in C (where you dont expect magic like this)

Because of the "g" flag, regular expression state is saved (in C it was saved in the global variable, i assume in js there is some kind of closure)

sipayRT · 2024-12-04T12:14:37Z

test/src/collect-ubuntu-browser-dependencies/shared-object.ts

+    describe("getBinarySharedObjectDependencies", () => {
+        it("should return binary direct shared object deps", async () => {
+            readElfStub.resolves(`
+Dynamic section at offset 0xb00 contains 26 entries:


KuznetsovRoman · 2024-12-06T02:26:22Z

src/browser-installer/chrome/index.ts

-
-    const [chromeDriverPath] = await Promise.all([
+    const [chromeDriverPath, randomPort, chromeDriverEnv] = await Promise.all([
        installChromeDriver(chromeVersion),
-        installChrome(chromeVersion),
-        shouldInstallUbuntuPackageDependencies ? installUbuntuPackageDependencies() : null,
+        getPort(),
+        isUbuntu()
+            .then(isUbuntu => (isUbuntu ? getUbuntuLinkerEnv() : null))
+            .then(extraEnv => (extraEnv ? { ...process.env, ...extraEnv } : process.env)),
    ]);


Now we dont install browser and ubuntu packages here, and instead we assume it is already installed by "browser-installer/run.ts", which runs "browser-installer/install.ts" before calling "run*Driver"

KuznetsovRoman · 2024-12-06T02:27:39Z

src/browser-installer/run.ts

 export const runBrowserDriver = async (
-    driverName: SupportedDriver,
+    browserName: SupportedBrowser,
    browserVersion: string,
    { debug = false } = {},
 ): Promise<{ gridUrl: string; process: ChildProcess; port: number }> => {
-    switch (driverName) {
-        case Driver.CHROMEDRIVER:
+    const installBrowserOpts = { shouldInstallWebDriver: true, shouldInstallUbuntuPackages: true };
+
+    await installBrowser(browserName, browserVersion, installBrowserOpts);


As now we need to install browser before running its driver, we expect "browserName" instead of "driverName" in order to pass "browserName" to "installBrowser"

KuznetsovRoman · 2024-12-06T02:30:33Z

src/browser-installer/utils.ts

-export const getDriverNameForBrowserName = (browserName: SupportedBrowser): SupportedDriver | null => {
-    if (browserName === Browser.CHROME || browserName === Browser.CHROMIUM) {
-        return Driver.CHROMEDRIVER;
+export const getNormalizedBrowserName = (


As now we dont map "browserName" to "driverName" for "runBrowserDriver", we dont need "getDriverNameForBrowserName", and instead i created this function, so in "installBrowser" we can be sure it only receives valid "browserName" of SupportedBrowser type

This way we can handle cases with invalid browser name outside of "install", so we know, what exactly user wanted to do, and we can provide relevant error message:

invalid browser name was used in order to run local browser

invalid browser name was used in order to download a browser

KuznetsovRoman · 2024-12-06T02:34:15Z

src/browser-pool/webdriver-pool.ts

    async getWebdriver(
-        browserName: SupportedBrowser,
-        browserVersion: string,
+        browserName?: string,
+        browserVersion?: string,
        { debug = false } = {},
    ): ReturnType<typeof this.createWebdriverProcess> {
-        const driverName = getDriverNameForBrowserName(browserName);
+        const browserNameNormalized = getNormalizedBrowserName(browserName);

-        if (!driverName) {
+        if (!browserNameNormalized) {
            throw new Error(


As we now dont want to map "browserName" to "driverName", we still need to check if "browserName" is valid (because this method is called inside NewBrowser with its config's browserName, which is user provided value, which can be invalid)

KuznetsovRoman · 2024-12-06T02:36:48Z

src/browser/new-browser.ts

        const executablePath = await installBrowser(
-            this._config.desiredCapabilities?.browserName as SupportedBrowser,
-            this._config.desiredCapabilities?.browserVersion as string,
+            normalizedBrowserName,


"installBrowser" now only wants normalized browser name (so we could pull out "invalid browser name" handling in order to throw relevant errors)

KuznetsovRoman · 2024-12-06T02:38:05Z

src/browser/new-browser.ts

-            this._config.desiredCapabilities?.browserVersion as string,
+            normalizedBrowserName,
+            this._config.desiredCapabilities?.browserVersion,
+            { shouldInstallWebDriver: false, shouldInstallUbuntuPackages: true },


Explicitly setting "should install" options, as they are all set to "false" now by default.

In "_addExecutablePath" we dont need to install webdriver as user might want to run browser in "devtools" protocol

KuznetsovRoman · 2024-12-09T23:38:29Z

src/browser-installer/registry/cli-progress-bar.ts

+        const downloadProgressCallback: DownloadProgressCallback = (done, total = 100) => {
            if (!bar) {
-                const totalMB = Math.round((totalBytes / BYTES_PER_MEGABYTE) * 100) / 100;
-                bar = progressBar.create(totalMB, 0, { filename: `${browserName}@${browserVersion}` });
+                bar = progressBar.create(100, 0, { filename: `${browserName}@${browserVersion}` });


Now use percents as more versatile progress bar unit.
This way we can also use the same progress bar for ubuntu packages download

KuznetsovRoman · 2024-12-09T23:52:45Z

src/browser-installer/utils.ts

@@ -129,13 +142,42 @@ const getCacheDir = (envValueOverride = process.env.TESTPLANE_BROWSERS_PATH): st
 export const getRegistryPath = (envValueOverride?: string): string =>
    path.join(getCacheDir(envValueOverride), "registry.json");

+export const readRegistry = (registryPath: string): Registry => {


I decided to extend registry so it would store not only binary paths, but could store anything else
This way we can manage os packages by this registry file too.

src/browser-installer/install.ts

sipayRT · 2024-12-09T09:50:08Z

src/browser-installer/utils.ts

-        return Driver.CHROMEDRIVER;
+export const getNormalizedBrowserName = (
+    browserName?: string,
+): Exclude<SupportedBrowser, typeof Browser.CHROMIUM> | null => {


why do we need exclude here?

We dont need to, i just wanted to show "chromium can't be returned here", because chromium maps to chrome, as installing "chrome" and "chromium" has a single entry point, being "chrome", as "chromium" is not a valid W3C browserName

test/src/browser-installer/run.ts

src/browser-installer/install.ts

src/browser-installer/utils.ts

src/browser-installer/registry/index.ts

…r-installer/ubuntu-packages/

KuznetsovRoman commented Nov 29, 2024

View reviewed changes

KuznetsovRoman force-pushed the TESTPLANE-352.ubuntu_deps branch 4 times, most recently from dbacdc9 to 6f9aeaf Compare December 1, 2024 20:59

sipayRT reviewed Dec 4, 2024

View reviewed changes

KuznetsovRoman force-pushed the TESTPLANE-352.ubuntu_deps branch from 642e41f to 4c3fd27 Compare December 6, 2024 02:23

KuznetsovRoman commented Dec 6, 2024

View reviewed changes

KuznetsovRoman force-pushed the TESTPLANE-352.ubuntu_deps branch 2 times, most recently from 907c990 to 8c6d6c6 Compare December 9, 2024 23:51

KuznetsovRoman commented Dec 9, 2024

View reviewed changes

KuznetsovRoman force-pushed the TESTPLANE-352.ubuntu_deps branch from 8c6d6c6 to bd17127 Compare December 10, 2024 10:54

sipayRT approved these changes Dec 16, 2024

View reviewed changes

KuznetsovRoman force-pushed the TESTPLANE-352.ubuntu_deps branch 2 times, most recently from 27c49dd to e43d792 Compare December 17, 2024 00:35

KuznetsovRoman added 12 commits December 17, 2024 03:37

feat: auto ubuntu packages download for local browsers

9d1d380

fix: just log warning if ubuntu version is not supported

ef2b78b

chore: rename revisions '.ts' -> '.json'

5a02cad

chore: move src/collect-ubuntu-browser-dependencies inside src/browse…

6c744ea

…r-installer/ubuntu-packages/

chore: assert browser is installed before running its driver

61ce0bf

chore: polish gh workflow

6312a75

chore: minor changes

495fe22

chore: manage packages installation by registry

3a83de3

chore: clear progress bar on complete

3498cd3

refactor: implement registry as class

86cab81

refactor: move install driver/ubuntu packages into browser scope

90088e8

fix: remove registry v0 support

bde19e0

KuznetsovRoman force-pushed the TESTPLANE-352.ubuntu_deps branch from e43d792 to bde19e0 Compare December 17, 2024 00:38

fix: aquire chromium versions

90c0be2

KuznetsovRoman merged commit 25895ea into master Dec 17, 2024
2 checks passed

KuznetsovRoman deleted the TESTPLANE-352.ubuntu_deps branch December 17, 2024 01:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: auto ubuntu packages download for local browsers #1036

feat: auto ubuntu packages download for local browsers #1036

KuznetsovRoman commented Nov 29, 2024 •

edited

Loading

KuznetsovRoman Nov 29, 2024

KuznetsovRoman Nov 29, 2024

KuznetsovRoman Nov 29, 2024

sipayRT Dec 3, 2024

KuznetsovRoman Dec 5, 2024

KuznetsovRoman Nov 29, 2024

KuznetsovRoman Nov 29, 2024

KuznetsovRoman Nov 29, 2024

KuznetsovRoman Nov 29, 2024

KuznetsovRoman Nov 29, 2024

KuznetsovRoman Nov 29, 2024

KuznetsovRoman Nov 29, 2024

KuznetsovRoman Nov 29, 2024

sipayRT Dec 3, 2024

sipayRT Dec 3, 2024

KuznetsovRoman Dec 4, 2024 •

edited

Loading

sipayRT Dec 4, 2024

KuznetsovRoman Dec 4, 2024

sipayRT Dec 4, 2024

KuznetsovRoman Dec 4, 2024

sipayRT Dec 4, 2024

KuznetsovRoman Dec 4, 2024 •

edited

Loading

sipayRT Dec 4, 2024

KuznetsovRoman Dec 6, 2024

KuznetsovRoman Dec 6, 2024

KuznetsovRoman Dec 6, 2024

KuznetsovRoman Dec 6, 2024

KuznetsovRoman Dec 6, 2024

KuznetsovRoman Dec 6, 2024

KuznetsovRoman Dec 6, 2024

KuznetsovRoman Dec 9, 2024

KuznetsovRoman Dec 9, 2024

sipayRT Dec 9, 2024

KuznetsovRoman Dec 16, 2024 •

edited

Loading

		await fs.outputJSON(this._sharedObjectsMapPath, sortObject(sharedObjectsMap), { spaces: 4 });
		await fs.outputJSON(this._processedBrowsersCachePath, sortObject(processedBrowsers), { spaces: 4 });

		// Those are couldn't be seen with readelf -d
		export const EXTRA_FIREFOX_SHARED_OBJECTS = ["libdbus-glib-1.so.2", "libXt.so.6"];


		logger.log(`Fetched ${browserVersions.length} browser milestones`);

		const browsersToDownload = cache.filterProcessedBrowsers(browserVersions);

feat: auto ubuntu packages download for local browsers #1036

feat: auto ubuntu packages download for local browsers #1036

Conversation

KuznetsovRoman commented Nov 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KuznetsovRoman Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KuznetsovRoman Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KuznetsovRoman Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

KuznetsovRoman commented Nov 29, 2024 •

edited

Loading

KuznetsovRoman Dec 4, 2024 •

edited

Loading

KuznetsovRoman Dec 4, 2024 •

edited

Loading

KuznetsovRoman Dec 16, 2024 •

edited

Loading