Introduce `NativeLink Web Bridge` #1289

SchahinRohani · 2024-08-28T15:26:18Z

Description

This is a fast prototype for subscribing to the redis/dragonflydb "build_events" channel and
decode them properly via protobuf and fires them via websocket to the browser.

It's a plain prototype without any error handling and formatting

Included: latency calculation from bazel to bridge

Type of change

New feature (non-breaking change which adds functionality)
This change requires a documentation update

How Has This Been Tested?

In 4 different shells inside the nix flake environment on MacOS.

Checklist

Updated documentation if needed
Tests added/amended
bazel test //... passes locally
PR is contained in a single commit, using git amend see some docs

This change is

aaronmondal

Reviewed 1 of 2 files at r4, 15 of 15 files at r5, all commit messages.
Reviewable status: 0 of 1 LGTMs obtained, and 19 of 43 files reviewed, and 10 discussions need to be resolved

web/bridge/.gitignore line 1 at r5 (raw file):

# Logs

nit: Could remove everything from this file that we don't actually use?

web/bridge/.env.example line 3 at r5 (raw file):

REDIS_URL=redis://localhost:6379
NATIVELINK_PUB_SUB_CHANNEL=build_event
POSTGRES_URL=

nit: Consider setting a default postres url

web/bridge/image.nix line 7 at r5 (raw file):

}: let
  description = "A simple Bun environment image";
  title = "Bun Environment";

Let's inline the title and description.

web/bridge/README.md line 73 at r3 (raw file):

Previously, SchahinRohani (Schahin) wrote…

And how can we use a custom .bazelrc with the hello-lre example?

I don't think we'd need one. Instead of the bazelrc we can just putt all these flags into the bazelrc and leave a "tip" mentioning thath the flags can be put into a bazelrc as well. Maybe something like this, similar to how we do it in https://www.nativelink.com/docs/deployment-examples/kubernetes:

bazel clean && bazel build \
    --remote_cache=grpc://$CACHE \
    --remote_executor=grpc://$SCHEDULER \
    --THE_OTHER_FLAGS... \
    //local-remote-execution/examples:hello_lre

web/bridge/src/index.ts line 14 at r5 (raw file):

  // NativeLink URL
  const nativelinkRepo = "TraceMachina/nativelink"
  const nativelinkBranch = "main"

I'm just noticing that we can't use branches like "main" or "master" here, as it would cause unpredictable behavior if the branches got updated. Instead, let's pin all these URLs to specific commits. We can also remove the variables etc. Let's use regular URLs which are easier to copy-paste.

web/bridge/src/protobuf.ts line 6 at r5 (raw file):

    console.log("Loading Remote Proto Files");

    // Create a new Root instance

Remove all the comments that explain trivial lines of code.

web/bridge/src/protobuf.ts line 43 at r5 (raw file):

    // Handle googleapis imports
    if (importPath.startsWith("google/api") || importPath.startsWith("google/devtools/build/v1")) {
        return `https://raw.githubusercontent.com/googleapis/googleapis/master/${importPath}`;

As mentioned above, we'll need to keep these paths globally and use pinned urls.

web/bridge/src/utils.ts line 1 at r5 (raw file):

type ParsedMessage = {

Everything in this file seems to be used just by the eventhandler.ts. Let's move these functions there.

web/bridge/src/websocket.ts line 6 at r5 (raw file):

export const startWebSocket = () => {
  console.log('\nWebSocket server is running on ws://localhost:8080\n');

This probably shouldn't be hardcoded

web/bridge/src/websocket.ts line 8 at r5 (raw file):

  console.log('\nWebSocket server is running on ws://localhost:8080\n');
  Bun.serve({
  port: 8080,

Let's not hardcode ports.

SchahinRohani

Reviewable status: 0 of 1 LGTMs obtained, and 8 of 43 files reviewed, and pending CI: Web Platform Deployment / ubuntu-24.04, pre-commit-checks, and 8 discussions need to be resolved

web/bridge/image.nix line 7 at r5 (raw file):