Feature/lxc container runtime (#2562)

* wip(fix): Dependencies * wip: context * wip(fix) Sorta auth * wip: warnings * wip(fix): registry/admin * wip(fix) marketplace * wip(fix) Some more converted and fixed with the linter and config * wip: Working on the static server * wip(fix)static server * wip: Remove some asynnc * wip: Something about the request and regular rpc * wip: gut install Co-authored-by: J H <[email protected]> * wip: Convert the static server into the new system * wip delete file * test * wip(fix) vhost does not need the with safe defaults * wip: Adding in the wifi * wip: Fix the developer and the verify * wip: new install flow Co-authored-by: J H <[email protected]> * fix middleware * wip * wip: Fix the auth * wip * continue service refactor * feature: Service get_config * feat: Action * wip: Fighting the great fight against the borrow checker * wip: Remove an error in a file that I just need to deel with later * chore: Add in some more lifetime stuff to the services * wip: Install fix on lifetime * cleanup * wip: Deal with the borrow later * more cleanup * resolve borrowchecker errors * wip(feat): add in the handler for the socket, for now * wip(feat): Update the service_effect_handler::action * chore: Add in the changes to make sure the from_service goes to context * chore: Change the * refactor service map * fix references to service map * fill out restore * wip: Before I work on the store stuff * fix backup module * handle some warnings * feat: add in the ui components on the rust side * feature: Update the procedures * chore: Update the js side of the main and a few of the others * chore: Update the rpc listener to match the persistant container * wip: Working on updating some things to have a better name * wip(feat): Try and get the rpc to return the correct shape? * lxc wip * wip(feat): Try and get the rpc to return the correct shape? * build for container runtime wip * remove container-init * fix build * fix error * chore: Update to work I suppose * lxc wip * remove docker module and feature * download alpine squashfs automatically * overlays effect Co-authored-by: Jade <[email protected]> * chore: Add the overlay effect * feat: Add the mounter in the main * chore: Convert to use the mounts, still need to work with the sandbox * install fixes * fix ssl * fixes from testing * implement tmpfile for upload * wip * misc fixes * cleanup * cleanup * better progress reporting * progress for sideload * return real guid * add devmode script * fix lxc rootfs path * fix percentage bar * fix progress bar styling * fix build for unstable * tweaks * label progress * tweaks * update progress more often * make symlink in rpc_client * make socket dir * fix parent path * add start-cli to container * add echo and gitInfo commands * wip: Add the init + errors * chore: Add in the exit effect for the system * chore: Change the type to null for failure to parse * move sigterm timeout to stopping status * update order * chore: Update the return type * remove dbg * change the map error * chore: Update the thing to capture id * chore add some life changes * chore: Update the loging * chore: Update the package to run module * us From for RpcError * chore: Update to use import instead * chore: update * chore: Use require for the backup * fix a default * update the type that is wrong * chore: Update the type of the manifest * chore: Update to make null * only symlink if not exists * get rid of double result * better debug info for ErrorCollection * chore: Update effects * chore: fix * mount assets and volumes * add exec instead of spawn * fix mounting in image * fix overlay mounts Co-authored-by: Jade <[email protected]> * misc fixes * feat: Fix two * fix: systemForEmbassy main * chore: Fix small part of main loop * chore: Modify the bundle * merge * fixMain loop" * move tsc to makefile * chore: Update the return types of the health check * fix client * chore: Convert the todo to use tsmatches * add in the fixes for the seen and create the hack to allow demo * chore: Update to include the systemForStartOs * chore UPdate to the latest types from the expected outout * fixes * fix typo * Don't emit if failure on tsc * wip Co-authored-by: Jade <[email protected]> * add s9pk api * add inspection * add inspect manifest * newline after display serializable * fix squashfs in image name * edit manifest Co-authored-by: Jade <[email protected]> * wait for response on repl * ignore sig for now * ignore sig for now * re-enable sig verification * fix * wip * env and chroot * add profiling logs * set uid & gid in squashfs to 100000 * set uid of sqfs to 100000 * fix mksquashfs args * add env to compat * fix * re-add docker feature flag * fix docker output format being stupid * here be dragons * chore: Add in the cross compiling for something * fix npm link * extract logs from container on exit * chore: Update for testing * add log capture to drop trait * chore: add in the modifications that I make * chore: Update small things for no updates * chore: Update the types of something * chore: Make main not complain * idmapped mounts * idmapped volumes * re-enable kiosk * chore: Add in some logging for the new system * bring in start-sdk * remove avahi * chore: Update the deps * switch to musl * chore: Update the version of prettier * chore: Organize' * chore: Update some of the headers back to the standard of fetch * fix musl build * fix idmapped mounts * fix cross build * use cross compiler for correct arch * feat: Add in the faked ssl stuff for the effects * @dr_bonez Did a solution here * chore: Something that DrBonez * chore: up * wip: We have a working server!!! * wip * uninstall * wip * tes * misc fixes * fix cli * replace interface with host * chore: Fix the types in some ts files * chore: quick update for the system for embassy to update the types * replace br-start9 with lxcbr0 * split patchdb into public/private * chore: Add changes for config set * Feat: Adding some debugging for the errors * wip: Working on getting the set config to work * chore: Update and fix the small issue with the deserialization * lightning, masked, schemeOverride, invert host-iface relationship * feat: Add in the changes for just the sdk * feat: Add in the changes for the new effects I suppose for now --------- Co-authored-by: J H <[email protected]> Co-authored-by: J H <[email protected]> Co-authored-by: J H <[email protected]> Co-authored-by: Matt Hill <[email protected]>
Start9Labs · Feb 22, 2024 · 089199e · 089199e
1 parent d7bc7a2
commit 089199e
Show file tree

Hide file tree

Showing 58 changed files with 1,057 additions and 3,057 deletions.
diff --git a/container-runtime/src/Adapters/HostSystemStartOs.ts b/container-runtime/src/Adapters/HostSystemStartOs.ts
@@ -38,7 +38,10 @@ export class HostSystemStartOs implements Effects {
 
   constructor(readonly callbackHolder: CallbackHolder) {}
   id = 0
-  rpcRound(method: string, params: unknown) {
+  rpcRound<K extends keyof Effects | "getStore" | "setStore">(
+    method: K,
+    params: unknown,
+  ) {
     const id = this.id++
     const client = net.createConnection({ path: SOCKET_PATH }, () => {
       client.write(
@@ -74,7 +77,7 @@ export class HostSystemStartOs implements Effects {
                   console.error("Debug: " + res.error.data.debug)
                 }
               }
-              reject(new Error(message))
+              reject(new Error(`${message}@${method}`))
             } else if (testRpcResult(res)) {
               resolve(res.result)
             } else {
@@ -91,13 +94,7 @@ export class HostSystemStartOs implements Effects {
       })
     })
   }
-  started =
-    // @ts-ignore
-    this.method !== MAIN
-      ? null
-      : () => {
-          return this.rpcRound("started", null)
-        }
+
   bind(...[options]: Parameters<T.Effects["bind"]>) {
     return this.rpcRound("bind", options) as ReturnType<T.Effects["bind"]>
   }
@@ -131,9 +128,9 @@ export class HostSystemStartOs implements Effects {
       T.Effects["exportAction"]
     >
   }
-  exportServiceInterface(
-    ...[options]: Parameters<T.Effects["exportServiceInterface"]>
-  ) {
+  exportServiceInterface: Effects["exportServiceInterface"] = (
+    ...[options]: Parameters<Effects["exportServiceInterface"]>
+  ) => {
     return this.rpcRound("exportServiceInterface", options) as ReturnType<
       T.Effects["exportServiceInterface"]
     >
@@ -158,31 +155,24 @@ export class HostSystemStartOs implements Effects {
       T.Effects["getContainerIp"]
     >
   }
-  getHostnames: any = (...[allOptions]: any[]) => {
+  getHostInfo: Effects["getHostInfo"] = (...[allOptions]: any[]) => {
     const options = {
       ...allOptions,
       callback: this.callbackHolder.addCallback(allOptions.callback),
     }
-    return this.rpcRound("getHostnames", options) as ReturnType<
-      T.Effects["getHostnames"]
-    >
+    return this.rpcRound("getHostInfo", options) as ReturnType<
+      T.Effects["getHostInfo"]
+    > as any
   }
-  getInterface(...[options]: Parameters<T.Effects["getInterface"]>) {
-    return this.rpcRound("getInterface", {
+  getServiceInterface(
+    ...[options]: Parameters<T.Effects["getServiceInterface"]>
+  ) {
+    return this.rpcRound("getServiceInterface", {
       ...options,
       callback: this.callbackHolder.addCallback(options.callback),
-    }) as ReturnType<T.Effects["getInterface"]>
-  }
-  getIPHostname(...[]: Parameters<T.Effects["getIPHostname"]>) {
-    return this.rpcRound("getIPHostname", null) as ReturnType<
-      T.Effects["getIPHostname"]
-    >
-  }
-  getLocalHostname(...[]: Parameters<T.Effects["getLocalHostname"]>) {
-    return this.rpcRound("getLocalHostname", null) as ReturnType<
-      T.Effects["getLocalHostname"]
-    >
+    }) as ReturnType<T.Effects["getServiceInterface"]>
   }
+
   getPrimaryUrl(...[options]: Parameters<T.Effects["getPrimaryUrl"]>) {
     return this.rpcRound("getPrimaryUrl", {
       ...options,
@@ -196,14 +186,6 @@ export class HostSystemStartOs implements Effects {
       T.Effects["getServicePortForward"]
     >
   }
-  getServiceTorHostname(
-    ...[interfaceId, packageId]: Parameters<T.Effects["getServiceTorHostname"]>
-  ) {
-    return this.rpcRound("getServiceTorHostname", {
-      interfaceId,
-      packageId,
-    }) as ReturnType<T.Effects["getServiceTorHostname"]>
-  }
   getSslCertificate(
     ...[packageId, algorithm]: Parameters<T.Effects["getSslCertificate"]>
   ) {
@@ -223,11 +205,13 @@ export class HostSystemStartOs implements Effects {
       callback: this.callbackHolder.addCallback(options.callback),
     }) as ReturnType<T.Effects["getSystemSmtp"]>
   }
-  listInterface(...[options]: Parameters<T.Effects["listInterface"]>) {
-    return this.rpcRound("listInterface", {
+  listServiceInterfaces(
+    ...[options]: Parameters<T.Effects["listServiceInterfaces"]>
+  ) {
+    return this.rpcRound("listServiceInterfaces", {
       ...options,
       callback: this.callbackHolder.addCallback(options.callback),
-    }) as ReturnType<T.Effects["listInterface"]>
+    }) as ReturnType<T.Effects["listServiceInterfaces"]>
   }
   mount(...[options]: Parameters<T.Effects["mount"]>) {
     return this.rpcRound("mount", options) as ReturnType<T.Effects["mount"]>
@@ -304,17 +288,4 @@ export class HostSystemStartOs implements Effects {
         T.Effects["store"]["set"]
       >,
   }
-
-  /**
-   * So, this is created
-   * @param options
-   * @returns
-   */
-  embassyGetInterface(options: {
-    target: "tor-key" | "tor-address" | "lan-address"
-    packageId: string
-    interface: string
-  }) {
-    return this.rpcRound("embassyGetInterface", options) as Promise<string>
-  }
 }
diff --git a/container-runtime/src/Adapters/RpcListener.ts b/container-runtime/src/Adapters/RpcListener.ts
@@ -11,6 +11,7 @@ import {
   matches,
   any,
   shape,
+  anyOf,
 } from "ts-matches"
 
 import { types as T } from "@start9labs/start-sdk"
@@ -24,16 +25,28 @@ import { HostSystem } from "../Interfaces/HostSystem"
 import { jsonPath } from "../Models/JsonPath"
 import { System } from "../Interfaces/System"
 type MaybePromise<T> = T | Promise<T>
-type SocketResponse = { jsonrpc: "2.0"; id: IdType } & (
-  | { result: unknown }
-  | {
-      error: {
-        code: number
-        message: string
-        data: { details: string; debug?: string }
-      }
-    }
+export const matchRpcResult = anyOf(
+  object({ result: any }),
+  object({
+    error: object(
+      {
+        code: number,
+        message: string,
+        data: object(
+          {
+            details: string,
+            debug: any,
+          },
+          ["details", "debug"],
+        ),
+      },
+      ["data"],
+    ),
+  }),
 )
+export type RpcResult = typeof matchRpcResult._TYPE
+type SocketResponse = { jsonrpc: "2.0"; id: IdType } & RpcResult
+
 const SOCKET_PARENT = "/media/startos/rpc"
 const SOCKET_PATH = "/media/startos/rpc/service.sock"
 const jsonrpc = "2.0" as const
@@ -186,23 +199,11 @@ export class RpcListener {
             input: params.input,
             timeout: params.timeout,
           })
-          .then((result) =>
-            "ok" in result
-              ? {
-                  jsonrpc,
-                  id,
-                  result: result.ok === undefined ? null : result.ok,
-                }
-              : {
-                  jsonrpc,
-                  id,
-                  error: {
-                    code: result.err.code,
-                    message: "Package Root Error",
-                    data: { details: result.err.message },
-                  },
-                },
-          )
+          .then((result) => ({
+            jsonrpc,
+            id,
+            ...result,
+          }))
           .catch((error) => ({
             jsonrpc,
             id,

diff --git a/container-runtime/src/Adapters/Systems/SystemForEmbassy/DockerProcedureContainer.ts b/container-runtime/src/Adapters/Systems/SystemForEmbassy/DockerProcedureContainer.ts
@@ -33,8 +33,11 @@ export class DockerProcedureContainer {
           await overlay.mount({ type: "assets", id: mount }, mounts[mount])
         } else if (volumeMount.type === "certificate") {
           volumeMount
-          const certChain = await effects.getSslCertificate()
-          const key = await effects.getSslKey()
+          const certChain = await effects.getSslCertificate(
+            null,
+            volumeMount["interface-id"],
+          )
+          const key = await effects.getSslKey(null, volumeMount["interface-id"])
           await fs.writeFile(
             `${path}/${volumeMount["interface-id"]}.cert.pem`,
             certChain.join("\n"),

diff --git a/container-runtime/src/Adapters/Systems/SystemForEmbassy/index.ts b/container-runtime/src/Adapters/Systems/SystemForEmbassy/index.ts
@@ -28,6 +28,9 @@ import {
 import { HostSystemStartOs } from "../../HostSystemStartOs"
 import { JsonPath, unNestPath } from "../../../Models/JsonPath"
 import { HostSystem } from "../../../Interfaces/HostSystem"
+import { RpcResult, matchRpcResult } from "../../RpcListener"
+import { ServiceInterface } from "../../../../../sdk/dist/cjs/lib/types"
+import { createUtils } from "../../../../../sdk/dist/cjs/lib/util"
 
 type Optional<A> = A | undefined | null
 function todo(): never {
@@ -68,24 +71,22 @@ export class SystemForEmbassy implements System {
       input: unknown
       timeout?: number | undefined
     },
-  ): Promise<ExecuteResult> {
+  ): Promise<RpcResult> {
     return this._execute(effects, options)
       .then((x) =>
         matches(x)
           .when(
             object({
               result: any,
             }),
-            (x) => ({
-              ok: x.result,
-            }),
+            (x) => x,
           )
           .when(
             object({
               error: string,
             }),
             (x) => ({
-              err: {
+              error: {
                 code: 0,
                 message: x.error,
               },
@@ -96,20 +97,34 @@ export class SystemForEmbassy implements System {
               "error-code": tuple(number, string),
             }),
             ({ "error-code": [code, message] }) => ({
-              err: {
+              error: {
                 code,
                 message,
               },
             }),
           )
-          .defaultTo({ ok: x }),
+          .defaultTo({ result: x }),
       )
-      .catch((error) => ({
-        err: {
-          code: 0,
-          message: "" + error,
-        },
-      }))
+      .catch((error: unknown) => {
+        if (error instanceof Error)
+          return {
+            error: {
+              code: 0,
+              message: error.name,
+              data: {
+                details: error.message,
+                debug: `${error?.cause ?? "[noCause]"}:${error?.stack ?? "[noStack]"}`,
+              },
+            },
+          }
+        if (matchRpcResult.test(error)) return error
+        return {
+          error: {
+            code: 0,
+            message: String(error),
+          },
+        }
+      })
   }
   async exit(effects: HostSystemStartOs): Promise<void> {
     if (this.currentRunning) await this.currentRunning.clean()
@@ -157,6 +172,7 @@ export class SystemForEmbassy implements System {
             return this.dependenciesAutoconfig(effects, procedures[2], input)
         }
     }
+    throw new Error(`Could not find the path for ${options.procedure}`)
   }
   private async init(
     effects: HostSystemStartOs,
@@ -864,6 +880,7 @@ async function updateConfig(
 ) {
   if (!dictionary([string, unknown]).test(spec)) return
   if (!dictionary([string, unknown]).test(mutConfigValue)) return
+  const utils = createUtils(effects)
   for (const key in spec) {
     const specValue = spec[key]
 
@@ -890,11 +907,18 @@ async function updateConfig(
       mutConfigValue[key] = configValue
     }
     if (matchPointerPackage.test(specValue)) {
-      mutConfigValue[key] = await effects.embassyGetInterface({
-        target: specValue.target,
-        packageId: specValue["package-id"],
-        interface: specValue["interface"],
-      })
+      const filled = await utils.serviceInterface
+        .get({
+          packageId: specValue["package-id"],
+          id: specValue.interface,
+        })
+        .once()
+      if (specValue.target === "tor-key")
+        throw new Error("This service uses an unsupported target TorKey")
+      mutConfigValue[key] =
+        specValue.target === "lan-address"
+          ? filled.addressInfo.localHostnames[0]
+          : filled.addressInfo.onionHostnames[0]
     }
   }
 }
diff --git a/container-runtime/src/Adapters/Systems/SystemForStartOs.ts b/container-runtime/src/Adapters/Systems/SystemForStartOs.ts
@@ -3,6 +3,7 @@ import { unNestPath } from "../../Models/JsonPath"
 import { string } from "ts-matches"
 import { HostSystemStartOs } from "../HostSystemStartOs"
 import { Effects } from "../../Models/Effects"
+import { RpcResult } from "../RpcListener"
 const LOCATION = "/usr/lib/startos/package/startos"
 export class SystemForStartOs implements System {
   private onTerm: (() => Promise<void>) | undefined
@@ -30,8 +31,8 @@ export class SystemForStartOs implements System {
       input: unknown
       timeout?: number | undefined
     },
-  ): Promise<ExecuteResult> {
-    return { ok: await this._execute(effects, options) }
+  ): Promise<RpcResult> {
+    return { result: await this._execute(effects, options) }
   }
   async _execute(
     effects: Effects,

diff --git a/container-runtime/src/Interfaces/System.ts b/container-runtime/src/Interfaces/System.ts
@@ -1,6 +1,7 @@
 import { types as T } from "@start9labs/start-sdk"
 import { JsonPath } from "../Models/JsonPath"
 import { HostSystemStartOs } from "../Adapters/HostSystemStartOs"
+import { RpcResult } from "../Adapters/RpcListener"
 export type ExecuteResult =
   | { ok: unknown }
   | { err: { code: number; message: string } }
@@ -17,7 +18,7 @@ export interface System {
       input: unknown
       timeout?: number
     },
-  ): Promise<ExecuteResult>
+  ): Promise<RpcResult>
   // sandbox(
   //   effects: Effects,
   //   options: {

diff --git a/container-runtime/src/Models/CallbackHolder.ts b/container-runtime/src/Models/CallbackHolder.ts
@@ -7,7 +7,9 @@ export class CallbackHolder {
     return this.root + (this.inc++).toString(36)
   }
   addCallback(callback: Function) {
-    return this.callbacks.set(this.newId(), callback)
+    const id = this.newId()
+    this.callbacks.set(id, callback)
+    return id
   }
   callCallback(index: string, args: any[]): Promise<unknown> {
     const callback = this.callbacks.get(index)