Can STUNner be used with vCluster?

[ryansburgoyne]

I am trying to use STUNner on CoreWeave, which limits my access to the cluster to a specific namespace and prevents installing CRDs. But they do have support for vCluster which creates a virtual cluster in the namespace. I was able to install the gateway operator, as well as all the resources described in the README. I have successfully configured STUNner on a different cluster where I have full access. But in vCluster, no udp-gateway service was created.

[rg0now]

I see three options:

• Install the STUNner dataplane manually into the CoreWeave cluster. This way you lose the ease-of-use provided by the operator so you must configure stunnerd through the config file, but at least you do not need to fight against CoreWeave policies.
• Go with vCluster and we'll try to debug the problem together. For this we'll need all the operator logs.
• Theoretically you can manage the STUNner dataplane from a remote cluster, so you can install the operator into a regular Kubernetes cluster, deploy the CRDs and configure STUNner in that cluster, and then deploy the STUNner dataplane into the CodeWeave cluster and configure it to take its running config from the remote cluster. This solution is totally untested and may come with strange side-effects, but at least you get the full story.

[ryansburgoyne]

This is my config:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: stunner-gatewayclass
spec:
  controllerName: "stunner.l7mp.io/gateway-operator"
  parametersRef:
    group: "stunner.l7mp.io"
    kind: GatewayConfig
    name: stunner-gatewayconfig
    namespace: stunner
  description: "STUNner is a WebRTC media gateway for Kubernetes"

---
apiVersion: stunner.l7mp.io/v1
kind: GatewayConfig
metadata:
  name: stunner-gatewayconfig
  namespace: stunner
spec:
  realm: stunner.l7mp.io
  authRef:
    name: stunner-auth-secret
    namespace: stunner

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: udp-gateway
  namespace: stunner
  annotations:
    metallb.universe.tf/address-pool: public-ord1
    metallb.universe.tf/allow-shared-ip: default
spec:
  gatewayClassName: stunner-gatewayclass
  listeners:
    - name: udp-listener
      port: 3478
      protocol: TURN-UDP

---
apiVersion: stunner.l7mp.io/v1
kind: UDPRoute
metadata:
  name: stunner-headless
  namespace: stunner
spec:
  parentRefs:
    - name: udp-gateway
  rules:
    - backendRefs:
        - group: stunner.l7mp.io
          kind: StaticService
          name: wildcard

---
apiVersion: stunner.l7mp.io/v1alpha1
kind: StaticService
metadata:
  name: wildcard
  namespace: stunner
spec:
  prefixes:
    - "0.0.0.0/0"

[ryansburgoyne]

For option 1, I found https://docs.l7mp.io/en/v0.15.1/OBSOLETE/. Is that the approach you would recommend?

[rg0now]

Basically yes, but the config file format has changed a lot since then. Here is the process that should work (fully untested):

• Start with a fresh cluster, install STUNner, deploy and configure the CRDs, and let the operator render a config for you.
• Compile stunnerctl (make build in the root dir) and issue bin/stunnerctl -n <my-namespace> config <my-gateway> -o yaml to obtain the running config for the gateway you want to clone into the CoreWeave cluster. The manual gives a sample config.
• Load the config into a ConfigMap in the CoreWeave cluster and edit the parameters you want to change.
• Deploy the dataplane in standalone mode (the l7mp/stunner Helm chart may work, but it is fully untested since we default to the managed mode).
• Map the ConfigMap into the file-system of the stunnerd container, rewrite the command line args in the contained manifest so that stunnerd takes the config from the config file we have just mapped into the file-system (see the manual for details), set the environment variables you want to set, and restart the STUNner dataplane.
• Test and reiterate until you get something workable.

I understand this is a terrible PITA, but as long as CoreWeave gives you a crippled Kubernetes experience (frankly, I fail to see why they believe that Kubernetes would be usable without CRDs to any reasonable extent) that's the best I can come up with.

Option 3 is still there to get a managed STUNner, but it requires testing.

[ryansburgoyne]

OK this is what I was able to figure out by adapting https://github.com/l7mp/stunner/blob/main/deploy/manifests/static/stunner-manifests.yaml:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: stunner-config-watcher-serviceaccount
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: stunner-config-watcher-role
rules:
- apiGroups: [""]
  resources: ["configmaps", "secrets"]
  verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: stunner-config-watcher-rolebind
roleRef:
  kind: Role
  name: stunner-config-watcher-role
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: stunner-config-watcher-serviceaccount
---
apiVersion: v1
kind: Service
metadata:
  name: stunner
  labels:
    app: stunner
  annotations:
    # To get a CoreWeave public IP
    metallb.universe.tf/address-pool: public-ord1
spec:
  selector:
    app: stunner
  ports:
    - port: 3478
      protocol: UDP
      name: stunner
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: stunner
  annotations:
    app: stunner
    helm.sh/chart: stunner-0.19.0
    app.kubernetes.io/version: "0.19.0"
spec:
  selector:
    matchLabels:
      app: stunner
      app.kubernetes.io/name: stunner
      app.kubernetes.io/instance: stunner
  replicas: 1
  template:
    metadata:
      labels:
        app: stunner
        app.kubernetes.io/name: stunner
        app.kubernetes.io/instance: stunner
    spec:
      serviceAccountName: stunner-config-watcher-serviceaccount
      volumes:
        - name: stunnerd-config-volume
      terminationGracePeriodSeconds: 3600
      hostNetwork: false
      containers:
      - name: stunnerd
        image: "l7mp/stunnerd:0.19.0"
        imagePullPolicy: IfNotPresent
        command: ["stunnerd"]
        args: ["-w", "-c", "file:///etc/stunnerd/stunnerd.conf", "--udp-thread-num=16"]
        env:
          - name: STUNNER_ADDR
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.podIP
          - name: STUNNER_USERNAME
            valueFrom:
              secretKeyRef:
                name: stunner-auth-secret
                key: username
          - name: STUNNER_PASSWORD
            valueFrom:
              secretKeyRef:
                name: stunner-auth-secret
                key: password
        volumeMounts:
          - name: stunnerd-config-volume
            mountPath: /etc/stunnerd
            readOnly: true
        resources:
          limits:
            cpu: "2"
            memory: 512Mi
          requests:
            cpu: 500m
            memory: 128Mi
        livenessProbe:
          httpGet:
            path: /live
            port: 8086
          initialDelaySeconds: 0
          periodSeconds: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: 8086
          initialDelaySeconds: 0
          periodSeconds: 5
        ports:
          - name: web
            containerPort: 8080
      - name: config-watcher
        image: kiwigrid/k8s-sidecar:latest
        imagePullPolicy: IfNotPresent
        volumeMounts:
          - name: stunnerd-config-volume
            mountPath: /etc/stunnerd
        env:
        - name: LABEL
          value: app
        - name: LABEL_VALUE
          value: stunner
        - name: FOLDER
          value: /etc/stunnerd
        - name: RESOURCE
          value: configmap
        resources:
          limits:
            cpu: 500m
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 128Mi
      nodeSelector:
        kubernetes.io/os: linux
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              # To schedule in the same region as the load balancer
              - key: topology.kubernetes.io/region
                operator: In
                values:
                  - ORD1

I had to deploy stunner-auth-secret as well. Then I created:

apiVersion: v1
kind: ConfigMap
metadata:
  name: stunner
  labels:
    app: stunner
data:
  stunnerd.conf: |-
    version: v1alpha1
    admin:
      name: stunner
      logLevel: all:INFO
      realm: stunner.l7mp.io
    auth:
      type: static
      credentials:
        username: ${STUNNER_USERNAME}
        password: ${STUNNER_PASSWORD}
    listeners:
      - name: stunnerd-udp
        address: ${STUNNER_ADDR}
        protocol: turn-udp
        port: 3478
        routes:
          - default
    clusters:
      - name: default
        endpoints:
          - "0.0.0.0/0"

This setup lets me re-apply the config without having to restart stunnerd. A couple of questions:

1. The auth variables in the config aren't working. I have to hard-code the credentials into the ConfigMap. But $STUNNER_ADDR does seem to work. From the stunnerd README, it seemed like environment variables would be replaced on initialization.
2. I am using STUNner for Unreal Engine pixel streaming. I have a StatefulSet with one replica per user of my app. Since it is a StatefulSet, it uses a headless service—there is no single IP to point the endpoint to. Is there a better way to configure this than 0.0.0.0/0?

Regarding CoreWeave, it is annoying that I can't install CRDs, but I don't know how else they could provide access to their cluster, which is running on bare metal. The only alternative I can think of would be to have a cluster per tenant, and then we would have access to a much more limited node pool. They've pre-installed the most popular apps in their cluster (ArgoCD, Traefik, etc.).

[rg0now]

The auth variables in the config aren't working. I have to hard-code the credentials into the ConfigMap. But $STUNNER_ADDR does seem to work. From the stunnerd README, it seemed like environment variables would be replaced on initialization.

We switch environment variable substitution off when parsing the auth credentials. The reason is that some users reported isues for username/password pairs that contain a $ sign (in many use cases the credentials are random generated). I'm afraid you will have to hardcode the credentials into the config file...