local-development.md

HelixML Local Development guide

Welcome to the local development documentation for Helix.ml! This guide will help you set up your development environment, understand the project structure, run the application locally, and contribute to the Helix.

Prerequisites

Before you start, ensure you have the following software installed on your machine:

• docker
• golang
• Node.js and npm

Setting Up the Development Environment

1. Clone the Repository

   git clone [email protected]:helixml/helix.git
   cd helix

   If you are an external contributor, consider working out of a forked repository of Helix.

2. Set Up Environment Variables

   Create an .env file with settings based on the example values and edit it:

   cp .env.example-prod .env
   

   The default values for settings are optimised for local development.

Project Structure

Here is an overview of the project structure:

helix/
├── Dockerfiles         # Dockerfiles for various environments
├── api/                # Main Control Plane API directory
│   ├── cmd/            # Standard golang project structure within here
│   ├── pkg/            #
│   ├── main.go         #
├── llamaindex/         # llamaindex
│   └── src/            #
│   └── ...             # Other app-specific files
├── unstructured        # Python Unstructured for parsing content
├── scripts             # Scripts to get stuff done
├── runner              # Runner configurations
├── frontend/           # Frontend in React, ts
│   ├── package.json    # npm dependencies
│   └── src/            # Source files for the frontend
└── .env                # Environment variables file

Running the Application

1. Bring up the Helix stack

./stack up

This will bring up the control plane which serves the front-end and various other components on the stack. Refer Helix architecture docs

The control comes up on http://localhost:8080 by default.

Sanity check your environment with

docker ps

This should show you the running containers

$ docker ps
IMAGE                                       PORTS                                       NAMES
ankane/pgvector                             0.0.0.0:5433->5432/tcp, :::5433->5432/tcp   helix-pgvector-1
helix-frontend                              0.0.0.0:8081->8081/tcp, :::8081->8081/tcp   helix-frontend-1
helix-gptscript_runner                                                                  helix-gptscript_runner-1
registry.helix.ml/helix/llamaindex:latest                                               helix-llamaindex-1
webhookrelay/webhookrelayd                                                              helix-webhook_relay_github-1
webhookrelay/webhookrelayd                                                              helix-webhook_relay_stripe-1
helix-api                                   0.0.0.0:8080->80/tcp, :::8080->80/tcp       helix-api-1
postgres:12.13-alpine                       0.0.0.0:5432->5432/tcp, :::5432->5432/tcp   helix-postgres-1
quay.io/keycloak/keycloak:23.0              8080/tcp, 8443/tcp                          helix-keycloak-1

2. Attach a runner

Follow the instructions on the docs to attach a runner

If you're local machine isn't able to host a runner, you have a few options:

• use a VSCode remote SSH session to develop within a machine that does have the resources.
• spin up a remote runner in Luke's bunker (or equivalent) and connect it back to your localhost via an SSH tunnel.
• spin up a remote runner in runpod (or equivalent) and use webhookrelay to connect that machine back to your localhost.

Connecting to a Runner via SSH

To connect your localhost to a remote runner via an SSH tunnel, follow these steps:

1. In a separate window, SSH into a remote machine and open a connection from the remote back to local:

   ssh -p $SSH_PORT -R 8080:localhost:8080 [email protected]

   Where 8080 is the port that your local API is running on.

2. On the remote: git clone https://github.com/helixml/helix somewhere

3. On the remote create a .env file with the following settings:

   SERVER_PORT=9080 # By default, the runner runs on 8080, so use another port.
   API_HOST=http://localhost:8080 # You've just forwarded this port back to your local machine
   API_TOKEN=oh-hallo-insecure-token # This should match the control plane

4. On the remote start the runner: docker compose -f docker-compose.runner.yaml up -d

5. Now go back to your local machine and browse to /dashboard in Helix. You should see the runner. If not, take a look at the runner logs on the remote.

Connecting to a Runner via WebhookRelay

To connect to a runner via WebhookRelay:

1. Create a free account with https://webhookrelay.com/

2. Create a new access token and make a note of it: https://my.webhookrelay.com/tokens

3. Create a new "tunnel": https://my.webhookrelay.com/tunnels. Give it a name of helix, based in the eu, with a destination of http://localhost:8080. Click create. Make a note of the "Host" that looks something like: https://9hxxxxxxxxx.webrelay.io

4. Add or edit your docker-compose.dev.yaml to include the following configuration. Here we're using host network mode so it can connect to the api which is exposed on 8080. You could change the destination in the previous step to point to somewhere else (like api:80 for example).

   webhook_relay_localhost:
       image: webhookrelay/webhookrelayd
       network_mode: "host"
       environment:
       - KEY=${WEBHOOK_RELAY_KEY:-}
       - SECRET=${WEBHOOK_RELAY_SECRET:-}
       command: ["--mode", "tunnel", "-t", "helix"]

5. Start your remote runner with the API_HOST variable set to the webrelay.io url, e.g.: --api-host https://xxxx.webrelay.io.

6. Watch the runner logs to make sure it's able to connect to your local helix.

3. (Optional) Expose a Github Webhook

If you're testing, developing or working with Apps then you will need to connect a Github OAuth app to be able to read from user's repositories. The current way of doing this is via https://webhookrelay.com/.

1. Create a free account with https://webhookrelay.com/
2. Create a "bucket", with "forward to an internal location: https://my.webhookrelay.com/new-internal-destination
3. Set the Destination URL to: http://localhost:8080/api/v1/github/webhook
4. This should produce a url that looks something like https://xxxx.hooks.webhookrelay.com
5. Go to the Github oauth app settings and find the HelixML app called Helix apps development. Generate a new secret and grab the client ID.
6. Set the following environmental variables in the .env file:

GITHUB_INTEGRATION_ENABLED=true
GITHUB_INTEGRATION_CLIENT_ID=6d04xxxxxxxx
GITHUB_INTEGRATION_CLIENT_SECRET=xxxxxx
GITHUB_INTEGRATION_WEBHOOK_URL=https://3nmlcxxxxx.hooks.webhookrelay.com/api/v1/github/webhook

7. Restart the API and create an app.
8. If you have any problems check the logs in webhook relay bucket.

4. Rebuild individual components

./stack up --build <component>

If you're familiar with tmux you will find it useful to do ./stack start and ./stack stop instead.

Both the frontend and the api have hot-reloads when in development mode. Rebuilds should only be required when adding libraries.

5. Running linter

Important

You must install golangci-lint See here for installation options.

./stack lint

6. Running tests

./stack test

7. Tear down the Helix stack

Bring down the stack

./stack stop

Debugging

• View all Docker logs

  docker logs <container-name>
  

Contributing

1. Branching Strategy

   • Create a new branch for each feature or bugfix:

     git checkout -b feature/your-feature-name

2. Code Style

   • Format all code with standard language formatters.
   • Follow the project's coding guidelines for the frontend.
   • Run Go linter

3. Commit Messages

   • Write clear and concise commit messages.

4. Pull Requests

   • Submit a pull request to the main branch for review.

Further Reading

• Helix Documentation

Happy coding! If you have any questions or run into issues, feel free to reach out to the maintainers on 👥 Discord.