docs: update dev docs with buck / nix info

.github/workflows/buck2-test.yaml

@@ -23,10 +23,10 @@ jobs:
           for LABEL in $(jq -r '.[]' < labels.json); do
             case "$LABEL" in
               dashboard|consent)
-                ARGS+=" //apps/$LABEL:test"
+                ARGS+=" //apps/$LABEL:test-unit"
                 ;;
               core)
-                ARGS+=" //core/api:test"
+                ARGS+=" //core/api:test-unit"
                 ;;
             esac
           done

README.md

@@ -3,29 +3,29 @@
 [![Mattermost](https://img.shields.io/badge/chat-on%20mattermost-blue?style=social&logo=mattermost)](https://chat.galoy.io)
 [![GitHub Repo stars](https://img.shields.io/github/stars/GaloyMoney/galoy?style=social)](https://github.com/GaloyMoney/galoy/stargazers)
 [![Twitter Follow](https://img.shields.io/twitter/follow/GaloyMoney?style=social)](https://twitter.com/GaloyMoney)
-### 💡 Get help
+## 💡 Get help
 [Q&A](https://github.com/GaloyMoney/galoy/discussions) or [Mattermost 💬](https://chat.galoy.io)
 
-### TLDR
+## TLDR
 
 Galoy is an opinionated bitcoin banking platform.
 
 This repo represents the main api that brings all functionality together.
-Take a look at the [Quickstart](./quickstart) if you want to take it for a spin.
 
-### Responsible disclosure 
+
+## Responsible disclosure 
 
 Found critical bugs/vulnerabilities?
 Please email them [email protected] Thanks!
 
-### Get Started
+## Get Started
 
-Want to try it out and contribute? Checkout the [dev documentation](./DEV.md) to deploy locally with a docker compose script.
+Want to try it out and contribute? Checkout the [dev documentation](./docs/DEVELOPMENT_ENVIRONMENT.md) to deploy locally with a docker compose script.
 
 If you have questions, you can [join our Workspace](https://chat.galoy.io)
 
 For an overview of all relevant repository checkout [awesome-galoy](https://github.com/GaloyMoney/awesome-galoy).
-### Galoy-Backend features
+## Galoy-Backend features
 
 - GraphqlAPI:
   - Public API following industry best practices
@@ -80,13 +80,90 @@ For an overview of all relevant repository checkout [awesome-galoy](https://gith
   - Mobile clients can receive notifications of balance changes in real-time
   - Daily notification of balance for active end users
 
-### Tech Stack
+## Local Development Setup
+
+Running the Galoy software locally can be done in a variety of ways, but this abbreviated section will focus on a single method for
+getting your environment ready to run the stack.
+For more information and options on running Galoy locally, see the [development environment documentation](./docs/DEVELOPMENT_ENVIRONMENT.md).
+
+### (1) Choose a Supported Platform
+
+Currently developming on MacOS and Linux is supported.
+
+### (2) Install Dependencies
+
+Install dependencies on your chosen platform.
+
+- **1)** [`nix` with flakes enabled](https://github.com/DeterminateSystems/nix-installer)
+- **2)** `docker` from [Docker Desktop](https://www.docker.com/products/docker-desktop/) or [Docker Engine](https://docs.docker.com/engine/)
+- **3a)** [`direnv`](https://direnv.net) version `>= 2.30` installed
+- **3b)** [`direnv` hooked into your shell](https://direnv.net/docs/hook.html)
+
+For `nix`, we highly recommend using the [Determinate Nix Installer](https://github.com/DeterminateSystems/nix-installer).
+
+For `docker`, the Docker Desktop version corresponding to your native architecture should be used.
+
+For `direnv`, you can install it with [your package manager of choice](https://direnv.net/docs/installation.html).
+However, if you're unsure which installation method to use or your package manager does not provide a compatible version,
+you can use `nix` itself (e.g. `nix profile install nixpkgs#direnv`).
+
+> We recommend using [the upstream docs for hooking `direnv` into your shell](https://direnv.net/docs/hook.html), but here is an example on how to do it
+> on a system where `zsh` is the default shell.
+> In this example, the following is added to the end of `~/.zshrc`.
+>
+> ```zsh
+> if [ $(command -v direnv) ]; then
+>    eval "$(direnv hook zsh)"
+> fi
+> ```
+
+### (3) Enter the Repository Directory
+
+All commands need to be run from the `nix` environment.
+Since `direnv` is installed _and_ hooked into your shell, you can `cd` into
+the repository and `nix` will boostrap the environment for you using the flake.
+
+_Please note: you may notice a large download of dependencies when entering the repository for the first time._
+
+### (4) Running the Stack
 
-- GCP, Kubernetes, Terraform, Helm, Concourse, Docker
-- Opentelemetry, Prometheus
-- Bitcoind, LND, Specter, RideTheLightning, Loop, Lndmon, Pool
-- PostgreSQL, MongoDB, Redis
-- NodeJS
-- Typescript
-- GraphQL
-- React + React Native
+We use [**buck2**](https://github.com/facebook/buck2) to run the stack, run and build individual services and libraries, perform lints and tests, etc.
+
+_Before continuing, you should stop any locally running services to avoid conflicting ports with the stack.
+Some of the services that will run include, but are not limited to the following: PostgreSQL, OpenTelemetry, MongoDB.
+
+Check if you are ready to run the stack before continuing.
+
+```bash
+buck2 run dev:healthcheck
+```
+
+You may notice some checks related to resource limits.
+On macOS and in WSL2 in particular, we recommend significantly increasing the file descriptor limit for `buck2` to work as intended (e.g. `ulimit -n 10240`).
+_Please note: the new file descriptor limit may not persist to future sessions._
+
+Once ready, we can build relevant services and run the entire stack locally.
+
+```bash
+buck2 run dev:up
+```
+
+Once Tilt starts, you can check on the status of all services by accessing the UI through the given port on your local host (e.g. [http://localhost:10350/](http://localhost:10350/)).
+Every service should eventually have a green checkmark next to them, which ensures that they are in "ready" states.
+
+### (6) Troubleshooting in Tilt
+
+If some services failed to start, you can restart them on the Tilt dashboard.
+
+### (7) Tearing Down the Stack
+
+The following command will stop all running services and containers.
+It will also remove the containers and, consequentially, the data held in them.
+
+```bash
+buck2 run dev:down
+```
+
+## Run integration tests with galoy as a dependency?
+
+Take a look at the [Quickstart](./quickstart) if you want to take it for a spin.

apps/consent/BUCK

@@ -18,7 +18,7 @@ dev_pnpm_task_binary(
 )
 
 dev_pnpm_task_test(
-  name = "cypress",
+  name = "test-integration",
   command = "cypress:run",
 )
 
@@ -68,7 +68,7 @@ eslint(
 )
 
 test_suite(
-    name = "test",
+    name = "test-unit",
     tests = [
         ":check-lint",
     ],

apps/dashboard/BUCK

@@ -52,7 +52,7 @@ eslint(
 )
 
 test_suite(
-    name = "test",
+    name = "test-unit",
     tests = [
         ":check-lint",
     ],

core/api/BUCK

@@ -101,7 +101,7 @@ madge_check(
 )
 
 test_suite(
-    name = "test",
+    name = "test-unit",
     tests = [
         ":check-lint",
         ":check-type",

dev/BUCK

@@ -9,3 +9,9 @@ tilt_up(
 tilt_down(
     name = "down",
 )
+
+python_bootstrap_binary(
+    name = "healthcheck",
+    main = "healthcheck.py",
+    visibility = ["PUBLIC"],
+)

dev/Tiltfile

@@ -30,7 +30,7 @@ groups = {
     ],
 }
 
-consent_test_target = "//apps/consent:cypress"
+consent_test_target = "//apps/consent:test-integration"
 local_resource(
   "consent-test",
   labels = ["test"],

dev/healthcheck.py

@@ -0,0 +1,165 @@
+#!/usr/bin/env python3
+"""
+Detects and reports the max number of open files allowed per process.
+"""
+import argparse
+import subprocess
+import shutil
+import sys
+
+MIN_OPENFILES_VALUE = 1024
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.parse_args()
+
+    remediations = 0
+
+    section("Running health check on system for developing in the SI project")
+
+    docker_cmd = detect_docker_command()
+    remediations += docker_cmd
+    if docker_cmd == 0:
+        remediations += detect_docker_engine()
+        remediations += detect_docker_compose()
+    remediations += detect_nix_bash()
+    remediations += detect_openfiles_soft_limit()
+
+    blank()
+    if remediations > 0:
+        section(f"Health check complete with _{remediations}_ "
+                "suggested remediation(s)")
+    else:
+        section(f"Health check complete with **no** suggested remediations.")
+
+    # If there are remediations, exit non-zero so scripts can detect that the
+    # current setup may not be healthy
+    return remediations
+
+
+def detect_docker_command() -> int:
+    result = detect_command("docker")
+    return result
+
+
+def detect_docker_engine() -> int:
+    result = subprocess.run(["docker", "info"], capture_output=True)
+    if result.returncode == 0:
+        info("Detected running Docker Engine")
+        return 0
+    else:
+        warn("Failed to detect running Docker Engine")
+        indent("Output from `docker info` stderr:")
+        indent("-----")
+        for line in result.stderr.splitlines():
+            indent(line.decode("ascii"))
+        indent("-----")
+        indent("Ensure that the Docker Engine is running and try again")
+        return 1
+
+
+def detect_docker_compose() -> int:
+    result = subprocess.run(
+        [
+            "docker",
+            "compose",
+            "version",
+        ],
+        capture_output=True,
+    )
+    if result.returncode == 0:
+        info("Detected Docker Compose is installed")
+        return 0
+    else:
+        warn("Failed to detect Docker Compose installation")
+        indent("Output from `docker compose version` stderr:")
+        indent("-----")
+        for line in result.stderr.splitlines():
+            indent(line.decode("ascii"))
+        indent("-----")
+        indent("Ensure that the Docker Compose is running and try again")
+        return 1
+
+
+def detect_nix_bash() -> int:
+    cmd = "bash"
+    result = shutil.which(cmd)
+    if result and result.startswith("/nix/store/"):
+        info(f"Found `{cmd}` in Nix environment")
+        return 0
+    elif result:
+        warn(f"Failed to find `{cmd}` in Nix environment (cmd={result})")
+        indent("Ensure that your direnv setup is correct or that you have ran "
+               "`nix develop`")
+        return 1
+    else:
+        warn(f"Failed to find `{cmd}` in Nix environment or on system")
+        indent("Ensure that your direnv setup is correct or that you have ran "
+               "`nix develop`")
+        return 1
+
+
+def detect_openfiles_soft_limit() -> int:
+    soft_limit = current_openfiles_soft_limit()
+
+    if soft_limit < MIN_OPENFILES_VALUE:
+        new = MIN_OPENFILES_VALUE
+        warn("Low value for max open files soft limit detected in "
+             f"current shell (current={soft_limit})")
+        blank()
+        indent("To set a value for this shell session **only** run:")
+        indent(f"ulimit -Sn {new}", amount=2)
+        blank()
+        indent(
+            "To make a **permanent** change for all new shell sessions run:")
+        indent(f"echo 'ulimit -Sn {new}' >>\"$HOME/.profile\"", amount=2)
+        return 1
+    else:
+        info("Reasonable value for max open files soft limit detected in "
+             f"current shell (current={soft_limit})")
+        return 0
+
+
+def detect_command(cmd: str) -> int:
+    result = shutil.which(cmd)
+    if result:
+        info(f"Detected `{cmd}` command (cmd={result})")
+        return 0
+    else:
+        warn(f"Failed to find `{cmd}` on PATH and is required for development")
+        return 1
+
+
+def current_openfiles_soft_limit() -> int:
+    result = subprocess.run(
+        ["bash", "-c", "ulimit -Sn"],
+        capture_output=True,
+    )
+    result.check_returncode()
+    return int(result.stdout.strip().decode("ascii"))
+
+
+def section(msg: str):
+    print(f"--- {msg}")
+
+
+def info(msg: str):
+    print(f"  - {msg}")
+
+
+def warn(msg: str):
+    print(f"  x {msg}")
+
+
+def indent(msg: str, amount=1):
+    print("{:>{width}}{}".format("", msg, width=amount * 4))
+
+
+def blank():
+    print("")
+
+
+if __name__ == "__main__":
+    sys.exit(main())
+