From 762e0ee2b4ffe010611d4c633b865f7f9b5667ea Mon Sep 17 00:00:00 2001 From: bodymindarts Date: Thu, 26 Oct 2023 11:14:41 +0200 Subject: [PATCH] docs: update dev docs with buck / nix info --- .github/workflows/buck2-test.yaml | 4 +- README.md | 109 +++++++++++++++++--- apps/consent/BUCK | 4 +- apps/dashboard/BUCK | 2 +- core/api/BUCK | 2 +- dev/BUCK | 6 ++ dev/Tiltfile | 2 +- dev/healthcheck.py | 165 ++++++++++++++++++++++++++++++ docs/BUCK2.md | 120 ++++++++++++++++++++++ docs/DEVELOPMENT_ENVIRONMENT.md | 145 ++++++++++++++++++++++++++ DEV.md => docs/LEGACY_DEV.md | 0 11 files changed, 536 insertions(+), 23 deletions(-) create mode 100644 dev/healthcheck.py create mode 100644 docs/BUCK2.md create mode 100644 docs/DEVELOPMENT_ENVIRONMENT.md rename DEV.md => docs/LEGACY_DEV.md (100%) diff --git a/.github/workflows/buck2-test.yaml b/.github/workflows/buck2-test.yaml index 43225d4c2a7..c77e6a1c097 100644 --- a/.github/workflows/buck2-test.yaml +++ b/.github/workflows/buck2-test.yaml @@ -23,10 +23,10 @@ jobs: for LABEL in $(jq -r '.[]' < labels.json); do case "$LABEL" in dashboard|consent) - ARGS+=" //apps/$LABEL:test" + ARGS+=" //apps/$LABEL:test-unit" ;; core) - ARGS+=" //core/api:test" + ARGS+=" //core/api:test-unit" ;; esac done diff --git a/README.md b/README.md index c4298460ee4..9a67c769dd7 100644 --- a/README.md +++ b/README.md @@ -3,29 +3,29 @@ [![Mattermost](https://img.shields.io/badge/chat-on%20mattermost-blue?style=social&logo=mattermost)](https://chat.galoy.io) [![GitHub Repo stars](https://img.shields.io/github/stars/GaloyMoney/galoy?style=social)](https://github.com/GaloyMoney/galoy/stargazers) [![Twitter Follow](https://img.shields.io/twitter/follow/GaloyMoney?style=social)](https://twitter.com/GaloyMoney) -### 💡 Get help +## 💡 Get help [Q&A](https://github.com/GaloyMoney/galoy/discussions) or [Mattermost 💬](https://chat.galoy.io) -### TLDR +## TLDR Galoy is an opinionated bitcoin banking platform. This repo represents the main api that brings all functionality together. -Take a look at the [Quickstart](./quickstart) if you want to take it for a spin. -### Responsible disclosure + +## Responsible disclosure Found critical bugs/vulnerabilities? Please email them security@galoy.io Thanks! -### Get Started +## Get Started -Want to try it out and contribute? Checkout the [dev documentation](./DEV.md) to deploy locally with a docker compose script. +Want to try it out and contribute? Checkout the [dev documentation](./docs/DEVELOPMENT_ENVIRONMENT.md) to deploy locally with a docker compose script. If you have questions, you can [join our Workspace](https://chat.galoy.io) For an overview of all relevant repository checkout [awesome-galoy](https://github.com/GaloyMoney/awesome-galoy). -### Galoy-Backend features +## Galoy-Backend features - GraphqlAPI: - Public API following industry best practices @@ -80,13 +80,90 @@ For an overview of all relevant repository checkout [awesome-galoy](https://gith - Mobile clients can receive notifications of balance changes in real-time - Daily notification of balance for active end users -### Tech Stack +## Local Development Setup + +Running the Galoy software locally can be done in a variety of ways, but this abbreviated section will focus on a single method for +getting your environment ready to run the stack. +For more information and options on running Galoy locally, see the [development environment documentation](./docs/DEVELOPMENT_ENVIRONMENT.md). + +### (1) Choose a Supported Platform + +Currently developming on MacOS and Linux is supported. + +### (2) Install Dependencies + +Install dependencies on your chosen platform. + +- **1)** [`nix` with flakes enabled](https://github.com/DeterminateSystems/nix-installer) +- **2)** `docker` from [Docker Desktop](https://www.docker.com/products/docker-desktop/) or [Docker Engine](https://docs.docker.com/engine/) +- **3a)** [`direnv`](https://direnv.net) version `>= 2.30` installed +- **3b)** [`direnv` hooked into your shell](https://direnv.net/docs/hook.html) + +For `nix`, we highly recommend using the [Determinate Nix Installer](https://github.com/DeterminateSystems/nix-installer). + +For `docker`, the Docker Desktop version corresponding to your native architecture should be used. + +For `direnv`, you can install it with [your package manager of choice](https://direnv.net/docs/installation.html). +However, if you're unsure which installation method to use or your package manager does not provide a compatible version, +you can use `nix` itself (e.g. `nix profile install nixpkgs#direnv`). + +> We recommend using [the upstream docs for hooking `direnv` into your shell](https://direnv.net/docs/hook.html), but here is an example on how to do it +> on a system where `zsh` is the default shell. +> In this example, the following is added to the end of `~/.zshrc`. +> +> ```zsh +> if [ $(command -v direnv) ]; then +> eval "$(direnv hook zsh)" +> fi +> ``` + +### (3) Enter the Repository Directory + +All commands need to be run from the `nix` environment. +Since `direnv` is installed _and_ hooked into your shell, you can `cd` into +the repository and `nix` will boostrap the environment for you using the flake. + +_Please note: you may notice a large download of dependencies when entering the repository for the first time._ + +### (4) Running the Stack -- GCP, Kubernetes, Terraform, Helm, Concourse, Docker -- Opentelemetry, Prometheus -- Bitcoind, LND, Specter, RideTheLightning, Loop, Lndmon, Pool -- PostgreSQL, MongoDB, Redis -- NodeJS -- Typescript -- GraphQL -- React + React Native +We use [**buck2**](https://github.com/facebook/buck2) to run the stack, run and build individual services and libraries, perform lints and tests, etc. + +_Before continuing, you should stop any locally running services to avoid conflicting ports with the stack. +Some of the services that will run include, but are not limited to the following: PostgreSQL, OpenTelemetry, MongoDB. + +Check if you are ready to run the stack before continuing. + +```bash +buck2 run dev:healthcheck +``` + +You may notice some checks related to resource limits. +On macOS and in WSL2 in particular, we recommend significantly increasing the file descriptor limit for `buck2` to work as intended (e.g. `ulimit -n 10240`). +_Please note: the new file descriptor limit may not persist to future sessions._ + +Once ready, we can build relevant services and run the entire stack locally. + +```bash +buck2 run dev:up +``` + +Once Tilt starts, you can check on the status of all services by accessing the UI through the given port on your local host (e.g. [http://localhost:10350/](http://localhost:10350/)). +Every service should eventually have a green checkmark next to them, which ensures that they are in "ready" states. + +### (6) Troubleshooting in Tilt + +If some services failed to start, you can restart them on the Tilt dashboard. + +### (7) Tearing Down the Stack + +The following command will stop all running services and containers. +It will also remove the containers and, consequentially, the data held in them. + +```bash +buck2 run dev:down +``` + +## Run integration tests with galoy as a dependency? + +Take a look at the [Quickstart](./quickstart) if you want to take it for a spin. diff --git a/apps/consent/BUCK b/apps/consent/BUCK index 2dd0dd0702d..57fff3406db 100644 --- a/apps/consent/BUCK +++ b/apps/consent/BUCK @@ -18,7 +18,7 @@ dev_pnpm_task_binary( ) dev_pnpm_task_test( - name = "cypress", + name = "test-integration", command = "cypress:run", ) @@ -68,7 +68,7 @@ eslint( ) test_suite( - name = "test", + name = "test-unit", tests = [ ":check-lint", ], diff --git a/apps/dashboard/BUCK b/apps/dashboard/BUCK index 27471b9f6c6..b6b6821b6fb 100644 --- a/apps/dashboard/BUCK +++ b/apps/dashboard/BUCK @@ -52,7 +52,7 @@ eslint( ) test_suite( - name = "test", + name = "test-unit", tests = [ ":check-lint", ], diff --git a/core/api/BUCK b/core/api/BUCK index aa16a26d443..4e668980d2d 100644 --- a/core/api/BUCK +++ b/core/api/BUCK @@ -101,7 +101,7 @@ madge_check( ) test_suite( - name = "test", + name = "test-unit", tests = [ ":check-lint", ":check-type", diff --git a/dev/BUCK b/dev/BUCK index f19d31eb18c..c080b693f46 100644 --- a/dev/BUCK +++ b/dev/BUCK @@ -9,3 +9,9 @@ tilt_up( tilt_down( name = "down", ) + +python_bootstrap_binary( + name = "healthcheck", + main = "healthcheck.py", + visibility = ["PUBLIC"], +) diff --git a/dev/Tiltfile b/dev/Tiltfile index f03f9b37656..1d4d4800bf6 100644 --- a/dev/Tiltfile +++ b/dev/Tiltfile @@ -30,7 +30,7 @@ groups = { ], } -consent_test_target = "//apps/consent:cypress" +consent_test_target = "//apps/consent:test-integration" local_resource( "consent-test", labels = ["test"], diff --git a/dev/healthcheck.py b/dev/healthcheck.py new file mode 100644 index 00000000000..f11ff1f5fb0 --- /dev/null +++ b/dev/healthcheck.py @@ -0,0 +1,165 @@ +#!/usr/bin/env python3 +""" +Detects and reports the max number of open files allowed per process. +""" +import argparse +import subprocess +import shutil +import sys + +MIN_OPENFILES_VALUE = 1024 + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.parse_args() + + remediations = 0 + + section("Running health check on system for developing in the SI project") + + docker_cmd = detect_docker_command() + remediations += docker_cmd + if docker_cmd == 0: + remediations += detect_docker_engine() + remediations += detect_docker_compose() + remediations += detect_nix_bash() + remediations += detect_openfiles_soft_limit() + + blank() + if remediations > 0: + section(f"Health check complete with _{remediations}_ " + "suggested remediation(s)") + else: + section(f"Health check complete with **no** suggested remediations.") + + # If there are remediations, exit non-zero so scripts can detect that the + # current setup may not be healthy + return remediations + + +def detect_docker_command() -> int: + result = detect_command("docker") + return result + + +def detect_docker_engine() -> int: + result = subprocess.run(["docker", "info"], capture_output=True) + if result.returncode == 0: + info("Detected running Docker Engine") + return 0 + else: + warn("Failed to detect running Docker Engine") + indent("Output from `docker info` stderr:") + indent("-----") + for line in result.stderr.splitlines(): + indent(line.decode("ascii")) + indent("-----") + indent("Ensure that the Docker Engine is running and try again") + return 1 + + +def detect_docker_compose() -> int: + result = subprocess.run( + [ + "docker", + "compose", + "version", + ], + capture_output=True, + ) + if result.returncode == 0: + info("Detected Docker Compose is installed") + return 0 + else: + warn("Failed to detect Docker Compose installation") + indent("Output from `docker compose version` stderr:") + indent("-----") + for line in result.stderr.splitlines(): + indent(line.decode("ascii")) + indent("-----") + indent("Ensure that the Docker Compose is running and try again") + return 1 + + +def detect_nix_bash() -> int: + cmd = "bash" + result = shutil.which(cmd) + if result and result.startswith("/nix/store/"): + info(f"Found `{cmd}` in Nix environment") + return 0 + elif result: + warn(f"Failed to find `{cmd}` in Nix environment (cmd={result})") + indent("Ensure that your direnv setup is correct or that you have ran " + "`nix develop`") + return 1 + else: + warn(f"Failed to find `{cmd}` in Nix environment or on system") + indent("Ensure that your direnv setup is correct or that you have ran " + "`nix develop`") + return 1 + + +def detect_openfiles_soft_limit() -> int: + soft_limit = current_openfiles_soft_limit() + + if soft_limit < MIN_OPENFILES_VALUE: + new = MIN_OPENFILES_VALUE + warn("Low value for max open files soft limit detected in " + f"current shell (current={soft_limit})") + blank() + indent("To set a value for this shell session **only** run:") + indent(f"ulimit -Sn {new}", amount=2) + blank() + indent( + "To make a **permanent** change for all new shell sessions run:") + indent(f"echo 'ulimit -Sn {new}' >>\"$HOME/.profile\"", amount=2) + return 1 + else: + info("Reasonable value for max open files soft limit detected in " + f"current shell (current={soft_limit})") + return 0 + + +def detect_command(cmd: str) -> int: + result = shutil.which(cmd) + if result: + info(f"Detected `{cmd}` command (cmd={result})") + return 0 + else: + warn(f"Failed to find `{cmd}` on PATH and is required for development") + return 1 + + +def current_openfiles_soft_limit() -> int: + result = subprocess.run( + ["bash", "-c", "ulimit -Sn"], + capture_output=True, + ) + result.check_returncode() + return int(result.stdout.strip().decode("ascii")) + + +def section(msg: str): + print(f"--- {msg}") + + +def info(msg: str): + print(f" - {msg}") + + +def warn(msg: str): + print(f" x {msg}") + + +def indent(msg: str, amount=1): + print("{:>{width}}{}".format("", msg, width=amount * 4)) + + +def blank(): + print("") + + +if __name__ == "__main__": + sys.exit(main()) + diff --git a/docs/BUCK2.md b/docs/BUCK2.md new file mode 100644 index 00000000000..2a5225880b8 --- /dev/null +++ b/docs/BUCK2.md @@ -0,0 +1,120 @@ +## [buck2](https://github.com/facebook/buck2) + +This document contains information on using `buck2` within this repository. +We recommend using the `buck2` binary provided by our [Nix flake](../flake.nix) to ensure compatible versioning. + +## Terminology + +- A "target" is an instantiation of a rule +- A "rule" is a library-esque function that can be buildable, runnable and/or testable +- A "buildable" rule (`buck2 build`) only runs when affected sources are changed, and _ignores_ environment variables and passed down command-line arguments +- A "runnable" rule (`buck2 run`) runs upon every invocation, and _accepts_ environment variables and passed down command-line arguments +- A "testable" rule (`buck2 test`) runs upon every invocation and is similar to a runnable rule, but collects test metadata and is intended for sandboxed environments (e.g. CI) + +## The Shape of `buck2` Commands + +All `buck2` commands follow similar syntax. + +```shell + buck2 : -- +``` + +You can use pseudo-relative pathing to access targets. +You cannot use relative parent directories (e.g. `../../path/to/directory/with/BUCK`), but you can +use child relative directories, like in the example below. + +```shell +buck2 run apps/consent:test-integration +``` + +However, you can always use the `//` prefix to start from the root, regardless of your current +working directory in the repository. + +```bash +# Let's change our current working directory to somewhere in the repository. +cd apps/dashboard + +# Now, let's build using a BUCK file somewhere else in the repository. +buck2 build //apps/consent +``` + +You may have noticed in the example above, we could build `apps/consent` without writing `apps/consent:consent`. +If the target shares the same name as the directory, you do not have to write the name. + +```bash +# This... +buck2 build apps/consent + +# is the same as this... +buck2 build apps/consent:consent + +# and this... +buck2 build //apps/consent + +# is the same as this. +buck2 build //apps/consent:consent +``` + +## I don't have syntax highlighting when viewing `BUCK` or prelude files. Help! + +Both `BUCK` files and the rules it uses are written in [Starlark](https://github.com/bazelbuild/starlark), which is a superset of Python. + +There is an [upstream VS Code extension](https://github.com/facebook/buck2/tree/main/starlark-rust/vscode) +that you can build locally using `npm`. +After building it, you can install the `vsix` file as an extension in VS Code. + +One great thing about the extension is that you can use "Go To Definition" in VS Code to follow the path of where a rule +comes from and how it's written. + +## Where do the rules come from? + +There are two directories where the rules come from: + +- **[prelude](../prelude):** the vendored, [upstream standard library](https://github.com/facebook/buck2-prelude) with rules for common use cases and programming languages + - **Common use case example:** `sh_binary` is provided as a way to run shell scripts + - **Programming language example:** `rust_library` is a provided as a way to add buildable Rust library targets + - **Side note:** this must be kept up to date in conjunction with the `buck2` binary +- **[toolchains](../toolchains/*):** our custom toolchains to use pnpm (which is not supported out of the box) + - **Example:** `tsc_build` uses tsc to output compiled javascript + +## How do I view all targets available for a given directory? + +Run the following command, but do not forget the `:` at the end. + +```shell +buck2 targets : +``` + +Here is an example: + +```shell +buck2 targets //core/api +``` + +## Why isn't this new file I added available during builds? + +Expanding on the [terminology](#terminology) section, "buildable" targets only use files that are explicitly provided. +If your new file isn't available during builds, you likely need to do one of two things: + +- Use the `export_file` rule to ensure the file is available for builds +- Check the `srcs` attribute of a rule, if applicable, to ensure the file is in the sources tree + +## How do I run Rust tests with `buck2`? + +Check out the [RUNNING_RUST_TESTS](./RUNNING_RUST_TESTS.md) guide! +Essentially, you'll use the following pattern: + +```shell +# Pattern for unit tests + buck2 run :test-unit -- + +# Pattern for integration tests + buck2 run :test-integration -- +``` + +## Where are `buck2` users hanging out? + +[![Discord Server](https://img.shields.io/badge/discord-5865F2?style=for-the-badge&logo=discord&logoColor=white)](https://discord.gg/P5Tbrt735m) + +If you are looking to find other `buck2` users and/or ask questions, share ideas and share experiences related to `buck2`, check out the unofficial ["Buck2 Fans" Discord server](https://discord.gg/P5Tbrt735m). + diff --git a/docs/DEVELOPMENT_ENVIRONMENT.md b/docs/DEVELOPMENT_ENVIRONMENT.md new file mode 100644 index 00000000000..918045cb7c6 --- /dev/null +++ b/docs/DEVELOPMENT_ENVIRONMENT.md @@ -0,0 +1,145 @@ +# Development Environment + +Developing galoy locally can be done in a variety of ways, but the officially supported method is to use the [Nix Flake](../flake.nix) +at the root of the repository. + +## Supported Platforms + +Using the flake requires using one of the below platforms. +It is possible that the System Initiative software can be developed on even more platforms, but these platforms have +been validated to work with `nix` and the corresponding flake. + +### macOS + +macOS (Darwin) is supported on both x86_64 (amd64) (also known as "Intel") and aarch64 (arm64) (also known as +"Apple Silicon") architectures. +We do not specify the minimum version of macOS that must be used, so we recommend looking at the [Dependencies](#dependences) +section for more information. + +On macOS, you will likely hit the [file descriptor limit](#file-descriptor-limit) problem, which requires user intervention. + +### Linux + +Linux (GNU) is supported. + +### Windows + +Using native Windows is not supported at this time. + +### File Descriptor Limit + +On some systems, you may need to significantly increasing the file descriptor limit for `buck2`. +This is because `buck2` opens many more files than either `cargo` or `pnpm` do. +Not only that, but when using Tilt to build and run concurrent services, even more files are opened than they would be for sequential builds. + +Increasing the file descriptor limit is possible via the `ulimit` command. +To see all limits, execute the following command: + +```bash +ulimit -a +``` + +Here is an example of a significant limit increase, where the argument provided after the flag represents the new desired number of file descriptors: + +```bash +ulimit -n +``` + +To find an acceptable limit, run the health check command. + +```bash +buck2 run dev:healthcheck +``` + +## Dependencies + +For all supported platforms, there are two dependencies that must be installed, `nix` (preferably via the [Determinate Nix Installer](https://github.com/DeterminateSystems/nix-installer)) and `docker`. + +### Nix + +We use `nix` as our package manager for the repository. +It ensures that our developers are all using the same versions of all packages and libraries for developing Galoy. + +Regardless of how `nix` is installed, it must have the [flakes](https://nixos.wiki/wiki/Flakes) feature enabled. +We highly recommend using the [Determinate Nix Installer](https://github.com/DeterminateSystems/nix-installer) over the +official installer; one reason being that the former will enable flakes by default. + +> You can use `direnv` (version >= 2.30) with our [Nix flake](../flake.nix) for both ease of running commands +> and for editor integration. +> +> For more information, see the **Direnv** section. + +### Docker + +We use `docker` to run our dependent services for the galoy stack. +It can either be installed via [Docker Desktop](https://www.docker.com/products/docker-desktop/) or +directly via [Docker Engine](https://docs.docker.com/engine/). + +For Docker Desktop, the version corresponding to your native architecture should be used (e.g. install the aarch64 +(arm64) version on a Apple-Silicon-equipped MacBook Pro). + +WSL2 users should be able to use either Docker Desktop for WSL2 or Docker Engine (i.e. installing and using +`docker` within the distro and not interacting with the host). + +Regardless of platform, you may need to configure credentials in `~/.local/share`. + +#### Rancher Desktop + +Since [Rancher Desktop](https://rancherdesktop.io/) provides the ability to use [moby](https://github.com/moby/moby), +you can use it to run and develop the System Initiative software. +However, it is untested, and you may need to further configuration depending on your platform. + +### (Optional) Direnv + +[Direnv](https://direnv.net/) (version >= 2.30) with [nix-direnv](https://github.com/nix-community/nix-direnv) can +automatically set up your shell, which means you don't need to enter a subshell with `nix develop`, or prefix all +commands with `nix develop --command`. + +You can install it with [your package manager of choice](https://direnv.net/docs/installation.html), but if you're +unsure which installation method to use or your package manager does not provide a compatible version, you +can use `nix` itself (e.g. `nix profile install nixpkgs#direnv`). + +We recommend using [the upstream docs for hooking `direnv` into your shell](https://direnv.net/docs/hook.html), but here +is an example on how to do it on a system where `zsh` is the default shell. +In this example, the following is added to the end of `~/.zshrc`. + +```zsh +if [ $(command -v direnv) ]; then + eval "$(direnv hook zsh)" +fi +``` + +There are also plugins to integrate `direnv` with common editors. + +**Editor plugin support:** + +- CLion: [Direnv integration](https://plugins.jetbrains.com/plugin/15285-direnv-integration), + [Better Direnv](https://plugins.jetbrains.com/plugin/19275-better-direnv) +- Emacs: [emacs-direnv](https://github.com/wbolster/emacs-direnv) +- (Neo)Vim: [direnv.vim](https://github.com/direnv/direnv.vim) +- Visual Studio Code: [direnv](https://marketplace.visualstudio.com/items?itemName=mkhl.direnv) + +## How to Run Commands + +All commands need to be run from the `nix` environment. +There are two primary options to do so: + +1. If `direnv` is installed _and_ hooked into your shell, you can `cd` into + the repository and `nix` will boostrap the environment for you using the flake. +2. Otherwise, you can execute `nix develop` to enter the environment, `nix develop --command ` to + execute a command, or use the environment in whatever way your prefer. + +## Troubleshooting Potential Service Conflicts + +Galoy uses external services in conjunction with its native components. +These external services are deployed via [`docker compose`](https://docs.docker.com/compose/) and are configured to stick to their default settings as +closely as possible, including port settings. +Thus, it is worth checking if you are running these services to avoid conflicts when running Galoy. +Potentially conflicting services include, but are not limited to, the following: + +* PostgreSQL DB +* OpenTelemetry +* MongoDB + +In the case of a port conflict, a good strategy is to temporarily disable the host service until Galoy is no longer being +run. diff --git a/DEV.md b/docs/LEGACY_DEV.md similarity index 100% rename from DEV.md rename to docs/LEGACY_DEV.md