diff --git a/README.md b/README.md index a1eed32..849d310 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ Bemi automatically tracks database changes ensuring 100% reliability and a compr - High performance without affecting runtime execution - Easy-to-use without changing table structures - Time travel querying and ability to easily filter changes -- Optional application-specific context by using [ORM packages](https://docs.bemi.io/#supported-nodejs-orms) +- Optional application-specific context by using [ORM packages](https://docs.bemi.io/#supported-orms) ## Use cases @@ -108,7 +108,7 @@ UPDATE _bemi_migrations SET executed_at = NOW() WHERE id = 1; This will add a new record in the `changes` table within the same database after a few seconds. -To optionally automatically enhance these low-level database changes with application-specific context (e.g., user ID, API endpoint, etc.), check out our compatible [ORM packages](https://docs.bemi.io/#supported-nodejs-orms). +To optionally automatically enhance these low-level database changes with application-specific context (e.g., user ID, API endpoint, etc.), check out our compatible [ORM packages](https://docs.bemi.io/#supported-orms). ## Architecture @@ -118,7 +118,7 @@ Bemi consists of three main parts: 1. [Debezium](https://github.com/debezium/debezium), a very flexible tool for implementing Change Data Capture that is written in Java. It is used by many companies that need to implement ETL such as [Airbyte](https://github.com/airbytehq/airbyte) and [Materialize](https://github.com/MaterializeInc/materialize). We rely on it to be able to connect to PostgreSQL replication log, perform logical decoding, and send raw data to a data sink. 2. [NATS JetStream](https://github.com/nats-io/nats-server), a cloud-native messaging system written in Go. Debezium is historically designed to send data to Kafka, but it can be also re-configured to send data to NATS JetStream. It is much more lightweight and easy to manage while being very performant and having over 45 clients for different programming languages. -3. Bemi Worker, a process responsible for stitching data change with app context sent via our open-source [ORM packages](https://docs.bemi.io/#supported-nodejs-orms) and storing data changes. It is written in TypeScript and uses the `core` that we rely on for our [Bemi](https://bemi.io/) cloud platform. +3. Bemi Worker, a process responsible for stitching data change with app context sent via our open-source [ORM packages](https://docs.bemi.io/#supported-orms) and storing data changes. It is written in TypeScript and uses the `core` that we rely on for our [Bemi](https://bemi.io/) cloud platform. The described architecture and the `worker` code in this repository are a simplified version that can be easily run without much overhead. If you want to self-host it in a production environment, see our [self-hosting docs](https://docs.bemi.io/self-hosting). diff --git a/docs/docs/home.md b/docs/docs/home.md index 8e695d3..edb4f21 100644 --- a/docs/docs/home.md +++ b/docs/docs/home.md @@ -48,11 +48,17 @@ This allows tracking all database changes with 100% accuracy by reusing built-in This allows automatically enhancing low-level database changes with application-specific context by using our open-source libraries built for popular ORMs. For example, see all recent changes made by a user, revert all data changes made within an API request, see all processes that made changes by a record, etc. -## Supported Node.js ORMs +## Supported ORMs + +#### Node.js * **[Prisma](/orms/prisma)** * **[TypeORM](/orms/typeorm)** +#### Ruby + +* **[Ruby on Rails](/orms/rails)** + ## Architecture overview Bemi is designed to be lightweight and secure. It takes a practical approach to achieving the benefits of [event sourcing](https://martinfowler.com/eaaDev/EventSourcing.html) without requiring rearchitecting existing code, switching to highly specialized databases, or using unnecessary git-like abstractions on top of databases. We want your system to work the way it already does with your existing database to allow keeping things as simple as possible. diff --git a/docs/docs/orms/prisma.md b/docs/docs/orms/prisma.md index 44be0a9..807c428 100644 --- a/docs/docs/orms/prisma.md +++ b/docs/docs/orms/prisma.md @@ -6,7 +6,7 @@ [Bemi](https://bemi.io) plugs into [Prisma](https://github.com/prisma/prisma) and PostgreSQL to track database changes automatically. It unlocks robust context-aware audit trails and time travel querying inside your application. -This package is an recommended Prisma integration, enabling you to pass application-specific context when performing database changes. This can include context such as the 'where' (API endpoint, worker, etc.), 'who' (user, cron job, etc.), and 'how' behind a change, thereby enriching the information captured by Bemi. +This package is a recommended Prisma integration, enabling you to pass application-specific context when performing database changes. This can include context such as the 'where' (API endpoint, worker, etc.), 'who' (user, cron job, etc.), and 'how' behind a change, thereby enriching the information captured by Bemi. ## Prerequisites @@ -21,7 +21,7 @@ This package is an recommended Prisma integration, enabling you to pass applicat npm install @bemi-db/prisma ``` -2. Generate a Prisma migration file to add lightweight [PostgreSQL triggers](https://www.postgresql.org/docs/current/plpgsql-trigger.html) for inserting application context into replication logs. +2. Generate a Prisma migration file to add lightweight [PostgreSQL triggers](https://www.postgresql.org/docs/current/plpgsql-trigger.html) for passing application context with all data changes into PostgreSQL replication log ```sh npx bemi migration:create @@ -53,11 +53,11 @@ import { PrismaClient } from '@prisma/client'; const prisma = withPgAdapter(new PrismaClient()); ``` -Now you can specify custom application context that will be passed with all data changes by following the code examples below. Application context: +Now you can specify custom application context that will be automatically passed with all data changes by following the code examples below. Application context: * Is bound to the current asynchronous runtime execution context, for example, an HTTP request. * Is used only with `INSERT`, `UPDATE`, `DELETE` SQL queries performed via Prisma. Otherwise, it is a no-op. -* Is passed directly into PG Write-Ahead Log with data changes without affecting the SQL queries or DB schema. +* Is passed directly into PG [Write-Ahead Log](https://www.postgresql.org/docs/current/wal-intro.html) with data changes without affecting the structure of the database and SQL queries. ### Express.js @@ -189,6 +189,8 @@ Password for user u_9adb30103a55: 27 | todo | DELETE | {"id": 27, "task": "Eat", "isCompleted": false} | {} | {"userId": 187234, "endpoint": "/todo/27", "params": {"id": 27}} | 2023-12-11 17:09:18+00 ``` +See [Destination Database](/postgresql/destination-database) for more details. + ## Data change querying Lastly, connect to the Bemi PostgreSQL destination database to easily query change data from your application. diff --git a/docs/docs/orms/rails.md b/docs/docs/orms/rails.md new file mode 100644 index 0000000..53074a6 --- /dev/null +++ b/docs/docs/orms/rails.md @@ -0,0 +1,101 @@ +# Ruby on Rails + +BemiHQ/bemi-rails +
+
+ +[Bemi](https://bemi.io) plugs into [Ruby on Rails](https://github.com/rails/rails) with Active Record and PostgreSQL to track database changes automatically. It unlocks robust context-aware audit trails and time travel querying inside your application. + +This Ruby gem is a recommended Ruby on Rails integration, enabling you to pass application-specific context when performing database changes. This can include context such as the 'where' (API endpoint, worker, etc.), 'who' (user, cron job, etc.), and 'how' behind a change, thereby enriching the information captured by Bemi. + +## Prerequisites + +- PostgreSQL 14+ +- Ruby on Rails + +## Installation + +1. Install the gem by adding it to your `Gemfile` + +``` +gem 'bemi-rails' +``` + +2. Generate a Rails migration file to add lightweight [PostgreSQL triggers](https://www.postgresql.org/docs/current/plpgsql-trigger.html) for passing application context with all data changes into PostgreSQL replication log + +```sh +bin/rails g bemi:migration +``` + +3. Run the rails migration + +```sh +bin/rails db:migrate +``` + +## Usage + +Now you can easily specify custom application context that will be automatically passed with all data changes. + +```rb +class ApplicationController < ActionController::Base + before_action :set_bemi_context + + private + + def set_bemi_context + Bemi.set_context( + user_id: current_user&.id, + endpoint: "#{request.method} #{request.path}", + method: "#{self.class}##{action_name}", + ) + end +end +``` + +Application context: + +* Is bound to the current Ruby thread. So it is isolated to a single HTTP request or background job. +* Is used only with `INSERT`, `UPDATE`, `DELETE` SQL queries performed via Active Record. Otherwise, it is a no-op. +* Is passed directly into PG [Write-Ahead Log](https://www.postgresql.org/docs/current/wal-intro.html) with data changes without affecting the structure of the database and SQL queries. + + +## Data change tracking + +Connect your PostgreSQL source database on [bemi.io](https://bemi.io) to start ingesting and storing all data changes stitched together with application-specific context. The database connection details can be securely configured through the [dashboard UI](https://dashboard.bemi.io/log-in?ref=rails) in a few seconds. + +![dashboard](/img/dashboard.png) + +Once your destination PostgreSQL database has been fully provisioned, you'll see a "Connected" status. You can now test the connection after making database changes in your connected source database: + +``` +psql -h us-west-1-prod-destination-pool.ctbxbtz4ojdc.us-west-1.rds.amazonaws.com -p 5432 -U u_9adb30103a55 -d db_9adb30103a55 -c \ + 'SELECT "primary_key", "table", "operation", "before", "after", "context", "committed_at" FROM changes;' +Password for user u_9adb30103a55: + + primary_key | table | operation | before | after | context | committed_at +-------------+--------+-----------+-------------------------------------------------+--------------------------------------------------+------------------------------------------------------------------------------------------+------------------------ + 26 | todos | CREATE | {} | {"id": 26, "task": "Sleep", "completed": false} | {"user_id": 187234, "endpoint": "POST /todos", "method": "TodosController#create"} | 2023-12-11 17:09:09+00 + 27 | todos | CREATE | {} | {"id": 27, "task": "Eat", "completed": false} | {"user_id": 187234, "endpoint": "POST /todos", "method": "TodosController#create"} | 2023-12-11 17:09:11+00 + 28 | todos | CREATE | {} | {"id": 28, "task": "Repeat", "completed": false} | {"user_id": 187234, "endpoint": "POST /todos", "method": "TodosController#create"} | 2023-12-11 17:09:13+00 + 26 | todos | UPDATE | {"id": 26, "task": "Sleep", "completed": false} | {"id": 26, "task": "Sleep", "completed": true} | {"user_id": 187234, "endpoint": "POST /todos/26", "method": "TodosController#update"} | 2023-12-11 17:09:15+00 + 27 | todos | DELETE | {"id": 27, "task": "Eat", "completed": false} | {} | {"user_id": 187234, "endpoint": "DELETE /todos/27", "method": "TodosController#destroy"} | 2023-12-11 17:09:18+00 +``` + +See [Destination Database](/postgresql/destination-database) for more details. + +## Alternative gems + +| | [Bemi](https://github.com/BemiHQ/bemi-rails)           | [PaperTrail](https://github.com/paper-trail-gem/paper_trail) | [Audited](https://github.com/collectiveidea/audited)     | [Logidze](https://github.com/palkan/logidze)     | +|----------------------------|------|------|------|------| +| Open source | ✅ | ✅ | ✅ | ✅ | +| Capturing record deletions | ✅ | ✅ | ✅ | ❌ | +| Reliability and accuracy | ✅ | ❌ | ❌ | ❌ | +| Scalability | ✅ | ❌ | ❌ | ❌ | +| No performance impact | ✅ | ❌ | ❌ | ❌ | +| Easy-to-use UI | ✅ | ❌ | ❌ | ❌ | + +## License + +Distributed under the terms of the [LGPL-3.0](https://github.com/BemiHQ/bemi-rails/blob/main/LICENSE). +If you need to modify and distribute the code, please release it to contribute back to the open-source community. diff --git a/docs/docs/orms/typeorm.md b/docs/docs/orms/typeorm.md index 6a102a4..9aed811 100644 --- a/docs/docs/orms/typeorm.md +++ b/docs/docs/orms/typeorm.md @@ -6,7 +6,7 @@ [Bemi](https://bemi.io/) plugs into [TypeORM](https://github.com/typeorm/typeorm) and PostgreSQL to track database changes automatically. It unlocks robust context-aware audit trails and time travel querying inside your application. -This package is an recommended TypeORM integration, enabling you to pass application-specific context when performing database changes. This can include context such as the 'where' (API endpoint, worker, etc.), 'who' (user, cron job, etc.), and 'how' behind a change, thereby enriching the information captured by Bemi. +This package is a recommended TypeORM integration, enabling you to pass application-specific context when performing database changes. This can include context such as the 'where' (API endpoint, worker, etc.), 'who' (user, cron job, etc.), and 'how' behind a change, thereby enriching the information captured by Bemi. See [this repo](https://github.com/BemiHQ/bemi-typeorm-example) as an Todo app example with TypeORM that automatically tracks all changes. @@ -14,7 +14,6 @@ See [this repo](https://github.com/BemiHQ/bemi-typeorm-example) as an Todo app e - PostgreSQL 14+ - TypeORM -- Express (Fastify support coming soon) ## Installation @@ -24,7 +23,7 @@ See [this repo](https://github.com/BemiHQ/bemi-typeorm-example) as an Todo app e npm install @bemi-db/typeorm ``` -2. Generate a TypeORM compatible migration file to add lightweight [PostgreSQL triggers](https://www.postgresql.org/docs/current/plpgsql-trigger.html) for inserting application metadata into replication logs. +2. Generate a TypeORM migration file to add lightweight [PostgreSQL triggers](https://www.postgresql.org/docs/current/plpgsql-trigger.html) for passing application context with all data changes into PostgreSQL replication log ```sh npx bemi migration:create ./path-to-migrations-dir @@ -38,7 +37,7 @@ npx typeorm migration:run ## Usage -Add an [Express](https://expressjs.com/) middleware. Here is an example of how to pass application context with all underlying data changes within an HTTP request: +Add an [Express](https://expressjs.com/) middleware to pass application context with all underlying data changes within an HTTP request: ```ts import { setContext } from "@bemi-db/typeorm"; @@ -59,6 +58,12 @@ app.use( AppDataSource.initialize() // initialize TypeORM connection as normal ``` +Application context: + +* Is bound to the current asynchronous runtime execution context, for example, an HTTP request. +* Is used only with `INSERT`, `UPDATE`, `DELETE` SQL queries performed via TypeORM. Otherwise, it is a no-op. +* Is passed directly into PG [Write-Ahead Log](https://www.postgresql.org/docs/current/wal-intro.html) with data changes without affecting the structure of the database and SQL queries. + ## Data change tracking Connect your PostgreSQL source database on [bemi.io](https://bemi.io) to start ingesting and storing all data changes stitched together with application-specific context. @@ -82,6 +87,8 @@ Password for user u_9adb30103a55: 27 | todo | DELETE | {"id": 27, "task": "Eat", "isCompleted": false} | {} | {"userId": 187234, "endpoint": "/todo/27", "params": {"id": 27}} | 2023-12-11 17:09:18+00 ``` +See [Destination Database](/postgresql/destination-database) for more details. + ## Data change querying Lastly, connect directly to the Bemi PostgreSQL database to easily query change data from your application. diff --git a/docs/docs/postgresql/destination-database.md b/docs/docs/postgresql/destination-database.md index 6e1803e..6c204ae 100644 --- a/docs/docs/postgresql/destination-database.md +++ b/docs/docs/postgresql/destination-database.md @@ -24,7 +24,7 @@ Changes performed by creating, updating, or deleting each row are stored in a ta | `operation` | `text` | Enum that can be either `CREATE`, `UPDATE`, or `DELETE` | | `before` | `jsonb` | Record's values before the change | | `after` | `jsonb` | Record's values after the change | -| `context` | `jsonb` | App context passed by using our recommended [ORM packages](/#supported-nodejs-orms) | +| `context` | `jsonb` | App context passed by using our recommended [ORM packages](/#supported-orms) | | `committed_at` | `timestamptz(0)` | When the record was changed | | `queued_at` | `timestamptz(0)` | When the changed record was ingested from WAL | | `created_at` | `timestamptz(0)` | When the change record was stored in the database | @@ -33,7 +33,7 @@ Changes performed by creating, updating, or deleting each row are stored in a ta ## Querying Changes -You can query changes by using our [ORM packages](/#supported-nodejs-orms) or by directly connecting and executing SQL queries. +You can query changes by using our [ORM packages](/#supported-orms) or by directly connecting and executing SQL queries. For example, if you need to find when and how a user record with ID `b7267340-5011-40f4-ab9a-902b68fc5b25` had its email updated to `new@example.com` in the last 3 months: ```sql diff --git a/docs/docs/self-hosting.md b/docs/docs/self-hosting.md index ea14cbe..171ee53 100644 --- a/docs/docs/self-hosting.md +++ b/docs/docs/self-hosting.md @@ -10,7 +10,7 @@ Bemi consists of three main parts: 1. [Debezium](https://github.com/debezium/debezium), a very flexible tool for implementing Change Data Capture that is written in Java. It is used by many companies that need to implement ETL such as [Airbyte](https://github.com/airbytehq/airbyte) and [Materialize](https://github.com/MaterializeInc/materialize). We rely on it to be able to connect to PostgreSQL replication log, perform logical decoding, and send raw data to a data sink. 2. [NATS JetStream](https://github.com/nats-io/nats-server), a cloud-native messaging system written in Go. Debezium is historically designed to send data to Kafka, but it can be also re-configured to send data to NATS JetStream. It is much more lightweight and easy to manage while being very performant and having over 45 clients for different programming languages. -3. Bemi Worker, a process responsible for stitching data change with app context sent via our open-source [ORM packages](https://docs.bemi.io/#supported-nodejs-orms) and storing data changes. It is written in TypeScript and uses the [`core`](https://github.com/BemiHQ/bemi) that we rely on for our [Bemi](https://bemi.io/) cloud platform. +3. Bemi Worker, a process responsible for stitching data change with app context sent via our open-source [ORM packages](https://docs.bemi.io/#supported-orms) and storing data changes. It is written in TypeScript and uses the [`core`](https://github.com/BemiHQ/bemi) that we rely on for our [Bemi](https://bemi.io/) cloud platform. If you want to self-host our solution in a production environment, please [contact us](mailto:hi@bemi.io), and we'll be happy to provide you with a Docker image and assist with configuring the system in exchange for your feedback :) diff --git a/docs/docusaurus.config.ts b/docs/docusaurus.config.ts index b8e194c..f969a72 100644 --- a/docs/docusaurus.config.ts +++ b/docs/docusaurus.config.ts @@ -97,6 +97,10 @@ const config: Config = { label: "TypeORM", href: "https://github.com/BemiHQ/bemi-typeorm", }, + { + label: "Ruby on Rails", + href: "https://github.com/BemiHQ/bemi-rails", + }, ], }, { diff --git a/docs/package.json b/docs/package.json index 13e9645..937b4dd 100644 --- a/docs/package.json +++ b/docs/package.json @@ -4,7 +4,7 @@ "private": true, "scripts": { "docusaurus": "docusaurus", - "start": "docusaurus start", + "start": "docusaurus start --port 4004", "build": "docusaurus build", "swizzle": "docusaurus swizzle", "deploy": "docusaurus deploy", diff --git a/docs/sidebars.ts b/docs/sidebars.ts index cae5cf6..e89d18e 100644 --- a/docs/sidebars.ts +++ b/docs/sidebars.ts @@ -30,6 +30,7 @@ const sidebars: SidebarsConfig = { items: [ 'orms/prisma', 'orms/typeorm', + 'orms/rails', ], }, 'alternatives',