From 512994e456d1d05720e2e5391196786eea1b0833 Mon Sep 17 00:00:00 2001 From: Mike Alfare <13974384+mikealfare@users.noreply.github.com> Date: Thu, 25 Jul 2024 03:52:16 -0400 Subject: [PATCH] Add docs for using `psycopg2` with `dbt-postgres` (#5707) ## What are you changing in this pull request and why? We changed how `dbt-postgres` is installed with regards to `psycopg2` versus `psycopg2-binary`. In prior versions you could configure it with an environment variable. In 1.8, that's not an option because we changed our build tooling to a more modern stack. We initially tried an OS-driven approach, but that had issues mainly due to CI workflows that normally run on linux images. So in 1.8.2 we reverted back to installing `psycopg2-binary` by default and prescribed the manual workaround of uninstalling `psycopg2-binary` and reinstalling `psycopg2`. ## Checklist - [x] Review the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [x] For [docs versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#about-versioning), review how to [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content). - [x] Add a checklist item for anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." --------- Co-authored-by: Mila Page <67295367+VersusFacit@users.noreply.github.com> Co-authored-by: Colin Rogers <111200756+colin-rogers-dbt@users.noreply.github.com> Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- .../connect-data-platform/postgres-setup.md | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/website/docs/docs/core/connect-data-platform/postgres-setup.md b/website/docs/docs/core/connect-data-platform/postgres-setup.md index ea645bb22df..7720e82844d 100644 --- a/website/docs/docs/core/connect-data-platform/postgres-setup.md +++ b/website/docs/docs/core/connect-data-platform/postgres-setup.md @@ -93,3 +93,62 @@ If the database closes its connection while dbt is waiting for data, you may see If `dbt-postgres` encounters an operational error or timeout when opening a new connection, it will retry up to the number of times configured by `retries`. The default value is 1 retry. If set to 2+ retries, dbt will wait 1 second before retrying. If set to 0, dbt will not retry at all. + +### `psycopg2` vs `psycopg2-binary` + +`psycopg2-binary` is installed by default when installing `dbt-postgres`. +Installing `psycopg2-binary` uses a pre-built version of `psycopg2` which may not be optimized for your particular machine. +This is ideal for development and testing workflows where performance is less of a concern and speed and ease of install is more important. +However, production environments will benefit from a version of `psycopg2` which is built from source for your particular operating system and archtecture. In this scenario, speed and ease of install is less important as the on-going usage is the focus. + + + +To use `psycopg2`: +1. Install `dbt-postgres` +2. Uninstall `psycopg2-binary` +3. Install the equivalent version of `psycopg2` + +```bash +pip install dbt-postgres +if [[ $(pip show psycopg2-binary) ]]; then + PSYCOPG2_VERSION=$(pip show psycopg2-binary | grep Version | cut -d " " -f 2) + pip uninstall -y psycopg2-binary && pip install psycopg2==$PSYCOPG2_VERSION +fi +``` + + + + + +To ensure your dbt installation uses `psycopg2`, prefix all `dbt-postgres` installation commands with `DBT_PSYCOPG2_NAME=psycopg2`. +For example: +```bash +DBT_PSYCOPG2_NAME=psycopg2 pip install dbt-postgres +``` + + + +Installing `psycopg2` often requires OS level dependencies. +These dependencies may vary across operating systems and architectures. + +For example, on Ubuntu, you need to install `libpq-dev` and `python-dev`: +```bash +sudo apt-get update +sudo apt-get install libpq-dev python-dev +``` +whereas on Mac, you need to install `postgresql`: +```bash +brew install postgresql +pip install psycopg2 +``` +Your OS may have its own dependencies based on your particular scenario. + + + +#### Limitations + +In versions 1.8.0 and 1.8.1, `psycopg2-binary` is installed on MacOS and Windows operating systems and `psycopg2` is installed on Linux operating systems. +This has the side effect of requiring the OS dependencies identified above to install `dbt-postgres` on Linux. +Users will either need to update their workflows to install these dependencies, or upgrade to 1.8.2. + +