From fdb3067c4aeee7ccac17477287b24e046751a0f7 Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Thu, 27 Jun 2024 10:54:14 -0400 Subject: [PATCH 1/6] add docs for using psycopg2 with dbt-postgres --- .../connect-data-platform/postgres-setup.md | 54 +++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/website/docs/docs/core/connect-data-platform/postgres-setup.md b/website/docs/docs/core/connect-data-platform/postgres-setup.md index ea645bb22df..d96a2c4432b 100644 --- a/website/docs/docs/core/connect-data-platform/postgres-setup.md +++ b/website/docs/docs/core/connect-data-platform/postgres-setup.md @@ -93,3 +93,57 @@ If the database closes its connection while dbt is waiting for data, you may see If `dbt-postgres` encounters an operational error or timeout when opening a new connection, it will retry up to the number of times configured by `retries`. The default value is 1 retry. If set to 2+ retries, dbt will wait 1 second before retrying. If set to 0, dbt will not retry at all. + +### `psycopg2` vs `psycopg2-binary` + +`psycopg2-binary` is installed by default when installing `dbt-postgres`. +This is ideal for development and testing workflows where performance is less of a concern. +However, production environments will benefit from `psycopg2`, which is built from source for that particular operating system and archtecture. + + + +To use `psycopg2`, simply uninstall `psycopg2-binary` and install the equivalent version of `psycopg2` after installing `dbt-postgres`: +```bash +pip install dbt-postgres +if [[ $(pip show psycopg2-binary) ]]; then + PSYCOPG2_VERSION=$(pip show psycopg2-binary | grep Version | cut -d " " -f 2) + pip uninstall -y psycopg2-binary + pip install psycopg2==$PSYCOPG2_VERSION +fi +``` + + + + + +To use `psycopg2`, ensure that `DBT_PSYCOPG2_NAME=psycopg2` when installing `dbt-postgres`. +The easiest way to do this is to set it inline while installing: +```bash +DBT_PSYCOPG2_NAME=psycopg2 pip install dbt-postgres +``` + + + +Installing `psycopg2` requires OS level dependencies, which is one of the reasons users opt for `psycopg2-binary` (the other reason being a much longer installation time). +These dependencies vary across operating systems and architectures. +For example, on Ubuntu, you need to install `libpq-dev` and `python-dev`: +```bash +sudo apt-get update +sudo apt-get install libpq-dev python-dev +``` +whereas on Mac, you need to install `postgresql`: +```bash +brew install postgresql +pip install psycopg2 +``` +You should research which OS dependencies are needed for your particular scenario. + + + +#### Limitations + +In versions 1.8.0 and 1.8.1, `psycopg2-binary` is installed on MacOS and Windows operating systems and `psycopg2` is installed on Linux operating systems. +This has the side effect of requiring the OS dependencies identified above to install `dbt-postgres` on Linux. +Users will either need to update their workflows to install these dependencies, or upgrade to 1.8.2. + + From 8a24f1430fd411d9237519ceca24714af2d9d87a Mon Sep 17 00:00:00 2001 From: Mike Alfare <13974384+mikealfare@users.noreply.github.com> Date: Fri, 12 Jul 2024 11:54:46 -0400 Subject: [PATCH 2/6] Update website/docs/docs/core/connect-data-platform/postgres-setup.md Co-authored-by: Mila Page <67295367+VersusFacit@users.noreply.github.com> --- .../docs/docs/core/connect-data-platform/postgres-setup.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/website/docs/docs/core/connect-data-platform/postgres-setup.md b/website/docs/docs/core/connect-data-platform/postgres-setup.md index d96a2c4432b..99d1e441e53 100644 --- a/website/docs/docs/core/connect-data-platform/postgres-setup.md +++ b/website/docs/docs/core/connect-data-platform/postgres-setup.md @@ -102,7 +102,10 @@ However, production environments will benefit from `psycopg2`, which is built fr -To use `psycopg2`, simply uninstall `psycopg2-binary` and install the equivalent version of `psycopg2` after installing `dbt-postgres`: +To use `psycopg2`: +1. install `dbt-postgres` +2. uninstall `psycopg2-binary` +3. install the equivalent version of `psycopg2` ```bash pip install dbt-postgres if [[ $(pip show psycopg2-binary) ]]; then From 09ea6a52a369db6cc3cc41d67a76e35f58178f0f Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Fri, 12 Jul 2024 12:02:14 -0400 Subject: [PATCH 3/6] incorporate feedback --- .../connect-data-platform/postgres-setup.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/website/docs/docs/core/connect-data-platform/postgres-setup.md b/website/docs/docs/core/connect-data-platform/postgres-setup.md index 99d1e441e53..71989167036 100644 --- a/website/docs/docs/core/connect-data-platform/postgres-setup.md +++ b/website/docs/docs/core/connect-data-platform/postgres-setup.md @@ -97,21 +97,22 @@ If `dbt-postgres` encounters an operational error or timeout when opening a new ### `psycopg2` vs `psycopg2-binary` `psycopg2-binary` is installed by default when installing `dbt-postgres`. -This is ideal for development and testing workflows where performance is less of a concern. -However, production environments will benefit from `psycopg2`, which is built from source for that particular operating system and archtecture. +Installing `psycopg2-binary` uses a pre-built version of `psycopg2` which may not be optimized for your particular machine. +This is ideal for development and testing workflows where performance is less of a concern and speed and ease of install is more important. +However, production environments will benefit from a version of `psycopg2` which is built from source for your particular operating system and archtecture. In this scenario, speed and ease of install is less important as the on-going usage is the focus. To use `psycopg2`: -1. install `dbt-postgres` -2. uninstall `psycopg2-binary` -3. install the equivalent version of `psycopg2` +1. Install `dbt-postgres` +2. Uninstall `psycopg2-binary` +3. Install the equivalent version of `psycopg2` + ```bash pip install dbt-postgres if [[ $(pip show psycopg2-binary) ]]; then PSYCOPG2_VERSION=$(pip show psycopg2-binary | grep Version | cut -d " " -f 2) - pip uninstall -y psycopg2-binary - pip install psycopg2==$PSYCOPG2_VERSION + pip uninstall -y psycopg2-binary && pip install psycopg2==$PSYCOPG2_VERSION fi ``` @@ -139,7 +140,7 @@ whereas on Mac, you need to install `postgresql`: brew install postgresql pip install psycopg2 ``` -You should research which OS dependencies are needed for your particular scenario. +Your OS may have its own dependencies based on your particular scenario. From 15a86cb1b5c74b7605f6eb3769d08bd2a78093c0 Mon Sep 17 00:00:00 2001 From: Mike Alfare <13974384+mikealfare@users.noreply.github.com> Date: Fri, 12 Jul 2024 12:02:41 -0400 Subject: [PATCH 4/6] Update website/docs/docs/core/connect-data-platform/postgres-setup.md Co-authored-by: Mila Page <67295367+VersusFacit@users.noreply.github.com> --- website/docs/docs/core/connect-data-platform/postgres-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/core/connect-data-platform/postgres-setup.md b/website/docs/docs/core/connect-data-platform/postgres-setup.md index 71989167036..501cbec9a4f 100644 --- a/website/docs/docs/core/connect-data-platform/postgres-setup.md +++ b/website/docs/docs/core/connect-data-platform/postgres-setup.md @@ -120,7 +120,7 @@ fi -To use `psycopg2`, ensure that `DBT_PSYCOPG2_NAME=psycopg2` when installing `dbt-postgres`. +To ensure your dbt installation uses `psycopg2`, prefix all `dbt-postgres` installation commands with `DBT_PSYCOPG2_NAME=psycopg2`. The easiest way to do this is to set it inline while installing: ```bash DBT_PSYCOPG2_NAME=psycopg2 pip install dbt-postgres From 3ff68832d5f03fa0bc686e4d188d35711e84ac07 Mon Sep 17 00:00:00 2001 From: Mike Alfare <13974384+mikealfare@users.noreply.github.com> Date: Fri, 12 Jul 2024 12:02:50 -0400 Subject: [PATCH 5/6] Update website/docs/docs/core/connect-data-platform/postgres-setup.md Co-authored-by: Mila Page <67295367+VersusFacit@users.noreply.github.com> --- website/docs/docs/core/connect-data-platform/postgres-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/core/connect-data-platform/postgres-setup.md b/website/docs/docs/core/connect-data-platform/postgres-setup.md index 501cbec9a4f..f3d6b2bfb18 100644 --- a/website/docs/docs/core/connect-data-platform/postgres-setup.md +++ b/website/docs/docs/core/connect-data-platform/postgres-setup.md @@ -121,7 +121,7 @@ fi To ensure your dbt installation uses `psycopg2`, prefix all `dbt-postgres` installation commands with `DBT_PSYCOPG2_NAME=psycopg2`. -The easiest way to do this is to set it inline while installing: +For example: ```bash DBT_PSYCOPG2_NAME=psycopg2 pip install dbt-postgres ``` From cf3f1e961f7fb7fd68727f4de9b8a45417a61b0f Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Fri, 12 Jul 2024 12:08:43 -0400 Subject: [PATCH 6/6] update wording for os dependencies --- .../docs/docs/core/connect-data-platform/postgres-setup.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/website/docs/docs/core/connect-data-platform/postgres-setup.md b/website/docs/docs/core/connect-data-platform/postgres-setup.md index f3d6b2bfb18..7720e82844d 100644 --- a/website/docs/docs/core/connect-data-platform/postgres-setup.md +++ b/website/docs/docs/core/connect-data-platform/postgres-setup.md @@ -128,8 +128,9 @@ DBT_PSYCOPG2_NAME=psycopg2 pip install dbt-postgres -Installing `psycopg2` requires OS level dependencies, which is one of the reasons users opt for `psycopg2-binary` (the other reason being a much longer installation time). -These dependencies vary across operating systems and architectures. +Installing `psycopg2` often requires OS level dependencies. +These dependencies may vary across operating systems and architectures. + For example, on Ubuntu, you need to install `libpq-dev` and `python-dev`: ```bash sudo apt-get update