Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docs for using psycopg2 with dbt-postgres #5707

Merged
merged 11 commits into from
Jul 25, 2024
59 changes: 59 additions & 0 deletions website/docs/docs/core/connect-data-platform/postgres-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,3 +93,62 @@ If the database closes its connection while dbt is waiting for data, you may see

If `dbt-postgres` encounters an operational error or timeout when opening a new connection, it will retry up to the number of times configured by `retries`. The default value is 1 retry. If set to 2+ retries, dbt will wait 1 second before retrying. If set to 0, dbt will not retry at all.


### `psycopg2` vs `psycopg2-binary`

`psycopg2-binary` is installed by default when installing `dbt-postgres`.
Installing `psycopg2-binary` uses a pre-built version of `psycopg2` which may not be optimized for your particular machine.
This is ideal for development and testing workflows where performance is less of a concern and speed and ease of install is more important.
However, production environments will benefit from a version of `psycopg2` which is built from source for your particular operating system and archtecture. In this scenario, speed and ease of install is less important as the on-going usage is the focus.

<VersionBlock firstVersion="1.8">

To use `psycopg2`:
1. Install `dbt-postgres`
2. Uninstall `psycopg2-binary`
3. Install the equivalent version of `psycopg2`

```bash
pip install dbt-postgres
if [[ $(pip show psycopg2-binary) ]]; then
mikealfare marked this conversation as resolved.
Show resolved Hide resolved
PSYCOPG2_VERSION=$(pip show psycopg2-binary | grep Version | cut -d " " -f 2)
pip uninstall -y psycopg2-binary && pip install psycopg2==$PSYCOPG2_VERSION
fi
```

</VersionBlock>

<VersionBlock lastVersion="1.7">

To ensure your dbt installation uses `psycopg2`, prefix all `dbt-postgres` installation commands with `DBT_PSYCOPG2_NAME=psycopg2`.
For example:
```bash
DBT_PSYCOPG2_NAME=psycopg2 pip install dbt-postgres
```

</VersionBlock>

Installing `psycopg2` often requires OS level dependencies.
These dependencies may vary across operating systems and architectures.

For example, on Ubuntu, you need to install `libpq-dev` and `python-dev`:
```bash
sudo apt-get update
sudo apt-get install libpq-dev python-dev
```
whereas on Mac, you need to install `postgresql`:
```bash
brew install postgresql
pip install psycopg2
```
Your OS may have its own dependencies based on your particular scenario.

<VersionBlock firstVersion="1.8">

#### Limitations

In versions 1.8.0 and 1.8.1, `psycopg2-binary` is installed on MacOS and Windows operating systems and `psycopg2` is installed on Linux operating systems.
This has the side effect of requiring the OS dependencies identified above to install `dbt-postgres` on Linux.
Users will either need to update their workflows to install these dependencies, or upgrade to 1.8.2.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Given that we recommend users always install the latest (patch) version, no one should be installing v1.8.0 or v1.8.1 (instead of v1.8.2+) — but given you've already written it, I'm happy to have it in here :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on the recommendation. I'm trying to be cognizant of the fact that dbt-postgres tends to be used more as a base adapter than other adapters (e.g. what we used to do with dbt-redshift). I thought it might be more likely in those scenarios that folks are hard pinning the version.


</VersionBlock>
Loading