Skip to content

Commit

Permalink
Add docs for using psycopg2 with dbt-postgres (#5707)
Browse files Browse the repository at this point in the history
## What are you changing in this pull request and why?

We changed how `dbt-postgres` is installed with regards to `psycopg2`
versus `psycopg2-binary`.
In prior versions you could configure it with an environment variable.
In 1.8, that's not an option because we changed our build tooling to a
more modern stack. We initially tried an OS-driven approach, but that
had issues mainly due to CI workflows that normally run on linux images.
So in 1.8.2 we reverted back to installing `psycopg2-binary` by default
and prescribed the manual workaround of uninstalling `psycopg2-binary`
and reinstalling `psycopg2`.

## Checklist
- [x] Review the [Content style
guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md)
so my content adheres to these guidelines.
- [x] For [docs
versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#about-versioning),
review how to [version a whole
page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version)
and [version a block of
content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content).
- [x] Add a checklist item for anything that needs to happen before this
PR is merged, such as "needs technical review" or "change base branch."

---------

Co-authored-by: Mila Page <[email protected]>
Co-authored-by: Colin Rogers <[email protected]>
Co-authored-by: Mirna Wong <[email protected]>
  • Loading branch information
4 people authored Jul 25, 2024
1 parent 561b767 commit 512994e
Showing 1 changed file with 59 additions and 0 deletions.
59 changes: 59 additions & 0 deletions website/docs/docs/core/connect-data-platform/postgres-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,3 +93,62 @@ If the database closes its connection while dbt is waiting for data, you may see

If `dbt-postgres` encounters an operational error or timeout when opening a new connection, it will retry up to the number of times configured by `retries`. The default value is 1 retry. If set to 2+ retries, dbt will wait 1 second before retrying. If set to 0, dbt will not retry at all.


### `psycopg2` vs `psycopg2-binary`

`psycopg2-binary` is installed by default when installing `dbt-postgres`.
Installing `psycopg2-binary` uses a pre-built version of `psycopg2` which may not be optimized for your particular machine.
This is ideal for development and testing workflows where performance is less of a concern and speed and ease of install is more important.
However, production environments will benefit from a version of `psycopg2` which is built from source for your particular operating system and archtecture. In this scenario, speed and ease of install is less important as the on-going usage is the focus.

<VersionBlock firstVersion="1.8">

To use `psycopg2`:
1. Install `dbt-postgres`
2. Uninstall `psycopg2-binary`
3. Install the equivalent version of `psycopg2`

```bash
pip install dbt-postgres
if [[ $(pip show psycopg2-binary) ]]; then
PSYCOPG2_VERSION=$(pip show psycopg2-binary | grep Version | cut -d " " -f 2)
pip uninstall -y psycopg2-binary && pip install psycopg2==$PSYCOPG2_VERSION
fi
```

</VersionBlock>

<VersionBlock lastVersion="1.7">

To ensure your dbt installation uses `psycopg2`, prefix all `dbt-postgres` installation commands with `DBT_PSYCOPG2_NAME=psycopg2`.
For example:
```bash
DBT_PSYCOPG2_NAME=psycopg2 pip install dbt-postgres
```

</VersionBlock>

Installing `psycopg2` often requires OS level dependencies.
These dependencies may vary across operating systems and architectures.

For example, on Ubuntu, you need to install `libpq-dev` and `python-dev`:
```bash
sudo apt-get update
sudo apt-get install libpq-dev python-dev
```
whereas on Mac, you need to install `postgresql`:
```bash
brew install postgresql
pip install psycopg2
```
Your OS may have its own dependencies based on your particular scenario.

<VersionBlock firstVersion="1.8">

#### Limitations

In versions 1.8.0 and 1.8.1, `psycopg2-binary` is installed on MacOS and Windows operating systems and `psycopg2` is installed on Linux operating systems.
This has the side effect of requiring the OS dependencies identified above to install `dbt-postgres` on Linux.
Users will either need to update their workflows to install these dependencies, or upgrade to 1.8.2.

</VersionBlock>

0 comments on commit 512994e

Please sign in to comment.