Support Poetry for local development. Add local development guide. (#193

)

Closes #192

These changes add support for managing the Pinecone Python client via
[Poetry](https://python-poetry.org/), a virtualenv and dependency
manager.

These changes also add a new CONTRIBUTING guide which explains how to develop the Pinecone Python client locally even across multiple shells using Poetry.

The intent of this guide is to significantly ease local development of
Pinecone in scenario like Jupyter Notebooks and Python applications
where `pinecone` and supporting libraries such as `langchain` are often
imported after being `pip install`ed.

## Type of Change

- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [x] This change requires a documentation update
- [ ] Infrastructure change (CI configs, etc)
- [ ] Non-code change (docs, etc)
- [ ] None of the above: (explain here)

---------

Co-authored-by: Austin DeNoble <[email protected]>

.github/workflows/publish-to-pypi.yaml

@@ -71,6 +71,10 @@ jobs:
             exit 1
           fi
 
+      - name: Poetry bump pyproject toml version
+        run: |
+          poetry version ${{ inputs.releaseLevel }}
+
       - name: Set up Git
         run: |
           git config --global user.name "Pinecone CI"
@@ -99,7 +103,16 @@ jobs:
       - name: Commit changes, if not prerelease
         if: ${{ inputs.isPrerelease == false }}
         run: |
+          # Add the original pinecone client version file to git
+          # Even though Poetry is now the preferred means of working
+          # with this project, since this __version__ file has been the 
+          # one source of truth for our release process. We need to maintain
+          # both files for the time being, and they should always contain the
+          # identical package version
           git add pinecone/__version__
+          # Add also the pyproject.toml, which is Poetry's source of truth, so
+          # that we maintain the exact same version across the two files 
+          git add pyproject.toml
           git commit -m "[skip ci] Bump version to ${{ steps.bump.outputs.VERSION_TAG }}"
 
       - name: Tag version
@@ -116,4 +129,4 @@ jobs:
 
       - name: Push tags (production release)
         if: ${{ inputs.isPrerelease == false }}
-        run: git push --follow-tags
+        run: git push --follow-tags

.github/workflows/testing.yaml

@@ -19,17 +19,30 @@ jobs:
       with:
         python-version: ${{ matrix.python-version }}
 
+    - name: Install Poetry
+      run: |
+        curl -sSL https://install.python-poetry.org | python -
+
     - name: Install dependencies
       run: |
-        python -m pip install --upgrade pip
-        pip install tox==3.27.0
-        pip install tox tox-gh-actions==2.10.0
+        poetry install
+
+    # We separate the gRPC dependencies from the REST client's install 
+    # behavior, so we test installing the grpc dependencies here as well
+    # The dependencies that are specific to gRPC are defined in pyproject.toml
+    # under tool.poetry.extras
+    - name: Install gRPC dependencies 
+      run: |
+        poetry install --extras "grpc"
 
     - name: Run tests
-      run: make tests TOX_PARALLEL_NO_SPINNER=1
+      run: make tests
 
     - name: Build Python client
       run: make package
 
     - name: Build docs
       run: make docs
+
+    - name: Build Project
+      run: poetry build

CONTRIBUTING.md

@@ -0,0 +1,127 @@
+# Contributing 
+
+## Developing locally with Poetry 
+
+[Poetry](https://python-poetry.org/) is a tool that combines [virtualenv](https://virtualenv.pypa.io/en/latest/) usage with dependency management, to provide a consistent experience for project maintainers and contributors who need to develop the pinecone-python-client
+as a library. 
+
+A common need when making changes to the Pinecone client is to test your changes against existing Python code or Jupyter Notebooks that `pip install` the Pinecone Python client as a library. 
+
+Developers want to be able to see their changes to the library immediately reflected in their main application code, as well as to track all changes they make in git, so that they can be contributed back in the form of a pull request. 
+
+The Pinecone Python client therefore supports Poetry as its primary means of enabling a consistent local development experience. This guide will walk you through the setup process so that you can: 
+1. Make local changes to the Pinecone Python client that are separated from your system's Python installation
+2. Make local changes to the Pinecone Python client that are immediately reflected in other local code that imports the pinecone client
+3. Track all your local changes to the Pinecone Python client so that you can contribute your fixes and feature additions back via GitHub pull requests
+
+### Step 1. Fork the Pinecone python client repository
+
+On the [GitHub repository page](https://github.com/pinecone-io/pinecone-python-client) page, click the fork button at the top of the screen and create a personal fork of the repository: 
+
+![Create a GitHub fork of the Pinecone Python client](./docs/pinecone-python-client-fork.png)
+
+It will take a few seconds for your fork to be ready. When it's ready, **clone your fork** of the Pinecone python client repository to your machine. 
+
+Change directory into the repository, as we'll be setting up a virtualenv from within the root of the repository. 
+
+### Step 1. Install Poetry 
+
+Visit [the Poetry site](https://python-poetry.org/) for installation instructions. 
+
+### Step 2. Install dependencies 
+
+Run `poetry install` from the root of the project. 
+
+### Step 3. Activate the Poetry virtual environment and verify success
+
+Run `poetry shell` from the root of the project. At this point, you now have a virtualenv set up in this directory, which you can verify by running: 
+
+`poetry env info`
+
+You should see something similar to the following output: 
+
+```bash
+Virtualenv
+Python:         3.9.16
+Implementation: CPython
+Path:           /home/youruser/.cache/pypoetry/virtualenvs/pinecone-fWu70vbC-py3.9
+Executable:     /home/youruser/.cache/pypoetry/virtualenvs/pinecone-fWu70vbC-py3.9/bin/python
+Valid:          True
+
+System
+Platform:   linux
+OS:         posix
+Python:     3.9.16
+Path:       /home/linuxbrew/.linuxbrew/opt/[email protected]
+```
+If you want to extract only the path to your new virtualenv, you can run `poetry env info --path`
+
+## Loading your virtualenv in another shell 
+
+It's a common need when developing against this client to load it as part of some other application or Jupyter Notebook code, modify 
+it directly, see your changes reflected immediately and also have your changes tracked in git so you can contribute them back. 
+
+It's important to understand that, by default, if you open a new shell or terminal window, or, for example, a new pane in a tmux session, 
+your new shell will not yet reference the new virtualenv you created in the previous step. 
+
+### Step 1. Get the path to your virtualenv
+
+We're going to first get the path to the virtualenv we just created, by running: 
+
+```bash
+poetry env info --path
+```
+
+You'll get a path similar to this one:  `/home/youruser/.cache/pypoetry/virtualenvs/pinecone-fWu70vbC-py3.9/`
+
+### Step 2. Load your existing virtualenv in your new shell
+
+Within this path is a shell script that lives at `<your-virtualenv-path>/bin/activate`. Importantly, you cannot simply run this script, but you 
+must instead source it like so: 
+
+```bash
+source /home/youruser/.cache/pypoetry/virtualenvs/pinecone-fWu70vbC-py3.9/bin/activate
+```
+In the above example, ensure you're using your own virtualenv path as returned by `poetry env info --path`.
+
+### Step 3. Test out your virtualenv 
+
+Now, we can test that our virtualenv is working properly by adding a new test module and function to the `pinecone` client within our virtualenv 
+and running it from the second shell. 
+
+#### Create a new test file in pinecone-python-client
+In the root of your working directory of the `pinecone-python-client` where you first ran `poetry shell`, add a new file named `hello_virtualenv.py` under the `pinecone` folder. 
+
+In that file write the following: 
+
+```python
+def hello():
+    print("Hello, from your virtualenv!")
+```
+Save the file. 
+
+#### Create a new test file in your second shell 
+This step demonstrates how you can immediately test your latest Pinecone client code from any local Python application or Jupyter Notebook: 
+
+In your second shell, where you ran `source` to load your virtualenv, create a python file named `test.py` and write the following: 
+
+```python
+from pinecone import hello_virtualenv
+
+hello_virtualenv.hello()
+```
+
+Save the file. Run it with your Python binary. Depending on your system, this may either be `python` or `python3`: 
+
+```bash
+python3 test.py
+```
+
+You should see the following output: 
+
+```bash
+❯ python3 test.py
+Hello, from your virtualenv!
+```
+
+If you experience any issues please [file a new issue](https://github.com/pinecone-io/pinecone-python-client/issues/new).

Makefile

@@ -7,27 +7,23 @@ image:
 	MODULE=pinecone ../scripts/build.sh ./
 
 develop:
-	pip3 install -e .[grpc]
+	poetry install -E grpc
 
 tests:
-	# skipping flake8 for now
-	pip3 install --upgrade --quiet tox==3.27.0 && TOX_SKIP_ENV='flake|docs' tox
+	@echo "Installing dependencies..."
+	poetry install
+	@echo "Running tests..."
+	poetry run pytest --cov=pinecone --timeout=120 tests/unit
 
 version:
-	python3 setup.py --version
+	poetry version
 
 package:
-	pip3 install -U wheel && python3 setup.py sdist bdist_wheel
+	poetry build
 
 upload:
-	pip3 install --upgrade --quiet twine && \
-	twine upload \
-		--verbose \
-		--non-interactive \
-		--username ${PYPI_USERNAME} \
-		--password ${PYPI_PASSWORD} \
-		dist/* 
-
+	poetry publish --verbose --username ${PYPI_USERNAME} --password ${PYPI_PASSWORD}
+
 license:
 	# Add license header using https://github.com/google/addlicense.
 	# If the license header already exists in a file, re-running this command has no effect.

README.md

@@ -259,4 +259,6 @@ pinecone.init(api_key="YOUR_API_KEY", environment="us-west1-gcp")
 pinecone.delete_collection("example-collection")
 ```
 
+# Contributing 
 
+If you'd like to make a contribution, or get setup locally to develop the Pinecone python client, please see our [contributing guide](./CONTRIBUTING.md)

pinecone/__version__

@@ -1 +1 @@
-2.2.4
+2.2.4