Python instructions and new protos (#5)

.gitignore

@@ -2,4 +2,7 @@ examples/rust/target/**/*
 **/.env
 examples/rust/src/gen/**/*
 examples/go/gen/**/*
+examples/python/gen/**/*
+examples/python/build/**/*
+examples/python/venv/**/*
 !examples/rust/src/gen/mod.rs

README.md

@@ -3,13 +3,17 @@
 This repository contains protobufs (protocol buffers) for clients of Sift to interact directly with our gRPC service. Documentation for our protobufs can be found [here](https://docs.siftstack.com/ingestion/api).
 If certain protobufs are not yet available for a particular API, please refer to the documentation for our [REST API](https://docs.siftstack.com/api-docs/rest) which may expose those services via gRPC gateway.
 
+Please note that some protobufs that appear in our [protobuf documentation](https://docs.siftstack.com/ingestion/api) may not appear in this repository. Those protobufs are either actively being ported over or are
+in the process of being deprecated.
+
 For manual installation instructions for a particular programming language, click on one of the following links:
 - [Go](/docs/go.md)
 - [Rust](/docs/rust.md)
+- [Python](/docs/python.md)
 
-Please keep in mind that the manual installation instructions aims to be general, and users are encouraged to modify any of the steps that best suits the needs of their project. The instructions do not need to be strictly followed.
+Please keep in mind that the manual installation instructions aims to be general and do not need to be strictly followed. Users are encouraged to modify any of the steps or proceed with a custom setup if it better suits the needs of their project.
 
-For usage examples you may also refer to the [examples](examples/) directory which demonstrates basic usage of the code generated from compiling the protobufs.
+For usage examples you may also refer to the [examples](examples/) directory which demonstrates basic usage of the code generated from the protobufs.
 
 If there aren't instructions for your particular programming language consider one of the following options:
 - Request for the Sift team to include instructions for your language of choice. Keep in mind that there are varying degrees of support for each language throughout the protobuf ecosystem. Depending on the language, support might be totally unavailable.

buf_templates/buf.gen.python.yaml

@@ -0,0 +1,10 @@
+version: v1
+managed:
+  enabled: true
+plugins:
+  - plugin: buf.build/grpc/python:v1.62.1
+    out: gen
+  - plugin: buf.build/protocolbuffers/python
+    out: gen
+  - plugin: buf.build/protocolbuffers/pyi:v26.1
+    out: gen

docs/python.md

@@ -0,0 +1,150 @@
+# Sift Protobuf Installation for Python
+
+Before proceeding with installation, you will need to ensure that you have the [buf CLI](https://buf.build/docs/installation) installed.
+
+If `$ which buf` generates a path to the executable, you may proceed to the installation steps.
+
+Installing Sift protos into your project is done in two phases:
+- The first phase is generating a local Python package out of the code generated from the Sift protos.
+- The second phase is using `pip install` to add the Sift generated code as a dependency along with any other missing dependencies.
+
+To get started, make sure that you have `python3` installed as well as an empty directory that will serve as the place to build the
+Sift python package. Inside of that directory, it is recommended that you create a virtual environment. Example:
+
+```bash
+$ python -m venv venv
+```
+
+Once that is done, proceed with the following steps.
+
+1. Clone this repository onto your local machine and `cd` into it:
+
+```bash
+$ git clone https://github.com/sift-stack/sift
+$ cd sift
+```
+
+2. Assuming the path to the aforementioned empty directory where we'll build the package is `$PACKAGE_DIR`, run the following command in the `sift` directory that you just cloned:
+
+```bash
+$ buf export protos --output=$PACKAGE_DIR/protos --config protos/buf.yaml
+```
+
+The Sift protos can and its imports can now be found in your `$PACKAGE_DIR/protos` directory.
+
+3. Copy the `buf` template for Python to `$PACKAGE_DIR`
+
+```bash
+$ cp buf_templates/buf.gen.python.yaml $PACKAGE_DIR/buf.gen.yaml
+```
+
+Copy `setup.py` as well:
+
+```bash
+$ cp scripts/setup.py $PACKAGE_DIR/setup.py
+```
+
+4. `cd` into `$PACKAGE_DIR`.
+
+5. Once inside of `$PACKAGE_DIR`, ensure that `buf.gen.yaml` is at the root.
+
+6. Compile your protobufs.
+
+```bash
+$ buf generate protos
+```
+
+Your project up to this point should look like the following (full depth not shown and virtual env files omitted):
+
+```
+ python_example
+ ├─ buf.gen.yaml
+ ├─ gen
+ │  ├─ protoc_gen_openapiv2
+ │  │  └─ options
+ │  ├─ google
+ │  │  └─ api
+ │  └─ sift
+ │     ├─ notifications
+ │     ├─ data
+ │     ├─ runs
+ │     ├─ users
+ │     ├─ rules
+ │     ├─ assets
+ │     ├─ tags
+ │     ├─ calculated_channels
+ │     ├─ annotations
+ │     ├─ common
+ │     └─ annotation_logs
+ └─ protos
+    ├─ protoc-gen-openapiv2
+    │  └─ options
+    ├─ google
+    │  └─ api
+    └─ sift
+       ├─ common
+       ├─ notifications
+       ├─ tags
+       ├─ runs
+       ├─ assets
+       ├─ data
+       ├─ rules
+       ├─ users
+       ├─ calculated_channels
+       ├─ annotations
+       └─ annotation_logs
+```
+
+7. Execute the following script from your package root (i.e. `$PACKAGE_DIR`) to turn each directory in the generated code into a Python module:
+
+```bash
+for dir in $(find gen -type d); do
+  touch $dir/__init__.py
+done
+```
+
+8. Ensure that your virtual environment is active:
+
+```bash
+$ source venv/bin/activate
+```
+
+9. Install the following dependencies:
+
+```bash
+$  pip install build protobuf grpcio
+```
+
+10. Inspect `setup.py` and customize the metadata if necessary.
+
+11. Build the source distribution and wheel to generate the Python package:
+
+```bash
+$ python -m build --sdist && python -m build --wheel
+```
+
+11. Once that is complete, you should now have a `dist` in `$PACKAGE_DIR` which contains your Python package. For a given `setup.py` that looks like this:
+
+```python
+from setuptools import setup, find_packages
+
+setup(
+    name='sift_protos',
+    version='0.1',
+    author='Sift Stack',
+    author_email='[email protected]',
+    description='Sift generated protos',
+    packages=find_packages('gen'),
+    package_dir={'': 'gen'},
+)
+```
+
+the generated wheel file should be outputted into `$PACKAGE_DIR/dist/sift_protos-0.1-py3-none-any.whl`.
+
+12. Now from your actual Python project, you can install the newly generated package via `pip`:
+
+```bash
+$ pip install $PACKAGE_DIR/sift_protos-0.1-py3-none-any.whl
+```
+
+8. Now your project should be ready to use the generated Python code to interact with Sift's gRPC API. Please refer to the [example code](/examples/python) for usage.

examples/python/.env-example

@@ -0,0 +1,8 @@
+# If you are using the GovCloud environment
+# BASE_URI=https://gov.grpc-api.siftstack.com
+
+# If you are using the non GovCloud environment
+# BASE_URI=https://grpc-api.siftstack.com
+
+# Your sift api key
+SIFT_API_KEY=

examples/python/README.md

@@ -0,0 +1,51 @@
+# Sift Python Example
+
+To run this example ensure that you have Python and the [buf CLI](https://buf.build/docs/installation) installed as well as having your
+Sift API key ready. If you need a Sift API key please refer to [these instructions](https://help.siftstack.com/en/articles/8600475-api-keys).
+
+Once those are installed and your working directory is this project's root, compile the protobufs:
+
+```bash
+$ buf generate protos
+```
+
+Activate your virtual environment:
+
+```bash
+$ python -m venv venv
+$ source venv/bin/activate
+```
+
+Install dependencies:
+
+```bash
+$  pip install build protobuf grpcio
+```
+
+Build the generated code as a Python package:
+
+```bash
+$ python -m build --sdist && python -m build --wheel
+```
+
+Install the package:
+
+```bash
+$ pip install dist/sift_protos-0.1-py3-none-any.whl
+```
+
+Create your `.env` file:
+
+```bash
+$ cp .env-example .env
+```
+
+Be sure to set the appropriate environment variables in your `.env` file depending on the environment you're using. Comments
+meant to serve as guides can be found in the `.env-example` file.
+
+Now execute the program by providing the partial string of the annotations you wish to query. In the following example
+we'll be querying for all annotations whose name matches the `voltage` substring in a case-insensitive manner.
+
+```bash
+$ python main.py voltage
+```

examples/python/buf.gen.yaml

@@ -0,0 +1,11 @@
+# buf.gen.yaml
+version: v1
+managed:
+  enabled: true
+plugins:
+  - plugin: buf.build/grpc/python:v1.62.1
+    out: gen
+  - plugin: buf.build/protocolbuffers/python
+    out: gen
+  - plugin: buf.build/protocolbuffers/pyi:v26.1
+    out: gen

examples/python/main.py

@@ -0,0 +1,25 @@
+import sys
+from dotenv import load_dotenv
+import os
+from sift.annotations.v1.annotations_pb2_grpc import AnnotationServiceStub
+from sift.annotations.v1.annotations_pb2 import ListAnnotationsRequest
+from util.grpc import create_authenticated_connection
+
+if __name__ == "__main__":
+    load_dotenv()
+    API_KEY = os.getenv("SIFT_API_KEY")
+    BASE_URI = os.getenv("BASE_URI")
+    authorization_header = "authorization"
+    authorization_value = f"Bearer {API_KEY}"
+
+    if len(sys.argv) < 2:
+        print("Please provide a name.")
+        sys.exit(1)
+
+    name = sys.argv[1]
+    channel = create_authenticated_connection(BASE_URI, authorization_header, authorization_value)
+    annotation_service = AnnotationServiceStub(channel)
+    request = ListAnnotationsRequest(filter=f"name.matches(\"(?i){name}\")")
+    response = annotation_service.ListAnnotations(request)
+    print(response)
+    channel.close()

examples/python/protos/google/api/annotations.proto

@@ -0,0 +1,31 @@
+// Copyright 2015 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+syntax = "proto3";
+
+package google.api;
+
+import "google/api/http.proto";
+import "google/protobuf/descriptor.proto";
+
+option go_package = "google.golang.org/genproto/googleapis/api/annotations;annotations";
+option java_multiple_files = true;
+option java_outer_classname = "AnnotationsProto";
+option java_package = "com.google.api";
+option objc_class_prefix = "GAPI";
+
+extend google.protobuf.MethodOptions {
+  // See `HttpRule`.
+  HttpRule http = 72295728;
+}