pyintegrators

Integrators connect the information in your Pod. They import your data from external services using Importers (Gmail, WhatsApp, etc.), connect new data to the existing data using indexers (face recognition, spam detection, object detection), and execute actions (sending messages, uploading files).

Integrators for Memri have a single repository per language, this is the repository for Python integrators. Memri also has Node.js integrators. This repository is built with nbdev, which means that the repo structure has a few differences compared to a standard python repo. The documentation for this repository is available on pyintegrators.memri.io.

Installing

Pyintegrators can be installed for two purposes: 1) For local development, installation using pip is recommended 2) For deployment, installion using docker is recommended. Currently, the only way to call integrators from the memri clients is using docker, this will change soon.

Install with pip

To install the Python package, and correctly setup nbdev for development run:

pip install -e . && nbdev_install_git_hooks

The last command configures git to automatically clean metadata from your notebooks before a commit.

Install with Docker

The normal flow to run an integrator is from the client, by calling the pods run_integrator api. Subsequently, the Integrator is invoked by the Pod by launching a Docker container. To build the image for this container, run:

docker build -t memri-pyintegrators .

Overview

Pyintegrators currently provides the following integrators. Make sure to check out the documentation for tutorials and usage instructions.

Integrator	Description	Tests passing
EmailImporter	Imports emails over imap.
FaceClusteringIndexer	Clusters faces on photos.
GeoIndexer	Adds Countries and Cities to items with a location.
NotesListIndexer	Extracts lists from notes and categorizes them.

Nbdev & Jupyter Notebooks

The Python integrators are written in nbdev (video). With nbdev, it is encouraged to write code in Jupyter Notebooks. Nbdev syncs all the notebooks in /nbs with the python code in /integrators. Tests are written side by side with the code in the notebooks, and documentation is automatically generated from the code and markdown in the notebooks and exported into the /docs folder. Check out the nbdev quickstart for an introduction, watch the video linked above, or see the nbdev documentation for a all functionalities and tutorials.

Contributing

When you make a merge request, make sure that you used all the nbdev commands specified in the quickstart.

Requesting Integrators

If you are using memri, you can share which integrators you are currently missing the most. There is a topic on the memri forum in which you can request new integrators by posting a description, or upvote pending requests. Requests with upvotes will be prioritized.

Docs

New documentation will be deployed automatically when a new version is released to the prod branch. The CI will generate html files using:

nbdev_build_docs

Rendering Locally

To inspect changes in the documentation locally, you can render it on your local machine using installing Jekyll.

To build the documentation run:

cd docs
gem update --system 
bundle install

To serve the documentation run:

bundle exec jekyll serve