Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Housekeeping #90

Merged
merged 44 commits into from
Nov 22, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
b77cd52
initial commit
aMahanna Aug 5, 2022
60bce98
Update adapter.py
aMahanna Aug 5, 2022
8af9530
new: controller example in README
aMahanna Aug 5, 2022
79a2f79
new: `track_adb`, `track_nx`
aMahanna Aug 5, 2022
32cb4ff
Update utils.py
aMahanna Aug 5, 2022
67a5cf0
fix: isort
aMahanna Aug 5, 2022
fac0d19
cleanup: progress tracker
aMahanna Oct 31, 2022
1bd3e6c
reignite pr
aMahanna Jul 20, 2023
db27064
remove: python 3.7 support (eol)
aMahanna Jul 20, 2023
92b9440
temp: remove mypy for tests
aMahanna Jul 20, 2023
b80fda7
cleanup
aMahanna Jul 20, 2023
8ec493b
checkpoint
aMahanna Oct 11, 2023
5ba0ca5
fix: test for batch size (adb to nx)
aMahanna Oct 11, 2023
c231411
fix: black
aMahanna Oct 11, 2023
e3d727e
update notebook
aMahanna Oct 11, 2023
be9ca30
fix: `process_adb_cursor`
aMahanna Oct 11, 2023
0e5539c
fix: `abc`
aMahanna Oct 11, 2023
fb1209f
fix: `nx_graph`, typos
aMahanna Oct 11, 2023
875039b
cleanup
aMahanna Oct 11, 2023
0d08c53
cleanup: `rich`
aMahanna Oct 11, 2023
9356c25
update notebooks
aMahanna Oct 11, 2023
9da0bbd
fix docstring
aMahanna Oct 11, 2023
76904d4
update readme & notebooks
aMahanna Oct 11, 2023
5f46f53
more cleanup
aMahanna Oct 12, 2023
3d59007
cleanup
aMahanna Oct 18, 2023
15e8035
cleanup: `examples` directory
aMahanna Oct 19, 2023
f37fa9c
cleanup workflows
aMahanna Oct 19, 2023
5210feb
Update build.yml
aMahanna Oct 19, 2023
4fc9c4c
Update build.yml
aMahanna Oct 19, 2023
22ab583
update release action
aMahanna Oct 25, 2023
5aff8a7
Update README.md
aMahanna Oct 26, 2023
2586853
Update README.md
aMahanna Oct 26, 2023
42d34c3
Update README.md
aMahanna Oct 26, 2023
9ffc0a8
migrate to `pyproject.toml`
aMahanna Nov 13, 2023
0e0d334
Update build.yml
aMahanna Nov 13, 2023
d84628e
fix lint
aMahanna Nov 13, 2023
92372f6
fix lint
aMahanna Nov 13, 2023
911fa3f
disable python 3.12
aMahanna Nov 13, 2023
719578c
re-enable 3.12
aMahanna Nov 13, 2023
e22bc1f
flake8 extend ignore
aMahanna Nov 13, 2023
6ea0547
optimize: `__process_adb_vertex`
aMahanna Nov 16, 2023
a0d79a4
fix var name
aMahanna Nov 16, 2023
dd2bdf7
use `prepare_adb_vertex_method_is_empty` instead of `is_custom_contro…
aMahanna Nov 16, 2023
dc859c2
remove breakpoint
aMahanna Nov 16, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/analyze.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ jobs:

steps:
- name: Checkout repository
uses: actions/checkout@v2
uses: actions/checkout@v4

# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
Expand Down
26 changes: 19 additions & 7 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
name: build
on:
workflow_dispatch:
push:
branches: [ master ]
pull_request:
push:
branches: [ master ]
env:
PACKAGE_DIR: adbnx_adapter
Expand All @@ -13,34 +12,47 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python: ["3.7", "3.8", "3.9", "3.10"]
python: ["3.8", "3.9", "3.10", "3.11", "3.12"]
name: Python ${{ matrix.python }}
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4

- name: Setup Python ${{ matrix.python }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python }}
cache: 'pip'
cache-dependency-path: setup.py

- name: Set up ArangoDB Instance via Docker
run: docker create --name adb -p 8529:8529 -e ARANGO_ROOT_PASSWORD= arangodb/arangodb:3.9.1
run: docker create --name adb -p 8529:8529 -e ARANGO_ROOT_PASSWORD= arangodb/arangodb

- name: Start ArangoDB Instance
run: docker start adb

- name: Setup pip
run: python -m pip install --upgrade pip setuptools wheel

- name: Install packages
run: pip install .[dev]

- name: Run black
run: black --check --verbose --diff --color ${{env.PACKAGE_DIR}} ${{env.TESTS_DIR}}

- name: Run flake8
run: flake8 ${{env.PACKAGE_DIR}} ${{env.TESTS_DIR}}

- name: Run isort
run: isort --check --profile=black ${{env.PACKAGE_DIR}} ${{env.TESTS_DIR}}

- name: Run mypy
run: mypy ${{env.PACKAGE_DIR}} ${{env.TESTS_DIR}}

- name: Run pytest
run: pytest --cov=${{env.PACKAGE_DIR}} --cov-report xml --cov-report term-missing -v --color=yes --no-cov-on-fail --code-highlight=yes

- name: Publish to coveralls.io
if: matrix.python == '3.8'
if: matrix.python == '3.10'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: coveralls --service=github
68 changes: 13 additions & 55 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,76 +3,34 @@ on:
workflow_dispatch:
release:
types: [published]
env:
PACKAGE_DIR: adbnx_adapter
TESTS_DIR: tests
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
python: ["3.7", "3.8", "3.9", "3.10"]
name: Python ${{ matrix.python }}
steps:
- uses: actions/checkout@v2
- name: Setup Python ${{ matrix.python }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python }}
- name: Set up ArangoDB Instance via Docker
run: docker create --name adb -p 8529:8529 -e ARANGO_ROOT_PASSWORD= arangodb/arangodb:3.9.1
- name: Start ArangoDB Instance
run: docker start adb
- name: Setup pip
run: python -m pip install --upgrade pip setuptools wheel
- name: Install packages
run: pip install .[dev]
- name: Run black
run: black --check --verbose --diff --color ${{env.PACKAGE_DIR}} ${{env.TESTS_DIR}}
- name: Run flake8
run: flake8 ${{env.PACKAGE_DIR}} ${{env.TESTS_DIR}}
- name: Run isort
run: isort --check --profile=black ${{env.PACKAGE_DIR}} ${{env.TESTS_DIR}}
- name: Run mypy
run: mypy ${{env.PACKAGE_DIR}} ${{env.TESTS_DIR}}
- name: Run pytest
run: pytest --cov=${{env.PACKAGE_DIR}} --cov-report xml --cov-report term-missing -v --color=yes --no-cov-on-fail --code-highlight=yes
- name: Publish to coveralls.io
if: matrix.python == '3.8'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: coveralls --service=github

release:
needs: build
runs-on: ubuntu-latest
name: Release package
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4

- name: Fetch complete history for all tags and branches
run: git fetch --prune --unshallow

- name: Setup python
uses: actions/setup-python@v2
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.8"
python-version: "3.10"

- name: Install release packages
run: pip install setuptools wheel twine setuptools-scm[toml]

- name: Install dependencies
run: pip install .[dev]

- name: Build distribution
run: python setup.py sdist bdist_wheel

- name: Publish to PyPI Test
- name: Publish to Test PyPi
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.TWINE_PASSWORD_TEST }}
run: twine upload --repository testpypi dist/* #--skip-existing
- name: Publish to PyPI

- name: Publish to PyPi
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.TWINE_PASSWORD }}
Expand All @@ -83,7 +41,7 @@ jobs:
runs-on: ubuntu-latest
name: Update Changelog
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
with:
fetch-depth: 0

Expand All @@ -95,10 +53,10 @@ jobs:
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: Setup python
uses: actions/setup-python@v2
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.8"
python-version: "3.10"

- name: Install release packages
run: pip install wheel gitchangelog pystache
Expand All @@ -110,12 +68,12 @@ jobs:
run: gitchangelog ${{env.VERSION}} > CHANGELOG.md

- name: Make commit for auto-generated changelog
uses: EndBug/add-and-commit@v7
uses: EndBug/add-and-commit@v9
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
add: "CHANGELOG.md"
branch: actions/changelog
new_branch: actions/changelog
message: "!gitchangelog"

- name: Create pull request for the auto generated changelog
Expand Down
90 changes: 62 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

[![License](https://img.shields.io/github/license/arangoml/networkx-adapter?color=9E2165&style=for-the-badge)](https://github.com/arangoml/networkx-adapter/blob/master/LICENSE)
[![Code style: black](https://img.shields.io/static/v1?style=for-the-badge&label=code%20style&message=black&color=black)](https://github.com/psf/black)
[![Downloads](https://img.shields.io/badge/dynamic/json?style=for-the-badge&color=282661&label=Downloads&query=total_downloads&url=https://api.pepy.tech/api/projects/adbnx-adapter)](https://pepy.tech/project/adbnx-adapter)
[![Downloads](https://img.shields.io/badge/dynamic/json?style=for-the-badge&color=282661&label=Downloads&query=total_downloads&url=https://api.pepy.tech/api/v2/projects/adbnx-adapter)](https://pepy.tech/project/adbnx-adapter)

<a href="https://www.arangodb.com/" rel="arangodb.com">![](https://raw.githubusercontent.com/arangoml/networkx-adapter/master/examples/assets/logos/ArangoDB_logo.png)</a>
<a href="https://networkx.org/" rel="networkx.org">![](https://raw.githubusercontent.com/arangoml/networkx-adapter/master/examples/assets/logos/networkx_logo.svg)</a>
Expand Down Expand Up @@ -45,30 +45,40 @@ pip install git+https://github.com/arangoml/networkx-adapter.git
Also available as an ArangoDB Lunch & Learn session: [Graph & Beyond Course #2.9](https://www.arangodb.com/resources/lunch-sessions/graph-beyond-lunch-break-2-9-introducing-the-arangodb-networkx-adapter/)

```py
from arango import ArangoClient # Python-Arango driver
import networkx as nx

from arango import ArangoClient
from adbnx_adapter import ADBNX_Adapter, ADBNX_Controller

# Let's assume that the ArangoDB "fraud detection" dataset is imported to this endpoint
db = ArangoClient(hosts="http://localhost:8529").db("_system", username="root", password="")
# Connect to ArangoDB
db = ArangoClient().db()

# Instantiate the adapter
adbnx_adapter = ADBNX_Adapter(db)
```

### ArangoDB to NetworkX
```py
# 1.1: ArangoDB to NetworkX via Graph name
#######################
# 1.1: via Graph name #
#######################

nx_g = adbnx_adapter.arangodb_graph_to_networkx("fraud-detection")

# 1.2: ArangoDB to NetworkX via Collection names
#############################
# 1.2: via Collection names #
#############################

nx_g = adbnx_adapter.arangodb_collections_to_networkx(
"fraud-detection",
{"account", "bank", "branch", "Class", "customer"}, # Vertex collections
{"accountHolder", "Relationship", "transaction"} # Edge collections
)

# 1.3: ArangoDB to NetworkX via Metagraph
######################
# 1.3: via Metagraph #
######################

metagraph = {
"vertexCollections": {
"account": {"Balance", "account_type", "customer_id", "rank"},
Expand All @@ -79,10 +89,13 @@ metagraph = {
"accountHolder": {},
},
}

nx_g = adbnx_adapter.arangodb_to_networkx("fraud-detection", metagraph)

#######################################
# 1.4: with a custom ADBNX Controller #
#######################################

# 1.4: ArangoDB to NetworkX with a custom ADBNX Controller
class Custom_ADBNX_Controller(ADBNX_Controller):
"""ArangoDB-NetworkX controller.

Expand Down Expand Up @@ -113,7 +126,10 @@ nx_g = ADBNX_Adapter(db, Custom_ADBNX_Controller()).arangodb_graph_to_networkx("

### NetworkX to ArangoDB
```py
# 2.1: NetworkX Homogeneous graph to ArangoDB
#################################
# 2.1: with a Homogeneous Graph #
#################################

nx_g = nx.grid_2d_graph(5, 5)
edge_definitions = [
{
Expand All @@ -122,29 +138,44 @@ edge_definitions = [
"to_vertex_collections": ["Grid_Node"],
}
]

adb_g = adbnx_adapter.networkx_to_arangodb("Grid", nx_g, edge_definitions)

# 2.2: NetworkX Heterogeneous graph to ArangoDB with ArangoDB node IDs
edges = []
for i in range(1, 101):
for j in range(1, 101):
if j % i == 0:
# Notice that the NetworkX node IDs are following ArangoDB _id formatting standards (i.e `collection_name/node_key`)
edges.append((f"numbers_j/{j}", f"numbers_i/{i}", j / i))
#############################################################
# 2.2: with a Homogeneous Graph & a custom ADBNX Controller #
#############################################################

nx_g = nx.MultiDiGraph()
nx_g.add_weighted_edges_from(edges)
class Custom_ADBNX_Controller(ADBNX_Controller):
"""ArangoDB-NetworkX controller.

edge_definitions = [
{
"edge_collection": "is_divisible_by",
"from_vertex_collections": ["numbers_j"],
"to_vertex_collections": ["numbers_i"],
}
]
adb_g = adbnx_adapter.networkx_to_arangodb("Divisibility", nx_g, edge_definitions, keyify_nodes=True)
Responsible for controlling how nodes & edges are handled when
transitioning from ArangoDB to NetworkX, and vice-versa.
"""

def _prepare_networkx_node(self, nx_node: dict, col: str) -> None:
"""Prepare a NetworkX node before it gets inserted into the ArangoDB
collection **col**.

:param nx_node: The NetworkX node object to (optionally) modify.
:param col: The ArangoDB collection the node belongs to.
"""
nx_node["foo"] = "bar"

def _prepare_networkx_edge(self, nx_edge: dict, col: str) -> None:
"""Prepare a NetworkX edge before it gets inserted into the ArangoDB
collection **col**.

:param nx_edge: The NetworkX edge object to (optionally) modify.
:param col: The ArangoDB collection the edge belongs to.
"""
nx_edge["bar"] = "foo"

adb_g = ADBNX_Adapter(db, Custom_ADBNX_Controller()).networkx_to_arangodb("Grid", nx_g, edge_definitions)

###################################
# 2.3: with a Heterogeneous Graph #
###################################

# 2.3 NetworkX Heterogeneous graph to ArangoDB with non-ArangoDB node IDs
edges = [
('student:101', 'lecture:101'),
('student:102', 'lecture:102'),
Expand All @@ -160,7 +191,10 @@ edges = [
nx_g = nx.MultiDiGraph()
nx_g.add_edges_from(edges)

### Learn how this example is handled in Colab: https://colab.research.google.com/github/arangoml/networkx-adapter/blob/master/examples/ArangoDB_NetworkX_Adapter.ipynb#scrollTo=OuU0J7p1E9OM
# ...

# Learn how this example is handled in Colab:
# https://colab.research.google.com/github/arangoml/networkx-adapter/blob/master/examples/ArangoDB_NetworkX_Adapter.ipynb#scrollTo=OuU0J7p1E9OM
```

## Development & Testing
Expand Down
5 changes: 4 additions & 1 deletion adbnx_adapter/__init__.py
Original file line number Diff line number Diff line change
@@ -1,2 +1,5 @@
from adbnx_adapter.adapter import ADBNX_Adapter # noqa: F401
from adbnx_adapter.controller import ADBNX_Controller # noqa: F401
from adbnx_adapter.controller import ( # noqa: F401
ADBNX_Controller,
ADBNX_Controller_Full_Cycle,
)
Loading