Skip to content

Commit

Permalink
imporved-doc/included example to build binary from source code in Ubu…
Browse files Browse the repository at this point in the history 
…ntu 22.04, python3.10
  • Loading branch information
@kefeimo
kefeimo committed Jan 6, 2023
1 parent c4c9e04 commit 11867b7
Showing 1 changed file with 224 additions and 26 deletions.
250 changes: 224 additions & 26 deletions docs/Notes-on-Packaging.md
View file
Original file line number Diff line number Diff line change
@@ -1,40 +1,66 @@
# Notes on Packaging

## Background
* The dnp3-python version 0.2.0 is an extension and repackaging of [pydnp3](https://github.com/ChargePoint/pydnp3) version 0.1.0

* The dnp3-python version 0.2.0 is an extension and repackaging of [pydnp3](https://github.com/ChargePoint/pydnp3)
version 0.1.0
* The wrapper provides more user-friendly and out-of-the-box examples (under /src/dnp3demo)
* The wrapper enables features to install from [The Python Package Index (Pypi)](https://pypi.org/),
* The wrapper enables features to install from [The Python Package Index (Pypi)](https://pypi.org/),
while the original package requires installing from the source code.

## Overview

* While, building package from the source code is NOT needed when using the dnp3-python package,
it is still required for packing for distribution (i.e., building wheel and publishing to pypi.)
* The dependencies to build c++ binding code are cmake and pybind11.
* CMake >= 2.8.12 is required.
* a special version of pybind11 is used and located at /deps/pybind11.
* The c++ source code is based on [opendnp3](https://github.com/automatak/dnp3),
* CMake >= 2.8.12 is required.
* a special version of pybind11 is used and located at /deps/pybind11.
* The c++ source code is based on [opendnp3](https://github.com/automatak/dnp3),
forked and version-pinned at /deps/dnp3.
the binding code are located at /src and its sub-folders.
* setup.py describes the packaging configuration. (tested with with python==3.8.13, setuptools==63.4.1)
* to build and install the package locally, run `python setup.py install`
* to build wheel, run `python setup.py bdist_wheel [--plat-name=manylinux1_x86_64]`
* to build and install the package locally, run `python setup.py install`
* to build wheel, run `python setup.py bdist_wheel [--plat-name=manylinux1_x86_64]`

## Notes on working with recurse-submodules

* In order to build the binary (and the wheel) from the source code, the dnp3-python relies on pin-versioned "dnp3"
and "pybind11".
* Note that both the aforementioned source code repo need to present locally at the `/deps`.
* Both of them are included in git as submodules. (See [.gitmodules](../.gitmodules)). To include the submodule,
use `git clone <dnp3-python repo> --recurse-submodules`. The `--recurse-submodules` option makes sure nested
submodules are cloned. (Note that the submodule "dnp3" include another submodule "asio".)
* If the submodules were not cloned at the beginning, run `git submodule update --init --recursive`.
* To work with submodules, please see the following references:
* [Submodules - Git](https://git-scm.com/book/en/v2/Git-Tools-Submodules))
* [Pull latest changes for all git submodules](https://stackoverflow.com/questions/1030169/pull-latest-changes-for-all-git-submodules)
* [How to create Git submodules in GitHub and GitLab by example](https://www.theserverside.com/blog/Coffee-Talk-Java-News-Stories-and-Opinions/How-to-add-submodules-to-GitHub-repos)
* [Cleans and resets a git repo and its submodules - gists · GitHub](https://gist.github.com/nicktoumpelis/11214362)
* [How to checkout old git commit including all submodules recursively?](https://stackoverflow.com/questions/15124430/how-to-checkout-old-git-commit-including-all-submodules-recursively)

## Notes on Building native Python + CPP-binding Python Package

#### Intro

* Need to build + package a python project with cpp-binding code.
* Desired result
* pip install <package_name:dnp3_python>
* able to import cpp-binding modules, e.g., “from pydnp3 import opendnp3”
* able to import native python code, e.g., “from dnp3_python.dnp3station import master_new”
* pip install <package_name:dnp3_python>
* able to import cpp-binding modules, e.g., “from pydnp3 import opendnp3”
* able to import native python code, e.g., “from dnp3_python.dnp3station import master_new”
* Note:
* the package is a wrapper on “pydnp3” project by repackaging its cpp-binding functionality
and extended/redesigned API in Python.
* the package inherited “pydnp3” project’s root module naming,
i.e., “pydnp3” for cpp-binding related functionality.
* the extended method adopted root module naming, “dnp3_python”.
(There is a discussion at the end of this memo about the reason for using different root module name.)
* the package is a wrapper on “pydnp3” project by repackaging its cpp-binding functionality
and extended/redesigned API in Python.
* the package inherited “pydnp3” project’s root module naming,
i.e., “pydnp3” for cpp-binding related functionality.
* the extended method adopted root module naming, “dnp3_python”.
(There is a discussion at the end of this memo about the reason for using different root module name.)

#### Ingredients
* Reference: [Package Discovery and Namespace Packages](https://setuptools.pypa.io/en/latest/userguide/package_discovery.html)

*

Reference: [Package Discovery and Namespace Packages](https://setuptools.pypa.io/en/latest/userguide/package_discovery.html)

* Key configuration in `setup.py`
```
packages=find_namespace_packages(
Expand Down Expand Up @@ -74,21 +100,193 @@
```
#### Key takeaways
* Using find_namespace_packages to “automatically” find packages
* Using find_namespace_packages to “automatically” find packages
(assuming defined sub-modules properly, i.e., with `__init__.py`)
* Using trailing * to include submodules
* Using trailing * to include submodules
(e.g., dnp3_python* will include dnp3_python and dnp3_python/dnp3station)
* Verifying with artifact structure (e.g., whl structure or tar structure)
#### Discussion:
* dnp3_python is a package mixed with cpp binding binary and native Python source code.
* the cpp binding path is resolved by using dynamic binary (i.e., *.so file)
* the name space is called “pydnp3”
* To avoid namespace conflict, use different root namespace for python source code package.
* e.g., at one point, the pacakge adopted the structure /src/pydnp3/dnp3station,
with the attempt to achieve `from pydnp3.dnp3station.master_new import *`.
As a result, it will create a “pydnp3/dnp3station” dir at the site-package path.
* However, under the aforementioned structure, the cpp binding submodules are not resolvable,
e.g., not able to achieve “from pydnp3 import opendnp3”.
python will find the native “pydnp3/” first and ignore the package path linked to `*.so` file.
* e.g., at one point, the pacakge adopted the structure /src/pydnp3/dnp3station,
with the attempt to achieve `from pydnp3.dnp3station.master_new import *`.
As a result, it will create a “pydnp3/dnp3station” dir at the site-package path.
* However, under the aforementioned structure, the cpp binding submodules are not resolvable,
e.g., not able to achieve “from pydnp3 import opendnp3”.
python will find the native “pydnp3/” first and ignore the package path linked to `*.so` file.
## Example: building binary from C++ source code
* Demo OS: Ubuntu 22.04 -- use [OsBoxes VM](https://www.osboxes.org/ubuntu/) in this demo.
* Virtual Environment Tool (e.g., conda, virtualenv) -- use conda in this demo.
* Python version 3.10.
#### prepared the source code (including submodules) for build
```bash
$ git clone <dnp3-python-repo> --recurse-submodules
Cloning into 'dnp3-python'...
remote: Enumerating objects: 1633, done.
remote: Counting objects: 100% (15/15), done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 1633 (delta 2), reused 3 (delta 0), pack-reused 1618
Receiving objects: 100% (1633/1633), 12.09 MiB | 16.00 MiB/s, done.
Resolving deltas: 100% (1275/1275), done.
Submodule 'deps/dnp3' (https://github.com/kefeimo/opendnp3.git) registered for path 'deps/dnp3'
Submodule 'deps/pybind11' (https://github.com/Kisensum/pybind11.git) registered for path 'deps/pybind11'
Cloning into '/home/kefei/project/dnp3-python/deps/dnp3'...
remote: Enumerating objects: 96789, done.
remote: Counting objects: 100% (2074/2074), done.
remote: Compressing objects: 100% (834/834), done.
remote: Total 96789 (delta 1296), reused 1674 (delta 1152), pack-reused 94715
Receiving objects: 100% (96789/96789), 25.70 MiB | 2.01 MiB/s, done.
Resolving deltas: 100% (68129/68129), done.
Cloning into '/home/kefei/project/dnp3-python/deps/pybind11'...
remote: Enumerating objects: 9840, done.
remote: Total 9840 (delta 0), reused 0 (delta 0), pack-reused 9840
Receiving objects: 100% (9840/9840), 3.46 MiB | 1.99 MiB/s, done.
Resolving deltas: 100% (6648/6648), done.
Submodule path 'deps/dnp3': checked out '7d84673d165a4a075590a5f146ed1a4ba35d4e49'
Submodule 'deps/asio' (https://github.com/chriskohlhoff/asio.git) registered for path 'deps/dnp3/deps/asio'
Cloning into '/home/kefei/project/dnp3-python/deps/dnp3/deps/asio'...
remote: Enumerating objects: 62636, done.
remote: Counting objects: 100% (669/669), done.
remote: Compressing objects: 100% (269/269), done.
remote: Total 62636 (delta 462), reused 513 (delta 400), pack-reused 61967
Receiving objects: 100% (62636/62636), 25.34 MiB | 2.24 MiB/s, done.
Resolving deltas: 100% (45783/45783), done.
Submodule path 'deps/dnp3/deps/asio': checked out '28d9b8d6df708024af5227c551673fdb2519f5bf'
Submodule path 'deps/pybind11': checked out '338d615e12ce41ee021724551841de3cbe0bc1df'
Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'deps/pybind11/tools/clang'
Cloning into '/home/kefei/project/dnp3-python/deps/pybind11/tools/clang'...
remote: Enumerating objects: 368, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 368 (delta 3), reused 10 (delta 3), pack-reused 355
Receiving objects: 100% (368/368), 159.34 KiB | 1.20 MiB/s, done.
Resolving deltas: 100% (154/154), done.
Submodule path 'deps/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
```

#### install build-essential

```bash
$ sudo apt-get update && sudo apt-get install build-essential
Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
...
```

#### create a conda environment called dnp3-sandbox (Python==3.10)

```bash
$ conda create -n dnp3-sandbox python=3.10
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

environment location: /home/kefei/miniconda3/envs/dnp3-sandbox

added / updated specs:
- python=3.10


The following NEW packages will be INSTALLED:
...
```

#### activate the virtual environment and install required dependency for build (i.e., cmake)

```bash
$ conda activate dnp3-sandbox
(dnp3-sandbox) $ pip install cmake
Collecting cmake
Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.7/23.7 MB 2.1 MB/s eta 0:00:00
Installing collected packages: cmake
Successfully installed cmake-3.25.0
```

#### (at repo root path) run `python setup.py install`

Watch the artifact created at "build/" folder.
e.g., `lib.linux-x86_64-cpython-310/pydnp3.cpython-310-x86_64-linux-gnu.so`

```bash
running install
/home/kefei/miniconda3/envs/dnp3-sandbox/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/home/kefei/miniconda3/envs/dnp3-sandbox/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running bdist_egg
running egg_info
writing src/dnp3_python.egg-info/PKG-INFO
writing dependency_links to src/dnp3_python.egg-info/dependency_links.txt
writing entry points to src/dnp3_python.egg-info/entry_points.txt
writing top-level names to src/dnp3_python.egg-info/top_level.txt
reading manifest file 'src/dnp3_python.egg-info/SOURCES.txt'
adding license file 'LICENSE'
...
Processing dnp3_python-0.2.3b2-py3.10-linux-x86_64.egg
removing '/home/kefei/miniconda3/envs/dnp3-sandbox/lib/python3.10/site-packages/dnp3_python-0.2.3b2-py3.10-linux-x86_64.egg' (and everything under it)
creating /home/kefei/miniconda3/envs/dnp3-sandbox/lib/python3.10/site-packages/dnp3_python-0.2.3b2-py3.10-linux-x86_64.egg
Extracting dnp3_python-0.2.3b2-py3.10-linux-x86_64.egg to /home/kefei/miniconda3/envs/dnp3-sandbox/lib/python3.10/site-packages
dnp3-python 0.2.3b2 is already the active version in easy-install.pth
Installing dnp3demo script to /home/kefei/miniconda3/envs/dnp3-sandbox/bin

Installed /home/kefei/miniconda3/envs/dnp3-sandbox/lib/python3.10/site-packages/dnp3_python-0.2.3b2-py3.10-linux-x86_64.egg
Processing dependencies for dnp3-python==0.2.3b2
Finished processing dependencies for dnp3-python==0.2.3b2
```
(View full log
at [cmake-build-dnp3-python.log](https://gist.github.com/kefeimo/35b7536c235c31ba0abf0da7fa7c4125#file-cmake-build-dnp3-python-log))
#### run `python setup.py bdist_wheel --plat-name=manylinux1_x86_64` to build wheel
Watch the artifact created at "dist/" folder.
e.g., `dnp3_python-0.2.3b2-cp310-cp310-manylinux1_x86_64.whl`
```bash
$ python setup.py bdist_wheel --plat-name=manylinux1_x86_64
running bdist_wheel
running build
running build_py
running build_ext
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.

Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
...
adding 'dnp3_python-0.2.3b2.dist-info/METADATA'
adding 'dnp3_python-0.2.3b2.dist-info/WHEEL'
adding 'dnp3_python-0.2.3b2.dist-info/entry_points.txt'
adding 'dnp3_python-0.2.3b2.dist-info/top_level.txt'
adding 'dnp3_python-0.2.3b2.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
```
More information
about [building wheels](https://wheel.readthedocs.io/en/stable/user_guide.html?highlight=bdist_wheel#building-wheels),
[The Story of Wheel](https://wheel.readthedocs.io/en/stable/story.html?highlight=bdist_wheel#the-story-of-wheel),
[manylinux](https://github.com/pypa/manylinux) and `--plat-name`
in [Running setuptools commands](https://setuptools.pypa.io/en/latest/deprecated/commands.html).
## Useful Resource
* [Setup Tools Doc](https://setuptools.pypa.io/en/latest/)
* [wheel Doc](https://wheel.readthedocs.io/en/stable/index.html)

0 comments on commit 11867b7

Please sign in to comment.