From 11867b7cb793f72e8b214345ce55e54be705d8c9 Mon Sep 17 00:00:00 2001 From: Kefei Mo Date: Thu, 5 Jan 2023 23:48:37 -0600 Subject: [PATCH] imporved-doc/included example to build binary from source code in Ubuntu 22.04, python3.10 --- docs/Notes-on-Packaging.md | 250 +++++++++++++++++++++++++++++++++---- 1 file changed, 224 insertions(+), 26 deletions(-) diff --git a/docs/Notes-on-Packaging.md b/docs/Notes-on-Packaging.md index 1f0433c..d4f6c54 100644 --- a/docs/Notes-on-Packaging.md +++ b/docs/Notes-on-Packaging.md @@ -1,40 +1,66 @@ # Notes on Packaging ## Background -* The dnp3-python version 0.2.0 is an extension and repackaging of [pydnp3](https://github.com/ChargePoint/pydnp3) version 0.1.0 + +* The dnp3-python version 0.2.0 is an extension and repackaging of [pydnp3](https://github.com/ChargePoint/pydnp3) + version 0.1.0 * The wrapper provides more user-friendly and out-of-the-box examples (under /src/dnp3demo) -* The wrapper enables features to install from [The Python Package Index (Pypi)](https://pypi.org/), +* The wrapper enables features to install from [The Python Package Index (Pypi)](https://pypi.org/), while the original package requires installing from the source code. ## Overview + * While, building package from the source code is NOT needed when using the dnp3-python package, it is still required for packing for distribution (i.e., building wheel and publishing to pypi.) * The dependencies to build c++ binding code are cmake and pybind11. - * CMake >= 2.8.12 is required. - * a special version of pybind11 is used and located at /deps/pybind11. -* The c++ source code is based on [opendnp3](https://github.com/automatak/dnp3), + * CMake >= 2.8.12 is required. + * a special version of pybind11 is used and located at /deps/pybind11. +* The c++ source code is based on [opendnp3](https://github.com/automatak/dnp3), forked and version-pinned at /deps/dnp3. the binding code are located at /src and its sub-folders. * setup.py describes the packaging configuration. (tested with with python==3.8.13, setuptools==63.4.1) - * to build and install the package locally, run `python setup.py install` - * to build wheel, run `python setup.py bdist_wheel [--plat-name=manylinux1_x86_64]` + * to build and install the package locally, run `python setup.py install` + * to build wheel, run `python setup.py bdist_wheel [--plat-name=manylinux1_x86_64]` + +## Notes on working with recurse-submodules + +* In order to build the binary (and the wheel) from the source code, the dnp3-python relies on pin-versioned "dnp3" + and "pybind11". +* Note that both the aforementioned source code repo need to present locally at the `/deps`. +* Both of them are included in git as submodules. (See [.gitmodules](../.gitmodules)). To include the submodule, + use `git clone --recurse-submodules`. The `--recurse-submodules` option makes sure nested + submodules are cloned. (Note that the submodule "dnp3" include another submodule "asio".) +* If the submodules were not cloned at the beginning, run `git submodule update --init --recursive`. +* To work with submodules, please see the following references: + * [Submodules - Git](https://git-scm.com/book/en/v2/Git-Tools-Submodules)) + * [Pull latest changes for all git submodules](https://stackoverflow.com/questions/1030169/pull-latest-changes-for-all-git-submodules) + * [How to create Git submodules in GitHub and GitLab by example](https://www.theserverside.com/blog/Coffee-Talk-Java-News-Stories-and-Opinions/How-to-add-submodules-to-GitHub-repos) + * [Cleans and resets a git repo and its submodules - gists · GitHub](https://gist.github.com/nicktoumpelis/11214362) + * [How to checkout old git commit including all submodules recursively?](https://stackoverflow.com/questions/15124430/how-to-checkout-old-git-commit-including-all-submodules-recursively) ## Notes on Building native Python + CPP-binding Python Package + #### Intro + * Need to build + package a python project with cpp-binding code. * Desired result - * pip install - * able to import cpp-binding modules, e.g., “from pydnp3 import opendnp3” - * able to import native python code, e.g., “from dnp3_python.dnp3station import master_new” + * pip install + * able to import cpp-binding modules, e.g., “from pydnp3 import opendnp3” + * able to import native python code, e.g., “from dnp3_python.dnp3station import master_new” * Note: - * the package is a wrapper on “pydnp3” project by repackaging its cpp-binding functionality - and extended/redesigned API in Python. - * the package inherited “pydnp3” project’s root module naming, - i.e., “pydnp3” for cpp-binding related functionality. - * the extended method adopted root module naming, “dnp3_python”. - (There is a discussion at the end of this memo about the reason for using different root module name.) + * the package is a wrapper on “pydnp3” project by repackaging its cpp-binding functionality + and extended/redesigned API in Python. + * the package inherited “pydnp3” project’s root module naming, + i.e., “pydnp3” for cpp-binding related functionality. + * the extended method adopted root module naming, “dnp3_python”. + (There is a discussion at the end of this memo about the reason for using different root module name.) + #### Ingredients -* Reference: [Package Discovery and Namespace Packages](https://setuptools.pypa.io/en/latest/userguide/package_discovery.html) + +* + +Reference: [Package Discovery and Namespace Packages](https://setuptools.pypa.io/en/latest/userguide/package_discovery.html) + * Key configuration in `setup.py` ``` packages=find_namespace_packages( @@ -74,21 +100,193 @@ ``` #### Key takeaways -* Using find_namespace_packages to “automatically” find packages + +* Using find_namespace_packages to “automatically” find packages (assuming defined sub-modules properly, i.e., with `__init__.py`) -* Using trailing * to include submodules +* Using trailing * to include submodules (e.g., dnp3_python* will include dnp3_python and dnp3_python/dnp3station) * Verifying with artifact structure (e.g., whl structure or tar structure) #### Discussion: + * dnp3_python is a package mixed with cpp binding binary and native Python source code. * the cpp binding path is resolved by using dynamic binary (i.e., *.so file) * the name space is called “pydnp3” * To avoid namespace conflict, use different root namespace for python source code package. - * e.g., at one point, the pacakge adopted the structure /src/pydnp3/dnp3station, - with the attempt to achieve `from pydnp3.dnp3station.master_new import *`. - As a result, it will create a “pydnp3/dnp3station” dir at the site-package path. - * However, under the aforementioned structure, the cpp binding submodules are not resolvable, - e.g., not able to achieve “from pydnp3 import opendnp3”. - python will find the native “pydnp3/” first and ignore the package path linked to `*.so` file. - \ No newline at end of file + * e.g., at one point, the pacakge adopted the structure /src/pydnp3/dnp3station, + with the attempt to achieve `from pydnp3.dnp3station.master_new import *`. + As a result, it will create a “pydnp3/dnp3station” dir at the site-package path. + * However, under the aforementioned structure, the cpp binding submodules are not resolvable, + e.g., not able to achieve “from pydnp3 import opendnp3”. + python will find the native “pydnp3/” first and ignore the package path linked to `*.so` file. + +## Example: building binary from C++ source code + +* Demo OS: Ubuntu 22.04 -- use [OsBoxes VM](https://www.osboxes.org/ubuntu/) in this demo. +* Virtual Environment Tool (e.g., conda, virtualenv) -- use conda in this demo. +* Python version 3.10. + +#### prepared the source code (including submodules) for build + +```bash +$ git clone --recurse-submodules +Cloning into 'dnp3-python'... +remote: Enumerating objects: 1633, done. +remote: Counting objects: 100% (15/15), done. +remote: Compressing objects: 100% (15/15), done. +remote: Total 1633 (delta 2), reused 3 (delta 0), pack-reused 1618 +Receiving objects: 100% (1633/1633), 12.09 MiB | 16.00 MiB/s, done. +Resolving deltas: 100% (1275/1275), done. +Submodule 'deps/dnp3' (https://github.com/kefeimo/opendnp3.git) registered for path 'deps/dnp3' +Submodule 'deps/pybind11' (https://github.com/Kisensum/pybind11.git) registered for path 'deps/pybind11' +Cloning into '/home/kefei/project/dnp3-python/deps/dnp3'... +remote: Enumerating objects: 96789, done. +remote: Counting objects: 100% (2074/2074), done. +remote: Compressing objects: 100% (834/834), done. +remote: Total 96789 (delta 1296), reused 1674 (delta 1152), pack-reused 94715 +Receiving objects: 100% (96789/96789), 25.70 MiB | 2.01 MiB/s, done. +Resolving deltas: 100% (68129/68129), done. +Cloning into '/home/kefei/project/dnp3-python/deps/pybind11'... +remote: Enumerating objects: 9840, done. +remote: Total 9840 (delta 0), reused 0 (delta 0), pack-reused 9840 +Receiving objects: 100% (9840/9840), 3.46 MiB | 1.99 MiB/s, done. +Resolving deltas: 100% (6648/6648), done. +Submodule path 'deps/dnp3': checked out '7d84673d165a4a075590a5f146ed1a4ba35d4e49' +Submodule 'deps/asio' (https://github.com/chriskohlhoff/asio.git) registered for path 'deps/dnp3/deps/asio' +Cloning into '/home/kefei/project/dnp3-python/deps/dnp3/deps/asio'... +remote: Enumerating objects: 62636, done. +remote: Counting objects: 100% (669/669), done. +remote: Compressing objects: 100% (269/269), done. +remote: Total 62636 (delta 462), reused 513 (delta 400), pack-reused 61967 +Receiving objects: 100% (62636/62636), 25.34 MiB | 2.24 MiB/s, done. +Resolving deltas: 100% (45783/45783), done. +Submodule path 'deps/dnp3/deps/asio': checked out '28d9b8d6df708024af5227c551673fdb2519f5bf' +Submodule path 'deps/pybind11': checked out '338d615e12ce41ee021724551841de3cbe0bc1df' +Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'deps/pybind11/tools/clang' +Cloning into '/home/kefei/project/dnp3-python/deps/pybind11/tools/clang'... +remote: Enumerating objects: 368, done. +remote: Counting objects: 100% (13/13), done. +remote: Compressing objects: 100% (10/10), done. +remote: Total 368 (delta 3), reused 10 (delta 3), pack-reused 355 +Receiving objects: 100% (368/368), 159.34 KiB | 1.20 MiB/s, done. +Resolving deltas: 100% (154/154), done. +Submodule path 'deps/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' +``` + +#### install build-essential + +```bash +$ sudo apt-get update && sudo apt-get install build-essential +Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease +Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease +Hit:3 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease +Hit:4 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease +Reading package lists... Done +Reading package lists... Done +Building dependency tree... Done +Reading state information... Done +... +``` + +#### create a conda environment called dnp3-sandbox (Python==3.10) + +```bash +$ conda create -n dnp3-sandbox python=3.10 +Collecting package metadata (current_repodata.json): done +Solving environment: done + +## Package Plan ## + + environment location: /home/kefei/miniconda3/envs/dnp3-sandbox + + added / updated specs: + - python=3.10 + + +The following NEW packages will be INSTALLED: +... +``` + +#### activate the virtual environment and install required dependency for build (i.e., cmake) + +```bash +$ conda activate dnp3-sandbox +(dnp3-sandbox) $ pip install cmake +Collecting cmake + Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB) + ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.7/23.7 MB 2.1 MB/s eta 0:00:00 +Installing collected packages: cmake +Successfully installed cmake-3.25.0 +``` + +#### (at repo root path) run `python setup.py install` + +Watch the artifact created at "build/" folder. +e.g., `lib.linux-x86_64-cpython-310/pydnp3.cpython-310-x86_64-linux-gnu.so` + +```bash +running install +/home/kefei/miniconda3/envs/dnp3-sandbox/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. + warnings.warn( +/home/kefei/miniconda3/envs/dnp3-sandbox/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools. + warnings.warn( +running bdist_egg +running egg_info +writing src/dnp3_python.egg-info/PKG-INFO +writing dependency_links to src/dnp3_python.egg-info/dependency_links.txt +writing entry points to src/dnp3_python.egg-info/entry_points.txt +writing top-level names to src/dnp3_python.egg-info/top_level.txt +reading manifest file 'src/dnp3_python.egg-info/SOURCES.txt' +adding license file 'LICENSE' +... +Processing dnp3_python-0.2.3b2-py3.10-linux-x86_64.egg +removing '/home/kefei/miniconda3/envs/dnp3-sandbox/lib/python3.10/site-packages/dnp3_python-0.2.3b2-py3.10-linux-x86_64.egg' (and everything under it) +creating /home/kefei/miniconda3/envs/dnp3-sandbox/lib/python3.10/site-packages/dnp3_python-0.2.3b2-py3.10-linux-x86_64.egg +Extracting dnp3_python-0.2.3b2-py3.10-linux-x86_64.egg to /home/kefei/miniconda3/envs/dnp3-sandbox/lib/python3.10/site-packages +dnp3-python 0.2.3b2 is already the active version in easy-install.pth +Installing dnp3demo script to /home/kefei/miniconda3/envs/dnp3-sandbox/bin + +Installed /home/kefei/miniconda3/envs/dnp3-sandbox/lib/python3.10/site-packages/dnp3_python-0.2.3b2-py3.10-linux-x86_64.egg +Processing dependencies for dnp3-python==0.2.3b2 +Finished processing dependencies for dnp3-python==0.2.3b2 +``` + +(View full log +at [cmake-build-dnp3-python.log](https://gist.github.com/kefeimo/35b7536c235c31ba0abf0da7fa7c4125#file-cmake-build-dnp3-python-log)) + +#### run `python setup.py bdist_wheel --plat-name=manylinux1_x86_64` to build wheel + +Watch the artifact created at "dist/" folder. +e.g., `dnp3_python-0.2.3b2-cp310-cp310-manylinux1_x86_64.whl` + +```bash +$ python setup.py bdist_wheel --plat-name=manylinux1_x86_64 +running bdist_wheel +running build +running build_py +running build_ext +CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required): + Compatibility with CMake < 2.8.12 will be removed from a future version of + CMake. + + Update the VERSION argument value or use a ... suffix to tell + CMake that the project does not need compatibility with older versions. +... +adding 'dnp3_python-0.2.3b2.dist-info/METADATA' +adding 'dnp3_python-0.2.3b2.dist-info/WHEEL' +adding 'dnp3_python-0.2.3b2.dist-info/entry_points.txt' +adding 'dnp3_python-0.2.3b2.dist-info/top_level.txt' +adding 'dnp3_python-0.2.3b2.dist-info/RECORD' +removing build/bdist.linux-x86_64/wheel +``` + +More information +about [building wheels](https://wheel.readthedocs.io/en/stable/user_guide.html?highlight=bdist_wheel#building-wheels), +[The Story of Wheel](https://wheel.readthedocs.io/en/stable/story.html?highlight=bdist_wheel#the-story-of-wheel), +[manylinux](https://github.com/pypa/manylinux) and `--plat-name` +in [Running setuptools commands](https://setuptools.pypa.io/en/latest/deprecated/commands.html). + +## Useful Resource + +* [Setup Tools Doc](https://setuptools.pypa.io/en/latest/) +* [wheel Doc](https://wheel.readthedocs.io/en/stable/index.html) \ No newline at end of file