-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Using wheels to distribute Python packages
If your SPK uses Python packages you can use the 'wheel' format to distribute the packages together with your SPK. Read up on the format here.
Generally speaking, there are tree types of Python packages:
- Pure-python packages. Wheels are platform independent and are self-contained for the most part in terms of dependencies;
- Packages with (optional) C-extensions. These packages have to be compiled with GCC, and require a (cross-)compiled Python to be available;
- Packages forced using api/abi limited that may need to limit API compatibility to Python 3.x (
cp3x
) and ABI to Python 3 (abi3
).
For spksrc specifically, we also define a fourth type of package:
- Packages with C-extensions, which depend on other cross-packages at build time;
- Packages that need patches to be applied in order to create a working wheel.
This type of package requires a new cross package to be created. Generally speaking, these packages will also require a (cross-)compiled Python to be available.
By default spksrc consider all packages to be of cross-compiled type thus allowing you to use any given name for the requirement file (although default normally is requirements.txt
). Otherwise, spksrc uses 3x distinct requirement files to handle wheels:
-
requirements-cross.txt
is used for cross-compiled packages -
requirements-pure.txt
is user for pure-python packages -
requirements-abi3.txt
is for api/abi limited packages
Add the package requirements-cross|pure|abi3.txt
file to the WHEELS
variable in the SPKs Makefile. In order to create reproducible builds, all the required packages should be frozen to a specific version (e.g. docutils==0.12
).
spksrc will store a requirements-cross|pure|abi3.txt
in $(WORK_DIR)/wheelhouse
. From there it will compile each of the needed types of wheels and then store the original in $(WORK_DIR)/wheelhouse
. Finally, spksrc will rename and copy the wheel to $(INSTALL_DIR)/$(INSTALL_PREFIX)/share/wheelhouse
for packaging along with creating a consolidated requirements.txt
that will include all cross, pure and abi3 resulting wheels.
Using the requirements-pure.txt
file, spksrc will create a pure-python wheel by clearing all the build flags and using the native/python310
python interpreter. The resulting wheel is stored in $(WORK_DIR)/wheelhouse
for later processing.
- It mandatory requires to add
BUILD_DEPENDS += cross/python310
to the SPKs Makefile. This ensures that Python is cross-compiled and other requirements are setup correctly to create cross-compiled wheels. At the end of thecross/python310
compilation it generate a fullcrossenv
that includespip
,wheel
,setuptools
,cffi
andcryptography
andpoetry
. This crossenv is key to generate cross-compiled wheels later-on in the build process; - Add the requirement filename to the
WHEELS
variable in spk/Makefile. We suggest using the defaultrequirements.txt
filename when there only are cross-compiled wheels OR always userequirements-cross.txt
when there also is pure|abi3 wheels to be created. In order to create reproducible builds, all the required packages are frozen to a specific version (e.g.mercurial==4.0.1
); - spksrc will create a cross-compiled wheel by including all the build flags and using the
crossenv
python interpreter; - The resulting wheel is stored in
$(WORK_DIR)/wheelhouse
for later processing; - Finally, spksrc it will rename the wheel so it matches the
uname -m
of the target DSM and copy to$(INSTALL_DIR)/$(INSTALL_PREFIX)/share/wheelhouse
for packaging.
Usage of cross/
python wheels can often be circumvented by:
- Adding the needed
DEPENDS +=
to the SPKs Makefile (ex:DEPENDS += cross/c-ares
); - Including in the shell environment the needed arguments (ex:
ENV += PYCARES_USE_SYSTEM_LIB=1
); - Adding the required packages frozen to a specific version (e.g.
pycares==4.1.2
)
When that is insufficient we must then use a cross package:
- Create a new package in cross/ with the correct details. Add an include to
spksrc.python-wheel.mk
in the Makefile so spksrc knows how to build the package; - Add
BUILD_DEPENDS = cross/python
to the SPKs Makefile. This ensures that Python is cross-compiled and that thecrossenv
requirements are setup correctly to create cross-compiled wheels; - Add the new cross package to
DEPENDS
in the SPKs Makefile; - In contrast to the other two types, this type of package should normally not be included in any
requirements-cross|pure|abi3.txt
.
spksrc will cross-compile Python, then process python-cc.mk
. Due to include ../../mk/spksrc.python-wheel.mk
, spksrc creates a cross-compiled wheel. The resulting wheel is stored in $(WORK_DIR)/wheelhouse
. Finally, spksrc will rename and copy the wheel to $(INSTALL_DIR)/$(INSTALL_PREFIX)/share/wheelhouse
for further processing.
After the above, spksrc will resume its normal activities to build the SPK.
- To include
$(INSTALL_DIR)/$(INSTALL_PREFIX)/share/wheelhouse
in the SPK itself, addrsc:share/wheelhouse
to the SPKs PLIST. - In addition, the installer should contain a line to install the wheels in a Python virtualenv. The generic format to create a virtualenv and install wheels for SynoCommunity packages is:
# Create a Python virtualenv
${VIRTUALENV} --system-site-packages ${INSTALL_DIR}/env > /dev/null
# Install the wheels
${INSTALL_DIR}/env/bin/pip install --no-deps --no-index -U --force-reinstall -f ${INSTALL_DIR}/share/wheelhouse ${INSTALL_DIR}/share/wheelhouse/*.whl > /dev/null 2>&1
The Mercurial SPK contains two Python packages: Mercurial itself, and Docutils, which is a dependency of Mercurial.
Mercurial needs cross-compiling because it contains C-extensions. In addition, it also has to be patched to ensure a working wheel is created. That means it's a package of the third type as previously described. Docutils on the other hand is a pure-python package.
Starting off with Docutils:
Mercurials Makefile sets WHEELS = src/requirements.txt
. This requirements file contains docutils==0.17.1
as its only entry.
As this only package builds properly using cross-compiling set by default the requirement filename is using default requirements.txt
. This is all that needs to be done to create a Docutils wheel.
For Mercurial itself, a bit more is needed:
-
spksrc/cross/mercurial/Makefile
is created with the correct content. There's no need for dependencies in this case, as docutils is handled via the requirements file. - The Makefile's include is set to create cross-compiled wheels:
include ../../mk/spksrc.python-wheel.mk
. - The appropriate patches for Mercurial are added to the
patches
directory. - A digests file should be created, to ensure the file download is not corrupted.
- The SPKs Makefile then needs the following:
BUILD_DEPENDS = cross/python
to cross-compile Mercurial.BUILD_DEPENDS
also containscross/mercurial
(although it could also be added toDEPENDS
as there's nothing in the PLIST) - The last step is to add
rsc:share/wheelhouse
to the SPKs PLIST.
Building the SPK via make arch-$(ARCH) should now result in two wheels in $(WORK_DIR)/wheelhouse
. The wheels are also stored in $(INSTALL_DIR)/$(INSTALL_PREFIX)/share/wheelhouse
, but with a uname -m
DSM architecture matching naming format to ensure the wheels are recognized as valid on the target device.
During the processing of the SPKs PLIST, the wheelhouse directory is copied to $(STAGING_DIR)/share/wheelhouse
.
Once the Python packages are successfully created and included in the package, you'll need to make sure the wheels are installed.
- In Mercurial installer, include the generic command to first create a Python virtual environment:
${VIRTUALENV} --system-site-packages ${INSTALL_DIR}/env > /dev/null
. Note that in some cases you might not want to use--system-site-packages
. - Install all available wheels into the virtual environment as follows:
${INSTALL_DIR}/env/bin/pip install --no-deps --no-index -U --force-reinstall -f ${INSTALL_DIR}/share/wheelhouse ${INSTALL_DIR}/share/wheelhouse/*.whl > /dev/null 2>&1
- Generally speaking, you should start with the assumption that all the required Python packages are pure-python. When building a pure-python wheel fails, the build process will halt with an error, after which you can decide what to do. A good next step is to assume that one or more packages should be cross-compiled, which means adding
BUILD_DEPENDS = cross/python
and see if that works better. - To identify if a package pure-python or not, in most cases the wheels name can tell you. If the wheel package is uploaded on PyPI you can search for the package. The following also applies to wheels created by spksrc:
- While debugging and finding the best configuration it is possible to use a unique
requirements.txt
file and append apure:
,cross:
orabi3:
prefix to the needed wheels. From there usingmake spkclean
will clean-up the wheelhouse in order for it to be regenerated at make time. Note that a final package must not use any prefixes. - It is required to pin packages to a specific version. Example:
mercurial==4.0.1
. Other version specifiers are not allowed. - It is required to add all requirements to the package. Upstream maintainers sometimes only list so-called top-level requirements for their packages, and rely on
pip
to process dependencies during installation. This can cause issues during installation of SynoCommunity packages. To make sure all the requirements are included in your final requirements.txt, runpip install -r requirements.txt
(without specifying--no-deps
) in a separate virtualenv. After pip has processed all the requirements, runpip freeze
, and use that output as starting point for the finalrequirements.txt
. - References to
setuptools
,pip
orwheel
should not be included in requirements.txt or be commented out. - Python packages that are processed as
DEPENDS
, or cross packages, should not be included in requirements.txt or be commented out. - Errors such as
command 'gcc' failed with exit status 1
means cross-compiling is required. - In some cases, wheels appear to build successfully as a pure-python wheel, but fail to install or work correctly on the target. Make sure to test the package, and if you run into issues, try to cross-compile the wheel instead.
- Some native wheel code may not compile without CFLAGS=-Wno-error=format-security
- Some wheel code archive like gevent does not include generated C code. Prefer to download wheel source archives PyPi.org from https://pypi.org/project/gevent/1.4.0/#files
- Home
-
Packages
- Adminer
- Aria2
- Beets
- BicBucStriim
- Borgmatic
- cloudflared
- Comskip
- Debian Chroot
- Deluge
- Duplicity
- dnscrypt-proxy
- FFmpeg
- FFsync
- Flexget
- Gstreamer
- Google Authenticator
- Home Assistant Core
- Jellyfin
- Kiwix
- [matrix] Synapse homeserver
- MinIO
- Mono
- Mosh
- Mosquitto
- Node-Exporter
- Radarr/Sonarr/Lidarr/Jackett
- SaltStack
- SickBeard Custom
- SynoCLI-Disk
- SynoCLI-Devel
- SynoCLI-File
- SynoCLI-Kernel
- SynoCLI-Misc.
- SynoCLI-Monitor
- SynoCLI-NET
- Synogear
- Concepts
- Development
- Resources