-
Notifications
You must be signed in to change notification settings - Fork 19
Yocto Python Recipe Guide for Non Python People
This is a guide to creating an entire custom meta-pythonapplication layer for a yocto nanbield distribution!
By Nicholas Buckley 12/13/2023 Original Document Link
Start with creating your top level meta layer! This layer will hold at minimum a recipe for your application and the recipes for your python application dependencies. After sourcing your environment, now in your newly created build environment. You can now use the following command to create your application layer.
bitbake create-layer ../meta-application
This creates the following folder structure…
meta-application
conf
Layer.conf
README
recipes-example
example
Example_0.1.bb
So now we have a layer where we can add the application recipes and the python recipes. This is where things can get time consuming, but with a little prep you can make this as painless as possible.
My recommendation is to first gather and understand the requirements and limitations of the application. This means understanding compatibility with python versions and dependencies as well as their specific version, as every single version has to be specific or very close when it comes to the world of python applications.
Starting with the most important dependency, python version!
Due to the continuous “upgrades” made by python many many functions change syntax and depreciate older use of them when changing python revisions. This means python applications, if not constantly updated, will likely only function on a subset of versions. So it is key to make sure your python recipe is in line with your python applications requirements. For open source applications this can normally be found in a requirements, the readme, or in the toml requirements file. Let's say we wanted to add the well known scipy application. In the toml file it looks to currently support 3.10 - 3.13 shown here. This means we will need a python3 recipe that provides a version in that range. Luckily poky/meta/recipes-devtools/ provides a recipe for python3, at the time of writing this version is 3.11.6, within the application range! So when you are creating your bb for scipy all you would need to do is add
DEPENDS += "python3 python3-core"
And you will meet the requirements!
However this will not always be the case and sometimes you will need to revert to a specific, hopefully not very old version of python to match your application. For me, a good way to do this is to clone a previous version of the poky repository which had a recipe for the older version of python you require. Then you need to possibly update the syntax of that recipe, depending on how old it is, to match your current yocto release version. Note that this could be the hardest step in the process of making your application layer, since if the python version required is old enough many python package dependencies provided by poky won’t work and will need to changed or removed to allow that python version to be ran in your image.
Assuming you find a python3_3.x.x.bb and it’s patch files for your applications, you then need to create your recipes-python directory to hold your recipes. For python3 you will then add a python3 directory which will contain the .bb and patch files copied. Looking like the tree below
meta-application
recipes-python
python3
python3_3.x.x.bb
python3
0001_x_x.patch
…
0021_x_x.patch
If done properly, with some luck, and ensuring your layer’s BBFILE_PRIORITY is higher then other python3.bb’s in your layer.conf file, this will allow your python version to be the one you specified here in your image. If it does not compile it would be worth trying downgrading yocto revisions to maybe get more of pythons dependencies in line otherwise my advice is one by one fix or remove the recipes it complains about when bitbaking with your python3 version. Being an optimist though I’m sure you now are able to use your new version of python3 and now you just need to add the dependencies!
To add python dependencies for your applications you can go with either again looking at the requirements and toml file for your python application and extracting it’s dependencies and versions from there, or the fastest way, installing the application locally and pip freezing it to get the exact list! To do the later simply setup a virtual environment with the version of python and pip of your recipe that you just created, then pip install theApplication, assuming it installs and builds all the dependencies, you can now do pip freeze theApplication and it lists everything just installed and their version, the more complex the application, the longer the list… It’ll look like this:
pythonPack0==1.0.0
pythonPack1==1.2.0
…
pythonPack20==6.4.0
Now that you have the list you can create python recipes for each package to complete Exodia!!! ( I mean your applications dependencies to add when making the DEPENDS += "x x1" for your application’s bb file)
I found a few methods to go about this but all of them will require some leg work and elbow grease. The first and my personal favorite way to get these packages is via an already made openembedded or poky pythonPackage.bb file of the version you need. Use the yocto layer search to search for recipes. If a bb file and recipe are made for you, I would say that is more development time for you on the application, so skip the leg work and you can just add the layer and add that package to your depends and you are done!
If you are unable to find an existing recipe for a package, then you must create a recipe/bb for the package_version you need. A good starting point is using a tool called pipoe. This is installed with
pip install pipoe
And can be used to attempt to automate the creation of python package bb files! Running
pipoe -p package –python python3 -v 1.2.3
Will create something like the example below…
python3-frozendict_2.3.10.bb
SUMMARY = "A simple immutable dictionary"
HOMEPAGE = "https://github.com/Marco-Sulla/python-frozendict"
AUTHOR = "Marco Sulla <[email protected]>"
LICENSE = "LGPL-3.0"
LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=3000208d539ec061b899bce1d9ce9404"
SRC_URI = "https://files.pythonhosted.org/packages/c8/0b/5892fa56dca4a3f133f317f5e1e59d99916d649b4d47b7408bae7ac26b1a/frozendict-2.3.10.tar.gz"
SRC_URI[md5sum] = "14b5ebac02d85261315668ffcd24b21a"
SRC_URI[sha256sum] = "aadc83510ce82751a0bb3575231f778bc37cbb373f5f05a52b888e26cbb92f79"
S = "${WORKDIR}/frozendict-2.3.10"
RDEPENDS_${PN} = ""
inherit setuptools3
Note that this is likely close to the final version of this bb, but for newer distributions of yocto such as nanbield, this will not work as expected or depending on python package or license it may not work at all. This is a great starting point though especially when just starting so the tool has it’s uses still.
Going through this bb though there is a subtle error in syntax that could cause issues if not dealt with, the RDEPENDS_${PN}. The underscore, _, for any branches after dunfell, should replaced with a :. If you use _ the RDEPENDS will be ignored! Possibly not allowing your package to build and seem broken when it’s just a syntax error. This is an easy thing to miss and should be the first thing you update when making bb’s with this tool.
The other issues I’ve experienced are license and wheel building issues. For licensing issues, indicated by the packaging step failing on the bitbake, for a quick fix set the license to CLOSED and remove the license checksum, for the real fix you must match the license path, checksum, and license title in the package. This is easier said than done but doable if you look in your build/tmp/work/path/to/package/ then you can collect the information to update the bb with. Now for the biggy, wheel building//compiling issues.
With a lot of time under my belt now working with these I have a few options I would point to for varies python packages. First ensure your SRC_URI and sha’s are correct, this information can be found on pypi under releases and download files (view hashes). If these are correct and still not working attempt to use the setuptools3 inherit with the pypi package finder instead of using SRC_URI, an example of this type of package is below...
Python3-flask-assets_2.1.0.bb
SUMMARY = "Flask webassets integration."
HOMEPAGE = "http://flask-assets.readthedocs.org/en/latest/"
PYPI_PACKAGE = "Flask-Assets"
LICENSE = "BSD-3-Clause"
LIC_FILES_CHKSUM = "file://LICENSE;md5=2c0c455026ee819a9365678bea8b2c82"
SRC_URI[sha256sum] = "f84d6532ffe59c9ff352885e8740ff4da25c0bcfacd805f0a806815e44354813"
inherit pypi setuptools3
RDEPENDS:${PN} = "python3-webassets"
This package is installed via the PYPI_PACKAGE line and checked against the sha256 and then it also now inherits pypi. Pretty straightforward updates, but this is a good more modern way to install and build a python package via a bb, however sometimes setuptools3 does not properly build newer or very specific python packages. This shows in the form of the compile step failing for a package. If you see this then some further intervention will be requires in the tools inherited to build the application. In the next example the python package required a build tool called python_hatchling.
SUMMARY = "Low-level CFFI bindings for Argon2"
HOMEPAGE = "https://github.com/hynek/argon2-cffi-bindings"
AUTHOR = "Hynek Schlawack <[email protected]>"
LICENSE = "CLOSED"
#LIC_FILES_CHKSUM = "file://LICENSE;md5=4642dfcbd13c1cc49e9f99df9de51ba1"
inherit python_hatchling pypi
PYPI_PACKAGE = “argon2-cffi-bindings”
SRC_URI += "file://0001-Ensure1-it-compiles-for-non-x86.patch"
SRC_URI[sha256sum] = "bb89ceffa6c791807d1305ceb77dbfacc5aa499891d2c55661c6459651fc39e3"
S = "${WORKDIR}/argon2-cffi-bindings-21.2.0"`
DEPENDS += "\
${PYTHON_PN}-cffi \
${PYTHON_PN}-cffi-native \
python3-argon2-cffi \
"
This package required this specific inherited builder and a patch to allow proper compiling for a non x86 system, but it did get built and it works. With my low understanding of the specifics of python package building, I can’t tell you which inherits are specifically needed for every package although I’m sure someone with more python knowledge would know, here is a short list of the inherits that I found to work on all of the package combinations that could work with various different packages.
setuptools3, python_hatchling, python_poetry_core , and python_setuptools_build_meta
If none of these allow the package to compile, then some manual patching of the recipe may be required, with the easiest way to do that being devtool modify recipename.
Again being an optimist though, you can now rejoice as you have gotten all of your python packages turned into building bitbake recipes! So now we just need the application!
Now we need to add your application bb file! For this guide’s sake the only thing that needs to be done is to ensure your newly created recipes are all added to the DEPENDS of your application. So assuming all of your recipes are creating and compiling and packaging, along with your new application of course, your application should now be included in your image ready to run! Hopefully this was helpful