Python 3 support 🙲 locking of dependencies

[cerna]

Changing the supported version of Python from 2 to 3 in Machinekit-HAL dependency (in machinekit/machinekit-hal#303) caused that EMCApplication Debian packages are installable, but the software is actuall unusable.

This is because EMCApplication has Machinekit-HAL as a dependency, but is comfortable with anything >0.4. In hindsight this is naïve as the development of Machinekit-HAL (and other dependencies) can go at paces different from EMCApplication, so (minimally when creating binary packages) the dependencies should be locked at known version.

In case with Machinekit-HAL, the version with which the EMCApplication was build with.

[jallwine]

Didn't LinuxCNC finish porting from Python 2 to 3? After installing EMCApplication on top of MachineKit-HAL, I can only import linuxcnc from Python 2.

[cerna]

@jallwine,
I don't think they fully finished it. But yes, the LinuxCNC should be able to run on Python3. The problem described here has multiple vectors:

1. I made mistake in presumption that binary packages created from this repository (Debian) will be able to run with any version of Machinekit-HAL. What I should have done is to pin the required package of Machinekit-HAL libraries in the EMCApplication binary package.

2. Debian packaging in LinuxCNC does not actually take Python 3 into account right now. It is still building packages based on Python 2. (Look for example on this line.) And there is no switch I can toggle to make it use Python 3. There is actually discussion how to make the switch, so I was waiting on this. (As everything I do here should be minimal.) But it looks like I will have to solve it and when it is decided in LinuxCNC upstream, revert it back/bend it to the decision.

[jallwine]

When I build LinuxCNC from source using this configure command, I do get a python3 version of the linuxcnc module:

./configure --with-realtime=uspace --with-python=python3 --with-boost-python=boost_python-py35

[cerna]

@jallwine,
yes, sorry, I was not precise. I am talking about Debian (and Ubuntu) packaging. You can of course build in-tree version with Python 3.

Looks like @rene-dev is busy working and actually has support for Python 3 based packages in LinuxCNC/linuxcnc#943.

[jallwine]

Gotcha, so following the instructions in the EMCApplication README I can build it from source to have Python 3 support?

Copied mostly from the README, but with the python flags:

git clone https://github.com/machinekit/emcapplication.git
cd emcapplication
sudo apt install build-essential fakeroot devscripts python
debian/configure machinekit-hal no-docs
mk-build-deps -irs sudo
cd src
./autogen.sh
./configure --with-hal=machinekit-hal --with-python=python3 --with-boost-python=boost_python-py35
make
sudo make install
cd ..
source ./scripts/rip-environment

[rene-dev]

linuxcnc master works with python3, but no python3 packages have been released, and it is currently not the default.
Main reason is that the only UI that works is axis.
my branch fixes the UIs, and packaging. so if you want UIs other than axis, use my branch.
is EMCApplication just linuxcnc with updated build system to accept MK HAL? if yes, how large is the diff?

[cerna]

@jallwine,

Gotcha, so following the instructions in the EMCApplication README I can build it from source to have Python 3 support?

Yes. (At least I have just tried it with update PR #4.)

Copied mostly from the README, but with the python flags:

Yup, this should work!

[cerna]

is EMCApplication just linuxcnc with updated build system to accept MK HAL? if yes, how large is the diff?

It is basically patched LinuxCNC@master branch to build and work against Machinekit-HAL repository (RuntimeAPI, HAL, modules etc.). Only the EMC application (+ at least one GUI) is expected to work.

The whole idea is to have a bunch of Featured Applications running against Machinekit-HAL and support that way the modularized approach. (Another one is the HAL ROS integration.)

my branch fixes the UIs, and packaging. so if you want UIs other than axis, use my branch.

Thanks, but the idea behind EMCApplication is to have minimal patching and LinuxCNC@master tracking. So I will have to wait until it is merged to master.

[rene-dev]

Is there any reason the updated build system can't be merged into linuxcnc master?

[zultron]

Is there any reason the updated build system can't be merged into linuxcnc master?

It would be fantastic if the LCNC project could merge the changes for building EMC against an external HAL. When I wrote them, I deliberately made them as minimal as possible, both to make them easier to merge into updated LCNC, and also in hopes they would be easier to review for LCNC inclusion.

@cerna knows best the latest status, but once the EMCApplication is building smoothly in CI and we've worked out any remaining issues, I'd consider the changes ready for inclusion upstream.

[cerna]

Is there any reason the updated build system can't be merged into linuxcnc master?

Because I didn't think (@zultron probably too) that it would fly. I still don't. This is basically packaging parts of the LinuxCNC for use in other programs, it could be a precursor to splitting the LinuxCNC into HAL, CNC and GUIs parts. And I just cannot see the community going for it.

Or you mean the Python 3 packaging for Debian? Well, I wouldn't be too much considerate to the non-Python 3 supporting parts. One GUI is good enough for me. And now you, heavyweight in LinuxCNC development community, are doing it.

The status is that now the CI part must be done. Given that the Machinekit wants to use Machinekit-CI repository for most CI related tasks, I need to abstract away the differences between all repositories to some useable and not-too-stupid level.

Then there are the tests, which not all pass because of environment/executable mismatch.

I haven't really spent any time on this recently and considered it lower priority. (It was working). So, so far I would consider good plan to implement machinekit-hal package locking and rebuild all images with Python2. It won't help @jallwine or others needing newer versions of HAL, but at least it will be useable.

[jallwine]

@rene-dev how much more is there to do to get your changes into LinuxCNC master? It seems like the move to Python 3 is actively progressing and rolling back to Python 2 isn't the best plan.

I would love to see LinuxCNC split up into HAL, CNC and GUIs. Encapsulation is the name of the game in software development. It's more maintainable, understandable and flexible. We don't use any of the GUIs, so its frustrating that they seem to be the main thing holding back progress. Python 2 has been EOL for almost a year now. I understand open source projects can lag behind due to the nature of volunteers driving development, and it can be difficult to get buy-in on large changes due to the limited resources to implement them, but if LinuxCNC was split up into more manageable pieces, it seems like it would be better for long term development. @cerna, @zultron what do you think the main resistance to splitting up would be?

[zultron]

@zultron what do you think the main resistance to splitting up would be?

Of course, this question should be asked in the LCNC project, but I do remember the idea being broached in the past.

I recall seeing only tepid support. Speculating, perhaps a lack of enthusiasm stems from not understanding the benefits, or a low benefit to effort ratio. (FWIW, I'm obviously in favor of splitting!)

The only concrete reason opposing a split that I recall concerned a fear that users submitting issues would be confused about which repo to file the issue in, for example the UI, the EMC or the HAL repo. IMO, if that's actually a significant problem, there are ways to mitigate it.

[cerna]

@jallwine,

(...) what do you think the main resistance to splitting up would be?

Thinking about it for a couple of days, I consider the main hurdles:

1. Version mismatch between depending-on packages

• This is basically the core of the issue Require specific Machinekit-HAL package version #6, you have Machinekit-HAL which has some development vector and you have LinuxCNC with its own development vector. And you somehow must keep these in sync, or better said you need to track which CNC package version is satisfied with which HAL package (and then you have the GUI problem, where both this is valid plus some GUI packages can be depending only on HAL [HAL<-GUI] or on CNC [HAL<-CNC<-GUI]). With adding packages to the mix, the dependency graph is becoming quite complex.

2. Build system recipes are quite monolithic and tangled

• This is actually connected to 1., but LinuxCNC (and Machinekit so far too) is using a weird combination of make, autoconf and KBuild. It is not using encapsulated namespaced objects (targets) as far as I know, so you cannot take one part out and put it somewhere else. Systems like Modern CMake or Meson have it thought out much better. Machinekit-HAL and Machinekit-CNC were not much better at first and the separation was (still kinda is) bloody.

3. CI and CD infrastructure

• LinuxCNC is building on quite precarious system based on old version of Buildbot software. This is mainly because of support for number of very old distributions. But the dark side is that it is more often down than up (I have a script which checks this out). Any deep change like this would need to be reflected in the build environment and for this I think it would need to be fully reimplemented.

4. Support for extremely ancient and out-of-date distributions

• This one is quite startling, because LinuxCNC is using Debian as base distro and Debian community is maintaining mostly only the latest and greatest version (Sid) to be up-to date, everything else is only basic security updates. (I was once told in Debian developer channel to either stop asking questions about Stretch or go fishing.) But LinuxCNC is preparing its latest release for even the older distribution. (Machinekit is doing it too, but at least I slashed the Jessie support the day it was EOLed.) This is maintenance nightmare, because you are using software with bugs which were plugged long ago in mainline. It also goes against the software release status-quo, so you are pretty much on your own. (Everybody else thinks you are insane.)

5. Community is extremely conservative

• From time to time (more often lately) you see in LinuxCNC channels that people start to think about how development of LinuxCNC should progress in the future (with more or less finesse). Most lately for example in this thread. These discussions are usually quickly shut down with the argument that what the proposing party wants goes against the core values of LinuxCNC (whatever that may be). No abstracting of the threads of presented idea, not throwing stuff on the wall and looking at what sticks with glass of wine and interested individuals. If it isn't immediately visible to machining crowd, it is not going to get support. The software engineering aspect is not important that much. I guess that it is kind of like cook asking physicist the usefulness of infinite dimensional vector spaces. Given the amount of work needed, I am not surprised that nobody wants to do work (for free) which is not wanted here. There is old thread at emc-developers where it was presented and the mood was very meh. Actually, the main LinuxCNC repository is taking in more parts (GUI, modules, configs) then shedding them.

• BTW, don't take this like I am trashing the LinuxCNC community, it's just completely different mindset. One not particularly inclined to software development issues.

[jallwine]

It sounds like LinuxCNC has now fully implemented Python 3 support. Perhaps this issue can now be resolved?

Also, I'd like to continue our conversation about the possibility of merging EMCApplication upstream, but I'm not sure the best place to do that (so I'm going to here for now). It would be great to have the conversation somewhere both LinuxCNC an MachineKit developers could weigh in (maybe a pull request?).

I've been in contact with a LinuxCNC developer (@SebKuzminsky) as well as their sponsor into Debian (@smoe) (they are in the process of integrating packages officially into Debian, see LinuxCNC/linuxcnc#1269). The impression I get is they are in favor of splitting things out where it makes sense (HAL, GUIs, etc.) and of course the amount of work that is involved is a key factor. It seems like it would be fairly straight forward to merge EMCApplication upstream, but there could be complications about dependencies and package management? @cerna, I've seen a lot of work on your part with respect to package management and the build system (machinekit/machinekit-hal#349 and machinekit/machinekit-hal#353). Any thoughts? Another concern is providing an environment for GUI developers to be able to focus on GUI concerns and less about package management concerns. Like you mentioned, the dependency graph could get complicated, so ensuring there's a reasonable development workflow is key. Anyway, I definitely think bringing these forks back together somehow would have benefits for everyone!

For what it's worth, we recently released software for all Pocket NC machines that includes EMCApplication+MachineKit-HAL. A major reason was to get the best of both worlds: the HAL improvements from MachineKit-HAL and the CNC improvements from LinuxCNC. If LinuxCNC started using MachineKit-HAL, we'd be onboard with using LinuxCNC+MachineKit-HAL.

[cerna]

It sounds like LinuxCNC has now fully implemented Python 3 support. Perhaps this issue can now be resolved?

Very probably yes. As I have been working on the buildsystem for Machinekit-HAL, I haven't looked into it yet.

Also, I'd like to continue our conversation about the possibility of merging EMCApplication upstream, but I'm not sure the best place to do that (so I'm going to here for now). It would be great to have the conversation somewhere both LinuxCNC an MachineKit developers could weigh in (maybe a pull request?).

Without any code, the issues are probably better than pull requests. Not sure if this one or something more neutral.

But the EMCApplication already is pretty much upstream, with only minor patches to allow building the EMC portion against externally packaged HAL.

As far as I know, @rene-dev is currently working on CMake port of the Makefile based buildsystem of LinuxCNC. I am not sure how far along this is, but in connection to my rework of Machinekit-HAL, it could (maybe) be fairly straightforward. Or just require some wrapping CMake targets.

I've been in contact with a LinuxCNC developer (@SebKuzminsky) as well as their sponsor into Debian (@smoe) (they are in the process of integrating packages officially into Debian, see LinuxCNC/linuxcnc#1269).

Interesting. I sent some feelers how hard it would be to include Machinekit-HAL into Debian proper and was told that there is almost zero chance with the current package structure et al. (Mixing executables, libraries and development files, Python and all into almost one.) Which is one reason why I started doing the restructuralization of packaging. (The current status can be seen here.) Using CMake/CPack and dh_cmake makes it fairly easy. It looks like Debian finally decided what to use for PEP517/518 when packaging Python wheels, thus also simplifying the problem and the Python stuff could be also disassembled into more packages. That in all could pass the Debian bar.

The impression I get is they are in favor of splitting things out where it makes sense (HAL, GUIs, etc.) and of course the amount of work that is involved is a key factor. It seems like it would be fairly straight forward to merge EMCApplication upstream, but there could be complications about dependencies and package management?

I am all in favor of splitting the stuff into multiple repositories and packages. Unfortunately, there is no hard standard for specifying dependencies. For the build itself, you can use the basic pkg-config or CMake export system which can be read by Meson or other tools. Dependencies can be solved by distro provided packages, package management systems like conan, CMake based superbuild and git repositories, combination of aforementioned... Bottom line, I don't think it is ever going to be easy.

From the GUI tools standpoint, these should just declare the dependency version they currently support but live on their own. Kind of like the group around the @kcjengr is currently doing with QtPyVCP or the @qtquickvcp.

To summarize, what I think is important:

• Kick the Debian exclusivity (and allow simple building for Fedora, Gentoo, Nix etc.)
  • This means kicking the current Makefile based buildsystem without sane installation procedure
• Separate parts of the code into its own projects/packages, so one change in one part would not start avalanche of errors in unrelated parts (example in Machinekit-HAL: the shotgun approach to includes)
  • This would also propagate a public API - developers would think more about change because it would not be just internal implementation

[smoe]

I need to exempt myself from this discussion until I understand more of LinuxCNC (and MachineKit-HAL). My hunch is that all @cerna outlined above will happen, albeit unlikely for 2.9 of LinuxCNC (now at 2.8.2). I am not sure if the modularisation needs to happen all in one go, and if not, which parts should be isolated first. The advantage of Make over all other build systems is that everyone thinks to understand it. And from what I oversee, there is no need to move away from it.

[rene-dev]

@zultron mentions a few problems with the current buld system here: machinekit/machinekit#336
you also mentioned a few while fixing the debian packages.
My cmake version already builds, and most tests are passing, it greatly simplifies things.
IMHO this has many advantages, the current buildsystem is very broken and hard to work with.
I also support breaking up hal and the UIs, and merging linuxcnc hal and mk hal back together.

[smoe]

We likely agree that it is all less about the merger of code than it is about the concentration of eyeballs and brain-power that the merge is helping with for the benefit of an accelerated development of LinuxCNC at large. Someone forking is not necessarily something negative - technically, every branch does that. So, forks shall be encouraged. And so should be merges for the benefit of all.

The modularisation is about helping forks to "fork small" and keep diffs humanely traceable, so merges are easier, too. The situation now is a very natural one of projects becoming big. If CMake is a way to help a partial merge with LinuxCNC's most prominent fork - so go for it, I'd say. I'll likely learn something.

@rene-dev , I presume you are in close contact with the other LinuxCNC devs, so you know better than me how to overcome last hurdles for a transition to CMake. Concerning the Debian packaging I happily adjust to what you core devs are agreeing on. To help the forking it may be reasonable to introduce sonames to package names to reflect the API versions. This would then be the formal representation of dependencies that @cerna asked for.

[jallwine]

Also, I'd like to continue our conversation about the possibility of merging EMCApplication upstream, but I'm not sure the best place to do that (so I'm going to here for now). It would be great to have the conversation somewhere both LinuxCNC an MachineKit developers could weigh in (maybe a pull request?).

Without any code, the issues are probably better than pull requests. Not sure if this one or something more neutral.

But the EMCApplication already is pretty much upstream, with only minor patches to allow building the EMC portion against externally packaged HAL.

@zultron @cerna @rene-dev, what concrete steps are necessary before a merge upstream is possible (ignoring developer buy-in for the moment)? @cerna, like you said, it seems like EMCApplication only has minimal changes, the main one being it separates the HAL side of LinuxCNC so it can be built on an external HAL. This is exactly what everyone here seems to be suggesting is a good idea. If the latest LinuxCNC changes were merged into EMCApplication, could we put together a pull request from EMCApplication back up to LinuxCNC? Obviously, there will be a big dependency change in doing so, but it seems like it would be straight forward to at least formalize the effort and move the conversation to a directly relevant place. @rene-dev you mentioned merging LinuxCNC HAL and MachineKit-HAL. Is there functionality in LinuxCNC that is missing from MachineKit-HAL? Or, do you see MachineKit-HAL as a suitable replacement for LinuxCNC HAL? Merging EMCApplication into LinuxCNC seems like the right first step from my perspective, but am I missing something obvious?

[cerna]

My hunch is that all @cerna outlined above will happen, albeit unlikely for 2.9 of LinuxCNC (now at 2.8.2). I am not sure if the modularisation needs to happen all in one go, and if not, which parts should be isolated first.

In Machinekit, the first step of modularization was to separate the Machinekit-HAL and Machinekit-CNC. After that the Machinekit-CNC was kind of dropped in favor of EMCApplication, which is mostly upstream LinuxCNC patched so it supports building against external HAL. This kind of separation is what I think would be interesting for LinuxCNC project as well.

What I have been trying to do with Machinekit-HAL is to separate the HAL portion even more - for example put each Python logical module into its own package and build a wheel from it. Or separate the NanoPB library, the PRU Assembler etc - basically things which were included in the past and which I think should not have been. CMake supports interesting pattern called Superbuild which allows you to download (or otherwise get) external dependencies and parts during the build itself, so you don't need to have them in the base repository.

@rene-dev,

My cmake version already builds, and most tests are passing, it greatly simplifies things.
IMHO this has many advantages, the current buildsystem is very broken and hard to work with.
I also support breaking up hal and the UIs, and merging linuxcnc hal and mk hal back together.

Do you have it somewhere public for me to take a look? Trying to look up branches named "cmake" or "buildsystem" in your fork was not very fruitful.

The modularisation is about helping forks to "fork small" and keep diffs humanely traceable, so merges are easier, too.

This! Trying to make changes one needs to keep track of all possible friction points. Having many small projects instead of one big one should help with this problem.

To help the forking it may be reasonable to introduce sonames to package names to reflect the API versions.

True. Small problem with this is that nobody really cared about it before, because everything was rebuild in one go. So it will require additional step (which might annoy developers).

@jallwine,

what concrete steps are necessary before a merge upstream is possible (ignoring developer buy-in for the moment)? (...) If the latest LinuxCNC changes were merged into EMCApplication, could we put together a pull request from EMCApplication back up to LinuxCNC?

I think the current changes could be merged pretty much as is upstream. Of course there could (and probably would) be some minor problems integrating it into the buildsystem. (Export what is in EMCApplication@machinekit/master into patch applicable against LinuxCNC@master.)

Is there functionality in LinuxCNC that is missing from MachineKit-HAL? Or, do you see MachineKit-HAL as a suitable replacement for LinuxCNC HAL?

I know it was not question for me, but I think that both HAL implementation will probably remain independent. It would be great to define some public API to standardize the cooperation, though.

[smoe]

@petterreinholdtsen is currently orchestrating the use of po4a for the internationalization of the LinuxCNC documentation. This requires changes to the build infrastructure and may be an opportunity for a functionally non-critical assessment of whatever plans there are emerging for the modularization. An early contact to @rene-dev is already established from what I have followed on IRC.

[cerna]

I PRed in the change to CMake based build-system into Machinekit-HAL. SO now is probably as good as any time to update the EMCApplication on current upstream master.

The buildsystem change created few problems which will need to be solved (read I will need to solve) on the side of EMCApplication:

• Machinekit-HAL had a single simple pkg-config file defining all libraries and includes, now the structure is workspace line, modularized, packageized (call it whatever) where each library (SHARED, STATIC, INTERFACE) has it's own binary and source (headers mostly) objects tightly coupled and exported in one *Target.cmake file. Option is to either create pkg-config alternative and then modify it on the EMCApplication side or just define it on EMCApplication side.
• The header files are now in Machinekit-HAL prefixed with the package name, as to not be complete mess and allow developers to name the files whatever they like (there were multiple hal.h files). So now to include HAL API, one needs #include <hal/hal.h>. This of course can be solved with just changing the -I option from .../include to .../include/hal for minimal changes.
• The Python modules were placed into namespace, so simple wrappers will be needed

As next task I will have to get familiar enough with these recent changes in upstream LinuxCNC.