Update pycoral for Python 3.11 and modern versions of Tensorflow

[feranick]

Description

Pycoral is available only with support for Python 3.9. Most distribution now no longer ship with that version, rather either 3.10 or 3.11. So there is factually no version of pycoral available to download. Furthermore, building pycoral from source is not compatible with Python 3.10 or 3.11 (again, not supported) and relies on an ancient version of tensorFlow (2.5.0) to work. While this al points to a lack of maintenance, it would be great to at least update so it can be used with, well, pretty much any modern version of OS.

Click to expand!

Issue Type

Build/Install

Operating System

Linux, Ubuntu, Mac OS

Coral Device

USB Accelerator, M.2 Accelerator A+E, M.2 Accelerator B+M, M.2 Accelerator with dual Edge TPU

Other Devices

Raspberry Pi 3, Rapsberry Pi 4

Programming Language

Other

Relevant Log Output

No response

[tranzmatt]

Has this project been officially abandoned?

[feranick]

There hasn't been any comment, action, or even statement in a very long time. The repos are frozen in time. Such a pity when every chip manufacturer is presenting NPUs as the next big thing, and Google (who pioneered it with the edgeTPU so long ago) decided to drop the ball. To make it worse, no official statement either, and a community that relies/invested on it is left it to its own devices (pun very much intended). And to make it worse, coral.ai still sells these products as fully supported when they are nothing but.

Sad, and infuriating.

[Emojibotbot]

I'm experiencing the same issue on 64 bit Raspberry Pi OS 12.4 running on a Pi 4 which has python 3.11.2. I tried to build pycoral myself and have not managed to make things work yet.

I guess all the layoffs have hit this team pretty hard.

[feranick]

I ended up forking the libedgetpu and bringing it up to date with compatibility to the latest version of TF. This is all I really needed (as I don't use pycoral, but directly access the edgeTPU withtflite_runtime) so I can fully use the edgeTPU. I took a look at pycoral but it seems that a whole lot more work is needed to bringing it up to date.

https://github.com/feranick/libedgetpu

[Emojibotbot]

I ended up forking the libedgetpu and bringing it up to date with compatibility to the latest version of TF. This is all I really needed (as I don't use pycoral, but directly access the edgeTPU withtflite_runtime) so I can fully use the edgeTPU. I took a look at pycoral but it seems that a whole lot more work is needed to bringing it up to date.

https://github.com/feranick/libedgetpu

I was able to install the edgetpu runtime without issues on my rig using instructions here (but not have tried to run it in anyway yet.) What made you need to update?

[feranick]

I was able to install the edgetpu runtime without issues on my rig (but not have tried to run it in anyway yet.) What made you need to update?

That is the point. The provided whl install fine if you have the correct Python environment (see below), but they segfault if you are running tlite_runtime > 2.11.1.

The wheels provided are only available for up to Python 3.9, su pretty useless in any modern Linux release (Raspbian/Debian-bookworm current uses 3.11, ubuntu 22.04 uses 3.10). It may work with "legacy" debian-bullseye which uses Python 3.9.

[feranick]

Plenty of discussion here:

google-coral/edgetpu#812

[feranick]

There is also another issue: the google-dkms gasket is incompatible with linux kernel >= 6.5 and therefore the edgeTPU won't be recognized. The solution is a recompile of the gasket itself, following instructions here:

google-coral/edgetpu#820

[Emojibotbot]

I see you have done a lot of work to update things already & nice job! I thought the issue was just that pycoral was not up to date, but the lack of updates much more systemic across all of google-coral.

[Emojibotbot]

Is it safe to assume your fork would not work? https://github.com/feranick/pycoral

[feranick]

Correct. If it will ever work, I will post binaries there, but I have limited time to spend on it at the moment. Thanks

…

On 2/16/24 12:21 PM, Emojibotbot wrote: Is it safe to assume your fork would not work? https://github.com/feranick/pycoral — Reply to this email directly, view it on GitHub <#137 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAIY77VBR5V37GYBMFAVQDYT6IQLAVCNFSM6AAAAABAT3RXKCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBYHEZTSNZVHA>. You are receiving this because you authored the thread.Message ID: ***@***.***>

--------------uAcACjLxWMBrvKW4lt9XPhZx Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit <!DOCTYPE html><html><head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body> Correct. If it will ever work, I will post binaries there, but I have limited time to spend on it at the moment. <br> <br> Thanks<br> <br> <div class="moz-cite-prefix">On 2/16/24 12:21 PM, Emojibotbot wrote:<br> </div> <blockquote type="cite" ***@***.***"> <p dir="auto">Is it safe to assume your fork would not work? <a href="https://github.com/feranick/pycoral" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/feranick/pycoral</a></p> <p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br> Reply to this email directly, <a href="#137 (comment)" moz-do-not-send="true">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AAAIY77VBR5V37GYBMFAVQDYT6IQLAVCNFSM6AAAAABAT3RXKCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBYHEZTSNZVHA" moz-do-not-send="true">unsubscribe</a>.<br> You are receiving this because you authored the thread.<img src="https://github.com/notifications/beacon/AAAIY75WOAKKEISPE2MYUP3YT6IQLA5CNFSM6AAAAABAT3RXKCWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTTUFJ264.gif" alt="" moz-do-not-send="true" width="1" height="1"><span style="color: transparent; font-size: 0; display: none; visibility: hidden; overflow: hidden; opacity: 0; width: 0; height: 0; max-width: 0; max-height: 0; mso-hide: all">Message ID: <span>&lt;google-coral/pycoral/issues/137/1948939758</span><span>@</span><span>github</span><span>.</span><span>com&gt;</span></span></p> <script type="application/ld+json">[ { ***@***.***": "http://schema.org", ***@***.***": "EmailMessage", "potentialAction": { ***@***.***": "ViewAction", "target": "#137 (comment)", "url": "#137 (comment)", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { ***@***.***": "Organization", "name": "GitHub", "url": "https://github.com" } } ]</script> </blockquote> <br> </body> </html>

--------------uAcACjLxWMBrvKW4lt9XPhZx--

[feranick]

Thanks to @pkgoogle and @Namburger, the libdgetpu repo/library has been updated, a big step forward. Next that should be updated is libcoral/pycoral, the frontend libraries that are (supposedly) the standard way to interact with the edgeTPU. My fork still does not compile, and several issues are at play. There is an open issue/PR for TF that is supposed to fix one of the issues: Change visibility status for lite/schema/schema_conversion_utils to allow compilation of external libraries. If there is a way for this PR to be merged, that'd be great, since in the current state, such visibility issue in TF (which is downloaded by Bazel in tmp_folder during the compilation of libcoral) hangs compilation of pycoral.

[feranick]

I finally managed to successfully compile libcoral through this new fork of mine.

I submitted a PR request: google-coral/libcoral#36

Of course, this needs the visibility issue to be fixed first. Yet, compilation can be successfully done manually, until the issue above is fixed. Two steps:

1. Change last line in WORKSPACE from:

path = "/usr/system_libs",

to

path = "/",

as the former is designed when operating with Docker.

2. Edit: nano ~/.cache/bazel/_bazel_XXXX/XXXXXXX/external/org_tensorflow/tensorflow/lite/schema/BUILD
   and remove visibility = [":utils_friends"],

from both:

cc_library(
    name = "schema_utils",
    srcs = ["schema_utils.cc"],
    hdrs = ["schema_utils.h"],
    compatible_with = get_compatible_with_portable(),
    visibility = [":utils_friends"],
    deps = [
        ":schema_fbs",
        "//tensorflow/lite/kernels/internal:compatibility",
        "@flatbuffers//:runtime_cc",
    ],
)

cc_library(
    name = "schema_conversion_utils",
    srcs = ["schema_conversion_utils.cc"],
    hdrs = ["schema_conversion_utils.h"],
    compatible_with = get_compatible_with_portable(),
    visibility = [":utils_friends"],
    deps = [
        ":schema_fbs",
        "//tensorflow/lite/kernels/internal:compatibility",
        "@flatbuffers",
    ],
)

[feranick]

A new fork for pycoral is now available, which relies on the libedgetpu and libcoral forks, and it has been tested to work. All three libraries are currently targeting the upcoming TensorFlow version 2.16.0 (currently in rc0) status, with the hope that the visibility issue is fixed in that release. Once TF 2.16.0 is officially released as stable, I will create a PR for the updated libedgetpu (which now targets TF2.15.0 in the official repo). Upon merging libedgetpu into the official repo (which currently targets TF 2.15.0), another PR is ready for the updated libcoral. Once that is merged, you guessed it, I will push a PR for pycoral based on this fork.

In the meantime, one can already build libedgetpu from the official repo (against TF 2.15.0) or the fork needed for this upcoming version of pycoral. With that, pycoral can be built as instructed in the README file in the forked repo, with the only caveat that the visibility issue needs to be manually addressed, as detailed in the comment above..

Thanks again to @Namburger and @pkgoogle for pushing forward the effort to modernize the EdgeTPU libraries.

[feranick]

A new fork for pycoral is now available, which relies on the libedgetpu and libcoral forks, and it has been tested to work. All three libraries are currently targeting the upcoming TensorFlow version 2.16.0 (currently in rc0) status, with the hope that the visibility issue is fixed in that release. Once TF 2.16.0 is officially released as stable, I will create a PR for the updated libedgetpu (which now targets TF2.15.0 in the official repo). Upon merging libedgetpu into the official repo (which currently targets TF 2.15.0), another PR is ready for the updated libcoral. Once that is merged, you guessed it, I will push a PR for pycoral based on this fork.

In the meantime, one can already build libedgetpu from the official repo (against TF 2.15.0) or the fork needed for this upcoming version of pycoral. With that, pycoral can be built as instructed in the README file in the forked repo, with the only caveat that the visibility issue needs to be manually addressed, as detailed in the comment above..

Thanks again to @Namburger and @pkgoogle for pushing forward the effort to modernize the EdgeTPU libraries.

[adelmiljkovic]

[A new fork for pycoral is now available]

I tested it and seems like something is wrong. When I run one of the examples I get INFO: Created TensorFlow Lite XNNPACK delegate for CPU. and I did some test and the inference times are the same for running the edgetpu models compared to the regular ones (e.g. movenet_single_pose_lightning_ptq_edgetpu.tflite and movenet_single_pose_lightning_ptq.tflite)

Anyone else experiencing this?

[feranick]

First, bazel 6.5.0 is required for TF 2.17.0. I'd stick to that.

If you want to do a native compilation (i.e. without docker), I simply run:

make pybind
make wheel

This with ubuntu:22.04. Works for ma.

[feranick]

I'd like to remind folks that want to compile these libraries to please read the READMEs. Versions and requirements are listed there.

https://github.com/feranick/TFlite-builds/blob/main/README.md
https://github.com/feranick/libedgetpu/blob/master/README.md
https://github.com/feranick/pycoral/blob/master/README.md

[virnik0]

No kidding. Childish mistake on my part, true. First of all, I was using already outdated approach. Second, I was using GCC instead of CLang. And last, I was using either too old (4.5.1) or too new (7.2.1) bazel

I've succeeded in building pycoral. I have uploaded its wheel to my repo for anyone dealing with the same thing to use.
Link bellow:
https://github.com/virnik0/pycoral_builds

[feranick]

Cool, glad it worked out.

[virnik0]

One thing though: I had to prepend desired TF support version, otherwise it wouldn't work correctly:

TF_PYTHON_VERSION=3.12 make pybind
TF_PYTHON_VERSION=3.12 make wheel

Anyway, thanks a lot for patience and will to assist. Your points were very helpful in realizing what I was doing wrong.

[feranick]

Thank you. Yes, that is needed, and will add to the README.md.

[virnik0]

I just have to rebuild edgetpu lib as well as tf now, as using what I have ends in segmentation fault from pycoral's side. Will report back once/if successful.

As for tensorflow_lite, cmake compilation seems not to work correctly (compilation crashes at 48%, will have to check the logs what exactly went wrong).

Bazel build seems to succeed.

After starting with clean slate again (removing libedge, tf, pycoral installs and installing from prior installs) freshly built packages built against Python 3.12, all works:

~/repositories/coral/pycoral# python3 examples/classify_image.py --model test_data/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite --labels test_data/inat_bird_labels.txt --input test_data/parrot.jpg ----INFERENCE TIME---- Note: The first inference on Edge TPU is slow because it includes loading the model into Edge TPU memory. 11.6ms 2.7ms 2.6ms 2.6ms 2.6ms -------RESULTS-------- Ara macao (Scarlet Macaw): 0.75781 ~/repositories/coral/pycoral#

I have uploaded all those packages to my github, at least for the time being until some more knowledgeable on this subject build better ones.

I've quickly tested both libcoral C+ layer, also works. ZMEventNotification server also accepts TPU like its own, and object detection works correctly. What I noticed is that using latest ubuntu and python, the detection is a bit faster (using max variant of libedgetpu lib) than it was on Ubuntu 20.04.

[sn0wst0rm]

@feranick I'm trying to build everything for MacOS Ventura (13) because the who you provided are for Python 3.11 and macOS 14 (cp311-cp311-macosx_14_0_x86_64.whl), while I have 13 and python 3.12. How did you compile them? If I use Bazel, I get to this error: Error in fail: Xcode version must be specified to use an Apple CROSSTOOL. If your Xcode version has changed recently, verify that "xcode-select -p" is correct and then try: "bazel shutdown" to re-run Xcode configuration.

I only have Xcode command line tools installed, do I also need entire Xcode?

Thanks for the advice, I just received a Coral TPU and I am really disappointed (to say the least) that it is practically unusable (when following the instructions in the official webpage)

[feranick]

Yes, you do need the full XCode. Follow through the error message.

Then you can follow the instruction on how to build what you need:

https://github.com/feranick/TFlite-builds/blob/main/README.md
https://github.com/feranick/libedgetpu/blob/master/README.md
https://github.com/feranick/pycoral/blob/master/README.md

[sn0wst0rm]

Yes, you do need the full XCode. Follow through the error message.

So @feranick I've managed to make bazze work, but it produced an artefact called tflite_runtime-2.17.0-cp312-cp312-macosx_14_0_x86_64.whl, while I was expecting macosx_13. When I try to install it with pip3 install, I get the following error:
ERROR: tflite_runtime-2.17.0-cp312-cp312-macosx_14_0_x86_64.whl is not a supported wheel on this platform.

I don't really know what I am doing wrong because nothing else fails and the only thing I changed was the TF_PYTHON_VERSION env variable that I set to 3.12.

Do you know how I can make pip install the wheel?

Thanks again for the support

[feranick]

I don't understand how you can get a MacOS 14 wheel while using macOS 13.... It's possible that you need a XCode version compatible with MacOS 13. I am afraid, I can't really say what's happening, as I don't have MacOS 13.

I have never used it on MacOS, but you can certainly try to run a build from a docker image consistent with your target...
https://hub.docker.com/r/sickcodes/docker-osx

Again, not sure what more I can help.

[feranick]

This is highly "last resort", but you can try and rename tflite_runtime-2.17.0-cp312-cp312-macosx_14_0_x86_64.whl as tflite_runtime-2.17.0-cp312-cp312-macosx_13_0_x86_64.whl.

Still might not work.

[feranick]

I pushed a few commits to the pycoralfork, to allow docker compilation under Ubuntu 24.04/Python3.12. Ubuntu 24.04 has defaulted in the use of new source files for apt that was not supported in the old dockerfile.

This has been tested to work for x86_64 and aarch64. Builds are available here:

https://github.com/feranick/pycoral/releases/tag/2.0.2TF2.17.0

[hinxx]

@feranick I tried using your prebuilt wheels for 3.10 (on ubuntu 22.04) and I get this error:

(venv) $ pip3 install --upgrade /data/data/coralai/tmp/pycoral-2.0.2-cp310-cp310-linux_x86_64.whl /data/data/coralai/tmp/tflite_runtime-2.17.0-cp310-cp310-linux_x86_64.whl 
Processing /data/data/coralai/tmp/pycoral-2.0.2-cp310-cp310-linux_x86_64.whl
Processing /data/data/coralai/tmp/tflite_runtime-2.17.0-cp310-cp310-linux_x86_64.whl
Requirement already satisfied: Pillow>=10.0.0 in /data/data/coralai/venv/lib/python3.10/site-packages (from pycoral==2.0.2) (10.4.0)
Requirement already satisfied: numpy>=1.24.0 in /data/data/coralai/venv/lib/python3.10/site-packages (from pycoral==2.0.2) (1.26.4)
Installing collected packages: tflite-runtime, pycoral
  Attempting uninstall: tflite-runtime
    Found existing installation: tflite-runtime 2.17.0.post1
    Uninstalling tflite-runtime-2.17.0.post1:
      Successfully uninstalled tflite-runtime-2.17.0.post1
Successfully installed pycoral-2.0.2 tflite-runtime-2.17.0

(venv) $ python3 examples/classify_image.py --model test_data/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite --labels test_data/inat_bird_labels.txt --input test_data/parrot.jpg 
Traceback (most recent call last):
  File "/data/data/coralai/feranick-pycoral/examples/classify_image.py", line 40, in <module>
    from pycoral.utils.edgetpu import make_interpreter
  File "/data/data/coralai/venv/lib/python3.10/site-packages/pycoral/utils/edgetpu.py", line 24, in <module>
    from pycoral.pybind._pywrap_coral import GetRuntimeVersion as get_runtime_version
ImportError: Python version mismatch: module was compiled for Python 3.11, but the interpreter version is incompatible: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0].

[feranick]

@hinxx Sorry for the trouble. I reuploaded the correct pycoral wheels, should work now.

[feranick]

I also pushed a few commits to allow correct matching of the Python version and the distribution.

@Namburger: Is there a chance we can merge libedgetpu, so we can merge pycoral as well?

[hinxx]

@feranick no trouble at all. Thank you for these packages!!

[larrylyle]

Yes, you do need the full XCode. Follow through the error message.

Then you can follow the instruction on how to build what you need:

https://github.com/feranick/TFlite-builds/blob/main/README.md https://github.com/feranick/libedgetpu/blob/master/README.md https://github.com/feranick/pycoral/blob/master/README.md

@feranick new to Raspberry Pi. I am trying to get my Coral Edge TPU up and running on my Raspberry Pi 5 w/ 8Gb so that I can use AI with Frigate. I have tried all the posts and I end up with this error:

RuntimeError: Error in device opening (/dev/apex_0)!

Will the above mention code correct this problem?

[feranick]

You need also this library:

https://github.com/feranick/gasket-driver/releases/tag/1.0-18.2

[larrylyle]

@feranick I know this may sound like a newby question, but how do I get bazel installed on my raspberry pi 5?

[feranick]

Assuming you are running Debian or Ubuntu 64 bit, in principle you can find it here:

https://github.com/bazelbuild/bazel/releases?q=6.5&expanded=true

However the arm64 version doesn't have an installer, which makes things more complex. As noted on the README.md in my repo (which people seems to always overlook), the easiest way is to build arm64 builds using docker on x86_64, which is what I do.

[larrylyle]

@feranick I'm so lost....it sucks being a newbee. I am so use to Windows and pre-built packages. Is there a possibility I could grease your pockets to complete for me?

[feranick]

Sorry, I don't use and have no interest in supporting Windows packages.

[Davros666]

I strongly suspect Goggle is criminally abandonning these local TPU devices, even though they used to sell them in order to drive 'customers' toward their online banks of those TPUs as a paid service.
The last thing a greedy corporation wants is puny peasants using their technology at home for free when they could instead be forced to purchase time for shekels-per-TFLOP-per-second on one of their Goggle servers.

[Techigami]

@feranick I'm trying to do a native compilation using bazel 6.5.0 to make pybind and wheel on my Raspberry Pi 5, but it always errors out:

"ERROR: home/XXXX/.cache/bazel/_bazel_XXXX/XXXX/external/org_tensorflow/tensorflow/lite/core/c/BUILD:321:38: Compiling tensorflow/lite/core/c/common.cc failed: (Exit 1): x86_64-linux-gnu-gcc failed: error executing command (from target @org_tensorflow//tensorflow/lite/core/c:common)"

"ERROR: /home/XXXX/pycoral/src/BUILD:67:17 Linking src/_pywrap_coral.so failed: (Exit 1): x86_64-linux-gnu-gcc failed: error executing command (from target @org_tensorflow//tensorflow/lite/core/c:common)"

I'm not sure why it's trying to use x86_64-linux-gnu-gcc on an ARM64 platform. Is there anyway I can fix this? Thanks.

[feranick]

Native build on arm is not recommended (docker builds are). Why not using the binaries I provide?

[Techigami]

@feranick The native compilation instructions are on your github page. I gave that up and tried the docker route which is still spitting error codes like 127 even though the bazel file with full perms is in the directory it's looking at. What am I doing wrong?

[feranick]

@Techigami To be clear, the native compilation is straight from the google repository. As far as I know it is was designed for the x86-64 platform. While I never dug deep into making it work natively in arm64, I can't say what may be the issue.

As for docker, for the same reasons, it is designed to be done on x86-64 to compile for arm7 and arm64.

More than that I can't say, as I use uniquely the docker route on x86_64 to compile for arm64. I'd like to remind people that I don't maintain the official repo, and only aim at providing binaries that may be of help, but doing it in the way that I find it easy and possible for me (which is the result of way too many hours of debugging an outdated repo). The rest is coming straight from the official repo. I welcome contribution from others that might compensate from what I purposefully didn't touch and have little interest in doing (including native/docker compilation uniquely on arm).

(pycoral for arm7 is broken regardless).