scikit-learn fails to build while installing zm event server

[hugalafutro]

Hi,
I'm using dlandon/zoneminder:latest with hooks enabled and everything was well for few weeks, but today I had to recreate the container and the zm event server install fails when installing some dependencies. This in turn renders the hooks unusable as testing the object detection now fails with

Traceback (most recent call last):
  File "/var/lib/zmeventnotification/bin/zm_detect.py", line 27, in <module>
    import zmes_hook_helpers.utils as utils
ModuleNotFoundError: No module named 'zmes_hook_helpers'

full log: https://pastebin.com/raw/8PDK4k8y
relevant part with errors:

zoneminder    |   Running setup.py bdist_wheel for scikit-learn: finished with status 'error'
zoneminder    |   Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-3jpm8ech/scikit-learn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpv4irtf__pip-wheel- --python-tag cp36:
zoneminder    |   Partial import of sklearn during the build process.
zoneminder    |   Traceback (most recent call last):
zoneminder    |     File "/tmp/pip-build-3jpm8ech/scikit-learn/setup.py", line 201, in check_package_status
zoneminder    |       module = importlib.import_module(package)
zoneminder    |     File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
zoneminder    |       return _bootstrap._gcd_import(name[level:], package, level)
zoneminder    |     File "<frozen importlib._bootstrap>", line 994, in _gcd_import
zoneminder    |     File "<frozen importlib._bootstrap>", line 971, in _find_and_load
zoneminder    |     File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
zoneminder    |   ModuleNotFoundError: No module named 'numpy'
zoneminder    |   Traceback (most recent call last):
zoneminder    |     File "<string>", line 1, in <module>
zoneminder    |     File "/tmp/pip-build-3jpm8ech/scikit-learn/setup.py", line 306, in <module>
zoneminder    |       setup_package()
zoneminder    |     File "/tmp/pip-build-3jpm8ech/scikit-learn/setup.py", line 292, in setup_package
zoneminder    |       check_package_status('numpy', min_deps.NUMPY_MIN_VERSION)
zoneminder    |     File "/tmp/pip-build-3jpm8ech/scikit-learn/setup.py", line 227, in check_package_status
zoneminder    |       .format(package, req_str, instructions))
zoneminder    |   ImportError: numpy is not installed.
zoneminder    |   scikit-learn requires numpy >= 1.13.3.
zoneminder    |   Installation instructions are available on the scikit-learn website: http://scikit-learn.org/stable/install.html
zoneminder    |
zoneminder    |
zoneminder    |   ----------------------------------------
zoneminder    |   Running setup.py clean for scikit-learn
zoneminder    |   Failed building wheel for scikit-learn
zoneminder    |   Running setup.py bdist_wheel for psutil: started
zoneminder    |   Running setup.py bdist_wheel for psutil: finished with status 'done'
zoneminder    |   Stored in directory: /root/.cache/pip/wheels/65/65/52/8543bfd86249ad90c3f5a3bac9e15332f2251f8967fd78fdcd
zoneminder    | Successfully built imageio-ffmpeg imutils pygifsicle pyzm psutil
zoneminder    | Failed to build scikit-learn
zoneminder    | Installing collected packages: Pillow, Shapely, numpy, imageio, imageio-ffmpeg, imutils, pygifsicle, SQLAlchemy, pytz, regex, tzlocal, python-dateutil, dateparser, protobuf, mysql-connector-python, portalocker, python-utils, progressbar2, psutil, certifi, chardet, urllib3, requests, websocket-client, pyzm, joblib, scipy, threadpoolctl, scikit-learn, zmes-hook-helpers
zoneminder    |   Running setup.py install for scikit-learn: started
zoneminder    |     Running setup.py install for scikit-learn: finished with status 'error'
zoneminder    |     Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-3jpm8ech/scikit-learn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-umtmd8km-record/install-record.txt --single-version-externally-managed --compile:
zoneminder    |     Partial import of sklearn during the build process.
zoneminder    |     Traceback (most recent call last):
zoneminder    |       File "/tmp/pip-build-3jpm8ech/scikit-learn/sklearn/_build_utils/__init__.py", line 27, in _check_cython_version
zoneminder    |         import Cython
zoneminder    |     ModuleNotFoundError: No module named 'Cython'
zoneminder    |
zoneminder    |     The above exception was the direct cause of the following exception:
zoneminder    |
zoneminder    |     Traceback (most recent call last):
zoneminder    |       File "<string>", line 1, in <module>
zoneminder    |       File "/tmp/pip-build-3jpm8ech/scikit-learn/setup.py", line 306, in <module>
zoneminder    |         setup_package()
zoneminder    |       File "/tmp/pip-build-3jpm8ech/scikit-learn/setup.py", line 302, in setup_package
zoneminder    |         setup(**metadata)
zoneminder    |       File "/usr/local/lib/python3.6/dist-packages/numpy/distutils/core.py", line 135, in setup
zoneminder    |         config = configuration()
zoneminder    |       File "/tmp/pip-build-3jpm8ech/scikit-learn/setup.py", line 186, in configuration
zoneminder    |         _check_cython_version()
zoneminder    |       File "/tmp/pip-build-3jpm8ech/scikit-learn/sklearn/_build_utils/__init__.py", line 30, in _check_cython_version
zoneminder    |         raise ModuleNotFoundError(message) from e
zoneminder    |     ModuleNotFoundError: Please install Cython with a version >= 0.28.5 in order to build a scikit-learn from source.
zoneminder    |
zoneminder    |     ----------------------------------------
zoneminder    | Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-3jpm8ech/scikit-learn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-umtmd8km-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-3jpm8ech/scikit-learn/

[joselito11]

I recreated container to add and test integrated gpu. Object detection do not work after that. Tried to recreate few times, but no luck. Any easy fix?

Manual run:

root@098a7d1a724b:/config/hook# ./zm_event_start.sh
Traceback (most recent call last):
  File "/var/lib/zmeventnotification/bin/zm_detect.py", line 27, in <module>
    import zmes_hook_helpers.utils as utils
ModuleNotFoundError: No module named 'zmes_hook_helpers'

root@098a7d1a724b:/config/hook# ./zm_detect.py 
Traceback (most recent call last):
  File "./zm_detect.py", line 27, in <module>
    import zmes_hook_helpers.utils as utils
ModuleNotFoundError: No module named 'zmes_hook_helpers'
root@098a7d1a724b:/config/hook#```

[joselito11]

root@098a7d1a724b:~# tail -F  /var/log/zm/zmeventnotification.log
12/25/20 09:29:41.674235 zmeventnotification[1084].INF [main:968] [PARENT: New event 184781 reported for Monitor:3 (Name:Terasa) External Motion[last processed eid:]]
12/25/20 09:29:44.025680 zmeventnotification[3450].INF [main:968] [|----> FORK:Terasa (3), eid:184781 Not sending over FCM as notify filters are on_success:all and on_fail:all]
12/25/20 09:29:44.031856 zmeventnotification[3450].INF [main:968] [|----> FORK:Terasa (3), eid:184781 Not sending over FCM as notify filters are on_success:all and on_fail:all]
12/25/20 09:29:44.035158 zmeventnotification[3450].INF [main:968] [|----> FORK:Terasa (3), eid:184781 Not sending over FCM as notify filters are on_success:all and on_fail:all]
12/25/20 09:29:44.038534 zmeventnotification[3450].INF [main:968] [|----> FORK:Terasa (3), eid:184781 Not sending over FCM as notify filters are on_success:all and on_fail:all]
12/25/20 09:29:44.041681 zmeventnotification[3450].INF [main:968] [|----> FORK:Terasa (3), eid:184781 Not sending over FCM as notify filters are on_success:all and on_fail:all]
12/25/20 09:29:44.044810 zmeventnotification[3450].INF [main:968] [|----> FORK:Terasa (3), eid:184781 Not sending over FCM as notify filters are on_success:all and on_fail:all]
12/25/20 09:29:46.676863 zmeventnotification[1084].INF [main:968] [PARENT: Re-loading monitors]
12/25/20 09:30:12.049500 zmeventnotification[3450].INF [main:968] [|----> FORK:Terasa (3), eid:184781 Event 184781 for Monitor 3 has finished]
12/25/20 09:30:16.059089 zmeventnotification[3450].INF [main:968] [|----> FORK:Terasa (3), eid:184781 Not sending event end alarm, as we did not send a start alarm for this, or start hook processing failed]
12/25/20 09:33:21.674205 zmeventnotification[1084].INF [main:968] [PARENT: New event 184782 reported for Monitor:5 (Name:Cesta) External Motion[last processed eid:184779]]
12/25/20 09:33:24.045112 zmeventnotification[3765].INF [main:968] [|----> FORK:Cesta (5), eid:184782 Not sending over FCM as notify filters are on_success:all and on_fail:all]
12/25/20 09:33:24.051209 zmeventnotification[3765].INF [main:968] [|----> FORK:Cesta (5), eid:184782 Not sending over FCM as notify filters are on_success:all and on_fail:all]
12/25/20 09:33:24.053708 zmeventnotification[3765].INF [main:968] [|----> FORK:Cesta (5), eid:184782 Not sending over FCM as notify filters are on_success:all and on_fail:all]
12/25/20 09:33:24.057018 zmeventnotification[3765].INF [main:968] [|----> FORK:Cesta (5), eid:184782 Not sending over FCM as notify filters are on_success:all and on_fail:all]
12/25/20 09:33:24.058173 zmeventnotification[3765].INF [main:968] [|----> FORK:Cesta (5), eid:184782 Not sending over FCM as notify filters are on_success:all and on_fail:all]
12/25/20 09:33:24.061226 zmeventnotification[3765].INF [main:968] [|----> FORK:Cesta (5), eid:184782 Not sending over FCM as notify filters are on_success:all and on_fail:all]
12/25/20 09:33:52.065682 zmeventnotification[3765].INF [main:968] [|----> FORK:Cesta (5), eid:184782 Event 184782 for Monitor 5 has finished]
12/25/20 09:33:56.073495 zmeventnotification[3765].INF [main:968] [|----> FORK:Cesta (5), eid:184782 Not sending event end alarm, as we did not send a start alarm for this, or start hook processing failed]

[joselito11]

It is missing folder: zmes_hook_helpers https://github.com/dlandon/zoneminder/tree/master/zmeventnotification/zmeventnotification/zmes_hook_helpers
copied the folder in doker container, restarted and it is wokring again

[hugalafutro]

@joselito11 Hi, could you please post more detailed steps how you fixed it? I'm really bad with docker I just like the portability of it. Many thanks!

[joselito11]

@hugalafutro Download all files, unzip and copy only the folder /zmes_hook_helpers in /mnt/Zoneminder/hook on host. I have mounted /config to /mnt/Zoneminder on host. Restart

[hugalafutro]

Thanks I got it 👍

Though the sci-kit error remains on rebuild of the container, but the object detection works after copying the directory in hook

[jmccoy555]

Hi, not only me then!

Things worked last week, when I first tried Zoneminder, then I had a play around and broke it. So started fresh and then this all began.

As well as copying the hook_helpers folder I ran

pip3 install Cython==0.29
pip3 install sklearn 

to fix the sklearn issue.

Also the same on the master version - dlandon/zoneminder.master:latest

Update: 01/01/21 - All now fixed on the master version, but you'll have database problems if you try to revert back!

[dlandon]

I cannot reproduce this issue. Did you set the 'INSTALL_HOOK' environment variable to '1' to install hook processing?

[jmccoy555]

I did yes. Repeted a number of times too, strange.

Did you fix the Cayton issue. I'd guess that's the real cause of the issue?

[dlandon]

What platform are you running on?

[jmccoy555]

What platform are you running on?

Debian 10 VM, but that shouldn't matter should it with Docker?

[dlandon]

Not really, but I am finding there are some differences between platforms.

When you start a fresh docker install with install hook processing enabled, does the log indicate any issues with the installation?

[jmccoy555]

That is odd.

Yes, during the build it moans about numpty (as mentioned on another issue), cython not being the right version for sklearn. I don't think the default is 0.29 as I think I got 0.28 without specifying.

[dlandon]

Take this issue to the Zoneminder forums. I think it is more of an ES issue, than with the Docker.

[jmccoy555]

I'm curious now.... If you have a straight build up, what version of Cython do you get? If not I'll check tomorrow.

Does your docker build / scripts install Cython?

[dlandon]

The hook installation is completely under the control of ES.

[jmccoy555]

Ah that makes sense then and probably explains why your docker that hadn't changed worked one week and not the next.

[jmccoy555]

Various updates to install.sh 12days ago which would fit with my timesces of this issue appearing.

pliablepixels/zmeventnotification@30dc285

Any chance you have something cached if you're not having this problem?

[dlandon]

No. There is a new version of ES coming and I suspect it will fix your issue. I'll update the docker once the ES version is stable.

[jmccoy555]

Ah yes. Your docker is quite complex to get your head around 🤔. Strange that something has change if your pinned to a version then. Ok thanks. I did try (well am actually running) your master docker with the same issue. Hopefully it will all sort itself out. Maybe I'll try and rebuild if I get a min. Thanks again.

If you do want me to test someone then just ping iit over.

[e10kstarfire]

I'm getting the same scikit and hook helpers issues on fresh new installs of this docker via unRAID with INSTALL_HOOK 1.

I did email pliablepixels (ES author) but he doesn't use Docker but don't believe he knew of this issue.

[dlandon]

There is a new version of ES coming soon that should solve this issue.

[jmccoy555]

@dlandon so should this ZoneMinder/zoneminder#3063 (comment) solve it, or is ES 6.1.0 not the new version?

[dlandon]

ES 6.1.0 is the new version but is not released as stable yet. I have updated to ES 6.1.0 pre-release in the master docker. Once it is released as stable, I will update this docker.

[jmccoy555]

Ok, getting lost in the versions.... Building now.

[jmccoy555]

@dlandon Yep, the latest master appears to solve all these issues. Strangely though there is still no zmes_hook_helpers folder, but the error has gone and object detection is working.

@e10kstarfire I guess this should get you up and running. Just change your docker image to dlandon/zoneminder.master obviously some caution as this is not pinned to a version and is for testing. Or get into the docker and do the fixes manually as per this thread.

[dlandon]

@jmccoy555 If all is working I will update this docker once ES 6.1.0 goes stable. If you move to the master docker, you will have a problem reverting back to this docker because the database will not be at the correct version and I don't believe it will downgrade.

[hugalafutro]

@jmccoy555 If all is working I will update this docker once ES 6.1.0 goes stable. If you move to the master docker, you will have a problem reverting back to this docker because the database will not be at the correct version and I don't believe it will downgrade.

I can confirm it will not downgrade, I went through this and had to restore backup. Best wait till the new ES makes it into stable imho.

[jmccoy555]

@dlandon Ah,I guess I already have the database problem then!! Not really done much setting up apart from adding 4 cameras so may wait or just drop back and do the manual fixes for now.

Do we have any idea when it will go stable?

Can I not just stick with master until 1.35 is released then jump back?

[dlandon]

@jmccoy555 Yes.

[dlandon]

There is a new docker available that should fix this. Look at the updated docker run for an adjustment in /dev/shm.