instability problem #62
Comments
|
Hi Markus, the first error message comes from That leaded to a RuntimeError in a thread started and hosted by the Which version of the Do you know a way to reproduce the error without waiting for hours? Thank you, |
|
Hi Ronald, python3-picamera2/stable,stable,now 0.3.17-1 all [installiert] I could uninstall it and use the libcamera build provided by Aaron with indi-allsky. it happened this morning again, unfortunately I do not know how to force it. The Zero 2 does nothing else and sits happily on my desk with the connected PiHQ camera. ``2024-02-22T12:52:31.739573+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:31: Driver indi_pylibcamera: indi_pylibcamera.indidevice-INFO- sending BLOB CS, Markus |
|
I can reproduce the error with my Pi Zero WH, running 32bit Bullseye and picamera2 0.3.12-2. |
|
The issue is difficult to debug. A few days ago I could reproduce the error in less than an hour. Than I added some more log messages to support debugging and now the driver runs without error since nearly a day. I wish it would fail more often. |
|
yeah, I restarted/power cycled my test setup last Saturday and no hickup since then. Strange, the week before it stopped every few hours. CS, Markus ok, about an hour later after my comment above, the next failure happened... |
|
I had only one error after adding more debug outputs. Maybe I am mislead by an accident, but the error happened after closing the camera. To allow an immediate aborting of a running exposure I used a stop-command which bypasses the camera event loop. Maybe that leaded to the error in the camera queue. I modified the code now to use the recommended stop-command during normal exposures. The risky immediate stop command is now used only when you abort a running exposure. It is still needed to use the risky command because it is the only way I know to abort a long term exposure without waiting until its end. I released v2.6.0 which hopefully fixed the issue. Would be very nice when you give me feedback if it works now reliable. Regards, |
|
Hi Ronald, sorry for the late answer. I restarted my trials a few weeks ago with a new RPi5 (8GB) intended for another project. With this system, I was able to use indi_pycamera for now more than 10days without a problem. Encouraged by this success, I created a new image (Raspbian Lite, bookworm) for the Zero 2 W and configured it exactly the same using the build scripts in the misc folder of indi-allsky for indi and libcamera. I can use libcamera-hello or qcam without a problem. But with indi_pylibcamera, the system now stops working after one or two pictures. Symptoms are that the indiserver process runs constantly at 100% and is almost unstoppable. Memory usage is still below 400MB and only about 100MB swap space is used. I'm a bit lib camera now. Is it really necessary to use a RPi4/5 for such a trivial task. IfI can try further stuff or send logs, let me know. CS, Markus |
|
Hi Markus, it would be a shame if it does not work on a Pi Zero! I will try to replicate your setup. On one of my telescopes I have a Pi Zero W with a HQ camera. It is not a Zero 2 W but this hopefully does not make a difference. I don't want to damage my installation and ordered a SD card (should arrive beginning of next week). When it arrives I will setup a fresh Bookworm and install indi-allsky. Do I understand right, you did Regards, |
|
Hi Ronald, CS, Markus |
|
I have seen on the indi-allsky page that the requirement for Pi Zero 2 (and Pi Zero 2) is Bullseye and not Bookworm. As far as I understand you used Bookworm. I will test Bullseye 32bit. Hopefully I tell you results tomorrow. |
|
Actually, I have tried all variants, 32bit Bullseye and 32/64bit bookworm. Makes no difference. Currently I have installed Ubuntu 24.4 64bit and it currently slowly compiles the libcamera stuff. Did not work with the system provided packages. CS, Markus |
|
indi_pylibcamera does not work with ubuntu 24.4 and self build libcamera or the system packages. rpicam-hello and qcam do work. Traceback (most recent call last): CS, Markus |
|
I newer tested Ubuntu. But the error message looks like the Hopefully this works also in Ubuntu. I try to setup indi-allsky on my Pi Zero W. It is compiling the indi library now since 8 hours. Maybe it will finish over night. |
|
the package is available and installed but it does not resolve the error. That was the reason I built libcamera using the build script inside the venv. I'm out of ideas. Ubuntu was an experiment. Last hope would be going back to bullseye if you are successful. CS, Markus |
|
ok, went back to Rasbian bullseye 32bit. Fresh install, full-upgrade, installed only indi-bin from the standard repo and python3 venv. Increased swap size to 1000MB and started the indiserver. It is running well since about a day now. Let's hope for more stability. Will keep you posted. CS, Markus |
|
I have now indi-allsky installed on my Pi Zero W. Afte reboot it runs automatically the indiserver: I installed indi-pylibcamera system wide. Do you know how I can make indi-allsky to start the new camera driver? I do not find a setting in the GUI. I tried this: |
|
you don't need a camera name at all if there is just one camera. It automatically takes the first camera it finds. It then shows the camera as indi_pylibcamera. I once changed the name to pylibcamera_Main in indi_pylibcamera.ini and it recognised that. CS, Markus |
|
How does indi-allsky know, which camera I have? The INDI library installed many camera driver. I do not believe that indi-allsky probed all these drivers for a connected camera. It just uses |
|
I also run |
|
argh, I misunderstood your question. Sorry my bad. You have to replace the indiserver setup. Either by rerunning setup.sh or by executing misc/change_camera.sh. After the selection it presents you a list of available camera drivers installed on the local system. CS, Markus |
|
Ha, that made the trick! Indiserver uses now indi_pylibcamera. By Pi Zero W immediately crashed, likely because of no memory. I will increase swap space. Using binning 2 will also help. I come back to you when I have more results. |
|
yep, you will need more swap as written before. I guess this is the major stability issue. If the software environment would be less wasteful of the resources, we wouldn't have a problem. CS, Markus |
|
indi-allsky is doing many memory consuming tasks on the Pi Zero. Even calculating a RGB image from the raw camera frame is pretty CPU and memory hungry. And allsky is doing much more with the images. I am surprised that it runs on my Pi Zero. But it is slow. Since about half an hour I get images from my HQ camera. I also set binning 2 in the allsky configuration. The images look underexposed but likely this can be fixed in the settings. |
|
Ronald, I never intended to run indi-allsky on the Pi Zero (2). The Pi Zero only delivers the pictures via your driver and indiserver to the indi-allsky instance running on my big server. Because it should/will be an unattended 24/7/365 operation, it must run very stable. I think Aaron did a lot to achieve this on the indi-allsky server side. Unfortunately, the current record with indi_pylibcamera on a Pi Zero 2 is 4 days and usually just one day (sitting on my office desk :-). I'm very thankful that you spent your time working on this issue and my special use case. CS, Markus |
|
Hi Markus, you are welcome. I am happy that other people also use the indi_pylibcamera. I realized that I misunderstood your setup. Am I right that your setup is like this:
I will build the same setup here. Do you think indi_allsky will run on a Pi 3B with 1GB RAM? I have 2 of them permanently running for other tasks and can install indi_allsky on one of them for test. Than I can run a camera for days Regards, |
|
yes, this is my setup. A Pi 3B with 1GB should be good enough. You don't need to generate time lapse videos, panoramas or store the data for a long time. CS, Markus |
|
3 days and still running without a hickup. I'm getting optimistic about this setup. CS, Markus |
|
I am happy to read this. Thank you for the update. |
|
I have indi-allsky now running on a Pi3 getting images from an indiserver/indi_pylibcamera running on a Pi Zero W. To make the load on the Pi Zero high I did not set the binning of the HQ camera to 2. I will let it run the next days. From next Saturday I will be in vacation for 2 weeks. I will not have access to my computer and I will not run the test unattended. After my vacation I can start the test again. |
|
7 days and still running without a glitch. Should be ready for the roof now. Ronald THANK YOU!!!! CS, Markus |
|
You did it without my help. Do you think the increasing of the swap space was the solution? I should make a note in the README. |
|
yes, I think about 500MB swap is needed instead of the default 100M. But you may have possibilities to reduce the memory footprint of the driver. CS, Markus |
I'm running indi_pylibcamera on a RPi Zero 2 W with Raspbian bookworm 64bit and a RPI HQ camera. Intended use is as an Allsky camera with indi-allsky. For various reasons, indi-allsky runs on a different server, the Zero 2 W only runs indiserver (v2.0.6) with the indi_pylibcamera driver. The driver was installed using pip but without a venv. Since about a week, this setup runs on my desk for test purposes. Unfortunately, the driver frequently crashes. Usually I get a run time of a few hours, the current record was about 24h. The error condition in the syslog looks always like this:
It can be resolved by restarting the indiserver
systemctl --user restart indiserver.serviceand it will happily run again for a few hours.
I know that this is difficult to diagnose and solve. Short term solution might be an automatic restart of the driver itself if he somehow diagnoses the error condition. Maybe I have to set an option in the ini file. I havn't created an ini file yet, lacking an example.
If I can be of any further help, I'm willing to provide any info, reinstall or whatever needs to be done :-)
CS, Markus
The text was updated successfully, but these errors were encountered: