Cyclic coding process

[SergejGrin]

Faced such a problem: when encoding video (MP4 1080HD - 290 MB) via a link from the network (a local file is similar), the encoding process goes to the end and starts again. Sometimes the process stops with an error (screenshot 1).

Avideo website and encoder hosted on a virtual machine on the server
CPU - 8 vCPUs
Memory - 8 GB
When encoding, the process uses up to 80% of the resources of the ENTIRE SERVER. Moreover, there is a strange display of information about the encoding process (see screenshot 2, 3).

I can not understand what the problem is, everything worked without problems before. I did not change the site configuration, only updated to version 12.2
I am also attaching a screenshot of the encoder log (screenshot 4).

Please advise what is wrong with my system.

[DanielnetoDotCom]

Are you using encoder or encoder network?

try to manually run this code and check the output

[SergejGrin]

Good evening, Daniel!
I use my encoder which is located on the same VM as the streamer. The encoder opens normally, starts processing the task - files in the /var/www/AVideo/upload/videos directory are created, but at the final stage, instead of the file transfer, the encoding process starts again. I noticed that if an error occurs at the stage of transition to the transfer (screenshot 1), then after pressing the button (screenshot 1-1)

the process starts again, but ends with success. As a result, 1-2 out of 10 attempts to encode video and add it to the streamer end in success. In other cases, after 2-3 coding attempts that go into "cyclicality", you have to delete the task and start over. An interesting and surprising situation, I am more and more inclined that the mistake is somewhere on my side. But where is it? Can not understand...

P.S. For the sake of interest, I tried to install AVideo on Ubuntu Server 22.04 with PHP 8.1.12 So far, not very successful. Question: is it worth switching to a new version of the system or continuing to work on Ubuntu Server 20.04?

[DanielnetoDotCom]

try to google for this error.

FYI, you are missing the php-gd lib

[SergejGrin]

You are amazing as always, Daniel!
I was sure that I installed all the modules for the PHP ... but no, this particular module was missing. Now on Ubuntu Server 22.04 everything is working. Thank you and God bless you. Now I'll try to upload and recode some video.

[SergejGrin]

Hello Daniel.
Still, I think the problem is with ffmpeg. I have two virtual machines with AVideo sites and encoders. Installed on one virtual machine:
ffmpeg version 4.2.7 (Ubuntu Server 20.04, PHP 7.4)
and on the second:
ffmpeg version 4.4.2 (Ubuntu Server 22.04, PHP 8.1)
Both sites have the same problem - video encoding cycles. At the same time, the situation is different for each video: 3-5 encoding cycles - an error - encoding restart and then, as a rule, a successful transfer of files to the streamer. Versions of AVideo sites updated to 12.2
And now I'm a bit confused - is the problem in the ffmpeg encoder or in the code of the site that triggers the encoding....?

[DanielnetoDotCom]

I am not sure about the FFmpeg, but you may be missing some FFmpeg library

that is why I asked you to run the command manually here

[SergejGrin]

I did not have time to run that code, since I deleted the log file and the task itself and started all over again.

[SergejGrin]

Hello Daniel!
I installed a clean system for the third time and "virtuously" (installed the streamer and encoder without a single error. There was not a single error, everything installed and started up great. Then I started encoding the video with which I had previously had problems. The cyclic encoding repeated itself. After four cycles encoder coding stopped with an error.

Video files were created in the upload folder, but the transfer to the video folder did not start.
I ran the command
ffmpeg -i "/var/www/AVideo/upload/videos/1_tmpFile.mp4"

The result is in the screenshot. I am attaching a link to the full encoder log file. The file is large and cannot be attached here.
https://wakesoft.info/index.php/s/fkNbp5Ep8iDHRSJ
What is the problem? Perhaps the video file has some strange format?
Here is a link to this file:
https://my.mail.ru//mail/bayan5804/video/85/14318.html
On previous installations, I also tried to encode this file from the link from YouTube, I tried to download the file from there and encode it. The result is exactly the same.

[DanielnetoDotCom]

If you want help please check your logs properly, and also execute the complete FFmpeg command manually

BTW I do not need a complete log. delete the log file, execute one file and share the log

[SergejGrin]

Thank you I'll try to do it tomorrow - it's already night.

[SergejGrin]

Hello Daniel!
I have to apologize for wasting your time.
Today I deleted the log file through the control panel, through
terminal deleted log files in streamer and encoder. I did nothing else - I wanted to get accurate fresh error records. Then I launched the encoder with a link from my.mail.ru, youtube.com and one local file, which yesterday gave an encoding cycle error. Today, all tasks went without errors and ended with complete success. Miracles! Thank you for your help.

[DanielnetoDotCom]

I see, maybe was a permission issue (maybe)

[SergejGrin]

I don't understand anything yet... I changed absolutely nothing - just deleted the log files and cleared the cache. I will look further - it became very interesting. If I find something I'll let you know

[SergejGrin]

Hello Daniel!
I'm at a loss... Yesterday I tried to encode test local files:

1. MPEG-4 AAC, H.264 - 640  x 480, 1.25 GB -- Success
2. MPEG-4 AAC, H.264 - 1280 x 720, 150.7 MB -- Success
3. MPEG-4 AAC, H.264 - 1920 x 1080, 464.3 MB -- Success
   Today I'm trying to encode another file:
   MPEG-4 AAC, H.264 - 1280 x 720, 1.22 GB - error and transition to cyclic encoding.

What could be wrong? What to pay attention to?
I am attaching the log file.
avideo.log_20221202-130341.zip

[SergejGrin]

I noticed the following pattern: if the encoding process on the server uses more than 40-50% of the CPU, then encoding will follow a cyclic scenario. If the CPU is used by 30-40%, then the process will end in success.

[SergejGrin]

This time I decided not to interrupt the encoding process, which lasted over 5 hours and was repeated 6 times. That is, the process reaches 100%, file verification begins, which is signaled by a green rapidly moving indicator, and ... encoding starts again from 0%. At the same time, the last sixth encoding process took place with a processor load of about 40% (which I already reported earlier) and at the end, the file transfer finally began.
The site has more than enough resources:
CPU - 6 vCPUs
Disk (SSD) - 600 GB
Memory - 8 GB
This begs the question: if the process eventually succeeded, why did it repeat itself so many times and take so long?
Maybe everything is as it should be and I'm raising a "storm in a glass" in vain?

[DanielnetoDotCom]

the only way to help you is if you share your encoder log

also due to the deadlock error, I suspect that your server does not have enough resources, or you are using a docker or a shared server that limits the resources access.

[SergejGrin]

The streamer and encoder are installed on one separate virtual machine with the parameters indicated above.
https://www.linuxbabe.com/ubuntu/install-avideo-youphptube-ubuntu-20-04-server

There are no restrictions, I can increase / decrease the amount of resources for a virtual machine - this is not a problem.
The log file is very large, posted on a cloud resource.
If you need to "cut" something out of it - this is also not a problem. I just don't know what is needed. The file was taken from the upload folder

https://wakesoft.info/index.php/s/ynoMRDHgz2KkqRa

[DanielnetoDotCom]

use this to install https://github.com/WWBN/AVideo-Encoder/wiki/How-to-install-LAMP,--FFMPEG-and-Git-on-a-fresh-Ubuntu-20.x---For-AVideo-Encoder

also increase the resources to at least 4 CPU and 8 GB ram each, double it if you can.

[SergejGrin]

Is it already worth it...? See my post above

Ресурсов на сайте более чем достаточно: CPU — 6 vCPU Диск (SSD) — 600 ГБ Память — 8 ГБ

[DanielnetoDotCom]

The description looks good , but seems to me you do not have all those resources available all the time or maybe your environment is not good.

I do not know em what you use to virtualize your servers but looks like it is not working good

FYI I do not use virtualized server for my self , I only use dedicated servers

[SergejGrin]

I have my own server ProLiant DL160 G6 12 CPUs x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz running VMWARE ESXi. I have been using your wonderful product for a long time, similarly on a separate virtual machine - everything worked great until PHP was replaced with version 8 and the new release 12.2

[DanielnetoDotCom]

I see, I also use php 8.1 on my servers , make sure you do not use php fpm , install php as an Apache module

[SergejGrin]

[DanielnetoDotCom]

So I do not know why you have the deadlock, maybe it is nothing (maybe)

But the restart happens when the video is corrupted, for now I do not know why it is corrupted, try to use the public encoders to see if it will work https://network.avideo.com/

[SergejGrin]

ok i will try your encoder for the same file

[SergejGrin]

Yes, everything went quickly and without problems on your encoder

TEST FILE-3 Your encoder https://network.avideo.com/ TEST FILE-2 My encoder https://youview.wakesoft.info/upload/

The only difference is that I enabled 5 video formats on my encoder, and you have 4. It turns out that the problem is with the encoder.

I think that maybe then I should try to create a separate virtual machine for the encoder network? That is, to separate the streamer and the encoder ...

[DanielnetoDotCom]

Encoder network will not make any difference.

Honestly I do not know how to resolve your problem , maybe is your hardware, maybe is your Apache/php installation

[SergejGrin]

I myself am at a loss. I installed everything carefully and not for the first time ...
However, I will try to install my own network of encoders. I want to make a completely independent platform. Times are like this...
Thank you Daniel for your help and support