Videos randomly disappearing #1453

rem0g · 2024-12-23T09:09:25Z

Some glosses has disappeared videos for example:

https://signbank.cls.ru.nl/dictionary/gloss/47361

https://signbank.cls.ru.nl/dictionary/gloss/47475

Some glosses has wrong still image from the video.

Also some glosses has wrong video perception.

Some glosses even has NMM video as gloss video.

This is happening everywhere, I have checked my scripts and everything looks fine at my end.

rem0g · 2024-12-23T10:39:58Z

Another example:

https://signbank.cls.ru.nl//dictionary/gloss/47336

When glos video has been deleted, then NME video has moved from NME to glos.

rem0g · 2024-12-23T11:29:49Z

At Glos AFBROKKELEN (https://signbank.cls.ru.nl//dictionary/gloss/46883), I am seeing this:

  {
                "ID": "211108",
                "Index": "0",
                "Description: Dutch": "afbrokkelen",
                "Description: English": "crumble",
                "Link": "https://signbank.cls.ru.nl//dictionary/protected_media/glossvideo/NGT/AF/AFBROKKELEN-46883.mp4"
            },
            {
                "ID": "208048",
                "Index": "1",
                "Description: Dutch": "afbrokkelen",
                "Description: English": "crumble",
                "Link": "https://signbank.cls.ru.nl//dictionary/protected_media/glossvideo/NGT/AF/AFBROKKELEN-46883.mp4"
            }
            
            
            Both has same link and that should not be the case.

susanodd · 2024-12-23T15:03:41Z

Can you increase the amount if time between updating/uploading and retrieving?

Like maybe increase it to 10 minutes? There was previously a problem that the time between operations was too frequent.
With videos in could be that a previous operation was not completed before a new one is performed. (It makes a difference if the file system is continuously writing new files. The transactions to create objects are faster than the file system.

susanodd · 2024-12-23T15:07:30Z

I had this problem locally on my own computer using iCloud for storage.
But I could not repeat it on the real server.
It was that the files had not been completely copied to iCloud. So they "disappeared" like that. They were actually on disc, but had names with "." before the filename, so not visible. This was on Apple's Unix.

rem0g · 2024-12-23T15:08:16Z

Yes I could do that but i would rather for Signbank side do something about the transactions, for example for incoming transactions I always enqueue them in a list instead of executing them immediately. When the server has finished processing a certain transaction then it could handle the next one.

Would that be something you can work on?

susanodd · 2024-12-23T16:53:19Z

@rem0g that sounds like an interesting approach. I will discuss that with @vanlummelhuizen how to implement that. He is the Django expert. A queuing mechanism.

susanodd · 2025-01-06T09:50:53Z

Another example:

https://signbank.cls.ru.nl//dictionary/gloss/47336

When glos video has been deleted, then NME video has moved from NME to glos.

@rem0g How are you deleting the video?

(There are some "signals" when objects are deleted. These may or may not move or delete video files. It would help in debugging to know what commands have been done.)
(When a gloss is deleted, the video files are not deleted.)

Theoretically, if you are uploading video files at rapid speed, the temporary files (that Unix is making) could end up being linked to the wrong object in Django. I suspect this for a long time, but cannot fix this myself. I will ask the others.)

I implemented a lot of code in November/December for managing the video files. There are pull requests for these. But nobody has reviewed them yet. The intention is that the dataset manager can inspect what is in the file system. That also allows to retrieve the videos from deleted glosses. The gloss IDs are not reused, so new videos should not have any interference with deleted glosses, since they always have the ID in the filename and these are not reused.)

susanodd · 2025-01-06T10:16:42Z

@rem0g for this gloss:

https://signbank.cls.ru.nl/dictionary/gloss/47361

the NME video is not in the correct format!

(On Firefox, it shows that it is not supported.)

Recall that you asked us to not test for MP4 anymore. Thus it can be that incorrect formats are causing problems.
(The images cannot be generated for videos in the wrong format. Hence, an old image will remain shown.)

susanodd · 2025-01-06T10:28:21Z

@rem0g here are the gloss video objects for AFBROKKELEN, yes, you can see that the same filename appears multiple times, for different GlossVideo objects with various perspectives and NME set.

susanodd · 2025-01-06T10:30:03Z

Is it possible something was wrong with the permissions on your source file? Or that it was a symbolic link?
It looks like something was wrong with the source file that it kept being bound to different objects.
Also, all of the files attached to the objects have very similar time stamps.

susanodd · 2025-01-06T10:30:57Z

@vanlummelhuizen can you help on this issue?

susanodd · 2025-01-06T11:50:55Z

@rem0g there are hundreds of videos with the wrong filename as you point out.

Did you change anything in your script?

I copied the newest database to signbank-test in order to inspect the filenames (in the objects).

https://signbank-test.cls.ru.nl/datasets/checks/5

(There are no files, but you can see the filenames in the objects)

The last time I checked filenames (end of November) everything was as expected.
It may have started when you can upload three videos at the same time in the API?

The problem is that all the gloss video objects of a gloss are indeed sharing a single video file. They are all pointing to the same file.

I can only think that this is being caused by an alias or something. That the file system is pointing to a single file during upload.

Babbling, but I know Django does not allow to upload multiple files in the Django Form Template. (We used to do this for the eaf files in the Dataset Manager, but when we updated to Django 4.2, the code had to be modified to only upload one file.)

Perhaps Django is somehow doing something here since multiple video files are in the same API request. (The Django feature was removed for security reasons from Django.)

susanodd · 2025-01-06T12:07:42Z

From Django manual

Together MemoryFileUploadHandler and TemporaryFileUploadHandler provide Django’s default file upload behavior of reading small files into memory and large ones onto disk.

You can write custom handlers that customize how Django handles files. You could, for example, use custom handlers to enforce user-level quotas, compress data on the fly, render progress bars, and even send data to another storage location directly without storing it locally. See Writing custom upload handlers for details on how you can customize or completely replace upload behavior.
Where uploaded data is stored¶

Before you save uploaded files, the data needs to be stored somewhere.

By default, if an uploaded file is smaller than 2.5 megabytes, Django will hold the entire contents of the upload in memory. This means that saving the file involves only a read from memory and a write to disk and thus is very fast.

However, if an uploaded file is too large, Django will write the uploaded file to a temporary file stored in your system’s temporary directory. On a Unix-like platform this means you can expect Django to generate a file called something like /tmp/tmpzfp6I6.upload. If an upload is large enough, you can watch this file grow in size as Django streams the data onto disk.

These specifics – 2.5 megabytes; /tmp; etc. – are “reasonable defaults” which can be customized as described in the next section.

susanodd · 2025-01-06T13:31:12Z

I need help with this issue.

One of the glosses that is messed up is STRUISVOGEL

Here, you see all the objects refer to the same file (the perspective videos have the same file name as the normal video):

Here you see a stats for the file:

stat ../writable/glossvideo/NGT/ST/STRUISVOGEL-45874.mp4
  File: ../writable/glossvideo/NGT/ST/STRUISVOGEL-45874.mp4
  Size: 1105797   	Blocks: 2158       IO Block: 131072 regular file
Device: 20007dh/2097277d	Inode: 864571      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/  ubuntu)   Gid: ( 1001/wwwsignbank)
Access: 2025-01-06 08:10:59.506318499 +0000
Modify: 2024-12-01 05:01:46.404138548 +0000
Change: 2024-12-20 13:58:20.693990022 +0000
 Birth: 2024-12-01 05:01:46.386138096 +0000

But in the file system, the timestamp on the file shows that it was not changed on 20 december.

-rw-r--r-- 1 ubuntu wwwsignbank 1105797 Dec 1 05:01 STRUISVOGEL-45874.mp4

susanodd · 2025-01-06T13:37:17Z

I don't know how to solve this.

Hopefully, @vanlummelhuizen can have a go.

rem0g · 2025-01-06T14:02:33Z

Hello, thank you for looking into the issue. For me the issue is not clear yet but now I know it's not caused by my script as everything relies on Signbank API. For uploading videos i do use this API:

/dictionary/api_update_gloss/{glossid}/video

That's it.

As for NME video upload i do use:

/dictionary/api_create_gloss_nmevideo/{datasetid}/{glossid}/}

And for deleting NME video:

/dictionary/api_delete_gloss_nmevideo/{datasetid}/{glossid}/{videoid}/

For every time if i want to upload a certain NME video i do execute api_delete_glos_nmevideo first, but for that i obtain unique ID from the nme video and then delete that and then upload the new NME video.

uklomp · 2025-01-07T09:16:08Z

Hi all, I added 'blocking' to indicate extra extra priority. If there's something on our end we can do, please let us know.

susanodd · 2025-01-07T09:38:37Z

I'm not able to solve this myself. I am aware of this problem for a long time, but it was on a local server running on iCloud. So I chalked that up to Apple quirks.

There are quite a few messed up video objects/ files now.

I'm playing with this locally, so I can inspect the file system and the admin without messing up anything.

Since multiple video objects refer to exactly the same file, it is not possible to "fix" this, other than to delete objects that point to the wrong file. But to do this, we need to turn off the "normal" process of deleting, otherwise the "correct" file may be deleted.

I made a command (pull request) for renaming backup video files, since that has been messed up for a long time.
But this is something different, since objects refer to the same file.

susanodd · 2025-01-07T10:09:11Z

@rem0g can you stop deleting the videos on your side? Because objects are referring to the same file, this is causing a shared file to be deleted.

I still think this is due to the API commands happening too fast for the file system. But once a file ends up shared by different objects, that escalates the problem, domino effect.

susanodd · 2025-01-07T12:55:38Z

@rem0g I put the database (per yesterday) onto signbank-susan where I have a pull request branch running with extra commands for the admin. (For filtering on NME, Perspective, and wrong filename).

Out of curiosity, I looked at the gloss video history. Here you can see the names of the files that were uploaded. The source file and the (desired) target file.

It looks like your naming convention with L, M, R in the filenames, that you are also using M for videos that are NME videos.

susanodd · 2025-01-08T11:11:44Z

I have a copy of the database on signbank-susan. (Per Monday. No videos, but you can browse the video objects / filenames / paths) in the admin.

I am working on filters to enhance the admin in order to detect and fix video problems. #1398

I added filters for NME and Perspective, plus a filter for "wrong filename".

Now it's possible to query on those wrong filenames.

For NME videos with the wrong filename, there are 1366 results!!!!!

For Perspective videos with the wrong filename, there are 4688 results!!

@Woseseltops @vanlummelhuizen this is a huge problem.

Since multiple objects are referring to the same video file (see the examples above), it is not possible to simply erase or delete anything.

I can make an "unlink file from object" command in order to uncouple the link. Then it would be possible to delete NME and Perspective objects that are pointing to the normal video file. (The files need to be either deleted or unlinked in order to delete the objects. But this runs the risk of deleting files that should not be deleted.) Moreover, when you do delete an object, this sets in motion lots of "signal" commands that move around backup video files.

vanlummelhuizen · 2025-01-09T15:59:12Z

I have looked into this problem, but without succes so far.

My hypothesis is that, because GlossVideoNME is a subclass of GlossVideo, the delete function of GlossVideoNME is executed as if it was a GlossVideo.

Things I found out about FRIKANDEL-A (gloss id 47336)

The contents of the corresponding video file were modified on December 18 (the file's birth), while it's metadata were changed on December 20 (change of file name, move to different location, etc):

root@signbank-new:/var/www/repo# stat ../writable/glossvideo/NGT/FR/FRIKANDEL-A-47336.mp4
  File: ../writable/glossvideo/NGT/FR/FRIKANDEL-A-47336.mp4
  Size: 1435252   	Blocks: 2816       IO Block: 131072 regular file
Device: 20007dh/2097277d	Inode: 623811      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/  ubuntu)   Gid: ( 1001/wwwsignbank)
Access: 2025-01-08 23:26:03.444677917 +0000
Modify: 2024-12-18 18:18:40.134785862 +0000
Change: 2024-12-20 13:59:33.609808729 +0000
 Birth: 2024-12-18 18:18:40.124785614 +0000

An NME video was deleted and created again on December 18:

sqlite> select * from dictionary_glossrevision where field_name like 'nmevideo_%' and gloss_id = 47336;
id|time|field_name|old_value|new_value|gloss_id|user_id
154385|2024-11-14 11:42:07.510864|nmevideo_create||FRIKANDEL-A-47336_nme_0.mp4|47336|323
155427|2024-11-14 12:40:33.165356|nmevideo_create||FRIKANDEL-A-47336_nme_1.mp4|47336|323
159004|2024-11-14 15:57:19.854011|nmevideo_delete|FRIKANDEL-A-47336_nme_0.mp4||47336|323
159005|2024-11-14 15:57:20.152481|nmevideo_create||FRIKANDEL-A-47336_nme_0.mp4|47336|323
159008|2024-11-14 15:57:20.377890|nmevideo_delete|FRIKANDEL-A-47336_nme_1.mp4||47336|323
160029|2024-11-14 16:41:34.021974|nmevideo_delete|FRIKANDEL-A-47336_nme_0.mp4||47336|323
160030|2024-11-14 16:41:34.234272|nmevideo_create||FRIKANDEL-A-47336_nme_0.mp4|47336|323
282341|2024-12-18 18:18:38.888102|nmevideo_delete|FRIKANDEL-A-47336_nme_0.mp4||47336|323
282342|2024-12-18 18:18:39.181953|nmevideo_create||FRIKANDEL-A-47336_nme_0.mp4|47336|323
282345|2024-12-18 18:18:39.881494|nmevideo_delete|FRIKANDEL-A-47336_nme_0.mp4||47336|323
282346|2024-12-18 18:18:40.147739|nmevideo_create||FRIKANDEL-A-47336_nme_0.mp4|47336|323

and in the logs

[pid: 337|app: 0|req: 358/3269] 10.208.155.251 () {60 vars in 1148 bytes} [Wed Dec 18 19:18:38 2024] GET /dictionary/api_delete_gloss_nmevideo/5/47336/26403/ => generated 61 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 147 bytes (1 switches on core 3)
[pid: 337|app: 0|req: 359/3270] 10.208.155.251 () {64 vars in 1234 bytes} [Wed Dec 18 19:18:39 2024] POST /dictionary/api_create_gloss_nmevideo/5/47336/ => generated 82 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 147 bytes (88 switches on core 1)
[pid: 350|app: 0|req: 1469/3271] 10.208.155.251 () {60 vars in 1148 bytes} [Wed Dec 18 19:18:39 2024] GET /dictionary/api_delete_gloss_nmevideo/5/47336/26403/ => generated 98 bytes in 14 msecs (HTTP/1.1 200) 5 headers in 147 bytes (1 switches on core 0)
(...)
[pid: 337|app: 0|req: 361/3274] 10.208.155.251 () {60 vars in 1151 bytes} [Wed Dec 18 19:18:39 2024] GET /dictionary/api_delete_gloss_nmevideo/5/47336/211214/ => generated 61 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 147 bytes (1 switches on core 3)
[pid: 350|app: 0|req: 1471/3275] 10.208.155.251 () {64 vars in 1234 bytes} [Wed Dec 18 19:18:40 2024] POST /dictionary/api_create_gloss_nmevideo/5/47336/ => generated 82 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 147 bytes (55 switches on core 2)
[pid: 341|app: 0|req: 557/3276] 10.208.155.251 () {60 vars in 1151 bytes} [Wed Dec 18 19:18:40 2024] GET /dictionary/api_delete_gloss_nmevideo/5/47336/211214/ => generated 98 bytes in 13 msecs (HTTP/1.1 200) 5 headers in 147 bytes (1 switches on core 3)

rem0g · 2025-01-09T17:03:23Z

NME videos are indeed deleted via API url:

url = f"https://signbank.cls.ru.nl/dictionary/api_delete_gloss_nmevideo/5/{gloss_id}/{videoid}/"

the videoid is obtained via:
url = f"https://signbank.cls.ru.nl/dictionary/get_gloss_data/5/{gloss_id}"

and then 

    nme_videos = gloss_info.get("NME Videos", [])

then we execute deletion via:

url = f"https://signbank.cls.ru.nl/dictionary/api_delete_gloss_nmevideo/5/{gloss_id}/{videoid}/"

When we want to replace NME video, we call that api_delete_gloss_nmevideo and then upload new nme_video again via:

url = f"https://signbank.cls.ru.nl/dictionary/api_create_gloss_nmevideo/5/{gloss_id}/"

vanlummelhuizen · 2025-01-10T07:42:01Z

@rem0g In the procedure you describe above you seem to delete a NME video twice via https://signbank.cls.ru.nl/dictionary/api_delete_gloss_nmevideo/5/{gloss_id}/{videoid}/. Is that the case? I would think only one call should be enough.

rem0g · 2025-01-10T07:58:00Z

It’s executed once just before new nme video creation. The first one is just a “title” of the workflow so it’s done one time though.

…

On Fri, Jan 10, 2025 at 08:42 Micha Hulsbosch ***@***.***> wrote: @rem0g <https://github.com/rem0g> In the procedure you describe above you seem to delete a NME video twice via https://signbank.cls.ru.nl/dictionary/api_delete_gloss_nmevideo/5/{gloss_id}/{videoid}/. Is that the case? I would think only one call should be enough. — Reply to this email directly, view it on GitHub <#1453 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACC4F7Q6HWJOWXEUQGLBWQL2J52WBAVCNFSM6AAAAABUCWGSK6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOBRHE3TKMZYGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Woseseltops · 2025-01-10T09:54:30Z

I'm not sure if we're all on the same page regarding this problem. Let me summarize:

For a gloss like FRIKANDEL-A, we would expect several video files, like the main FRIKANDEL-A-47336.mp4, but also perspective variants like FRIKANDEL-A-47336_left.mp4, and NME videos like FRIKANDEL-A-47336_nme_0.mp4
Signbank thinks both perspective videos and NME videos exist for FRIKANDEL_A... but when you request the file paths of these videos, it just returns the main video path FRIKANDEL-A-47336.mp4 .
On top of this, this main video file FRIKANDEL-A-47336.mp4 is NOT the original main video, but an NME video.
Looking at the video files at the server, we see a lot of backups (not sure of what exactly), but no FRIKANDEL-A-47336_left.mp4 or FRIKANDEL-A-47336_nme_0.mp4 .

Unlikely as it is, this seems to have happened:

An NME video has replaced the original main video
All video paths, for both perspective and NME videos, have been changed to link to this main video
All additional videos that once existed, but now were not linked to anything anymore, have been deleted

susanodd · 2025-01-10T10:25:25Z

Pointed out earlier, in the video history (admin), some video files that are uploaded are using the M... for the NME video as well, so name of a source video is the same for the normal video as an NME video. For videos larger than 2.5 mb (Django quote above) these are in temp storage. It could happen that somehow the uploaded file in temp storage is being used by multiple objects because it has the same name.

Woseseltops · 2025-01-10T11:04:26Z

Two new discoveries:

I just tried to reproduce the problem by adding and removing NME videos video the API a few times, but no luck... it all worked as intended :/ . Gomer must have done something differently, but I cannot imagine what.
The other example glosses given by Gomer, #BMW and #PSV, don't show perspective and NME videos, but that's because of an unrelated problem (the hashtag in the name). The actual videos linked show the same pattern as FRIKANDEL-A

Woseseltops · 2025-01-10T11:06:33Z

Pointed out earlier, in the video history (admin), some video files that are uploaded are using the M... for the NME video as well, so name of a source video is the same for the normal video as an NME video. For videos larger than 2.5 mb (Django quote above) these are in temp storage. It could happen that somehow the uploaded file in temp storage is being used by multiple objects because it has the same name.

I don't think this is what is happening here: mixing up files from temporary storage would lead to duplicates, all with correct file paths. Instead, we see that the URL paths stored in the database themselves are incorrect, and that the files at the correct path don't exist.

susanodd · 2025-01-10T11:27:34Z

Pointed out earlier, in the video history (admin), some video files that are uploaded are using the M... for the NME video as well, so name of a source video is the same for the normal video as an NME video. For videos larger than 2.5 mb (Django quote above) these are in temp storage. It could happen that somehow the uploaded file in temp storage is being used by multiple objects because it has the same name.

I don't think this is what is happening here: mixing up files from temporary storage would lead to duplicates, all with correct file paths. Instead, we see that the URL paths stored in the database themselves are incorrect, and that the files at the correct path don't exist.

Could it happen if a transaction doesn't complete? The object creations don't wait on the file system.

Or if the Django implementation (of Django) is somehow using symbolic links?

No idea, I'll stop. It's just weird.

When running it on iCloud, I had the same problem. It turned out the files were there, but with "." before them and had not been uploaded to iCloud yet.

rem0g · 2025-01-10T11:37:05Z

What would happen if you execute the transaction with 10 different glosses at short timespan? I am going to run the script on ten glosses soon. Gomer

…

On Fri, Jan 10, 2025 at 12:04 Wessel Stoop ***@***.***> wrote: Two new discoveries: - I just tried to reproduce the problem by adding and removing NME videos video the API a few times, but no luck... it all worked as intended :/ . Gomer must have done something differently, but I cannot imagine what. - The other example glosses given by Gomer, #BMW and #PSV, don't show perspective and NME videos, but that's because of an unrelated problem (the hashtag in the name). The actual videos linked show the same pattern as FRIKANDEL-A — Reply to this email directly, view it on GitHub <#1453 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACC4F7XXZAD3MIWAI2Q5OTD2J6SNBAVCNFSM6AAAAABUCWGSK6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOBSGQZTSOJZGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

rem0g · 2025-01-10T13:23:20Z

I am yet to see weird results but for this API I have encountered a bug:

I have to send POST update to api_update_gloss/{gloss}/video twice as the videos dont appear the first time i POST the videos, after 5 seconds again POST-ing it they appear again.

I don't know if that's related to this issue.

Gomer

rem0g · 2025-01-10T13:45:06Z

I am just going ahead and re-upload all base glos video files, but with sleep of 5 sec between uploads. If that's finished then i will start with NME video delete and re-uploads.

Woseseltops · 2025-01-10T13:53:42Z

What would happen if you execute the transaction with 10 different glosses at short timespan?

I expect nothing special, except maybe database locks if it's faster than the database can handle.

I have to send POST update to api_update_gloss/{gloss}/video twice as the videos dont appear the first time i POST the videos, after 5 seconds again POST-ing it they appear again.

I think unrelated. Respond to this comment with your fav emoji if you want me to create a separate issue for it :)

I am just going ahead and re-upload all base glos video files, but with sleep of 5 sec between uploads. If that's finished then i will start with NME video delete and re-uploads.

Apologies that this is needed, I realize the problem is on our side :(.

I think our best chance of fixing the problem in the current issue is being able to reproduce it. If you notice the disappearing/overwriting of videos again during the re-upload, would be great if you could (for one example gloss) explain exactly which commands you run so that I can try to make the behavior appear at will.

rem0g · 2025-01-10T14:03:00Z

Ok thank you! We will see what happens and no need to have seperate issue for that, for now :)

rem0g · 2025-01-11T16:13:58Z

Still reuploading as I also have script written to double check if video is there at gloss, if not then reupload again.

I came across this for example:

https://signbank.cls.ru.nl//dictionary/gloss/45834

Gloss video is in NME index, but it should not be there.

I have theory what is happening:

Gloss Video URL is https://signbank.cls.ru.nl/dictionary/protected_media/glossvideo/NGT/RA/RAT-A-45834.mp4?v=20250111170412

NME video URL is also: "Link": "https://signbank.cls.ru.nl//dictionary/protected_media/glossvideo/NGT/RA/RAT-A-45834.mp4"

When i run API to delete the NME video index 0, it will remove the gloss video.

I think that looks a reasonable cause.

My proposal for a bugfix:

Disable api_delete_gloss_nmevideo as we can simply overwrite existing NME video at index 0 for example.

API documentation states this:

Updates the details of a 'non-manual element' video for a gloss. To replace the video, delete it using api_delete_gloss_nmevideo and create a new NME video using api_create_gloss_nmevideo.

So that requirement will not be need anymore.

For now I have disabled NME video upload script, when the bug if recognized and fixed then I will re-activate it.

susanodd · 2025-01-13T11:42:55Z

[CODE BABBEL]

Yes, it is true that it overwrites the video with the same index number.
Originally, the indexing started at 1. I modified that to also allow index 0 since you were using that in your API and creating them with 0.
Originally, before the API, it was starting them at 1 because the template forms do this with number fields.
As you know, 0 and None and Null and False are all overloaded.
The test on whether an NME or Perspective video is one of these is a subclass builtin Django test.
But if some of the fields of the subclass are filled in or not filled in (for example, if the index is 0) it might not correctly record the type of the object. (That would be a Django bug or something very weird going on with coercion inside the code. Sometimes, the first instance of a variable determines the type of the variable in the rest of the code that follows.)

When you used to use the "zip" upload, the video files were overwritten. The backup system was avoided altogether.
The problems kind of started with the backup system came into use.

The normal videos do not delete the video files. But the subclasses do delete them.
Since somehow (we don't know how) the NME and perspective videos were being given the same file name inside the object as the normal video, then when you delete these, it was deleting the normal video file.

If you just stop using delete, it should just work, since it overwrites them. Did you try this?
Can you start numbering with index 1?
It could be something that the "0" index is somehow ending up matching "Null" and the code thinks the object is a normal video.
(If coercion on values is doing something.)

There could also be something messed up with the "str" coercion used on vide files. Normally, this yields the relative path.

Although while working on some new video admin commands, the __str__ method of GlossVideo (enhanced with print statements for if DEBUG_VIDEOS is turned on), the the str of the superclass is always being called, not the subclass str methods. (So internally, it looks like Django is ignoring the subclass methods when __str__ is used to obtain the path)

This is a concern (to me) because it means the wrong __str__ method is being called on a subclass object. And the __str__ method is being used to obtain the video path.

The type of the variables is dynamic and determined when the variable is first assigned. So it could be that somehow Django has "first" used a variable for an object of the supertype (normal video) and later in the code the type was already determined. The supertype methods also work on the subtype objects.
(Case in point, the __str__ method retrieves the relative path of the video.)

@vanlummelhuizen can write this more concisely. What I'm describing would kind of be a Django implementation of subclasses error.

I'm guessing we need to remove the "subclass" __str__ methods since this method is being used nonchalantly / callously to obtain the video relative path all over the place.

susanodd · 2025-01-14T06:42:26Z

Another try, can you make the index start at 1 of the NME video?

Using 0 can be dangerous, since it is also sometimes the same as False or None in the database. Because things are strings in the API it could be a problem with type coercion. Also, Django automatically converts objects to their "id" when they are passed around. They only remain an actual object in the template if it's via context variables.

It could possibly be that the 0 is somehow become a "Null" value, so Django thinks the field is not set in the object.

rem0g · 2025-01-14T17:17:12Z

Using 0 as replacement for gloss sounds weird as I understand the index number is for NME videos only.

Nothing has been sent to Signbank via API after last reupload of all videos before my vacation (happened past friday or so). NME autoupload and api_delete_gloss_nmevideo API call has been disabled since then too.

Per @uklomp comment everything was fine this morning, so something has happened on Signbank server outside of our scope? I hope @Woseseltops can solve this issue this week as we are nearing the end of our Signbank project.

rem0g assigned Woseseltops, rem0g, susanodd and uklomp Dec 23, 2024

This comment was marked as off-topic.

Sign in to view

susanodd assigned vanlummelhuizen Jan 6, 2025

susanodd mentioned this issue Jan 6, 2025

#1406: Enhancements to Dataset Manager -> Manage Video Storage functionality #1412

Open

susanodd added bug file system high priority labels Jan 6, 2025

uklomp added the blocking label Jan 7, 2025

susanodd mentioned this issue Jan 7, 2025

Admin commands for gloss videos #1398

Open

Woseseltops unassigned rem0g, vanlummelhuizen and uklomp Jan 9, 2025

uklomp mentioned this issue Jan 14, 2025

All neutral videos have been replaced with Right perspective #1457

Open

Videos randomly disappearing #1453

Videos randomly disappearing #1453

Comments

rem0g commented Dec 23, 2024 • edited Loading

rem0g commented Dec 23, 2024

rem0g commented Dec 23, 2024

This comment was marked as off-topic.

susanodd commented Dec 23, 2024

susanodd commented Dec 23, 2024

rem0g commented Dec 23, 2024

susanodd commented Dec 23, 2024

susanodd commented Jan 6, 2025 • edited Loading

susanodd commented Jan 6, 2025 • edited Loading

susanodd commented Jan 6, 2025

susanodd commented Jan 6, 2025

susanodd commented Jan 6, 2025

susanodd commented Jan 6, 2025 • edited Loading

susanodd commented Jan 6, 2025 • edited Loading

susanodd commented Jan 6, 2025 • edited Loading

susanodd commented Jan 6, 2025

rem0g commented Jan 6, 2025

uklomp commented Jan 7, 2025

susanodd commented Jan 7, 2025 • edited Loading

susanodd commented Jan 7, 2025

susanodd commented Jan 7, 2025

susanodd commented Jan 8, 2025 • edited Loading

vanlummelhuizen commented Jan 9, 2025

rem0g commented Jan 9, 2025 • edited Loading

vanlummelhuizen commented Jan 10, 2025

rem0g commented Jan 10, 2025 via email

Woseseltops commented Jan 10, 2025

susanodd commented Jan 10, 2025

Woseseltops commented Jan 10, 2025

Woseseltops commented Jan 10, 2025

susanodd commented Jan 10, 2025

rem0g commented Jan 10, 2025 via email

rem0g commented Jan 10, 2025

rem0g commented Jan 10, 2025

Woseseltops commented Jan 10, 2025 • edited Loading

rem0g commented Jan 10, 2025

rem0g commented Jan 11, 2025

susanodd commented Jan 13, 2025 • edited Loading

susanodd commented Jan 14, 2025 • edited Loading

rem0g commented Jan 14, 2025

rem0g commented Dec 23, 2024 •

edited

Loading

susanodd commented Jan 6, 2025 •

edited

Loading

susanodd commented Jan 6, 2025 •

edited

Loading

susanodd commented Jan 6, 2025 •

edited

Loading

susanodd commented Jan 6, 2025 •

edited

Loading

susanodd commented Jan 6, 2025 •

edited

Loading

susanodd commented Jan 7, 2025 •

edited

Loading

susanodd commented Jan 8, 2025 •

edited

Loading

rem0g commented Jan 9, 2025 •

edited

Loading

Woseseltops commented Jan 10, 2025 •

edited

Loading

susanodd commented Jan 13, 2025 •

edited

Loading

susanodd commented Jan 14, 2025 •

edited

Loading