#1406: Enhancements to Dataset Manager -> Manage Video Storage functionality

[susanodd]

THIS BRANCH IS ON SIGNBANK-TEST

It concerns improvements to the following page (follow this navigation):

Dataset Manager --> Dataset --> Manage Video Storage

• Improved layout with panels for different data
• Added links to unlinked videos to allow the dataset manager to retrieve deleted videos via a hyperlink

BUTTONS IN GLOSS VIDEOS PANEL DO THIS (PER GLOSS)

• Added command to renumber backup video versions (for use if they are numbered wrong, also renames bak bak sequences to correct format)
• Remove 7char Django strings in filenames (they appear when a file should overwrite an existing file, see elsewhere)
• Correct video file extension in filename to match video file encoding type (needed for backup files)

The paths in the Gloss Video objects are repaired as described above. If there is a matching file, it is fixed to the new path. Thus the commands can also be used on the development servers (where the files are missing), thus avoiding OS exceptions on the development servers.

One caveat, for the special case of multiple Gloss Video Version 0 objects, this removes objects without any video file, except for leaving one file. If more than one object has a file, only the most recent is left. This is likely a file with extra 7char Django sequence in it. That is fixed by the other code.

MODIFICATION IS ONLY DONE IF YOU USE A BUTTON. OTHERWISE NOTHING IS DONE. THE PATHS SHOWN ARE VISUALLY UPDATED SO YOU SEE WHAT WAS DONE. THIS IS DONE BY AJAX WITH JSON FEEDBACK TO THE VIEW

[susanodd]

It's working now!!

This command is here (via the Manage Datasets page):

The Rename button appears when there are backups or the file type does not match (more buttons to follow)

@Woseseltops any suggestions for using a search? Or tag the files in the list as per whether one of the "panel" categories applies. (There are thousands of video paths being displayed here. The button updates the one in place.) (Socket Socket!)

Ideally, I will put a fresh signbank.db on the server and copy some video archives in order to fully test this feature.

[susanodd]

I tested the newest code (that now has master merged into it) on signbank-test (with the database as per this morning).

I copied the video files from the server in glossvideo/NGT/#P onto signbank-test in order to test the renaming of the "wrong" backup files.
The code works correctly, the files are renamed to have to correct name and the correct version number.

As can be seen on signbank-test, numerous backup video file names (on the master server) are wrong.
(There are no video files on signbank-test, unless copied there. But you can rename the paths in the objects. It just doesn't rename any files unless they are found.)

The objective of this branch is to allow the dataset manager to repair messed up file names.

The perspective video files and the NME files may still have problems #1453. You can see that the names are the same as the normal video in the manage video page.
This branch only fixes normal backup videos. I can't do anything about objects pointing to wrong files if the right file does not exist.

[susanodd]

I made this be high priority because of #1453

If you look at the file system, there are lots of videos with wrong names. These are old backup files. But they keep being "touched" because all backup (objects) are "moved around" whenever a new video is uploaded. This cascades onto all the already existing files. (See old issues. Because there are thousands of these, nobody has cleaned them up. This command is to do that. Also the other pull request for removing them.)

[susanodd]

This branch can be put on master.

It does not harm anything. It only "does stuff" at the discretion of a user with permission. This is "per gloss" via a button on the page as shown above.

[susanodd]

I put the completed tag on the branch because it can be merged and it can be used already.

Improvements and more functionality can be added in the future. At the moment, it will solve bugs if we start using it now. The problems it addresses have existed since November.

[Woseseltops]

I really like your UI/UX approach here, giving end users the power to clean up! It might need search as well, as you mention, but lets wait until users ask for that. I also like that you renamed the Python file to better represent the content.

Some minor things:

• remove_backup_glossvideo_objects_with_no_file says it is no called, do we still need it?
• renumber_backup_videos could use a guard clause
• desired_filename = idgloss + '-' + str(gloss.id) + video_extension would benefit from an f-string

[susanodd]

Okay,

• I made the strings be f-strings
• I added guards (although not entirely sure what for guards,so I did what was possible given the argument)
• I removed the unused function (I think there needs to be an additional button for deleting the backups. But we can wait to see.)
• A search functionality would be useful too. But like you say, we will wait and see what the users want.

[vanlummelhuizen]

• I added guards (although not entirely sure what for guards,so I did what was possible given the argument)

To be honest, I also don't see the need for guard clauses here either. Without the guard clause you added, the code will handle an empty gloss variable rather elegantly, I think.

My interpretation of a guard clause...

If there is a `if..else` block with a longer if-part and a short else-part that exits the function or method (e.g. with `return` of `raise`), we can have a negated if that exits quickly and everything that was in the if-block below the new if statement.

function do_things(something):
    if something == True:
        print('something is true')
        # ... a lot more stuff
    else:
        return 'nothing to do'

=>

function do_things(something):
    if something != True:
        return 'nothing to do'
    print('something is true')
    # ... a lot more stuff

@susanodd Can you adopt the following?: if you don't understand exactly what @Woseseltops or I want you to change and why we want it changed, ask for clarification before commiting code.

[susanodd]

I'll wait and see what @Woseseltops says. Then just undo the "guards" if that's not what he meant.

In other issues in the past, he has meant to return earlier out of the function so it's not an indented block.