Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review all input validations #783

Open
rgaudin opened this issue May 17, 2023 · 6 comments
Open

Review all input validations #783

rgaudin opened this issue May 17, 2023 · 6 comments
Assignees

Comments

@rgaudin
Copy link
Member

rgaudin commented May 17, 2023

In order to prevent incorrect data to be entered in the Zimfarm, we want to review all recipes inputs and update their constraints.

All constraints are in https://github.com/openzim/zimfarm/blob/main/dispatcher/backend/src/common/schemas/fields.py and https://github.com/openzim/zimfarm/tree/main/dispatcher/backend/src/common/schemas/offliners

Beside Offliner Flags, recipe inputs are:

  • Recipe Name
  • Language
  • Tags
  • Category
  • Warehouse Path
  • Status
  • Periodicity
  • Offliner
  • Platform
  • Image Name
  • Image Tag
  • Monitoring
  • CPU
  • Memory
  • Disk
  • RAM fs

Please list below the list of changes you'd like to constraints on those fields or Offliner flags. If a change is to be applied to all offliners (albeit using their own scaper-specific names), please say so.

My proposal:

  • Recipe Name should enforce the Naming Convention
  • All fields setting Name metadata should enforce the Naming Convention
  • All fields setting Title metadata should be restricted to 30 chars max (as per our recommendations)
  • All fields setting the Description metadata should be restricted to 80 chars max
  • All fields setting the LongDescription metadata should be restricted to 400chars max
  • All fields setting the Language metadata should be checked for ISO-639-3 validity (as a comma-separated list).
@rgaudin
Copy link
Member Author

rgaudin commented May 19, 2023

  • All fields setting ZIM filename shoud enforce {Name}_{period}.zim

@kelson42 kelson42 pinned this issue May 24, 2023
@kelson42 kelson42 added the prio1 label Nov 1, 2023
@kelson42
Copy link
Contributor

devdocs.ios failing scenario because of this weakness https://farm.openzim.org/pipeline/cb6a074a-e33a-4552-9c45-932486a1dde9/debug

@rgaudin
Copy link
Member Author

rgaudin commented Jan 31, 2024

I believe it's a bit different: allowing arbitrary text in choice field or (more likely) a previously value choice that is not anymore and we did not update the recipe

@benoit74
Copy link
Collaborator

I think that #910 proves that we also need to regularly check that flags are still valid.

We should check:

  • does the flag still exists for given offliner?
  • is the flag value still valid?

I've opened a distinct issue #911 since this is a bit different than just ensuring that constraints of all offliners are appropriate.

@benoit74
Copy link
Collaborator

kiwix/operations#192 just shown how important it is to avoid unwanted chars in ZIM name / filename.

@benoit74
Copy link
Collaborator

Removed from zimit2 project unfortunately

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants