Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Validate dataset properties in jobs #1473

Draft
wants to merge 8 commits into
base: release-jobs
Choose a base branch
from

Conversation

sbliven
Copy link
Contributor

@sbliven sbliven commented Oct 30, 2024

Description

Allows jobs to validate properties of the dataset.

Motivation

The validate action currently allows operators to enforce attributes of the DTO.

However, many jobs depend on jobParams.datasetLists to link a job to certain datasets. This allows validation of properties on datasets associated with the dataset.

The motivating example would be checking datasetLifecycle properties for certain jobs. These are currently hard-coded for special job types. This would allow them to be applied to custom job types as well.

configVersion: v1.0
jobs:
  - jobType: custom_archive
    create:
      auth: "#datasetAccess"
      actions:
        - actionType: validate
          datasets:
            datasetlifecycle.archivable:
              const: true

Changes:

  • Add datasets property to validate actions (only in create operations) to validate linked dataset properties

Tests included

(WIP pending tests and documentation)

  • Included for each change/fix?
  • Passing?

Documentation

  • swagger documentation updated (required for API changes)
  • official documentation updated

official documentation info

Docs are included in SciCatProject/documentation#56

During a `create` operation, the `validate` action can take a
`datasets` option listing path/schema pairs that will be applied
to any datasets in jobParams.datasetList.

Datasets are fetched from the database during the DTO validation step
if needed.
If the session times out then only public jobs will be visible.
@sbliven sbliven self-assigned this Oct 30, 2024
@sbliven sbliven requested a review from despadam October 30, 2024 14:28
@sbliven
Copy link
Contributor Author

sbliven commented Nov 3, 2024

Documentation is now added to SciCatProject/documentation#56

@sbliven
Copy link
Contributor Author

sbliven commented Nov 5, 2024

The unit tests pass because I'm manually mocking the datasetService, so the core functionality works.

The mocha tests fail because the datasetService doesn't get injected properly by nest. I need to figure out how to properly initialize the ValidateCreateAction before merging this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant