Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alexla/upgrade kfp v1.8.16 #23

Open
wants to merge 225 commits into
base: feature/zg
Choose a base branch
from

Conversation

alexlatchford
Copy link
Collaborator

Description of your changes:

Pull in upstream changes from v1.8.16 to get us closer to the latest version!

We use v1.8.16 because this aligns with the major+minor KFP version that ships with KFP v1.5.0 (see here, it's v1.8.1). We don't want to use v1.8.1 directly as this PR is trying to address python dependency conflicts in the zillow-kfp SDK so the additional patch versions include requirements changes to make KFP compatible with the remainder of the python ecosystem it depends on.

Checklist:

neuromage and others added 30 commits October 28, 2021 14:25
…ubeflow#6804)

* Use google.protobuf.Value in v2 for passing parameters.

* retest samples.

* Fix tests.

* Update release, more cleanup.

* Use github.com/kubeflow/pipelines/api from same repo.

* Run go mod tidy

* chore: go mod tidy

* fix v2 compile error and clean up unused code

* pr comments.

* update goldens

* Fix metadata recording.

* Update kfp mlmd client.

* fix test again

* another try.

* chore: migrate v2 DAG driver input parameters to protobuf.Value + small refactorings

* fix v2 launcher + clean up

* fix a compile error

* fix a few more tests

* fix number parsing

* clean up

* disable cache_v2 test.

Co-authored-by: Yuan Gong <[email protected]>
* feat: upgrade argo to v3.1.14

* go mod tidy

* fix license versions
…#6825)

* Implement experimental `@component` component

* release note
…Protobuf.Value for import uri. (kubeflow#6827)

* fix importer

* release note

* remove error usage in test sample

* disable importer test
* added draft of create v2 pipeline run

* fixed broken UT and added UT for parsing template

* modified run apis to support v2 IR spec

* remove temporary patch

* fixed dependency

* fixed build failure

* finished draft

* finished create job and run

* refactor template and fixed broken UT

* updated go license

* fixed build failure

* fixed build

* added UT

* modified UT

* fixed build failure

* fixed license
* feat(sdk): add load_component_from_*

* address comments'
:

* update release notes
* fix display name for experimental tasks_group

* release note
* chore(sample): Update Katib examples for 0.12 release

* Run Early Stopping example
* feat(frontend): Allow setting AWS_S3_ENDPOINT

Allows setting the S3 endpoint used by minio to allow regions other than us-east-1

* feat(frontend): allow to configure s3 bucket region for artifacts

* fix formatting

* add integration test for non-default s3 bucket region
…r 0.1.9 release

PiperOrigin-RevId: 406876964
…eflow#6649 (kubeflow#6622)

* Update view-edit-cluster-roles.yaml

* Update view-edit-cluster-roles.yaml
…tform/ folder.

The image is not specific to aiplatform component.

PiperOrigin-RevId: 407182304
This change includes:
- Renaming custom job op
- Moving it to a utils module.
- Removing workerpoolspec override from the interface.
- Updating the sample for custom_job

PiperOrigin-RevId: 407603075
haoxins and others added 22 commits April 1, 2022 13:43
* chore(deps): Upgrade google-cloud-storage to v2

* Update RELEASE.md
…flow#7780)

* add README with project description

* modify language

* add links for python code and yaml
* expand accepted typing extensions versions

* update release notes
…==1.8 (kubeflow#7875)

* use kfp-pipeline-spec>=0.1.16

* add functions to v2 compiler utils

* add v2 compiler utils tests

* implement retry policy compilation logic

* add v2 compiler tests

* add retry policy sample test

* add release note
* bump version

* update release notes

* Update RELEASE.md

Co-authored-by: Chen Sun <[email protected]>
* add IteratorPolicy to v1 pipeline_spec proto

* revert proto changes as it is unnecessary

* bump pipeline_spec dependency to 0.1.16

* add compiler support for parallelism in ParallelFor

* added unit tests

* add one more test case

* add compiler cli tests for parallelfor with parallelism setting

* restore change to pipeline spec

* address PR comments: revert parallelism default to None, update error message, update tests

* update compiler_tests golden

* fixed function signature, simplified test function args

* add release.md message
* update README example

* use keyword arguments
…low#8226)

* add support for dynamic importer metadata

* fix non-deterministic dict comprehension processing

* use util function
* bump version

* update release notes
…ubeflow#8454)

* fix multiple workers bug on gcsfuse error

* add comment for context
* bump version

* update requirements.txt
* bump version

* update release notes

* be more precise in release notes
…8473)

* more google auth changes

* update release notes
@alexlatchford alexlatchford marked this pull request as ready for review December 9, 2022 01:37
Copy link

@aaron-arellano aaron-arellano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How did you test this?

@alexlatchford
Copy link
Collaborator Author

How did you test this?

To be honest I haven't yet, was going to pick folks brains on the best way to proceed here. I've got a PR for the aip-py upgrade but where do we need to test the resulting artifacts, seems like:

  • aip-workflow-example + integration tests.
  • metadata-writer
  • pipelines-ui

I got these from the list of artifacts zillow-kfp publishes (here). So I can start with those but if we find any others we can add them to the list. Thoughts?

@aaron-arellano
Copy link

I got these from the list of artifacts zillow-kfp publishes (here). So I can start with those but if we find any others we can add them to the list. Thoughts?

I am not sure 100% where this kfp client gets used but I would imagine any case that uses it. So specific projects that use this. If the case gets validated in integration tests then maybe that works.

@alexlatchford
Copy link
Collaborator Author

Blocked on Kserve upgrade to at least v0.9.0 due to non-overlapping constraints for google-api-core, plan is for us to tackle that upgrade in Q1 so we plan to leave this PR open until then. See this thread for context.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.