Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Datacatalog cache deletion #4655

Open
wants to merge 69 commits into
base: master
Choose a base branch
from
Open

[Feature] Datacatalog cache deletion #4655

wants to merge 69 commits into from

Commits on Dec 15, 2022

  1. Migrated CatalogClient from propeller to stdlib 

    Implemented new datacatalog functionality required for cache eviction
Updated to latest unreleased version of flyteidl and flyteplugins

Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Dec 15, 2022
    Configuration menu
    Copy the full SHA
    d5fe607 View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2023

  1. Added function for updating selected fields of NodeExecutionModel 

    Allows for fields to be explicity set/updated to nil

Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 4, 2023
    Configuration menu
    Copy the full SHA
    422eb46 View commit details
    Browse the repository at this point in the history

  2. Refactored listing of node and task executions to shared util 

    Allows for re-use by cache manager

Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 4, 2023
    Configuration menu
    Copy the full SHA
    47e9a00 View commit details
    Browse the repository at this point in the history

  3. Implemented CacheService 

    Added endpoint for evicting execution cache
Added endpoint for evicting task execution cache

Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 4, 2023
    Configuration menu
    Copy the full SHA
    25c187c View commit details
    Browse the repository at this point in the history

  4. Updated to latest unreleased versions of flytepropeller and flytestdlib 

    Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 4, 2023
    Configuration menu
    Copy the full SHA
    dc59334 View commit details
    Browse the repository at this point in the history

  5. Added function for deleting artifacts to catalog client interface 

    Extended reservation retrieval to allow querying via artifact tag in catalog client interface

Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 4, 2023
    Configuration menu
    Copy the full SHA
    4c041c1 View commit details
    Browse the repository at this point in the history

  6. Added method to release catalog reservation by artifact tag 

    Added method to delete catalog artifact by ID

Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 4, 2023
    Configuration menu
    Copy the full SHA
    56c7b31 View commit details
    Browse the repository at this point in the history

  7. Update to latest unreleased versions of flyteidl and flyteplugins 

    Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 4, 2023
    Configuration menu
    Copy the full SHA
    557a991 View commit details
    Browse the repository at this point in the history

  8. Updated to latest unreleased versions of flyteidl, flyteplugins, flyt… 

    …epropeller and flytestdlib

Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 4, 2023
    Configuration menu
    Copy the full SHA
    821467e View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2023

  1. Added datacatalog endpoint for deleting artifacts 

    Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    4dd0afd View commit details
    Browse the repository at this point in the history

  2. WIP: first draft for cache eviction of past executions 

    Added new CacheEvictionError message representing an error encountered during eviction of stored data
Added new UpdateTaskExecution endpoint for updating task executions, currently only supporting cache eviction

Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    7df2814 View commit details
    Browse the repository at this point in the history

  3. Updated to latest released version of flytestdlib 

    Ran go mod tidy

Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    6480f02 View commit details
    Browse the repository at this point in the history

  4. Refactored cache eviction to own service 

    Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    1dc2115 View commit details
    Browse the repository at this point in the history

  5. Refactored new cache service to share EvictCacheRequest for both endp… 

    …oints

Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    419570e View commit details
    Browse the repository at this point in the history

  6. Merged cache eviction endpoints into single RPC call 

    Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    ea6df4e View commit details
    Browse the repository at this point in the history

  7. Removed no longer relevant generated mocks 

    Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    da32ef0 View commit details
    Browse the repository at this point in the history

  8. Split CacheService.EvictCache into two separate endpoints 

    grpc-gateway parsing of URL params does not work for joined endpoint at the moment - fixed in major version upgrade
Added extra CacheEvictionErrorCode enum entries

Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    80e8e72 View commit details
    Browse the repository at this point in the history

  9. Add bulk endpoints for acquiring/releasing reservations and deleting … 

    …artifacts to datacatalog

Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    d7f5203 View commit details
    Browse the repository at this point in the history

  10. Implemented deletion of artifact including artifact data 

    Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    d7eda41 View commit details
    Browse the repository at this point in the history

  11. Deleting artifacts cleans up tags and partitions as well 

    Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    8890348 View commit details
    Browse the repository at this point in the history

  12. Updated to latest unreleased version of flyteidl and flytestdlib 

    Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    a2315f8 View commit details
    Browse the repository at this point in the history

  13. Updated to latest unreleased version of flytestdlib 

    Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    d2565a1 View commit details
    Browse the repository at this point in the history

  14. Updated to latest unreleased version fo flyteidl and flytestdlib 

    Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    0167db1 View commit details
    Browse the repository at this point in the history

  15. Implement acquiring/releasing of reserverations as bulk operation 

    Implement deleting of artifacts as bulk operation

Signed-off-by: Nick Müller <[email protected]>
    Nick Müller committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    8f2f414 View commit details
    Browse the repository at this point in the history

Commits on Dec 29, 2023

  1. Transfer commits 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Dec 29, 2023
    Configuration menu
    Copy the full SHA
    c2d5454 View commit details
    Browse the repository at this point in the history

Commits on Dec 30, 2023

  1. resolve conflicts 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Dec 30, 2023
    Configuration menu
    Copy the full SHA
    5cd55cf View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2024

  1. Merge remote-tracking branch 'blackshark-ai/flyteplugins/cache-evicti… 

    …on-past-executions' into feature/datacatalog-cache-deletion
    @pvditt
    pvditt committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    d79a634 View commit details
    Browse the repository at this point in the history

  2. resolve conflicts 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    adf6209 View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2024

  1. update import paths 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    23d2dc1 View commit details
    Browse the repository at this point in the history

  2. implement new catalog client methods 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    cda7c34 View commit details
    Browse the repository at this point in the history

  3. move catalog client to stdlib 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    5695f93 View commit details
    Browse the repository at this point in the history

  4. tidy 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    590c379 View commit details
    Browse the repository at this point in the history

Commits on Jan 5, 2024

  1. Merge remote-tracking branch 'blackshark-ai/flytestdlib/delete-artifa… 

    …ct' into feature/datacatalog-cache-deletion

Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 5, 2024
    Configuration menu
    Copy the full SHA
    143e160 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'blackshark-ai/flyteadmin/cache-eviction… 

    …-past-executions' into feature/datacatalog-cache-deletion

Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 5, 2024
    Configuration menu
    Copy the full SHA
    fac2e8e View commit details
    Browse the repository at this point in the history

Commits on Jan 6, 2024

  1. lint 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 6, 2024
    Configuration menu
    Copy the full SHA
    d48d9e8 View commit details
    Browse the repository at this point in the history

Commits on Jan 7, 2024

  1. update proto equals check 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 7, 2024
    Configuration menu
    Copy the full SHA
    751447a View commit details
    Browse the repository at this point in the history

  2. remove workflow executuion cache eviction endpoint 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 7, 2024
    Configuration menu
    Copy the full SHA
    147765d View commit details
    Browse the repository at this point in the history

  3. update idl service docs 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 7, 2024
    Configuration menu
    Copy the full SHA
    7c93bb6 View commit details
    Browse the repository at this point in the history

Commits on Jan 8, 2024

  1. Merge branch 'master' into feature/datacatalog-cache-deletion

    @pvditt
    pvditt committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    662355c View commit details
    Browse the repository at this point in the history

  2. only support cache eviction for single task 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    114a45e View commit details
    Browse the repository at this point in the history

Commits on Jan 9, 2024

  1. delete dataset along with artifact 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    2157d0d View commit details
    Browse the repository at this point in the history

  2. always attempt releasing reservation after it has been acquired 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    5fa9c8f View commit details
    Browse the repository at this point in the history

  3. move utility functions back to execution managers 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    80823c5 View commit details
    Browse the repository at this point in the history

  4. set db context on UpdateSelected 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    968fba6 View commit details
    Browse the repository at this point in the history

  5. Revert "update idl service docs" 

    This reverts commit 7c93bb6.

Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    4d422a6 View commit details
    Browse the repository at this point in the history

  6. manually remove EvictExecutionCache mentions from service.rst 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    77b1b2c View commit details
    Browse the repository at this point in the history

  7. update node execution closure after succeesful artifact deletion 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    2525232 View commit details
    Browse the repository at this point in the history

  8. don't remove catalog key from taskNodeMetadata to better support retries 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    5bfd6e9 View commit details
    Browse the repository at this point in the history

Commits on Jan 10, 2024

  1. Merge branch 'master' into feature/datacatalog-cache-deletion 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    d01a0e7 View commit details
    Browse the repository at this point in the history

  2. set node execution CacheStatus and taskNodeMetadata CacheStatus to ca… 

    …che evicted

Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    d7ac714 View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2024

  1. Merge branch 'master' into feature/datacatalog-cache-deletion 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    409e221 View commit details
    Browse the repository at this point in the history

  2. remove bulk endpoints 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    6aeaae1 View commit details
    Browse the repository at this point in the history

  3. remove bulk delete counters 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    706dc4e View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2024

  1. revert mod tidy chagnes 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    6ec0e88 View commit details
    Browse the repository at this point in the history

  2. replace flyteidl source in flytestdlib 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    3ce3742 View commit details
    Browse the repository at this point in the history

  3. add otelgrpc dependency 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    209e2aa View commit details
    Browse the repository at this point in the history

  4. fix flytestdlib dependencies 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    007be11 View commit details
    Browse the repository at this point in the history

  5. revert release reservation changes to support release multiple reserv… 

    …ations

Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    13a2f68 View commit details
    Browse the repository at this point in the history

  6. tidy 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    2f9c0b4 View commit details
    Browse the repository at this point in the history

  7. revert tidy 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    91d30e4 View commit details
    Browse the repository at this point in the history

  8. tidy 🤦 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    30eb267 View commit details
    Browse the repository at this point in the history

  9. Merge branch 'master' into feature/datacatalog-cache-deletion 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    a0d0aae View commit details
    Browse the repository at this point in the history

  10. tidy 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    7e8a19e View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2024

  1. merge master 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    cdbb950 View commit details
    Browse the repository at this point in the history

  2. cleanup 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    6e80529 View commit details
    Browse the repository at this point in the history

  3. cleanup 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    d858906 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2024

  1. update boiler plate linter 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    ae652a0 View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2024

  1. Merge branch 'master' into feature/datacatalog-cache-deletion 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    83263a8 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2024

  1. merge master 

    Signed-off-by: Paul Dittamo <[email protected]>
    @pvditt
    pvditt committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    9e322a8 View commit details
    Browse the repository at this point in the history