Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor Morpheus unit tests and plugin to the conda recipe for per-lib testing #1933

Merged
merged 25 commits into from
Oct 12, 2024
Merged

Refactor Morpheus unit tests and plugin to the conda recipe for per-lib testing #1933

merged 25 commits into from
Oct 12, 2024

Commits on Oct 11, 2024

  1. Organize tests by functional block 

    Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    29b49b9 View commit details
    Browse the repository at this point in the history

  2. Rely entirely on MORPHEUS_ROOT environment variable to setup TEST_DIRS 

    Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    6fa32e6 View commit details
    Browse the repository at this point in the history

  3. Make databricks tests dependent on databricks-connect being installed 

    databricks-connect is currently pip installed which makes it difficult
to add as a conda runtime dependency. so the tests for the functionality
have been made optional.

Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    4f33e7e View commit details
    Browse the repository at this point in the history

  4. Replace tmpdir_path fixture with tmp_path fixture 

    tmpdir is a legacy fixture that seems to only be available with
pytest_kafka

Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    59fe3ae View commit details
    Browse the repository at this point in the history

  5. Plugin per-lib tests into the morpheus conda recipe 

    Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    1657e9e View commit details
    Browse the repository at this point in the history

  6. Add lxml to the morpheus-core runtime dependencies 

    It is need by beautiful soup to fix the test failure in
test_rss_controller.py -
 bs4.FeatureNotFound: Couldn't find a tree builder with the features you
requested: xml.

Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    27a20c2 View commit details
    Browse the repository at this point in the history

  7. GPU is needed for building the conda libraries 

    conda buids runs the relevant unit tests to validate the library that
it just built and these tests require GPU(s).

Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    86dccc3 View commit details
    Browse the repository at this point in the history

  8. Update the ci_pipe to require a GPU for the conda package stage 

    GPU is a pre-requisite for testing the conda packages

Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    c2baf68 View commit details
    Browse the repository at this point in the history

  9. Fixup checks 

    Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    2b22b43 View commit details
    Browse the repository at this point in the history

  10. Use rapidsai/ci-conda as base image for morpheus conda-package 

    Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    429ace5 View commit details
    Browse the repository at this point in the history

  11. Update morpheus conda build dependencies to work with rapidsai/ci-conda 

    Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    b97c083 View commit details
    Browse the repository at this point in the history

  12. Use the location of the _utils file for setting the tests_dir 

    Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    7fb703b View commit details
    Browse the repository at this point in the history

  13. Add a conda_morpheus_llm key in dependencies.yaml 

    These are pip dependencies needed to run the llm unit tests at the end
of conda build

Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    8cb1f91 View commit details
    Browse the repository at this point in the history

  14. Add a fixture to skip databricks tests if databricks is not installed. 

    databricks units tests will be skipped in the morpheus-conda testing.

Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    79f1bdd View commit details
    Browse the repository at this point in the history

  15. Update the container used for ci_pipe:package stage to rapidsai/ci-conda 

    Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    dc4361d View commit details
    Browse the repository at this point in the history

  16. Change the imports in the examples dir to relative ones 

    This was detectected as an isort problem.

Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    b2cb54b View commit details
    Browse the repository at this point in the history

  17. Updates related to the rapids-24.10 upgrade 

    Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    a803ffc View commit details
    Browse the repository at this point in the history

  18. Change the container credentials for conda packaging 

    Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    79b2a9b View commit details
    Browse the repository at this point in the history

  19. Needed for tests that import modules under tests/morpheus without ins… 

    …tall

Example:
Without this change -
[pytest -v tests/morpheus] - works
[pytest -v tests/morpheus/pipeline] - fails

Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    bd8038e View commit details
    Browse the repository at this point in the history

  20. Include core dependencies in the morpheus-llm testing 

    Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    bd2ed83 View commit details
    Browse the repository at this point in the history

  21. Create _modules for files that need to be imported locally 

    Signed-off-by: Anuradha Karuppiah <[email protected]>
    @AnuradhaKaruppiah
    AnuradhaKaruppiah committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    7362b77 View commit details
    Browse the repository at this point in the history

  22. Update ci/scripts/run_ci_local.sh 

    Co-authored-by: David Gardner <[email protected]>
    @AnuradhaKaruppiah @dagardner-nv
    AnuradhaKaruppiah and dagardner-nv committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    3028977 View commit details
    Browse the repository at this point in the history

  23. Update tests/_utils/test_directories.py 

    Co-authored-by: David Gardner <[email protected]>
    @AnuradhaKaruppiah @dagardner-nv
    AnuradhaKaruppiah and dagardner-nv committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    3c1b408 View commit details
    Browse the repository at this point in the history

  24. Update tests/morpheus/conftest.py 

    Co-authored-by: David Gardner <[email protected]>
    @AnuradhaKaruppiah @dagardner-nv
    AnuradhaKaruppiah and dagardner-nv committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    3622020 View commit details
    Browse the repository at this point in the history

  25. Update tests/morpheus/conftest.py 

    Co-authored-by: David Gardner <[email protected]>
    @AnuradhaKaruppiah @dagardner-nv
    AnuradhaKaruppiah and dagardner-nv committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    d115b95 View commit details
    Browse the repository at this point in the history