Forward-merge branch-24.10 into branch-25.02 #2045
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…RC (nv-morpheus#1981) Depends on PR nv-morpheus/MRC#511 being completed, merged, and a new conda package uploaded before testing can begin. Fixes a small issue where MRC updated a function to be private to avoid incorrect usage of it. Authors: - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) - Christopher Harris (https://github.com/cwharris) URL: nv-morpheus#1981
Closes nv-morpheus#1973 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1975
* Adds `docs/source/basics/cpu_only_mode.md` Closes nv-morpheus#1959 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: nv-morpheus#1969
…tage` and `HttpClientSourceStage` (nv-morpheus#1834) * Add optional `message_type`, `task_type`, `task_payload` arguments to `HttpServerSourceStage` & `HttpClientSourceStage`. * Add new `ConfigurableOutputSource` base class for single output source stages which support both `MessageMeta` and `ControlMessage` as output types. * When `ControlMessage` output is selected, HTTP request headers will be added to the message's metadata. * Define new request handler callback method `request_handler_fn_t` which receives the boost asio tcp endpoint along with the request object. * `HttpEndpoint` now optionally receives a `request_handler_fn_t` method, and for compatibility the older `payload_parse_fn_t` method. * Add an optional `include_headers` argument to `HttpEndpointInterfaceProxy::init`, which when `False` (the default) uses the existing `payload_parse_fn_t` method, and `request_handler_fn_t` when `True`. This changes the function signature, but not the behavior when the `include_headers` argument is omitted, so technically not an API breaking change. * Move/rename `cm_task_t`-->`control_message_task_t` from `deserialize.hpp` to be shared with other stages * Don't use `accept_status` for live and ready endpoints, as these should always return 200 on success, while `accept_status` defaults to `201 CREATED` which isn't appropriate for a health check. Closes nv-morpheus#1811 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1834
This change was made because of check-style failures when tests were refactored but is causing import failures. Note: This commit also reverts the style fix and the local imports are mis-sorted. We may need to update the style checker to fix it post release. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: nv-morpheus#1978
…orpheus#1982) Docs update Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: nv-morpheus#1982
* Add missing `tensorrt-cu12` dependency to the `model-utils` environment. * Update `onnx-to-trt` to function with TensorRT v10, removed usage of attributes/methods which were deprecated in v8.4 and removed in v10. * Update documentation Closes nv-morpheus#1979 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) - https://github.com/hsin-c URL: nv-morpheus#1984
Update `val-run-all.sh` to run cpp pipeline only and do not run in Python. Closes nv-morpheus#1985 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Yuchen Zhang (https://github.com/yczhang-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: nv-morpheus#1986
…encies (nv-morpheus#1974) - Update `dependencies.yaml` and re-generate environment yaml's - Avoids install of `pytorch` cpu packages which causes examples like DFP try to use. Closes nv-morpheus#1943 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Eli Fajardo (https://github.com/efajardo-nv) - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1974
The utilities submodule commit is referencing something that doesn't exist on `nv-morphues/utilities`. This PR fixes that. ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1987
…ase container (nv-morpheus#1976) Move the data files needed for log-parsing from models to examples/data to avoid copying models to the release container Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: nv-morpheus#1976
* Agents: Show an actual runnable command that can be copy/pasted, not just the command usage. * Completion: Remove blank line in command block * RAG: Remove section about fetching data, RAG doesn't need any LFS files. * VDB Upload: Pipeline requires the `examples` dataset not the `datasets` dataset. * VDB Upload: Document how to run Milvus Closes nv-morpheus#1991 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: nv-morpheus#1992
Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: nv-morpheus#1998
- Dependencies for DFP container were recently moved from `examples/digital_fingerprinting/production/conda_env.yaml` to `conda/environments/examples_cuda-125_arch-x86_64.yaml` but `python-logging-loki` was not included. - Add `python-logging-loki` to `example-dfp-prod` in `dependencies.yaml`. - Remove `conda_env.yaml` since it's no longer being used anywhere. Closes nv-morpheus#1917 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Eli Fajardo (https://github.com/efajardo-nv) - David Gardner (https://github.com/dagardner-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: nv-morpheus#1977
…#1995) * Fixes an issue where an `int` is being returned in the DFP rolling window stage, likely caused by the recent update to the version of pandas being used. * Update file paths to reflect the move of the `dfp_duo_pipeline.py`, `dfp_azure_pipeline.py` & `dfp_integrated_training_batch_pipeline.py` scripts * Document installing the missing dependencies needed for the DFP Grafana example Closes nv-morpheus#1994 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: nv-morpheus#1995
…-morpheus#2004) Closes nv-morpheus#1934 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#2004
This is to account for the changes done during DFP refactoring Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#2003
* The deserialize stage no longer actually deserializes, but instead chunks incoming `MessageMeta` objects and converts to `ControlMessage` Closes nv-morpheus#2007 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: nv-morpheus#2009
Closes nv-morpheus#1999 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Yuchen Zhang (https://github.com/yczhang-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: nv-morpheus#2008
…v-morpheus#2006) Closes nv-morpheus#2002 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) - Christopher Harris (https://github.com/cwharris) URL: nv-morpheus#2006
…nv-morpheus#2016) Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: nv-morpheus#2016
Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: nv-morpheus#2017
…e correct rows for each batch (nv-morpheus#2015) * `DeserializeStage` now performs a copy of the rows needed for each batch, ensuring each output `ControlMessage` now has a unique `MessageMeta` and underlying `DataFrame` * Works-around issue where calling `SlicedMessageMeta::get_mutable_info().checkout_obj()` returns the entire `DataFrame` * This unfortunately means that `DeserializeStage` will require the GIL. * Remove unused `make_output_message` method * Avoid some redundant code in `SlicedMessageMeta` (unrelated but I was in this part of the code). Closes nv-morpheus#2002 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Christopher Harris (https://github.com/cwharris) URL: nv-morpheus#2015
…1971) Earlier the morpheus-pipeline stage was using the morpheus-runtime container as the base image. This change switches the base container to nvidia/cuda and installs the morpheus-dfp library. There is a minor docs fixup in conda_packages.md that I pull in based on the review comments. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: nv-morpheus#1971
…s#2001) * Updates the version of the Triton Inference server container being used by Morpheus from `23.06` to `24.09`. * Includes a re-trained `ransomw-model-short-rf` (PR nv-morpheus#2023), the original model was not compatible with Triton 24.09 * Update the output dimensions of the `log-parsing-onnx` model per nv-morpheus#1727 Closes nv-morpheus#2000 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) - Tad ZeMicheal (https://github.com/tzemicheal) Approvers: - https://github.com/raykallen - Christopher Harris (https://github.com/cwharris) - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#2001
Requires changes from MRC PR: nv-morpheus/MRC#502 Closes nv-morpheus#1519 Authors: - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: nv-morpheus#1963
This bug is caused by the transition from `MultiMessage` to `ControlMessage`. `inference_stage.py::InferenceStage::_build_single` calls `_convert_one_response` in a loop for a batch, and the argument is passing is the same for the whole batch, but inside `_convert_one_response` we're grabbing the tensors and assigning starting at zero, so the tensors are overwriting each other and causes the issue. Added `batch_offset` variable to keep track where the next incoming tensor should start writing to the output message. Closes nv-morpheus#2019 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Yuchen Zhang (https://github.com/yczhang-nv) - David Gardner (https://github.com/dagardner-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) - Christopher Harris (https://github.com/cwharris) URL: nv-morpheus#2024
* Use the index of the incoming DF as the source of the `doc` field not the first column of the sequence_ids * Fix typeo in docstring for `TableInfo::num_indices` (unrelated) * Remove declaration for unimplemtned/unused function in `python/morpheus/morpheus/_lib/include/morpheus/stages/deserialize.hpp` (unrelated) Closes nv-morpheus#2029 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: nv-morpheus#2031
- Fixes benchmark tests that were broken from latest code updates - Add `pynvml` dependency back to conda dev environment. It was removed when `nvtabular` was removed. - Remove broken E2E autoencoder test since Starter DFP is being removed. ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Eli Fajardo (https://github.com/efajardo-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: nv-morpheus#1958
…rpheus#2032) Closes nv-morpheus#2030 This is needed to run root_cause_analysis in the release container. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Christopher Harris (https://github.com/cwharris) URL: nv-morpheus#2032
See nv-morpheus#2026 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: nv-morpheus#2027
At some point these got out of sync where the CLI was using add-scores, but the Python API was using the filter stage, and some of the docs were using add-scores as well ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: nv-morpheus#2037
…rpheus#2036) Closes ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - https://github.com/raykallen URL: nv-morpheus#2036
…rors (nv-morpheus#2039) Closes nv-morpheus#2038 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: nv-morpheus#2039
This reverts commit 599e73f.
AnuradhaKaruppiah
approved these changes
Nov 5, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Manually resolving forward merge conflicts from PR #1983
By Submitting this PR I confirm: