Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a Vector Database Service to allow stages to read and write to VDBs #1225

Merged
merged 25 commits into from
Oct 13, 2023
Merged

Add a Vector Database Service to allow stages to read and write to VDBs #1225

merged 25 commits into from
Oct 13, 2023

Conversation

bsuryadevara
Copy link
Contributor

@bsuryadevara bsuryadevara commented Sep 26, 2023 •
edited
Loading

Description

Closes #1177

By Submitting this PR I confirm:

  • I am familiar with the Contributing Guidelines.
  • When the PR is ready for review, new or existing tests cover these changes.
  • When the PR is ready for review, the documentation is up to date with these changes.

@bsuryadevara bsuryadevara requested a review from a team as a code owner September 26, 2023 17:54
@bsuryadevara bsuryadevara marked this pull request as draft September 26, 2023 17:54
@bsuryadevara bsuryadevara changed the title Added milvus vdb prototype impl Vector Database Service Impl Sep 26, 2023
@bsuryadevara bsuryadevara self-assigned this Sep 26, 2023
@bsuryadevara bsuryadevara added the feature request New feature or request label Sep 26, 2023
@bsuryadevara
Copy link
Contributor Author

@mdemoret-nv here is the draft PR for MilvusVDB service. LlamaIndex is internally using OpenAI API's not sure how I can avoid that.

@mdemoret-nv mdemoret-nv changed the title Vector Database Service Impl Add a Vector Database Service to allow stages to read and write to VDBs Sep 27, 2023
mdemoret-nv
mdemoret-nv reviewed Sep 27, 2023
View reviewed changes
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good start. Lets move the vector_db_utils into a services submodule. And then create a stage which uses the VectorDbService instance.

morpheus/controllers/vector_db_controller.py Outdated Show resolved Hide resolved
morpheus/controllers/milvus_vector_db_controller.py Outdated Show resolved Hide resolved
morpheus/controllers/vector_db_controller.py Outdated Show resolved Hide resolved
morpheus/controllers/milvus_vector_db_controller.py Outdated Show resolved Hide resolved
morpheus/utils/vector_db_utils.py Outdated Show resolved Hide resolved
morpheus/controllers/milvus_llamaindex_controller.py Outdated Show resolved Hide resolved
@bsuryadevara bsuryadevara marked this pull request as ready for review October 11, 2023 18:48
@bsuryadevara bsuryadevara added the non-breaking Non-breaking change label Oct 11, 2023
@bsuryadevara bsuryadevara added the sherlock Issues/PRs related to Sherlock workflows and components label Oct 11, 2023
mdemoret-nv
mdemoret-nv approved these changes Oct 11, 2023
View reviewed changes
docker/conda/environments/cuda11.8_dev.yml Outdated Show resolved Hide resolved
morpheus/service/milvus_vector_db_service.py Outdated Show resolved Hide resolved
morpheus/service/milvus_vector_db_service.py Outdated Show resolved Hide resolved
morpheus/service/milvus_vector_db_service.py Outdated Show resolved Hide resolved
morpheus/service/milvus_vector_db_service.py Outdated Show resolved Hide resolved
tests/conftest.py Outdated Show resolved Hide resolved
tests/test_milvus_vector_db_service.py Outdated Show resolved Hide resolved
@bsuryadevara
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit c320903 into nv-morpheus:branch-23.11 Oct 13, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdemoret-nv mdemoret-nv mdemoret-nv approved these changes

Assignees

@bsuryadevara bsuryadevara

Labels
feature request New feature or request non-breaking Non-breaking change sherlock Issues/PRs related to Sherlock workflows and components
Projects
Morpheus Boards
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA]: Add Milvus vector DB service and sink
2 participants
@bsuryadevara @mdemoret-nv