-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add OpenSearch Binary & Byte data_type support #17476
base: main
Are you sure you want to change the base?
feat: Add OpenSearch Binary & Byte data_type support #17476
Conversation
@@ -269,7 +269,7 @@ class BaseNode(BaseComponent): | |||
id_: str = Field( | |||
default_factory=lambda: str(uuid.uuid4()), description="Unique ID of the node." | |||
) | |||
embedding: Optional[List[float]] = Field( | |||
embedding: Optional[List[Union[float, int]]] = Field( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not 100% sure how I feel about changing this. But I guess it makes sense. This will likely cause some mypy errors? Lets see the linting output
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a low-level change that I wasn't expecting to change. Any other changes to the core schema would be a more considerable breaking change if we wanted to add an embedding_type field and switch based on that.
Do you have any thoughts on how we should proceed? I suspect we'll see more need to add byte or binary support in the future.
Not sure whats up with tests... investigating UPDATE: fixed the testing weirdness UPDATE: jk still weird |
Description
This PR adds support for byte and binary vector storage in the OpenSearch vector store integration, leveraging OpenSearch 2.17's new vector capabilities. This enhancement enables more efficient vector storage by supporting 8-bit integers (-128 to 127) instead of only floats, significantly reducing storage requirements while maintaining search quality.
Fixes #17271
New Package?
Did I fill in the
tool.llamahub
section in thepyproject.toml
and provide a detailed README.md for my new integration or package?Version Bump?
Did I bump the version in the
pyproject.toml
file of the package I am updating? (Except for thellama-index-core
package)Type of Change
Please delete options that are not relevant.
How Has This Been Tested?
Your pull-request will likely not be merged unless it is covered by some form of impactful unit testing.
Added comprehensive test suite including:
Suggested Checklist:
make format; make lint
to appease the lint gods