Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(api): openai compliant annotations and vector_content retrieval #1164

Merged
merged 25 commits into from
Sep 30, 2024

Conversation

gphorvath
Copy link

@gphorvath gphorvath commented Sep 29, 2024

Description

Ensures our annotations pass conformance with OpenAI's API spec and provides a method of getting chunk/vector information via the LeapfrogAI namespace.

BREAKING CHANGES

Minor: Changed the format of annotation replacement text to match OpenAI spec.

CHANGES

  • Adds get_vector endpoint that can be used to retrieve rows from the vector_content table (contains chunk text, embeddings, etc.)
  • Adds crud operation for reading the rows of the vector_content table
  • Refactors composer and converter to use vector_content instead of file_id's to make it easier to return vector_id
  • Adds metadata type to help with converting dict --> object --> dict
  • Move Vector type out of crud into typedef
  • Adds a test client for testing api via rest calls (used for new tests in this PR)
  • Adds integration tests for leapfrogai/vector_stores
  • Updates tools/annotations conformance test to match on filename or source (sometimes OpenAI still returns source for some reason)

Related Issue

Addresses #766

Checklist before merging

@gphorvath gphorvath added the enhancement New feature or request label Sep 29, 2024
@gphorvath gphorvath self-assigned this Sep 29, 2024
@gphorvath gphorvath requested a review from a team as a code owner September 29, 2024 18:50
Copy link

netlify bot commented Sep 29, 2024

Deploy Preview for leapfrogai-docs canceled.

Name Link
🔨 Latest commit e3626ca
🔍 Latest deploy log https://app.netlify.com/sites/leapfrogai-docs/deploys/66facdaa0970540008560710

@gphorvath gphorvath changed the base branch from main to refactor-api-test-data-path September 29, 2024 22:49
justinthelaw
justinthelaw previously approved these changes Sep 30, 2024
Copy link
Contributor

@justinthelaw justinthelaw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as @andrewrisse is good with this API, then I am! Everything looks good to me.

jalling97
jalling97 previously approved these changes Sep 30, 2024
Copy link
Contributor

@jalling97 jalling97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed in a test UDS deployment that the vector endpoint works as expected and annotations are properly updated. Thanks for doing the legwork on this @gphorvath!

Base automatically changed from refactor-api-test-data-path to main September 30, 2024 15:44
@gphorvath gphorvath dismissed stale reviews from jalling97 and justinthelaw September 30, 2024 15:44

The base branch was changed.

@gphorvath gphorvath enabled auto-merge (squash) September 30, 2024 16:15
@gphorvath gphorvath merged commit 96d89f0 into main Sep 30, 2024
24 checks passed
@gphorvath gphorvath deleted the 766-annotations branch September 30, 2024 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat(api): return more annotation details
4 participants