Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement rag prototpye #22

Merged
merged 32 commits into from
Oct 23, 2024
Merged

Implement rag prototpye #22

merged 32 commits into from
Oct 23, 2024

Conversation

kevin-on
Copy link
Collaborator

No description provided.

@kevin-on
Copy link
Collaborator Author

kevin-on commented Oct 20, 2024

TODO Before Merging

  • Style Improvements
    • Update MarkdownReferenceBlock styles (MarkdownReferenceBlock.tsx)
    • Update QueryProgress component style (QueryProgress.tsx)
    • Update MentionableBadge styles (MentionableBadge.tsx)
    • Add folder and vault icons to MentionTypeaheadOption (MentionPlugin.tsx)
  • Decide vault index update strategy (ragEngine.ts)
  • Add Drizzle ORM for handling DB migration
  • Check ESbuild config
  • FIX MEMORY LEAK
  • Show update progress when re-indexing by command
  • Truncate long chat history in PromptGenerator

Backlog

  • Implement scoring logic for folders and vault in fuzzy search (fuzzy-search.ts)
  • Add similarity boosting for mentioned files/folders in RAG query processing (promptGenerator.ts)
  • Implement automatic vault re-indexing on settings change (ragEngine.ts)
  • Migrate to WebAssembly-based tiktoken for better performance and token-based chunking (token.ts, vector-db/manager.ts)
  • Create HNSW index for faster vector database search (vector-db/repository.ts)

@glowingjade
Copy link
Owner

Can you write a brief CONTRIBUTING.md file that includes instructions on how to use Drizzle, specifically regarding migration generation?
Please use this as a reference:
https://github.com/facebook/lexical/blob/main/CONTRIBUTING.md

src/constants.ts Outdated Show resolved Hide resolved
src/types/settings.ts Show resolved Hide resolved
src/main.ts Outdated Show resolved Hide resolved
src/main.ts Show resolved Hide resolved
drizzle/0001_create_vector_data_tables.sql Show resolved Hide resolved
src/utils/vector-db/manager.ts Outdated Show resolved Hide resolved
src/utils/vector-db/manager.ts Outdated Show resolved Hide resolved
src/utils/vector-db/manager.ts Outdated Show resolved Hide resolved
src/utils/ragEngine.ts Outdated Show resolved Hide resolved
src/utils/vector-db/repository.ts Show resolved Hide resolved
src/contexts/rag-context.tsx Outdated Show resolved Hide resolved
src/utils/vector-db/manager.ts Outdated Show resolved Hide resolved
src/utils/vector-db/repository.ts Show resolved Hide resolved
src/utils/promptGenerator.ts Show resolved Hide resolved
src/utils/promptGenerator.ts Outdated Show resolved Hide resolved
@glowingjade glowingjade merged commit 00c67d3 into main Oct 23, 2024
1 check passed
@glowingjade glowingjade deleted the kevin/rag branch October 23, 2024 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants