Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add button for Attachments (Image, PDF, etc.) in Text Chat #52

Open
wants to merge 13 commits into
base: insiders
Choose a base branch
from

Conversation

ashiquemukkil
Copy link

This PR introduces functionality for users to upload attachments (images, PDFs, etc.) along with their text input in the chat interface. The enhancement leverages GPT-4O's multimodal capabilities to effectively process and respond to multimedia queries.

Key Changes

  1. Frontend:
    Added an attachment upload button to the chat UI.
    Display of uploaded files as thumbnails or links in the chat.
  2. Backend:
    Upload API to upload file to blob storage.
Screenshot 2024-12-28 at 5 10 04 PM

Changes required at GPT-RAG-ORCHESTRATOR :

  • Extra plugging to process attachments
  • Need to maintain cache with attachment URL

@ashiquemukkil
Copy link
Author

@ashiquemukkil please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

  • (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@microsoft-github-policy-service agree
  • (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree

@ashiquemukkil ashiquemukkil marked this pull request as draft December 30, 2024 17:40
@ashiquemukkil ashiquemukkil marked this pull request as ready for review December 30, 2024 17:43
@placerda
Copy link
Collaborator

placerda commented Jan 7, 2025

Hi @ashiquemukkil could you list at a high-level what changes will this frontend feature requires in the orchestrator and bicep (for example attachment container creation, role assignments)?

@ashiquemukkil
Copy link
Author

Hi @ashiquemukkil could you list at a high-level what changes will this frontend feature requires in the orchestrator and bicep (for example attachment container creation, role assignments)?

Changes to the Orchestrator:

  1. Include the "file" field (representing the blob name) alongside the existing fields: conversation_id, answer, data_points, and thoughts orc endpoint. The value for "file" should be null if there is no attachment.
  2. Introduce an additional field in the Cosmos DB (cache) to store attachments.
  3. Add a new plugin to extract information from images.

bicep file change

  1. add one more container with the same privilege as 'documents', and add life cycle management to delete data after a day

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants