Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Qurator Omni: initial public release #4032

Merged
merged 121 commits into from
Oct 10, 2024
Merged
Changes from 1 commit
Commits
Show all changes
121 commits
Select commit Hold shift + click to select a range
595fbc9
scaffold
nl0 Jun 21, 2024
a1ab626
bump aws-sdk
nl0 Jun 23, 2024
02e0321
expose bedrock client
nl0 Jun 23, 2024
39b3179
chat with tool calling
nl0 Jun 23, 2024
599ab66
rename to Assistant
nl0 Jun 23, 2024
c5dcdf3
injectable context
nl0 Jun 23, 2024
bceb7df
assistant context for file page
nl0 Jun 24, 2024
a338236
chat -> button to trigger summarize prompt
nl0 Jun 24, 2024
8b1125c
add effect and @effect/schema
nl0 Jun 24, 2024
1a05a4d
refactor context management
nl0 Jun 24, 2024
f6dc833
stable context
nl0 Jun 24, 2024
6626385
render assistant ui above layout
nl0 Jun 24, 2024
0d03f72
multi-round tool use
nl0 Jun 24, 2024
2da0ad8
makeTool helper
nl0 Jun 24, 2024
5aa0a54
startSearch tool
nl0 Jun 24, 2024
21524ef
assistant: iterate
nl0 Jun 24, 2024
08e2f11
instrument search page
nl0 Jun 24, 2024
5ca7147
deploy catalog
nl0 Jun 24, 2024
c959d43
add @types/uuid
nl0 Jun 24, 2024
11c52e3
work on prompting
nl0 Jun 25, 2024
e2e4054
search: expose updateUrlState
nl0 Jun 26, 2024
8c5c004
adjust tools
nl0 Jun 26, 2024
994acbc
tsconfig: enable `downlevelIterators`
nl0 Jul 3, 2024
4103a4d
utils/useConstant: TS
nl0 Jul 3, 2024
05dcd49
lightweight Actor-like abstraction
nl0 Jul 3, 2024
a72c251
Actor: further work
nl0 Jul 3, 2024
79a7ccc
assistant model draft
nl0 Jul 5, 2024
58ce3bc
assistant ui scaffolding
nl0 Jul 5, 2024
47f3b26
use new assistant model and ui
nl0 Jul 5, 2024
e6712dd
some basic ui styling
nl0 Jul 8, 2024
6da6414
tool for getting search results
nl0 Jul 8, 2024
314af98
image and document preview tool
nl0 Jul 10, 2024
70ae971
update effect packages
nl0 Jul 11, 2024
72547d2
effect runtime w/ custom logger
nl0 Jul 11, 2024
d9eb7f4
use custom runtime
nl0 Jul 11, 2024
3a93ebf
use claude 3.5
nl0 Jul 12, 2024
b56e4f9
add stately inspector as a dev dependency
nl0 Jul 15, 2024
2cfe5d4
logging utils for effect
nl0 Jul 16, 2024
7e0749a
utils/Effect: remove logging logic
nl0 Jul 16, 2024
a6ddfca
further work, refactoring, instrumentation
nl0 Jul 16, 2024
b0b4af1
refactor a little
nl0 Jul 16, 2024
4cffe8b
preview tool: scoped logging
nl0 Jul 17, 2024
d7c0d0c
fix context / tools updates
nl0 Jul 18, 2024
19bb829
allow tool schemas with differing I and A types
nl0 Jul 18, 2024
13c937f
expose more comprehensive set of search params to the assistant
nl0 Jul 18, 2024
d4dc1ee
bump @types/react-router
nl0 Jul 19, 2024
aefad0e
refactor routes for more precision and less boilerplate
nl0 Jul 19, 2024
39c450e
add @effect/platform and upgrade other effect packages
nl0 Jul 24, 2024
3f10a3d
navigation tool wip
nl0 Jul 24, 2024
3d85f74
split navigation into reusable parts and instantiation
nl0 Jul 24, 2024
5f8c4fb
comprehensive search url schema
nl0 Jul 31, 2024
faef06d
move stuff around
nl0 Jul 31, 2024
8476e6f
expose route/location info to the assistant
nl0 Jul 31, 2024
495eb10
move stuff around
nl0 Aug 2, 2024
e7dfdb8
reimplement search context
nl0 Aug 2, 2024
ea31868
context: add markers
nl0 Aug 2, 2024
f7f7b78
add searchResultsReady marker
nl0 Aug 2, 2024
e6ea3a2
wait for markers
nl0 Aug 2, 2024
e7e2b96
move stuff around
nl0 Aug 3, 2024
5e12608
move more stuff around
nl0 Aug 3, 2024
a47d946
expose current time
nl0 Aug 3, 2024
713197d
use XML module more
nl0 Aug 3, 2024
99df3e0
mv buckets context to global
nl0 Aug 3, 2024
0c54a2d
graphql codegen config: exclude unnecessary files
nl0 Aug 5, 2024
2374019
fix user meta filters encoding
nl0 Aug 5, 2024
dd708b1
export routes, use XML
nl0 Aug 5, 2024
89b8d62
route en/decoding test
nl0 Aug 5, 2024
5555219
include extents for user meta facets into base search query
nl0 Aug 5, 2024
103c27f
iterate on ui
nl0 Aug 5, 2024
105cdca
discard messages
nl0 Aug 6, 2024
fa73d75
discard tool use; more timestamps
nl0 Aug 6, 2024
324d0f3
JsonDisplay: TSify, fix an issue on expansion (supposedly)
nl0 Aug 7, 2024
3fdec3c
remove obsolete comments
nl0 Aug 7, 2024
e614e7c
JsonDisplay: allow omitting name
nl0 Aug 9, 2024
a1e805b
rm obsolete @ts-expect-error
nl0 Aug 9, 2024
aa83eb2
abort calls; refactor
nl0 Aug 9, 2024
8ecac84
scroll to new messages
nl0 Aug 9, 2024
1209b8d
distinct tool use styles
nl0 Aug 9, 2024
6c1bbbe
utils/Navigation: exact & strict, fromPathParams
nl0 Aug 14, 2024
acf1da4
preview: treat ipynb as text; dedupe object names
nl0 Aug 14, 2024
5885d1b
instrument s3 object route; cleanup
nl0 Aug 14, 2024
0290883
switch back to calude 3 to support documents
nl0 Aug 14, 2024
63f8910
add path-to-regexp package
nl0 Aug 16, 2024
22d1b9f
utils/XML: export types
nl0 Aug 16, 2024
cc584f5
utils/Navigation: use path-to-regexp directly do work around encoding…
nl0 Aug 16, 2024
aef5ba1
more reliable marker handling
nl0 Aug 16, 2024
cb21ebc
FileAssistantContext: tsx -> ts
nl0 Aug 16, 2024
d5e84a0
instrument directory route
nl0 Aug 16, 2024
82ad7eb
feat: update Bedrock model to Claude 3.5 Sonnet
nl0 Oct 3, 2024
2ae30f5
update effect
nl0 Oct 4, 2024
00e91e5
update snapshots
nl0 Oct 4, 2024
783e689
feat: add indexedContent field to SearchHitObject type
nl0 Oct 4, 2024
672da1c
feat: truncate indexed content in search results
nl0 Oct 4, 2024
9b7d5b4
update effect
nl0 Oct 7, 2024
0ce8eec
style: capitalize titles and improve descriptions in search filters
nl0 Oct 7, 2024
4f64926
Merge branch 'master' into qurator-omni
nl0 Oct 7, 2024
e1dacb8
update snapshot
nl0 Oct 7, 2024
4b13363
style: remove background pattern from Assistant UI sidebar
nl0 Oct 8, 2024
13d37fd
rm bg
nl0 Oct 8, 2024
bb4449c
refactor: remove icons from message blocks
nl0 Oct 8, 2024
3d0e438
feat: add faint background to assistant messages
nl0 Oct 8, 2024
c251a5c
feat: update chat UI colors and styling
nl0 Oct 8, 2024
e7f884d
refactor: update MessageContainer to use color and align props
nl0 Oct 8, 2024
8837bc2
refactor: improve CSS specificity and simplify component props
nl0 Oct 8, 2024
ad5fa7b
refactor: move timestamps and actions outside message boxes
nl0 Oct 8, 2024
6bf0cef
style: align footer with colored content boxes in chat UI
nl0 Oct 8, 2024
7aa0264
style: adjust message container padding and action opacity
nl0 Oct 8, 2024
b09a6bd
style: adjust opacity for secondary text elements in JsonDisplay
nl0 Oct 8, 2024
d6ad3aa
style: redesign chat UI and input component
nl0 Oct 8, 2024
e2ba58e
style: update chat UI colors and input styling
nl0 Oct 8, 2024
1ba8262
style: adjust chat UI layout and colors
nl0 Oct 9, 2024
4b0e4aa
Merge branch 'master' into qurator-omni
nl0 Oct 10, 2024
61f5cd1
feat: add conditional rendering for Assistant features
nl0 Oct 10, 2024
856764c
docs: Update Qurator AI Assistant description to use Amazon Bedrock
nl0 Oct 10, 2024
98f3ebf
refactor: remove console.log statements from Context component
nl0 Oct 10, 2024
cd52f78
feat: add AI summarization button for package files
nl0 Oct 10, 2024
14f1a02
refactor: Remove commented out Qurator section in FileDisplay
nl0 Oct 10, 2024
07b7f6a
Merge branch 'master' into qurator-omni
nl0 Oct 10, 2024
e13ec46
changelog
nl0 Oct 10, 2024
89daea6
revert deployment changes
nl0 Oct 10, 2024
1e90cfa
Merge branch 'master' into qurator-omni
nl0 Oct 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
chat with tool calling
nl0 committed Jun 24, 2024
commit 39b31793ccb55438cf0d330540398796bbc61708
238 changes: 234 additions & 4 deletions catalog/app/components/Omni/Omni.tsx
Original file line number Diff line number Diff line change
@@ -1,15 +1,20 @@
import type { Types as BedrockTypes } from 'aws-sdk/clients/bedrockruntime'
import invariant from 'invariant'

import * as React from 'react'
import * as M from '@material-ui/core'

import Chat from 'components/Chat'
import * as style from 'constants/style'
import * as AWS from 'utils/AWS'

interface Omni {
interface OmniCtx {
isOpen: boolean
open: () => void
close: () => void
}

const Ctx = React.createContext<Omni | null>(null)
const Ctx = React.createContext<OmniCtx | null>(null)

export function Provider({ children }: React.PropsWithChildren<{}>) {
const [isOpen, setOpen] = React.useState(false)
@@ -19,10 +24,235 @@ export function Provider({ children }: React.PropsWithChildren<{}>) {
return <Ctx.Provider value={value}>{children}</Ctx.Provider>
}

const SYSTEM_PROMPT = `
You are Qurator -- Quilt Data's AI Assistant.
You are a conservative and creative scientist.
When asked a question about Quilt, refer to the documentation at https://docs.quiltdata.com.
For cross-account bucket policies, see https://docs.quiltdata.com/advanced/crossaccount.
Use GitHub flavored Markdown syntax for formatting when appropriate.

Use tools proactively, but don't mention it explicitly, so that it feels transparent.
`

const MODEL_ID = 'anthropic.claude-3-sonnet-20240229-v1:0'
// const MODEL_ID = 'anthropic.claude-3-5-sonnet-20240620-v1:0'

const stringifyContentBlock = (content: BedrockTypes.ContentBlock): string => {
if (content.text) return content.text
// if (content.toolUse) return `tool use: ${JSON.stringify(content.toolUse)}`
// if (content.toolResult) return `tool result: ${JSON.stringify(content.toolResult)}`
return JSON.stringify(content)
}

interface ToolDescriptor<I, O> {
description?: string
schema: {}
fn: (params: I) => O
}

const TOOLS: Record<string, ToolDescriptor<any, any>> = {
getContents: {
description: 'Get the contents of a file',
schema: {
type: 'object',
properties: {
filename: {
type: 'string',
},
},
required: ['filename'],
},
fn: async ({ filename }: { filename: string }) => {
// eslint-disable-next-line no-console
console.log('TOOL: getContents', filename)
return {
filename,
contents: `
This is a package containing DNA sequences that have been
aligned to the human genome.
`,
}
},
},
}

function useOmni() {
const bedrock = AWS.Bedrock.useClient()

const [history, setHistory] = React.useState([] as BedrockTypes.Messages)
const [loading, setLoading] = React.useState(false)

const getSystemPrompt = React.useCallback(
(): BedrockTypes.SystemContentBlocks => [
{ text: SYSTEM_PROMPT },
// TODO: extra context goes here
{
text: 'You are currently viewing the Quilt Catalog page for a file named README.md',
},
],
[],
)

const getToolConfig = React.useCallback(
(): BedrockTypes.ToolConfiguration => ({
tools: Object.entries(TOOLS).map(([name, { description, schema }]) => ({
toolSpec: {
name,
description,
inputSchema: { json: schema },
},
})),
// toolChoice:
}),
[],
)

const converse = React.useCallback(
async (messages: BedrockTypes.Messages) => {
// eslint-disable-next-line no-console
console.log('converse', messages)
const resp = await bedrock
.converse({
modelId: MODEL_ID,
system: getSystemPrompt(),
messages,
toolConfig: getToolConfig(),
// inferenceConfig?: InferenceConfiguration;
// guardrailConfig?: GuardrailConfiguration;
// additionalModelRequestFields?: Document;
// additionalModelResponseFieldPaths?: ConverseRequestAdditionalModelResponseFieldPathsList;
})
.promise()
// eslint-disable-next-line no-console
console.log('converse resp', resp)
return resp
},
[bedrock, getSystemPrompt, getToolConfig],
)

const callTool = React.useCallback(
async (name: string, input: any) => TOOLS[name].fn(input),
[],
)

const converseRec = React.useCallback(
async (messages: BedrockTypes.Messages) => {
const resp = await converse(messages)
if (resp.stopReason !== 'tool_use') return resp

const assistantMsg = resp.output.message
invariant(assistantMsg, 'expected assistant message to be non-empty')

setHistory((h) => h.concat(assistantMsg))

const tus = assistantMsg.content.reduce(
(acc, c) => (c.toolUse ? acc.concat(c.toolUse) : acc),
[] as BedrockTypes.ToolUseBlock[],
)

const toolResultsPs = tus.map(({ toolUseId, name, input }) =>
callTool(name, input)
.then((json) => ({
toolUseId,
content: [{ json }],
}))
.catch((err) => ({
toolUseId,
content: [{ text: `Error: ${err}` }],
status: 'error',
})),
)

const toolResults = await Promise.all(toolResultsPs)

const userMsg: BedrockTypes.Message = {
role: 'user',
content: toolResults.map((toolResult) => ({ toolResult })),
}

setHistory((h) => h.concat(userMsg))

return converse(messages.concat(assistantMsg, userMsg))
},
[converse, callTool],
)

const sendMessage = React.useCallback(
async (text: string) => {
if (loading) return
if (!text) return

const message: BedrockTypes.Message = { role: 'user' as const, content: [{ text }] }

const newHistory = history.concat(message)
setHistory(newHistory)

try {
setLoading(true)
const resp = await converseRec(newHistory)

// output: ConverseOutput;
// stopReason: StopReason;
// usage: TokenUsage;
// metrics: ConverseMetrics;
// additionalModelResponseFields?: Document;
// trace?: ConverseTrace;

const respMsg = resp.output.message || {
role: 'assistant' as const,
content: [{ text: `Empty response. Stop reason: ${resp.stopReason}` }],
}
setHistory((h) => h.concat(respMsg))
} catch (e) {
// eslint-disable-next-line no-console
console.error(e)
setHistory((h) =>
h.concat({
role: 'assistant' as const,
content: [{ text: `Error: ${e}` }],
}),
)
} finally {
setLoading(false)
}
},
[history, loading, converseRec],
)

const normalizedMessages = React.useMemo(
() =>
history.reduce(
(messages, m) =>
messages.concat(
m.content.map((c) => ({
role: m.role as 'user' | 'assistant',
content: stringifyContentBlock(c),
})),
),
[] as AWS.Bedrock.Message[],
),
[history],
)

return { history, sendMessage, loading, messages: normalizedMessages }
}

function Omni() {
const omni = useOmni()

return (
<Chat
history={{ messages: omni.messages }}
initializing={false}
onSubmit={omni.sendMessage}
/>
)
}

const useStyles = M.makeStyles((t) => ({
root: {
padding: t.spacing(2),
width: '40vw',
width: '50vw',
},
}))

@@ -37,7 +267,7 @@ export function Sidebar() {
<M.Drawer anchor="right" open={omni.isOpen} onClose={omni.close}>
<div className={classes.root}>
<M.Typography variant="h4">Qurator</M.Typography>
sup
<Omni />
</div>
</M.Drawer>
</M.MuiThemeProvider>