AI kit does not support Cross-region inference #8121
Comments
|
It's unclear in the docs, but you can solve it by putting the inference profile id in the resourcepath: but you also need to update your lambda (that is invoking bedrock) resource policy: Would be great if this can be adjusted |
|
Hey, thanks for the feedback and information on how this can be solved. Transferring the issue to the documentation repository for updates. |
|
Thanks for opening this @rpostulart. We'll add an example to the docs. In the meantime, here's an example of how you can do this directly within your Amplify backend. Add a custom conversation handlerAdd the In import { defineConversationHandlerFunction } from "@aws-amplify/backend-ai/conversation";
export const crossRegionModel = `eu.${model}`;
export const model = 'anthropic.claude-3-5-sonnet-20240620-v1:0';
export const conversationHandler = defineConversationHandlerFunction({
entry: "./conversationHandler.ts",
name: "conversationHandler",
models: [{ modelId: crossRegionModel }],
});
const schema = a.schema({
chat: a.conversation({
aiModel: {
resourcePath: crossRegionModel,
},
systemPrompt: 'You are a helpful assistant.',
handler: conversationHandler,
)}
.authorization((allow) => allow.owner())
});Create a new file import { handleConversationTurnEvent } from '@aws-amplify/backend-ai/conversation/runtime';
export const handler = handleConversationTurnEvent;In import { defineBackend } from "@aws-amplify/backend";
import { auth } from "./auth/resource";
import { data, conversationHandler, crossRegionModel, model } from "./data/resource";
import { PolicyStatement } from "aws-cdk-lib/aws-iam";
const backend = defineBackend({
auth,
data,
conversationHandler,
});
// This policy statement assumes that you're deploying in `eu-west-1`.
// If that's not the case, adjust the resources block in the policy statements accordingly.
backend.conversationHandler.resources.lambda.addToRolePolicy(
new PolicyStatement({
resources: [
`arn:aws:bedrock:eu-west-1:[account-number]:inference-profile/${crossRegionModel}`,
`arn:aws:bedrock:eu-west-1::foundation-model/${model}`,
`arn:aws:bedrock:eu-west-3::foundation-model/${model}`,
`arn:aws:bedrock:eu-central-1::foundation-model/${model}`,
],
actions: [
'bedrock:InvokeModelWithResponseStream'
],
})
); |
|
ok this is great. I will close the issue with your commitment you will update the docs :) |
Environment information
Describe the bug
In the schema I can only define model like
But i get an error in my region because it only allows access to Claude 3.5 via a inference profile. This is the error in de lambda:
Reproduction steps
Watched the cloudlogs errors, because I didn't get a response in the front end.
The text was updated successfully, but these errors were encountered: