Fix Blocking Memory Store Usage in Streamed Mode #1144
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Draft - I would like to implement some tests. |
florian-h05
approved these changes
Dec 9, 2024
|
BTW: I suspect using a |
geoand
approved these changes
Dec 10, 2024
|
@florian-h05 that's why I want to add some tests. I believe it's fine, as it uses a slightly weird manner to switch to another thread (loosing the context). I may adapt it at the same time. |
cescoffier
force-pushed
the
fix-blocking-memory-store-in-streamed-response
branch
from
43e09c6 to
d68fd30
Compare
.../java/io/quarkiverse/langchain4j/runtime/aiservice/AiServiceMethodImplementationSupport.java
Show resolved
Hide resolved
cescoffier
force-pushed
the
fix-blocking-memory-store-in-streamed-response
branch
from
d68fd30 to
28451d3
Compare
geoand
reviewed
Dec 12, 2024
| /** | ||
| * Utility class for streaming tests. | ||
| */ | ||
| public class StreamTestUtils { |
geoand
approved these changes
Dec 12, 2024
This comment has been minimized.
This comment has been minimized.
|
@geoand .... I may have to fix the ordering issue actually, the completion signal got sent before the items. |
|
Darn.... Happened sooner than we expected :( |
This commit addresses issues with using the blocking memory store in streamed responses. * Ensures the execution captures whether the caller is running on a worker thread. * Switches to worker threads for every emission and completion event when the caller is using a worker thread. * Relies on executeBlocking to propagate the context automatically when possible. Note: * The blocking memory store cannot be used when invoked on the event loop. It now requires that the caller must be on a worker thread.
cescoffier
force-pushed
the
fix-blocking-memory-store-in-streamed-response
branch
from
28451d3 to
0690a53
Compare
Status for workflow
|
cescoffier
deleted the
fix-blocking-memory-store-in-streamed-response
branch
|
@cescoffier The same issue occurs when using the retrieval augmentor: |
|
The retrieval augmented is called before we create the stream. Did you try
adding @Blocking?
Unfortunately, I don't believe I will have the time to revisit this soon.
…On Wed 8 Jan 2025 at 15:09, Florian Hotze ***@***.***> wrote:
@cescoffier <https://github.com/cescoffier> The same issue occurs when
using the retrieval augmentor:
org.jboss.resteasy.reactive.common.core.BlockingNotAllowedException
at org.jboss.resteasy.reactive.client.impl.InvocationBuilderImpl.unwrap(InvocationBuilderImpl.java:199)
at org.jboss.resteasy.reactive.client.impl.InvocationBuilderImpl.method(InvocationBuilderImpl.java:328)
at io.quarkiverse.langchain4j.openai.common.OpenAiRestApi$$QuarkusRestClientInterface.blockingEmbedding(Unknown Source)
at io.quarkiverse.langchain4j.openai.common.QuarkusOpenAiClient$5.execute(QuarkusOpenAiClient.java:339)
at io.quarkiverse.langchain4j.openai.common.QuarkusOpenAiClient$5.execute(QuarkusOpenAiClient.java:336)
at io.quarkiverse.langchain4j.azure.openai.AzureOpenAiEmbeddingModel.lambda$embedTexts$0(AzureOpenAiEmbeddingModel.java:119)
at dev.langchain4j.internal.RetryUtils$RetryPolicy.withRetry(RetryUtils.java:192)
at dev.langchain4j.internal.RetryUtils.withRetry(RetryUtils.java:229)
at io.quarkiverse.langchain4j.azure.openai.AzureOpenAiEmbeddingModel.embedTexts(AzureOpenAiEmbeddingModel.java:119)
at io.quarkiverse.langchain4j.azure.openai.AzureOpenAiEmbeddingModel.embedAll(AzureOpenAiEmbeddingModel.java:103)
at dev.langchain4j.model.embedding.EmbeddingModel.embed(EmbeddingModel.java:34)
at dev.langchain4j.model.embedding.EmbeddingModel.embed(EmbeddingModel.java:24)
—
Reply to this email directly, view it on GitHub
<#1144 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADCG7J53QIS7PT2HVRJ2CD2JUWRZAVCNFSM6AAAAABTJLTOASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZXG43DIMBWGM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix #1120
This commit addresses issues with using the blocking memory store in streamed responses.
Note: