You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
First of all: Amazing work. I love the concept and project. I played around a little and I tried the validation using validate_with_llm, but the validation always fails in my example:
defmoduleKeyInsightdouseEcto.SchemauseInstructor.Validator@doc""" A key insight a user had ## Fields: - summary: A short summary of the key insight, addressing the user in the second person singular. 50 words max. """@primary_keyfalseembedded_schemadofield(:summary,:string)end@impltruedefvalidate_changeset(changeset)dochangeset|>validate_with_llm(:summary,"Must contain the words 'you' or 'your'")endendInstructor.chat_completion(model: "gpt-4o",response_model: KeyInsight,max_retries: 3,messages: [%{role: "user",content:""" Find all key insights in this text. Your reply has to be in the second person singular, addressing the user. Text: Today I realized that using my phone so much just kills my productivity and I plan to keep the phone in a drawer while I'm working. I also want to understand the dopamine system and how it affects motivation better. """}])
Output:
15:21:26.325 [debug] Retrying LLM call for KeyInsight:
"summary - is invalid, The statement does not contain the words 'you' or 'your'."
15:21:27.930 [debug] Retrying LLM call for KeyInsight:
"summary - is invalid, The statement does not contain the words 'you' or 'your' as required."
15:21:29.824 [debug] Retrying LLM call for KeyInsight:
"summary - is invalid, The statement does not contain the words 'you' or 'your' as required by the rule."
{:error,
#Ecto.Changeset<
action: nil,
changes: %{
summary: "You see that excessive phone use damages your productivity, so you're planning to keep it in a drawer while you work. You're also eager to learn more about the dopamine system and its effect on your level of motivation."
},
errors: [summary: {"is invalid, The statement does not contain the words 'you' or 'your'.", []}],
data: #KeyInsight<>,
valid?: false,
...
>}
Any idea what is going on here? I think it is pretty clear that this validation should pass. Thank you!
The text was updated successfully, but these errors were encountered:
ospaarmann
changed the title
validate_with_llm doesn't properly work
validate_with_llm always fails for my example
Dec 7, 2024
First of all: Amazing work. I love the concept and project. I played around a little and I tried the validation using
validate_with_llm
, but the validation always fails in my example:Output:
Any idea what is going on here? I think it is pretty clear that this validation should pass. Thank you!
The text was updated successfully, but these errors were encountered: