Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

validate_with_llm always fails for my example #85

Open
ospaarmann opened this issue Dec 7, 2024 · 0 comments
Open

validate_with_llm always fails for my example #85

ospaarmann opened this issue Dec 7, 2024 · 0 comments

Comments

@ospaarmann
Copy link

First of all: Amazing work. I love the concept and project. I played around a little and I tried the validation using validate_with_llm, but the validation always fails in my example:

defmodule KeyInsight do
  use Ecto.Schema
  use Instructor.Validator

  @doc """
  A key insight a user had

  ## Fields:
  - summary: A short summary of the key insight, addressing the user in the second person singular. 50 words max.
  """
  @primary_key false
  embedded_schema do
    field(:summary, :string)
  end

  @impl true
  def validate_changeset(changeset) do
    changeset
    |> validate_with_llm(:summary, "Must contain the words 'you' or 'your'")
  end
end

Instructor.chat_completion(
  model: "gpt-4o",
  response_model: KeyInsight,
  max_retries: 3,
  messages: [
    %{
      role: "user",
      content:
        """
        Find all key insights in this text. Your reply has to be in the second person singular, 
        addressing the user.

        Text:

        Today I realized that using my phone so much just kills my productivity and I 
        plan to keep the phone in a drawer while I'm working. I also want to understand 
        the dopamine system and how it affects motivation better.
        """
        
    }
  ]
)

Output:


15:21:26.325 [debug] Retrying LLM call for KeyInsight:

 "summary - is invalid, The statement does not contain the words 'you' or 'your'."

15:21:27.930 [debug] Retrying LLM call for KeyInsight:

 "summary - is invalid, The statement does not contain the words 'you' or 'your' as required."

15:21:29.824 [debug] Retrying LLM call for KeyInsight:

 "summary - is invalid, The statement does not contain the words 'you' or 'your' as required by the rule."


{:error,
 #Ecto.Changeset<
   action: nil,
   changes: %{
     summary: "You see that excessive phone use damages your productivity, so you're planning to keep it in a drawer while you work. You're also eager to learn more about the dopamine system and its effect on your level of motivation."
   },
   errors: [summary: {"is invalid, The statement does not contain the words 'you' or 'your'.", []}],
   data: #KeyInsight<>,
   valid?: false,
   ...
 >}

Any idea what is going on here? I think it is pretty clear that this validation should pass. Thank you!

@ospaarmann ospaarmann changed the title validate_with_llm doesn't properly work validate_with_llm always fails for my example Dec 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant