Fix Infinite Repetition in JSON Schemas Using Integer and String

[lapp0]

Overview

The repetition problem of language models combined with patterns allowing for infinite-length fields results in broken JSON Schema outputs.

This was addressed previously for infinite whitespaces issues by setting a safe whitespace pattern as the default. In this PR, the safety of whitespaces is extended to Integer and String patterns.

Behavior

json_schema.to_regex now includes a kwarg safe_subset=True.

safe_subset=False

• Whitespace: r"[\n\t ]*"
• Integer: any number
• String: any string

safe_subset=True (default)

• Whitespace: r"[ ]?"
• Integer: (-1e19, 1e19)
• String: Any string of length (0, 256)

Fixes

Safe Integer

• Fixes llama_cpp - JSON fails to generate when using Pydantic model with models.llama_cpp #1110
• Fixes <Please write a descriptive title> #1099

Safe String

• Fixes The example provided by the official source has crashed. #1075
• Addresses Add a non_strict mode to deal with JSONDecodeError errors #985 (doesn't fix, they requested a non_strict mode)
• Fixes Accelerate not installed in Docker Image #1106

Further Work

• Important: In the resolved issues, the incorrect outputs are often caused by not applying a chat template. Let's help users get great completions. Examples should include chat templates, or user response quality will suffer. Format prompts Using Chat Templates in SequenceGeneratorAdapter #987

• Make code more failsafe: Add a non_strict mode to deal with JSONDecodeError errors #985

• number has no safe_subset implementation. It's likely the only unsafe primitive remaining without a safe_subset implementation. However, there aren't any open issues for an error caused by number.

[lapp0]

Probably should be at the top. We might also consider separating into multiple modules so it's easier to read

[rlouf]

Yes, I generally like to move helper functions to the end of the file.

[rlouf]

• Important: In the resolved issues, the incorrect outputs are often caused by not applying a chat template. Let's help users get great completions. Examples should include chat templates, or user response quality will suffer. Format prompts Using Chat Templates in SequenceGeneratorAdapter #987

@cpfiffer fyi

[lapp0]

We might want to hold off on this one actually. I did some profiling on get_str_pattern. Constrained strings have a large state and take a long time to compile.

>>> len(interegular.parse_pattern(STRING_INNER + "{,256}").to_fsm().reduce().states)
513
>>> len(interegular.parse_pattern(STRING_INNER + "*").to_fsm().reduce().states)
2

The better alternative is to

• now: Update this PR, enable safe_subset as an optional parameter non-default parameter
• soon: Reduce failure rate via
  • update docs, recommend pydantic constr by default, not str
  • Format prompts Using Chat Templates in SequenceGeneratorAdapter #987
  • Add a non_strict mode to deal with JSONDecodeError errors #985
• later: integrate this functionality into an automata which has a state stack, CFG or otherwise.

Alternatively we could reduce the size of safe_subset str to something like 20 instead of 256.

Let me know if this makes sense