JSON generation example failed with "gpt2"

[lightaime]

Thanks for the great work.

I tried the example - Efficient JSON generation following a Pydantic model in README.md. However, it failed with an error:

{
  "name": " role" { "",
  "age": 30,
  "armor": "chainmail",
  "weapon": "sword",
  "strength": 2
}
Traceback (most recent call last):
  File "/Users/lig/github/outlines/test.py", line 70, in <module>
    parsed = Character.model_validate_json(sequence)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lig/miniconda3/envs/outlines/lib/python3.11/site-packages/pydantic/main.py", line 523, in model_validate_json
    return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for Character
  Invalid JSON: expected `,` or `}` at line 2 column 19 [type=json_invalid, input_value='{\n  "name": " role" { "...d",\n  "strength": 2\n}', input_type=str]
    For further information visit https://errors.pydantic.dev/2.1/v/json_invalid

I also tried it with model = models.transformers("gpt2-medium") and it worked. All the seed I used is 42. Any idea?

I used Python 3.11.4 and here is the pip list:

Package            Version
------------------ ---------
annotated-types    0.5.0
asttokens          2.2.1
beartype           0.15.0
certifi            2023.7.22
charset-normalizer 3.2.0
cloudpickle        2.2.1
filelock           3.12.2
fsspec             2023.6.0
huggingface-hub    0.16.4
icontract          2.6.2
idna               3.4
interegular        0.3.2
Jinja2             3.1.2
lark               1.1.7
MarkupSafe         2.1.3
mpmath             1.3.0
networkx           3.1
numpy              1.25.2
outlines           0.0.8
packaging          23.1
perscache          0.6.1
Pillow             10.0.0
pip                23.2.1
pydantic           2.1.1
pydantic_core      2.4.0
PyYAML             6.0.1
regex              2023.8.8
requests           2.31.0
safetensors        0.3.2
scipy              1.11.1
setuptools         68.0.0
six                1.16.0
sympy              1.12
tenacity           8.2.3
tokenizers         0.13.3
torch              2.0.1
tqdm               4.66.1
transformers       4.31.0
typing_extensions  4.7.1
urllib3            2.0.4
wheel              0.41.1

[mondaychen]

I got another strange json output with "meta-llama/Llama-2-13b-chat-hf"

{
    "name":   "grittty",
    "age":  28,
    "a r mor" :   "p l a t e",
    "w e a p o n":   "b o w",
    "s t r e n g t h":  25
}

It's valid json but did not stick to schema

Not sure if caused by the same thing

[rlouf]

@mondaychen I couldn't reproduce this locally. How do you set the seed? As @mondaychen mentionned offline there is currently an issue with { not being escaped in the regexes that represent the JSON's structure. So it might be that, although I can only know for sure when I can reproduce the original error.

@mondaychen Could this be an issue with decoding the token ids? If you put a breakpoint right before the model decodes what do you get?

[andreamad8]

Same here:

>>> model = models.transformers("meta-llama/Llama-2-13b-chat-hf", device=torch.device('cuda'), torch_dtype=torch.float16, low_cpu_mem_usage=True)
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00,  9.06it/s]
>>> sequence = generate.json(model, Character)("Give me a character description")
>>> print(sequence)
{
    "name":   "grittty",
    "age":  28,
    "a r mor" :   "p l a t e",
    "w e a p o n":   "b o w",
    "s t r e n g t h":  25
}

[brandonwillard]

As with #267, this issue should be fixed by #243. The Llama-related spacing errors are a dupe of #273, so check there for progress.