[test] fix transformers neuronx integration test failure

[sindhuvahinis]

Description

EDIT: Even tiny models have higher context length. So, we trained model and tokenizer to have smaller context length 4k on runtime. I trained the model offline and uploaded it to s3. So the previous fix is not needed.

---

This should fix the tnx unit tests in integration tests failures. This is due to transformers getting upgraded to 4.45.2. This does not affect our handlers. This only affects, train_new_from_iterator in transformers.

Raised PR in transformers to fix it. huggingface/transformers#34661

[siddvenk]

Do we need 4.45.2 for neuron at the moment? (I think yes for multimodal?)

[siddvenk]

where does this come from?

[sindhuvahinis]

This comes from Llama 3.2 1B tokenizer.json. https://huggingface.co/meta-llama/Llama-3.2-1B/blob/main/tokenizer.json.
Here is the raw json version. Here i just converted into Object.

  "pre_tokenizer": {
    "type": "Sequence",
    "pretokenizers": [
      {
        "type": "Split",
        "pattern": {
          "Regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
        },
        "behavior": "Isolated",
        "invert": false
      },
      {
        "type": "ByteLevel",
        "add_prefix_space": false,
        "trim_offsets": true,
        "use_regex": false
      }
    ]
  }

[sindhuvahinis]

Do we need 4.45.2 for neuron at the moment? (I think yes for multimodal?)

Yes. For vllm 0.6.2 version, it is needed. #2518 MultiModal support is also based on vllm 0.6.2

[siddvenk]

spoke offline - we're going to look into whether we need to do the tokenizer training stuff we currently do. If we can just use the tiny llama tokenizer directly, then that simplifies things here.