ValueError: not enough values to unpack (expected 2, got 1).

[Itime-ren]

When I used load.py to load the meta tokenizer.model, I encountered the following error while running the function load_tiktoken_bpe: ValueError: not enough values to unpack (expected 2, got 1).
It's that the line.split() in the following code: for line in contents.splitlines() if line only produces a list with a length of 1 or 0.

File "/usr/local/lib/python3.10/site-packages/llama_models/llama3/api/tokenizer.py", line 77, in __init__
    mergeable_ranks = load_tiktoken_bpe(model_path)
  File "/usr/local/lib/python3.10/site-packages/tiktoken/load.py", line 145, in load_tiktoken_bpe
    return {
  File "/usr/local/lib/python3.10/site-packages/tiktoken/load.py", line 147, in <dictcomp>
    for token, rank in (line.split() for line in contents.splitlines() if line)
ValueError: not enough values to unpack (expected 2, got 1)
`

def load_tiktoken_bpe(tiktoken_bpe_file: str, expected_hash: str | None = None) -> dict[bytes, int]:
    # NB: do not add caching to this function
    contents = read_file_cached(tiktoken_bpe_file, expected_hash)
    return {
        base64.b64decode(token): int(rank)
        for token, rank in (line.split() for line in contents.splitlines() if line)

[Haroldhy]

I occurred same error

[Kamichanw]

I occurred same error

[JohnDoe2025-ai]

The ValueError: not enough values to unpack (expected 2, got 1) typically occurs when you attempt to unpack a single value into multiple variables. This often happens when you're trying to unpack a tuple or iterable that doesn't have the number of elements you expect.

For example, consider the following code:

x, y = (1,)

This will raise the ValueError: not enough values to unpack (expected 2, got 1) error because there is only one element in the tuple, but you're trying to unpack it into two variables (x and y).

Possible Causes and Solutions:

1. Single value when expecting multiple:

   • Ensure that the iterable (like a list or tuple) you're unpacking has the correct number of elements.

   Fix:

   x, y = (1, 2)  # Correct, as there are 2 elements

2. Returning a single value when expecting multiple:

   • Check the function or operation that you're unpacking. If it only returns a single value, you'll need to adjust the unpacking to expect one value, or modify the function to return multiple values.

   Example:

   result = some_function()
   x, y = result  # Error if result is a single value

   Fix:

   result = some_function()
   if isinstance(result, tuple) and len(result) == 2:
       x, y = result
   else:
       # Handle the case where result doesn't have 2 elements
       pass

3. Incorrect unpacking with functions like zip():

   • When using functions like zip(), ensure that the iterables being zipped have the correct number of elements. If zip() produces a tuple of a different size than expected, you will encounter this error.

   Example:

   a = [1, 2, 3]
   b = [4, 5]
   for x, y in zip(a, b):
       print(x, y)  # This will only iterate over the shortest iterable

4. Check for default values when unpacking:

   • If you're using a default value in case of missing items, make sure it matches the number of variables you're trying to unpack.

   Fix:

   a = [1, 2]
   x, y = a if len(a) == 2 else (a[0], None)  # Use default value if necessary

If you're working with specific code, feel free to share it, and I can help you pinpoint the issue!

[DenisBalan]

same issue, any upds?

[tadinhkien99]

@DenisBalan I had same issue and took my few days, then I realize that we need to install sentencepiece.
pip install sentencepiece==0.2.0