IndexError: list index out of range when running evaluation on certain datasets

[isaacna]

Has anyone ever ran into a similar issue to this while running evaluations on some tasks? For context, I'm running into this issue while running the DBPedia retrieval task, but am not seeing it for tasks such NFCorpus.

  File "/opt/anaconda3/envs/table/lib/python3.11/site-packages/mteb/evaluation/MTEB.py", line 461, in run
    raise e
  File "/opt/anaconda3/envs/table/lib/python3.11/site-packages/mteb/evaluation/MTEB.py", line 422, in run
    results, tick, tock = self._run_eval(
                          ^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/table/lib/python3.11/site-packages/mteb/evaluation/MTEB.py", line 299, in _run_eval
    results = task.evaluate(
              ^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/table/lib/python3.11/site-packages/mteb/abstasks/AbsTaskRetrieval.py", line 299, in evaluate
    scores[hf_subset] = self._evaluate_subset(
                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/table/lib/python3.11/site-packages/mteb/abstasks/AbsTaskRetrieval.py", line 308, in _evaluate_subset
    results = retriever(corpus, queries)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/table/lib/python3.11/site-packages/mteb/evaluation/evaluators/RetrievalEvaluator.py", line 501, in __call__
    return self.retriever.search(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/table/lib/python3.11/site-packages/mteb/evaluation/evaluators/RetrievalEvaluator.py", line 191, in search
    corpus_id = corpus_ids[corpus_start_idx + sub_corpus_id]
                ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: list index out of range

[isaacna]

Found the root cause. I didn't realize that the RetrievalEvaluator was doing some batching under the hood for encode_corpus calls. I was doing some caching locally to store embedding results but this was causing some mismatch issues with the batching