What is Sequence Group

[mikotohhh]

I am new to llm and trying to understand the source code of vllm, but feel confused about the concept of "sequence" and "sequence group".

I know every request will have its own seqgroup, and a seqgroup may have multiple sequences. But I don't understand the reasons for this mapping. Why not just having a single sequence that pair up with a request?

[liusitan]

@mikotohhh I am newbie in AI. but I have an explanation for that. As you can see from the paper, the sequence group is usually for beam search.

beam search is the reason why a request could have multiple sequences. in beam search, instead of choosing the token with top probability, it selects, let's say, 3 tokens with top 3 probability. at the first iteration, now you will get 3 new sequences because of the appended new token. then in the next iteration, let's say the number of tokens is 10000, it will select next 3 new tokens from the 3 * 10000 possibility. that's why you may get a group instead of a single sequence.
beam search tutorial https://www.youtube.com/watch?v=RLWuzLLSIgw&t=1s&ab_channel=DeepLearningAI