How do thread groups work in the CUDA kernel?

[vedantroy]

I'm trying to understand what's going on with thread groups in the CUDA kernel. Why are they used, etc. I've been studying it for a bit, but I don't understand how THREAD_GROUP_SIZE works. Why is it set the way it is?

vllm/csrc/attention/attention_kernels.cu

Line 116 in 1b290ac

constexpr int THREAD_GROUP_SIZE = MAX(WARP_SIZE / BLOCK_SIZE, 1);