Skip to content

Commit

Permalink
[Bugfix] Check chain_speculative_sampling before calling it (#11673)
Browse files Browse the repository at this point in the history
Signed-off-by: Lu Fang <[email protected]>
  • Loading branch information
houseroad authored Jan 3, 2025
1 parent 2f1e8e8 commit 07064cb
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion vllm/model_executor/layers/rejection_sampler.py
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ def forward(

# If use Flashinfer chain_speculative_sampling kernel
# for rejection sampling
if self.use_flashinfer:
if self.use_flashinfer and chain_speculative_sampling is not None:
batch_size, k, _ = draft_probs.shape
uniform_samples = self._create_uniform_samples(
seeded_seqs, batch_size, k, draft_probs.device)
Expand Down

0 comments on commit 07064cb

Please sign in to comment.