Skip to content
This repository has been archived by the owner on Oct 11, 2024. It is now read-only.

Add bias support for sparse layers #25

Merged
merged 4 commits into from
Feb 16, 2024
Merged

Conversation

mgoin
Copy link
Member

@mgoin mgoin commented Feb 16, 2024

Tested both unstructured and semi_structured - opt-125m has a bias which previous would assert:

  File "neuralmagic-vllm/vllm/model_executor/layers/sparsity/sparse_w16a16_linear_method.py", line 73, in apply_weights
    assert bias is None
from vllm import LLM, SamplingParams

model = LLM(
    "nm-testing/opt-125m-pruned2.4",
    sparsity="semi_structured_sparse_w16a16",
    enforce_eager=True,
)

sampling_params = SamplingParams(max_tokens=10, temperature=0)
outputs = model.generate("Hi", sampling_params=sampling_params)
print(outputs[0].outputs[0].text)
from vllm import LLM, SamplingParams

model = LLM(
    "nm-testing/opt-125m-pruned2.4",
    sparsity="sparse_w16a16",
    enforce_eager=True,
)

sampling_params = SamplingParams(max_tokens=10, temperature=0)
outputs = model.generate("Hi", sampling_params=sampling_params)
print(outputs[0].outputs[0].text)

@mgoin mgoin merged commit ab469e5 into main Feb 16, 2024
2 checks passed
@mgoin mgoin deleted the add-bias-support-for-sparse-layers branch February 16, 2024 22:02
@mgoin mgoin mentioned this pull request Feb 16, 2024
tlrmchlsmth pushed a commit that referenced this pull request Feb 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant