-
Notifications
You must be signed in to change notification settings - Fork 486
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add smooth quant #1398
base: main
Are you sure you want to change the base?
Add smooth quant #1398
Conversation
7af07ab
to
8ee658d
Compare
import importlib | ||
|
||
try: | ||
importlib.import_module("neural_compressor.adaptor.ox_utils.smooth_quant") | ||
except Exception as e: | ||
logging.error(f"{e}.") | ||
raise RuntimeError("Neural-compressor is required for SmoothQuant. Please install the library") from e | ||
|
||
import copy | ||
|
||
import onnx | ||
from neural_compressor.adaptor.ox_utils.smooth_quant import ORTSmoothQuant |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not recommended, reference: https://peps.python.org/pep-0008/#imports
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is something done by the ONNXRuntime source code too! Are u suggesting importing it at top level? https://github.com/microsoft/onnxruntime/blob/0f72739b6db129373d221483d61d6637ec11fb28/onnxruntime/python/tools/quantization/quantize.py#L421
What does this PR do?
Integrates the
SmoothQuant
into theOptimum ONNXRuntime Quantizer
.INC
behind the scenes for the quantisation.quantizer.apply_smooth_quant
before the calibration step in the regular quantization process.Usage
Before submitting