Intel® Extension for PyTorch* v2.1.100+cpu Release Notes

Highlights

• Improved the performance of BF16 LLM generation inference: #2253 #2251 #2236 #2278
• Added the optimization for Codegen: #2257
• Provided the dockerfile and updated the related doc to improve the UX for LLM users: #2229 #2195 #2299 #2315 #2283
• Improved the accuracy of the quantization path of LLMs: #2280 #2292 #2275 #2319
• Misc fix and enhancement: #2198 #2264 #2290

Full Changelog: v2.1.0+cpu...v2.1.100+cpu

Intel® Extension for PyTorch* v2.1.10+xpu Release Notes

2.1.10+xpu

We are pleased to announce the release of Intel® Extension for PyTorch* v2.1.10+xpu. This is the new Intel® Extension for PyTorch* release supports both CPU platforms and GPU platforms (Intel® Data Center GPU Flex Series, Intel® Data Center GPU Max Series and Intel® Arc™ A-Series Graphics) based on PyTorch* 2.1.0. It extends PyTorch* 2.1.0 with up-to-date features and optimizations on xpu for an extra performance boost on Intel hardware. Optimizations take advantage of AVX-512 Vector Neural Network Instructions (AVX512 VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel Xe Matrix Extensions (XMX) AI engines on Intel discrete GPUs. Moreover, through PyTorch* xpu device, Intel® Extension for PyTorch* provides easy GPU acceleration for Intel discrete GPUs with PyTorch*.

Please refer to the Installation Guide for the system requirements and steps to install and use Intel® Extension for PyTorch* 2.1.10+xpu. For more detailed tutorials and documentations describing features, APIs and technical details, please refer to Intel® Extension for PyTorch* Documentation.

Highlights

This release provides the following features:

• Large Language Model (LLM) optimizations for FP16 inference on Intel® Data Center GPU Max Series (Experimental): Intel® Extension for PyTorch* provides a lot of specific optimizations for LLM workloads on Intel® Data Center GPU Max Series in this release. In operator level, we provide highly efficient GEMM kernel to speedup Linear layer and customized fused operators to reduce HBM access and kernel launch overhead. To reduce memory footprint, we define a segment KV Cache policy to save device memory and improve the throughput. To better trade-off the performance and accuracy, low-precision solution e.g., weight-only-quantization for INT4 is enabled. Besides, tensor parallel can also be adopted to get lower latency for LLMs.

  • A new API function, ipex.optimize_transformers, is designed to optimize transformer-based models within frontend Python modules, with a particular focus on LLMs. It provides optimizations for both model-wise and content-generation-wise. You just need to invoke the ipex.optimize_transformers API instead of the ipex.optimize API to apply all optimizations transparently. More detailed information can be found at Large Language Model optimizations overview.
  • A typical usage of this new feature is quite simple as below:

    import torch
    import intel_extension_for_pytorch as ipex
    ...
    model = ipex.optimize_transformers(model, dtype=dtype)
    

• Torch.compile functionality on Intel® Data Center GPU Max Series (Experimental): Extends Intel® Extension for PyTorch* capabilities to support torch.compile APIs on Intel® Data Center GPU Max Series. And provides Intel GPU support on top of Triton* compiler to reach competitive performance speed-up over eager mode by default "inductor" backend of Intel® Extension for PyTorch*.

• Intel® Arc™ A-Series Graphics on WSL2, native Windows and native Linux are officially supported in this release. Intel® Arc™ A770 Graphic card has been used as primary verification vehicle for product level test.

• Other features are listed as following, more detailed information can be found in public documentation:

  • FP8 datatype support (Experimental): Add basic data type and FP8 Linear operator support based on emulation kernel.
  • Kineto Profiling (Experimental): An extension of PyTorch* profiler for profiling operators on Intel® GPU devices.
  • Fully Sharded Data Parallel (FSDP): Support new PyTorch* FSDP API which provides an industry-grade solution for large-scale model training.
  • Asymmetric INT8 quantization: Support asymmetric quantization to align with stock PyTorch* and provide better accuracy in INT8.

• CPU support has been merged in this release. CPU features and optimizations are equivalent to what has been released in Intel® Extension for PyTorch* v2.1.0+cpu release that was made publicly available in Oct 2023. For customers who would like to evaluate workloads on both GPU and CPU, they can use this package. For customers who are focusing on CPU only, we still recommend them to use Intel® Extension for PyTorch* v2.1.0+cpu release for smaller footprint, less dependencies and broader OS support.

Known Issues

Please refer to Known Issues webpage.

Intel® Extension for PyTorch* v2.1.0+cpu Release Notes

We are excited to announce the release of Intel® Extension for PyTorch* 2.1.0+cpu which accompanies PyTorch 2.1. This release mainly brings in our latest optimization on Large Language Model (LLM), torch.compile backend optimization as to leverage TorchInductor’s capability, performance optimization of static quantization under dynamic shape, together with a set of bug fixing and small optimization. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try this release and feedback as to improve further on this product.

Highlights

• Large Language Model (LLM) optimization (Experimental): Intel® Extension for PyTorch* provides a lot of specific optimizations for LLMs in this new release. In operator level, we provide highly efficient GEMM kernel to speedup Linear layer and customized operators to reduce the memory footprint. To better trade-off the performance and accuracy, different low-precision solutions e.g., smoothQuant for INT8 and weight-only-quantization for INT4 and INT8 are also enabled. Besides, tensor parallel can also be adopt to get lower latency for LLMs.

  A new API function, ipex.optimize_transformers, is designed to optimize transformer-based models within frontend Python modules, with a particular focus on Large Language Models (LLMs). It provides optimizations for both model-wise and content-generation-wise. You just need to invoke the ipex.optimize_transformers function instead of the ipex.optimize function to apply all optimizations transparently. More detailed information can be found at Large Language Model optimizations overview.

  Specifically, this new release includes the support of SmoothQuant and weight only quantization (both INT8 weight and INT4 weight) as to provide better performance and accuracy for low precision scenarios.

  A typical usage of this new feature is quite simple as below:

  import torch
  import intel_extension_for_pytorch as ipex
  ...
  model = ipex.optimize_transformers(model, dtype=dtype)

• torch.compile backend optimization with PyTorch Inductor (Experimental): We optimized Intel® Extension for PyTorch to leverage PyTorch Inductor’s capability when working as a backend of torch.compile, which can better utilize torch.compile’s power of graph capture, Inductor’s scalable fusion capability, and still keep customized optimization from Intel® Extension for PyTorch.

• performance optimization of static quantization under dynamic shape: We optimized the static quantization performance of Intel® Extension for PyTorch for dynamic shapes. The usage is the same as the workflow of running static shapes while inputs of variable shapes could be provided during runtime.

• Bug fixing and other optimization

  • Optimized the runtime memory usage #1563
  • Fixed the excessive size of the saved model #1677 #1688
  • Supported shared parameters in ipex.optimize #1664
  • Enabled the optimization of LARS fusion #1695
  • Supported dictionary input in ipex.quantization.prepare #1682
  • Updated oneDNN to v3.3 #2137

Intel® Extension for PyTorch* v2.0.110+xpu Release Notes

2.0.110+xpu

We are pleased to announce the release of Intel® Extension for PyTorch* v2.0.110+xpu. This is the new Intel® Extension for PyTorch* release supports both CPU platforms and GPU platforms (Intel® Data Center GPU Flex Series and Intel® Data Center GPU Max Series) based on PyTorch* 2.0.1. It extends PyTorch* 2.0.1 with up-to-date features and optimizations on xpu for an extra performance boost on Intel hardware. Optimizations take advantage of AVX-512 Vector Neural Network Instructions (AVX512 VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel Xe Matrix Extensions (XMX) AI engines on Intel discrete GPUs. Moreover, through PyTorch* xpu device, Intel® Extension for PyTorch* provides easy GPU acceleration for Intel discrete GPUs with PyTorch*.

Please refer to the Installation Guide for the system requirements and steps to install and use Intel® Extension for PyTorch* 2.0.110+xpu. For more detailed tutorials and documentations describing features, APIs and technical details, please refer to Intel® Extension for PyTorch* Documentation.

Highlights

This release introduces specific XPU solution optimizations on Intel discrete GPUs which include Intel® Data Center GPU Flex Series and Intel® Data Center GPU Max Series. Optimized operators and kernels are implemented and registered through PyTorch* dispatching mechanism for the xpu device. These operators and kernels are accelerated on Intel GPU hardware from the corresponding native vectorization and matrix calculation features. In graph mode, additional operator fusions are supported to reduce operator/kernel invocation overheads, and thus increase performance.

This release provides the following features:

• oneDNN 3.3 API integration and adoption
• Libtorch support
• ARC support on Windows, WSL2 and Ubuntu (Experimental)
• OOB models improvement
  • More fusion patterns enabled for optimizing OOB models
• CPU support is merged in this release:
  • CPU features and optimizations are equivalent to what has been released in Intel® Extension for PyTorch* v2.0.100+cpu release that was made publicly available in May 2023. For customers who would like to evaluate workloads on both GPU and CPU, they can use this package. For customers who are focusing on CPU only, we still recommend them to use Intel® Extension for PyTorch* v2.0.100+cpu release for smaller footprint, less dependencies and broader OS support.

This release adds the following fusion patterns in PyTorch* JIT mode for Intel GPU:

• add + softmax
• add + view + softmax

Known Issues

Please refer to Known Issues webpage.

Intel® Extension for PyTorch* v2.0.100+cpu Release Notes

Highlights

• Enhanced the functionality of Intel® Extension for PyTorch as a backend of torch.compile: #1568 #1585 #1590
• Fixed the Stable Diffusion fine-tuning accuracy issue #1587 #1594
• Fixed the ISA check on old hypervisor based VM #1513
• Addressed the excessive memory usage in weight prepack #1593
• Fixed the weight prepack of convolution when padding_mode is not 'zeros' #1580
• Optimized the INT8 LSTM performance #1566
• Fixed TransNetV2 calibration failure #1564
• Fixed BF16 RNN-T inference when AVX512_CORE_VNNI ISA is used #1592
• Fixed the ROIAlign operator #1589
• Enabled execution on designated numa nodes with launch script #1517

Full Changelog: v2.0.0+cpu...v2.0.100+cpu

Intel® Extension for PyTorch* v1.13.120+xpu Release Notes

1.13.120+xpu

We are pleased to announce the release of Intel® Extension for PyTorch* 1.13.120+xpu. This is the updated IPEX XPU release supports both CPU platforms and GPU platforms (Intel® Data Center GPU Flex Series and Intel® Data Center GPU Max Series) based on PyTorch 1.13.1. It extends PyTorch* 1.13.1 with up-to-date features and optimizations on xpu for an extra performance boost on Intel hardware. The optimizations take advantage of AVX-512 Vector Neural Network Instructions (AVX512 VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel Xe Matrix Extensions (XMX) AI engines on Intel discrete GPUs. Moreover, through PyTorch* xpu device, Intel® Extension for PyTorch* provides easy GPU acceleration for Intel discrete GPUs with PyTorch*.

Please refer to the Installation Guide for the system requirements and steps to install and use Intel® Extension for PyTorch* 1.13.120+xpu. For more detailed tutorials and documentations describing features, APIs and technical details, please refer to Intel® Extension for PyTorch* Documentation.

Highlights

This release introduces specific XPU solution optimizations on Intel discrete GPUs which include Intel® Data Center GPU Flex Series and Intel® Data Center GPU Max Series. Optimized operators and kernels are implemented and registered through PyTorch* dispatching mechanism for the xpu device. These operators and kernels are accelerated on Intel GPU hardware from the corresponding native vectorization and matrix calculation features. In graph mode, additional operator fusions are supported to reduce operator/kernel invocation overheads, and thus increase performance.

This release provides the following features:

• oneDNN 3.1 API integration and adoption
• OOB models improvement
  • More fusion patterns enabled for optimizing OOB models
• CPU support is merged in this release:
  • CPU features and optimizations are equivalent to what has been released in Intel® Extension for PyTorch* v1.13.100+cpu release that was made publicly available in Feb 2023. For customers who would like to evaluate workloads on both GPU and CPU, they can use this package. For customers who are focusing on CPU only, we still recommend them to use Intel® Extension for PyTorch* v1.13.100+cpu release for smaller footprint, less dependencies and broader OS support.

This release adds the following fusion patterns in PyTorch* JIT mode for Intel GPU:

• Matmul + UnaryOp(abs, sqrt, square, exp, log, round, Log_Sigmoid, Hardswish, HardSigmoid, Pow, ELU, SiLU, hardtanh, Leaky_relu)
• Conv2d + BinaryOp(add, sub, mul, div, max, min, eq, ne, ge, gt, le, lt)
• Linear + BinaryOp(add, sub, mul, div, max, min)
• Conv2d + mul + add
• Conv2d + mul + add + relu
• Conv2d + sigmoid + mul + add
• Conv2d + sigmoid + mul + add + relu

Known Issues

Please refer to Known Issues webpage.

Intel® Extension for PyTorch* v2.0.0+cpu Release Notes

We are pleased to announce the release of Intel® Extension for PyTorch* 2.0.0-cpu which accompanies PyTorch 2.0. This release mainly brings in our latest optimization on NLP, support of PyTorch 2.0's hero API –- torch.compile as one of its backend, together with a set of bug fixing and small optimization. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try this release and feedback as to improve further on this product.

Highlights

• Fast BERT optimization (Experimental): Intel introduced a new technique to speed up BERT workloads. Intel® Extension for PyTorch* integrated this implementation, which benefits BERT model especially training. A new API ipex.fast_bert is provided to try this new optimization. More detailed information can be found at Fast Bert Feature.

• MHA optimization with Flash Attention: Intel optimized MHA module with Flash Attention technique as inspired by Stanford paper. This brings less memory consumption for LLM, and also provides better inference performance for models like BERT, Stable Diffusion, etc.

• Work with torch.compile as an backend (Experimental): PyTorch 2.0 introduces a new feature, torch.compile, to speed up PyTorch execution. We've enabled Intel® Extension for PyTorch as a backend of torch.compile, which can leverage this new PyTorch API's power of graph capture and provide additional optimization based on these graphs.
  The usage of this new feature is quite simple as below:

import torch
import intel_extension_for_pytorch as ipex
...
model = ipex.optimize(model)
model = torch.compile(model, backend='ipex')

• Bug fixing and other optimization

  • Supported RMSNorm which is widely used in the t5 model of huggingface #1341
  • Optimized InstanceNorm #1330
  • Fixed the quantization of LSTM #1414 #1473
  • Fixed the correctness issue of unpacking non-contiguous Linear weight #1419
  • oneDNN update #1488

Known Issues

Please check at Known Issues webpage.

Intel® Extension for PyTorch* v1.13.100+cpu Release Notes

1.13.100+cpu

Highlights

• Quantization optimization with more fusion, op and auto channels last support #1318 #1353 #1328 #1355 #1367 #1384
• Installation and build enhancement #1295 #1392
• OneDNN graph and OneDNN update #1376
• Misc fix and enhancement #1373 #1338 #1391 #1322

Full Changelog: v1.13.0+cpu...v1.13.100+cpu

Intel® Extension for PyTorch* v1.13.10+xpu Release Notes

1.13.10+xpu

We are pleased to announce the release of Intel® Extension for PyTorch* 1.13.10+xpu, which is the first Intel® Extension for PyTorch* release supports both CPU platforms and GPU platforms (Intel® Data Center GPU Flex Series and Intel® Data Center GPU Max Series) based on PyTorch* 1.13. It extends PyTorch* 1.13 with up-to-date features and optimizations on xpu for an extra performance boost on Intel hardware. Optimizations take advantage of AVX-512 Vector Neural Network Instructions (AVX512 VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel Xe Matrix Extensions (XMX) AI engines on Intel discrete GPUs. Moreover, through PyTorch* xpu device, Intel® Extension for PyTorch* provides easy GPU acceleration for Intel discrete GPUs with PyTorch*.

Highlights

This release introduces specific XPU solution optimizations on Intel discrete GPUs which include Intel® Data Center GPU Flex Series and Intel® Data Center GPU Max Series. Optimized operators and kernels are implemented and registered through PyTorch* dispatching mechanism for the xpu device. These operators and kernels are accelerated on Intel GPU hardware from the corresponding native vectorization and matrix calculation features. In graph mode, additional operator fusions are supported to reduce operator/kernel invocation overheads, and thus increase performance.

This release provides the following features:

• Distributed Training on GPU:
  • support of distributed training with DistributedDataParallel (DDP) on Intel GPU hardware
  • support of distributed training with Horovod (experimental feature) on Intel GPU hardware
• Automatic channels last format conversion on GPU:
  • Automatic channels last format conversion is enabled. Models using torch.xpu.optimize API running on Intel® Data Center GPU Max Series will be converted to channels last memory format, while models running on Intel® Data Center GPU Flex Series will choose oneDNN block format.
• CPU support is merged in this release:
  • CPU features and optimizations are equivalent to what has been released in Intel® Extension for PyTorch* v1.13.0+cpu release that was made publicly available in Nov 2022. For customers who would like to evaluate workloads on both GPU and CPU, they can use this package. For customers who are focusing on CPU only, we still recommend them to use Intel® Extension for PyTorch* v1.13.0+cpu release for smaller footprint, less dependencies and broader OS support.

This release adds the following fusion patterns in PyTorch* JIT mode for Intel GPU:

• Conv2D + UnaryOp(abs, sqrt, square, exp, log, round, GeLU, Log_Sigmoid, Hardswish, Mish, HardSigmoid, Tanh, Pow, ELU, hardtanh)
• Linear + UnaryOp(abs, sqrt, square, exp, log, round, Log_Sigmoid, Hardswish, HardSigmoid, Pow, ELU, SiLU, hardtanh, Leaky_relu)

Known Issues

Please refer to Known Issues webpage.

Download wheel packages

Take wget as examples:

• Compatible with Python 3.10:

wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/intel_extension_for_pytorch-1.13.10%2Bxpu-cp310-cp310-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/torch_ccl/xpu/oneccl_bind_pt-1.13.100%2Bgpu-cp310-cp310-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/torch-1.13.0a0%2Bgitb1dde16-cp310-cp310-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/torchvision-0.14.1a0%2B0504df5-cp310-cp310-linux_x86_64.whl

• Compatible with Python 3.9:

wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/intel_extension_for_pytorch-1.13.10%2Bxpu-cp39-cp39-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/torch_ccl/xpu/oneccl_bind_pt-1.13.100%2Bgpu-cp39-cp39-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/torch-1.13.0a0%2Bgitb1dde16-cp39-cp39-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/torchvision-0.14.1a0%2B0504df5-cp39-cp39-linux_x86_64.whl

• Compatible with Python 3.8:

wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/intel_extension_for_pytorch-1.13.10%2Bxpu-cp38-cp38-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/torch_ccl/xpu/oneccl_bind_pt-1.13.100%2Bgpu-cp38-cp38-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/torch-1.13.0a0%2Bgitb1dde16-cp38-cp38-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/torchvision-0.14.1a0%2B0504df5-cp38-cp38-linux_x86_64.whl

• Compatible with Python 3.7:

wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/intel_extension_for_pytorch-1.13.10%2Bxpu-cp37-cp37m-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/torch_ccl/xpu/oneccl_bind_pt-1.13.100%2Bgpu-cp37-cp37m-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/torch-1.13.0a0%2Bgitb1dde16-cp37-cp37m-linux_x86_64.whl
wget https://intel-optimized-pytorch.s3.cn-north-1.amazonaws.com.cn/ipex_stable/xpu/torchvision-0.14.1a0%2B0504df5-cp37-cp37m-linux_x86_64.whl

Intel® Extension for PyTorch* v1.13.0+cpu Release Notes

We are pleased to announce the release of Intel® Extension for PyTorch* 1.13.0-cpu which accompanies PyTorch 1.13. This release is highlighted with quite a few usability features which help users to get good performance and accuracy on CPU with less effort. We also added a couple of performance features as always. Check out the feature summary below.

• Usability Features

1. Automatic channels last format conversion: Channels last conversion is now applied automatically to PyTorch modules with ipex.optimize by default. Users don't have to explicitly convert input and weight for CV models.
2. Code-free optimization (experimental): ipex.optimize is automatically applied to PyTorch modules without the need of code changes when the PyTorch program is started with the IPEX launcher via the new --auto-ipex option.
3. Graph capture mode of ipex.optimize (experimental): A new boolean flag graph_mode (default off) was added to ipex.optimize, when turned on, converting the eager-mode PyTorch module into graph(s) to get the best of graph optimization.
4. INT8 quantization accuracy autotune (experimental): A new quantization API ipex.quantization.autotune was added to refine the default IPEX quantization recipe via autotuning algorithms for better accuracy.
5. Hypertune (experimental) is a new tool added on top of IPEX launcher to automatically identify the good configurations for best throughput via hyper-parameter tuning.
6. ipexrun: The counterpart of torchrun, is a shortcut added for invoking IPEX launcher.

• Performance Features

1. Packed MKL SGEMM landed as the default kernel option for FP32 Linear, bringing up-to 20% geomean speedup for real-time NLP tasks.
2. DL compiler is now turned on by default with oneDNN fusion and gives additional performance boost for INT8 models.

Highlights

• Automatic channels last format conversion: Channels last conversion is now applied to PyTorch modules automatically with ipex.optimize by default for both training and inference scenarios. Users don't have to explicitly convert input and weight for CV models.

import intel_extension_for_pytorch as ipex
# No need to do explicitly format conversion
# m = m.to(format=torch.channels_last)
# x = x.to(format=torch.channels_last)
# for inference
m = ipex.optimize(m)
m(x)
# for training
m, optimizer = ipex.optimize(m, optimizer)
m(x)

• Code-free optimization (experimental): ipex.optimize is automatically applied to PyTorch modules without the need of code changes when the PyTorch program is started with the IPEX launcher via the new --auto-ipex option.

Example: QA case in HuggingFace

# original command
ipexrun --use_default_allocator --ninstance 2 --ncore_per_instance 28 run_qa.py \
  --model_name_or_path bert-base-uncased --dataset_name squad --do_eval \
  --per_device_train_batch_size 12 --learning_rate 3e-5 --num_train_epochs 2 \
  --max_seq_length 384 --doc_stride 128 --output_dir /tmp/debug_squad/

# automatically apply bfloat16 optimization (--auto-ipex --dtype bfloat16)
ipexrun --use_default_allocator --ninstance 2 --ncore_per_instance 28 --auto_ipex --dtype bfloat16 run_qa.py \
  --model_name_or_path bert-base-uncased --dataset_name squad --do_eval \
  --per_device_train_batch_size 12 --learning_rate 3e-5 --num_train_epochs 2 \
  --max_seq_length 384 --doc_stride 128 --output_dir /tmp/debug_squad/

• Graph capture mode of ipex.optimize (experimental): A new boolean flag graph_mode (default off) was added to ipex.optimize, when turned on, converting the eager-mode PyTorch module into graph(s) to get the best of graph optimization. Under the hood, it combines the goodness of both TorchScript tracing and TorchDynamo to get as max graph scope as possible. Currently, it only supports FP32 and BF16 inference. INT8 inference and training support are under way.

import intel_extension_for_pytorch as ipex
model = ...
model.load_state_dict(torch.load(PATH))
model.eval()
optimized_model = ipex.optimize(model, graph_mode=True)

• INT8 quantization accuracy autotune (experimental): A new quantization API ipex.quantization.autotune was added to refine the default IPEX quantization recipe via autotuning algorithms for better accuracy. This is an optional API to invoke (after prepare and before convert) for scenarios when the accuracy of default quantization recipe of IPEX cannot meet the requirement. The current implementation is powered by Intel Neural Compressor (INC).

import intel_extension_for_pytorch as ipex
# Calibrate the model
qconfig = ipex.quantization.default_static_qconfig
calibrated_model = ipex.quantization.prepare(model_to_be_calibrated, qconfig, example_inputs=example_inputs)
for data in calibration_data_set:
    calibrated_model(data)
# Autotune the model
calib_dataloader = torch.utils.data.DataLoader(...)
def eval_func(model):
    # Return accuracy value
    ...
    return accuracy
tuned_model = ipex.quantization.autotune(
                 calibrated_model, calib_dataloader, eval_func,
                 sampling_sizes=[100], accuracy_criterion={'relative': 0.01}, tuning_time=0
              )
# Convert the model to jit model
quantized_model = ipex.quantization.convert(tuned_model)
with torch.no_grad():
    traced_model = torch.jit.trace(quantized_model, example_input)
    traced_model = torch.jit.freeze(traced_model)
# Do inference
y = traced_model(x)

• Hypertune (experimental) is a new tool added on top of IPEX launcher to automatically identify the good configurations for best throughput via hyper-parameter tuning.

python -m intel_extension_for_pytorch.cpu.launch.hypertune --conf_file <your_conf_file> <your_python_script> [args]

Known Issues

Please check at Known Issues webpage.