From 604c9602303da575de49be3e8d0dbd9f6d918f50 Mon Sep 17 00:00:00 2001 From: Nikita Malinin Date: Wed, 26 Mar 2025 11:22:22 +0100 Subject: [PATCH 01/18] Release notes template --- ReleaseNotes.md | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index ba3aa699812..02352b9cd09 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -1,5 +1,53 @@ # Release Notes +## New in Release 2.16.0 + +Post-training Quantization: + +- Breaking changes: + - ... +- General: + - ... +- Features: + - ... +- Fixes: + - ... +- Improvements: + - ... +- Deprecations/Removals: + - ... +- Tutorials: + - ... +- Known issues: + - ... + +Compression-aware training: + +- Breaking changes: + - ... +- General: + - ... +- Features: + - ... +- Fixes: + - ... +- Improvements: + - ... +- Deprecations/Removals: + - ... +- Tutorials: + - ... +- Known issues: + - ... + +Deprecations/Removals: + +- ... + +Requirements: + +- ... + ## New in Release 2.15.0 Post-training Quantization: From 689ae888f21ae8e7881c047beb1537e4833a2c81 Mon Sep 17 00:00:00 2001 From: Nikita Savelyev Date: Thu, 27 Mar 2025 18:28:05 +0100 Subject: [PATCH 02/18] Update ReleaseNotes.md https://github.com/openvinotoolkit/nncf/pull/3292 https://github.com/openvinotoolkit/nncf/pull/3366 --- ReleaseNotes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index 02352b9cd09..66b9c1a3cb4 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -11,9 +11,9 @@ Post-training Quantization: - Features: - ... - Fixes: - - ... + - Fixed occasional failures of weight compression algorithm on ARM CPUs. - Improvements: - - ... + - Reduced the run time and peak memory of mixed precision assignment procedure during weight compression in the OpenVINO backend. Overall compression time reduction in mixed precision case is about 20-40%; peak memory reduction is about 20%. - Deprecations/Removals: - ... - Tutorials: From 47208f9728a80fe2e8718f36eaa96a98c0182abd Mon Sep 17 00:00:00 2001 From: Lyalyushkin Nikolay Date: Thu, 27 Mar 2025 23:08:54 +0100 Subject: [PATCH 03/18] Update ReleaseNotes.md https://github.com/openvinotoolkit/nncf/pull/3351 https://github.com/openvinotoolkit/nncf/pull/3348 https://github.com/openvinotoolkit/nncf/pull/3341 https://github.com/openvinotoolkit/nncf/pull/3337 https://github.com/openvinotoolkit/nncf/pull/3322 --- ReleaseNotes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index 66b9c1a3cb4..a43681aa5af 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -9,7 +9,7 @@ Post-training Quantization: - General: - ... - Features: - - ... + - (Torch) Introduced a novel weight compression method for Large Language Models (LLMs) that significantly improves accuracy with int4 weights. Leveraging Quantization-Aware Training (QAT) and absorbable LoRA adapters, this approach can achieve a 2x reduction in accuracy loss during compression compared to the best post-training weight compression technique in NNCF (Scale Estimation + AWQ + GPTQ). The `nncf.compress_weight` API now includes a new `compression_format` option, `CompressionFormat.FQ_LORA`, for this QAT method, and a sample compression pipeline with preview support is available [here](examples/llm_compression/torch/qat_with_lora). - Fixes: - Fixed occasional failures of weight compression algorithm on ARM CPUs. - Improvements: From df5d3e552246c44ce2c618db9182cbac845f7fd6 Mon Sep 17 00:00:00 2001 From: Lyalyushkin Nikolay Date: Thu, 27 Mar 2025 23:11:36 +0100 Subject: [PATCH 04/18] Update ReleaseNotes.md https://github.com/openvinotoolkit/nncf/pull/3330 --- ReleaseNotes.md | 1 + 1 file changed, 1 insertion(+) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index a43681aa5af..6b04f35052b 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -12,6 +12,7 @@ Post-training Quantization: - (Torch) Introduced a novel weight compression method for Large Language Models (LLMs) that significantly improves accuracy with int4 weights. Leveraging Quantization-Aware Training (QAT) and absorbable LoRA adapters, this approach can achieve a 2x reduction in accuracy loss during compression compared to the best post-training weight compression technique in NNCF (Scale Estimation + AWQ + GPTQ). The `nncf.compress_weight` API now includes a new `compression_format` option, `CompressionFormat.FQ_LORA`, for this QAT method, and a sample compression pipeline with preview support is available [here](examples/llm_compression/torch/qat_with_lora). - Fixes: - Fixed occasional failures of weight compression algorithm on ARM CPUs. + - (Torch) Fixed weight compression for float16/bfloat16 models. - Improvements: - Reduced the run time and peak memory of mixed precision assignment procedure during weight compression in the OpenVINO backend. Overall compression time reduction in mixed precision case is about 20-40%; peak memory reduction is about 20%. - Deprecations/Removals: From 28c66831f6e58eb9f061dbc6ee5ba714678cd36f Mon Sep 17 00:00:00 2001 From: Lyalyushkin Nikolay Date: Thu, 27 Mar 2025 23:15:29 +0100 Subject: [PATCH 05/18] Update ReleaseNotes.md reworked FQ + Lora --- ReleaseNotes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index 6b04f35052b..585c565d1a2 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -9,7 +9,7 @@ Post-training Quantization: - General: - ... - Features: - - (Torch) Introduced a novel weight compression method for Large Language Models (LLMs) that significantly improves accuracy with int4 weights. Leveraging Quantization-Aware Training (QAT) and absorbable LoRA adapters, this approach can achieve a 2x reduction in accuracy loss during compression compared to the best post-training weight compression technique in NNCF (Scale Estimation + AWQ + GPTQ). The `nncf.compress_weight` API now includes a new `compression_format` option, `CompressionFormat.FQ_LORA`, for this QAT method, and a sample compression pipeline with preview support is available [here](examples/llm_compression/torch/qat_with_lora). + - (Torch) Introduced a novel weight compression method to significantly improve the accuracy of Large Language Models (LLMs) with int4 weights. Leveraging Quantization-Aware Training (QAT) and absorbable LoRA adapters, this approach can achieve a 2x reduction in accuracy loss during compression compared to the best post-training weight compression technique in NNCF (Scale Estimation + AWQ + GPTQ). The `nncf.compress_weight` API now includes a new `compression_format` option, `CompressionFormat.FQ_LORA`, for this QAT method, and a sample compression pipeline with preview support is available [here](examples/llm_compression/torch/qat_with_lora). - Fixes: - Fixed occasional failures of weight compression algorithm on ARM CPUs. - (Torch) Fixed weight compression for float16/bfloat16 models. From 93ce251e0494eb6f60d4432745e424ad4e4604fc Mon Sep 17 00:00:00 2001 From: Aamir Nazir Date: Fri, 28 Mar 2025 12:29:25 +0400 Subject: [PATCH 06/18] Update ReleaseNotes.md --- ReleaseNotes.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index 585c565d1a2..645bc1f30e6 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -15,6 +15,7 @@ Post-training Quantization: - (Torch) Fixed weight compression for float16/bfloat16 models. - Improvements: - Reduced the run time and peak memory of mixed precision assignment procedure during weight compression in the OpenVINO backend. Overall compression time reduction in mixed precision case is about 20-40%; peak memory reduction is about 20%. + - (TorchFX, Experimental) Added quantization support for (TorchFX)[https://pytorch.org/docs/stable/fx.html] models exported with dynamic shapes. - Deprecations/Removals: - ... - Tutorials: @@ -47,7 +48,7 @@ Deprecations/Removals: Requirements: -- ... +- pdated PyTorch (2.6.0) and Torchvision (0.21.0) versions. ## New in Release 2.15.0 From c9426b605d20b8edd9818eb2f9fdf5528da95f05 Mon Sep 17 00:00:00 2001 From: Aamir Nazir Date: Fri, 28 Mar 2025 12:32:49 +0400 Subject: [PATCH 07/18] Update ReleaseNotes.md --- ReleaseNotes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index 645bc1f30e6..018b08f3955 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -48,7 +48,7 @@ Deprecations/Removals: Requirements: -- pdated PyTorch (2.6.0) and Torchvision (0.21.0) versions. +- Updated PyTorch (2.6.0) and Torchvision (0.21.0) versions. ## New in Release 2.15.0 From 230129839245ae6407756f0f8cb826db826a90b8 Mon Sep 17 00:00:00 2001 From: Aleksei Kashapov Date: Fri, 28 Mar 2025 09:44:26 +0100 Subject: [PATCH 08/18] Update ReleaseNotes.md --- ReleaseNotes.md | 1 + 1 file changed, 1 insertion(+) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index 018b08f3955..710725d8af3 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -10,6 +10,7 @@ Post-training Quantization: - ... - Features: - (Torch) Introduced a novel weight compression method to significantly improve the accuracy of Large Language Models (LLMs) with int4 weights. Leveraging Quantization-Aware Training (QAT) and absorbable LoRA adapters, this approach can achieve a 2x reduction in accuracy loss during compression compared to the best post-training weight compression technique in NNCF (Scale Estimation + AWQ + GPTQ). The `nncf.compress_weight` API now includes a new `compression_format` option, `CompressionFormat.FQ_LORA`, for this QAT method, and a sample compression pipeline with preview support is available [here](examples/llm_compression/torch/qat_with_lora). + - (Torch) Add support for 4-bit weight compression, along with AWQ and Scale Estimation data-aware methods to reduce quality loss after compression. - Fixes: - Fixed occasional failures of weight compression algorithm on ARM CPUs. - (Torch) Fixed weight compression for float16/bfloat16 models. From 7d99dbe3937a4a1785d651d0caad6b83626f3698 Mon Sep 17 00:00:00 2001 From: Liubov Talamanova Date: Fri, 28 Mar 2025 13:23:54 +0000 Subject: [PATCH 09/18] Add list of OV notebooks with NNCF to release notes --- ReleaseNotes.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index 710725d8af3..0215bcd3347 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -20,7 +20,18 @@ Post-training Quantization: - Deprecations/Removals: - ... - Tutorials: - - ... + - [Post-Training Optimization of Gemma3 Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/gemma3/gemma3.ipynb) + - [Post-Training Optimization of GLM4-V Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/glm4-v/glm4-v.ipynb) + - [Post-Training Optimization of Llasa Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llasa-speech-synthesis/llasa-speech-synthesis.ipynb) + - [Post-Training Optimization of YOLOv12 Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/yolov12-optimization/yolov12-object-detection.ipynb) + - [Post-Training Optimization of Phi-4-multimodal Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/phi-4-multimodal/phi-4-multimodal.ipynb) + - [Post-Training Optimization of Qwen2.5VL Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/qwen2.5-vl/qwen2.5-vl.ipynb) + - [Post-Training Optimization of DeepSeek-VL2 Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/deepseek-vl2/deepseek-vl2.ipynb) + - [Post-Training Optimization of FLUX.1 Fill Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/flux-fill/flux-fill.ipynb) + - [Post-Training Optimization of olmOCR Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/olmocr-pdf-vlm/olmocr-pdf-vlm.ipynb) + - [Post-Training Optimization of SmolDocling Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/smoldocling/smoldocling.ipynb) + - [Post-Training Optimization of SmolVLM2 Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/smolvlm2/smolvlm2.ipynb) + - [Post-Training Optimization of GOT-OCR 2.0 Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/got-ocr2/got-ocr2.ipynb) - Known issues: - ... From 81230600872b175595c601099f190fea7be9a4b3 Mon Sep 17 00:00:00 2001 From: Daniil Lyakhov Date: Tue, 1 Apr 2025 14:43:51 +0200 Subject: [PATCH 10/18] Update ReleaseNotes.md --- ReleaseNotes.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index 0215bcd3347..bf50500c150 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -16,7 +16,9 @@ Post-training Quantization: - (Torch) Fixed weight compression for float16/bfloat16 models. - Improvements: - Reduced the run time and peak memory of mixed precision assignment procedure during weight compression in the OpenVINO backend. Overall compression time reduction in mixed precision case is about 20-40%; peak memory reduction is about 20%. - - (TorchFX, Experimental) Added quantization support for (TorchFX)[https://pytorch.org/docs/stable/fx.html] models exported with dynamic shapes. + - The NNCF hardware config has been extended with the `narrow_range` parameter, enabling more combinations of quantization configurations in the MinMax quantization algorithm. + - (TorchFX, Experimental) Added quantization support for [TorchFX](https://pytorch.org/docs/stable/fx.html) models exported with dynamic shapes. + - (TorchFX, Experimental) The constant folding step is removed from the `quantize_pt2e` function and the `transform_for_annotation` method of the `OpenVINOQuantizer` to align with the `torch.ao` quantization implementation. - Deprecations/Removals: - ... - Tutorials: From 933f35b764f62d618d75074f35a5635f09a70024 Mon Sep 17 00:00:00 2001 From: Nikita Malinin Date: Thu, 3 Apr 2025 12:02:52 +0200 Subject: [PATCH 11/18] Minor fixes --- ReleaseNotes.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index bf50500c150..a009189fc18 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -9,13 +9,13 @@ Post-training Quantization: - General: - ... - Features: - - (Torch) Introduced a novel weight compression method to significantly improve the accuracy of Large Language Models (LLMs) with int4 weights. Leveraging Quantization-Aware Training (QAT) and absorbable LoRA adapters, this approach can achieve a 2x reduction in accuracy loss during compression compared to the best post-training weight compression technique in NNCF (Scale Estimation + AWQ + GPTQ). The `nncf.compress_weight` API now includes a new `compression_format` option, `CompressionFormat.FQ_LORA`, for this QAT method, and a sample compression pipeline with preview support is available [here](examples/llm_compression/torch/qat_with_lora). - - (Torch) Add support for 4-bit weight compression, along with AWQ and Scale Estimation data-aware methods to reduce quality loss after compression. + - (PyTorch) Introduced a novel weight compression method to significantly improve the accuracy of Large Language Models (LLMs) with int4 weights. Leveraging Quantization-Aware Training (QAT) and absorbable LoRA adapters, this approach can achieve a 2x reduction in accuracy loss during compression compared to the best post-training weight compression technique in NNCF (Scale Estimation + AWQ + GPTQ). The `nncf.compress_weights` API now includes a new `compression_format` option, `nncf.CompressionFormat.FQ_LORA`, for this QAT method, a sample compression pipeline with preview support is available [here](examples/llm_compression/torch/qat_with_lora). + - (PyTorch) Added support for 4-bit weight compression with AWQ and Scale Estimation data-aware methods to reduce quality loss. - Fixes: - - Fixed occasional failures of weight compression algorithm on ARM CPUs. - - (Torch) Fixed weight compression for float16/bfloat16 models. + - Fixed occasional failures of the weights compression algorithm on ARM CPUs. + - (PyTorch) Fixed weights compression for float16/bfloat16 models. - Improvements: - - Reduced the run time and peak memory of mixed precision assignment procedure during weight compression in the OpenVINO backend. Overall compression time reduction in mixed precision case is about 20-40%; peak memory reduction is about 20%. + - Reduced the run time and peak memory of the mixed precision assignment procedure during weight compression in the OpenVINO backend. Overall compression time reduction in the mixed precision case is about 20-40%; peak memory reduction is about 20%. - The NNCF hardware config has been extended with the `narrow_range` parameter, enabling more combinations of quantization configurations in the MinMax quantization algorithm. - (TorchFX, Experimental) Added quantization support for [TorchFX](https://pytorch.org/docs/stable/fx.html) models exported with dynamic shapes. - (TorchFX, Experimental) The constant folding step is removed from the `quantize_pt2e` function and the `transform_for_annotation` method of the `OpenVINOQuantizer` to align with the `torch.ao` quantization implementation. From 67b03e958327867092d946b28ee26d7796a7b603 Mon Sep 17 00:00:00 2001 From: Nikita Malinin Date: Thu, 3 Apr 2025 14:14:31 +0200 Subject: [PATCH 12/18] Update --- ReleaseNotes.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index a009189fc18..464d10651de 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -11,14 +11,19 @@ Post-training Quantization: - Features: - (PyTorch) Introduced a novel weight compression method to significantly improve the accuracy of Large Language Models (LLMs) with int4 weights. Leveraging Quantization-Aware Training (QAT) and absorbable LoRA adapters, this approach can achieve a 2x reduction in accuracy loss during compression compared to the best post-training weight compression technique in NNCF (Scale Estimation + AWQ + GPTQ). The `nncf.compress_weights` API now includes a new `compression_format` option, `nncf.CompressionFormat.FQ_LORA`, for this QAT method, a sample compression pipeline with preview support is available [here](examples/llm_compression/torch/qat_with_lora). - (PyTorch) Added support for 4-bit weight compression with AWQ and Scale Estimation data-aware methods to reduce quality loss. + - (PyTorch, Experimental) Introduced TorchFunctionMode support for MinMax, FastBiasCorrection, SmoothQuant, WeightCompression algorithms. - Fixes: - Fixed occasional failures of the weights compression algorithm on ARM CPUs. - - (PyTorch) Fixed weights compression for float16/bfloat16 models. + - Fixed GPTQ fails with per-channel int4 weights compression. + - (PyTorch, Experimental) Fixed weights compression for float16/bfloat16 models. + - (PyTorch, Experimental) Fixed several memory leak issues: non-detached tensors, extracted modules & graph building with gradients. - Improvements: - Reduced the run time and peak memory of the mixed precision assignment procedure during weight compression in the OpenVINO backend. Overall compression time reduction in the mixed precision case is about 20-40%; peak memory reduction is about 20%. - The NNCF hardware config has been extended with the `narrow_range` parameter, enabling more combinations of quantization configurations in the MinMax quantization algorithm. - (TorchFX, Experimental) Added quantization support for [TorchFX](https://pytorch.org/docs/stable/fx.html) models exported with dynamic shapes. - (TorchFX, Experimental) The constant folding step is removed from the `quantize_pt2e` function and the `transform_for_annotation` method of the `OpenVINOQuantizer` to align with the `torch.ao` quantization implementation. + - Optimized GPTQ algorithm behavior to decrease memory & time consumption by 2.71x and 1.16x, respectively. + - Added general support for optimization of models with FP8 and NF4 weights. - Deprecations/Removals: - ... - Tutorials: @@ -44,7 +49,7 @@ Compression-aware training: - General: - ... - Features: - - ... + - (PyTorch) Changed compression modules serialization API: `compressed_model.nncf.get_config` was changed to `nncf.torch.get_config`. The [documentation](/docs/usage/training_time_compression/quantization_aware_training/Usage.md#saving-and-loading-compressed-models) was updated to use the new API. - Fixes: - ... - Improvements: @@ -63,6 +68,9 @@ Deprecations/Removals: Requirements: - Updated PyTorch (2.6.0) and Torchvision (0.21.0) versions. +- Updated Transformers (>=4.48.0) version. +- Updated NumPy (<2.3.0) version support. +- Updated NetworkX (<3.5.0) version support. ## New in Release 2.15.0 From 9ba36da636b8c287cb6996faebb628238475dc3d Mon Sep 17 00:00:00 2001 From: Nikita Malinin Date: Thu, 3 Apr 2025 14:15:14 +0200 Subject: [PATCH 13/18] Cleanup --- ReleaseNotes.md | 26 -------------------------- 1 file changed, 26 deletions(-) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index 464d10651de..e26e9525c03 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -4,10 +4,6 @@ Post-training Quantization: -- Breaking changes: - - ... -- General: - - ... - Features: - (PyTorch) Introduced a novel weight compression method to significantly improve the accuracy of Large Language Models (LLMs) with int4 weights. Leveraging Quantization-Aware Training (QAT) and absorbable LoRA adapters, this approach can achieve a 2x reduction in accuracy loss during compression compared to the best post-training weight compression technique in NNCF (Scale Estimation + AWQ + GPTQ). The `nncf.compress_weights` API now includes a new `compression_format` option, `nncf.CompressionFormat.FQ_LORA`, for this QAT method, a sample compression pipeline with preview support is available [here](examples/llm_compression/torch/qat_with_lora). - (PyTorch) Added support for 4-bit weight compression with AWQ and Scale Estimation data-aware methods to reduce quality loss. @@ -24,8 +20,6 @@ Post-training Quantization: - (TorchFX, Experimental) The constant folding step is removed from the `quantize_pt2e` function and the `transform_for_annotation` method of the `OpenVINOQuantizer` to align with the `torch.ao` quantization implementation. - Optimized GPTQ algorithm behavior to decrease memory & time consumption by 2.71x and 1.16x, respectively. - Added general support for optimization of models with FP8 and NF4 weights. -- Deprecations/Removals: - - ... - Tutorials: - [Post-Training Optimization of Gemma3 Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/gemma3/gemma3.ipynb) - [Post-Training Optimization of GLM4-V Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/glm4-v/glm4-v.ipynb) @@ -39,31 +33,11 @@ Post-training Quantization: - [Post-Training Optimization of SmolDocling Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/smoldocling/smoldocling.ipynb) - [Post-Training Optimization of SmolVLM2 Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/smolvlm2/smolvlm2.ipynb) - [Post-Training Optimization of GOT-OCR 2.0 Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/got-ocr2/got-ocr2.ipynb) -- Known issues: - - ... Compression-aware training: -- Breaking changes: - - ... -- General: - - ... - Features: - (PyTorch) Changed compression modules serialization API: `compressed_model.nncf.get_config` was changed to `nncf.torch.get_config`. The [documentation](/docs/usage/training_time_compression/quantization_aware_training/Usage.md#saving-and-loading-compressed-models) was updated to use the new API. -- Fixes: - - ... -- Improvements: - - ... -- Deprecations/Removals: - - ... -- Tutorials: - - ... -- Known issues: - - ... - -Deprecations/Removals: - -- ... Requirements: From 10d109a141c15680d477cdabdc70b3ab30925ffa Mon Sep 17 00:00:00 2001 From: Nikita Malinin Date: Tue, 8 Apr 2025 10:07:05 +0200 Subject: [PATCH 14/18] Update ReleaseNotes.md --- ReleaseNotes.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index e26e9525c03..e436929a168 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -33,6 +33,8 @@ Post-training Quantization: - [Post-Training Optimization of SmolDocling Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/smoldocling/smoldocling.ipynb) - [Post-Training Optimization of SmolVLM2 Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/smolvlm2/smolvlm2.ipynb) - [Post-Training Optimization of GOT-OCR 2.0 Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/got-ocr2/got-ocr2.ipynb) + - [Post-Training Optimization of LTX-Video Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/ltx-video/ltx-video.ipynb) + - [Post-Training Optimization of OuteTTS Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/outetts-text-to-speech/outetts-text-to-speech.ipynb) Compression-aware training: From fdee9329957b0dfdd8eb810a7469645100647792 Mon Sep 17 00:00:00 2001 From: Nikita Malinin Date: Tue, 8 Apr 2025 10:16:47 +0200 Subject: [PATCH 15/18] Update ReleaseNotes.md --- ReleaseNotes.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index e436929a168..e091f560de0 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -35,6 +35,8 @@ Post-training Quantization: - [Post-Training Optimization of GOT-OCR 2.0 Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/got-ocr2/got-ocr2.ipynb) - [Post-Training Optimization of LTX-Video Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/ltx-video/ltx-video.ipynb) - [Post-Training Optimization of OuteTTS Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/outetts-text-to-speech/outetts-text-to-speech.ipynb) + - [Post-Training Optimization of SigLIP2 Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/siglip-zero-shot-image-classification/siglip-zero-shot-image-classification.ipynb) + - [Post-Training Optimization of OpenCLIP Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/open-clip/open-clip.ipynb) Compression-aware training: From 351f23e06a5984b76a815ff58dc74698ae3faa4b Mon Sep 17 00:00:00 2001 From: Alexander Suslov Date: Tue, 8 Apr 2025 15:24:25 +0400 Subject: [PATCH 16/18] Update ReleaseNotes.md --- ReleaseNotes.md | 1 + 1 file changed, 1 insertion(+) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index e091f560de0..df3a556378d 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -11,6 +11,7 @@ Post-training Quantization: - Fixes: - Fixed occasional failures of the weights compression algorithm on ARM CPUs. - Fixed GPTQ fails with per-channel int4 weights compression. + - Fixed weight compression fails for models with fp8 weights. - (PyTorch, Experimental) Fixed weights compression for float16/bfloat16 models. - (PyTorch, Experimental) Fixed several memory leak issues: non-detached tensors, extracted modules & graph building with gradients. - Improvements: From bd2dda7958528bc4c0b8cd4475536168fb25a060 Mon Sep 17 00:00:00 2001 From: Alexander Suslov Date: Tue, 8 Apr 2025 15:24:34 +0400 Subject: [PATCH 17/18] Update ReleaseNotes.md --- ReleaseNotes.md | 1 + 1 file changed, 1 insertion(+) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index df3a556378d..51fae5f634a 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -21,6 +21,7 @@ Post-training Quantization: - (TorchFX, Experimental) The constant folding step is removed from the `quantize_pt2e` function and the `transform_for_annotation` method of the `OpenVINOQuantizer` to align with the `torch.ao` quantization implementation. - Optimized GPTQ algorithm behavior to decrease memory & time consumption by 2.71x and 1.16x, respectively. - Added general support for optimization of models with FP8 and NF4 weights. + - Disable applying overflow fix for non 8-bit quantization. - Tutorials: - [Post-Training Optimization of Gemma3 Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/gemma3/gemma3.ipynb) - [Post-Training Optimization of GLM4-V Model](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/glm4-v/glm4-v.ipynb) From e19133cdf98e36701faf81a196ca81ecd94875f9 Mon Sep 17 00:00:00 2001 From: Lyalyushkin Nikolay Date: Tue, 8 Apr 2025 14:52:02 +0200 Subject: [PATCH 18/18] moved "tuning with LoRA" to QAT section --- ReleaseNotes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ReleaseNotes.md b/ReleaseNotes.md index 51fae5f634a..46d20a5d9c4 100644 --- a/ReleaseNotes.md +++ b/ReleaseNotes.md @@ -5,7 +5,6 @@ Post-training Quantization: - Features: - - (PyTorch) Introduced a novel weight compression method to significantly improve the accuracy of Large Language Models (LLMs) with int4 weights. Leveraging Quantization-Aware Training (QAT) and absorbable LoRA adapters, this approach can achieve a 2x reduction in accuracy loss during compression compared to the best post-training weight compression technique in NNCF (Scale Estimation + AWQ + GPTQ). The `nncf.compress_weights` API now includes a new `compression_format` option, `nncf.CompressionFormat.FQ_LORA`, for this QAT method, a sample compression pipeline with preview support is available [here](examples/llm_compression/torch/qat_with_lora). - (PyTorch) Added support for 4-bit weight compression with AWQ and Scale Estimation data-aware methods to reduce quality loss. - (PyTorch, Experimental) Introduced TorchFunctionMode support for MinMax, FastBiasCorrection, SmoothQuant, WeightCompression algorithms. - Fixes: @@ -43,6 +42,7 @@ Post-training Quantization: Compression-aware training: - Features: + - (PyTorch) Introduced a novel weight compression method to significantly improve the accuracy of Large Language Models (LLMs) with int4 weights. Leveraging Quantization-Aware Training (QAT) and absorbable LoRA adapters, this approach can achieve a 2x reduction in accuracy loss during compression compared to the best post-training weight compression technique in NNCF (Scale Estimation + AWQ + GPTQ). The `nncf.compress_weights` API now includes a new `compression_format` option, `nncf.CompressionFormat.FQ_LORA`, for this QAT method, a sample compression pipeline with preview support is available [here](examples/llm_compression/torch/qat_with_lora). - (PyTorch) Changed compression modules serialization API: `compressed_model.nncf.get_config` was changed to `nncf.torch.get_config`. The [documentation](/docs/usage/training_time_compression/quantization_aware_training/Usage.md#saving-and-loading-compressed-models) was updated to use the new API. Requirements: