add openvino VLM blog post #3071
Conversation
Co-authored-by: Helena Kloosterman <[email protected]>
Co-authored-by: Pedro Cuenca <[email protected]>
Co-authored-by: Pedro Cuenca <[email protected]>
Co-authored-by: Pedro Cuenca <[email protected]>
Co-authored-by: Pedro Cuenca <[email protected]>
Co-authored-by: Pedro Cuenca <[email protected]>
Co-authored-by: Pedro Cuenca <[email protected]>
Co-authored-by: Pedro Cuenca <[email protected]>
Co-authored-by: Pedro Cuenca <[email protected]>
Co-authored-by: Pedro Cuenca <[email protected]>
Co-authored-by: Pedro Cuenca <[email protected]>
* Blog updates * updates
|
Thanks a lot for the great review @pcuenca, didn't had time to include everything but will do in a second pass. The blog post is not ready for publication yet but once it is, I'll let you know |
|
Thanks a lot for your reviews @pcuenca @merveenoyan! The blog post is not ready yet (was set to draft but I should have clarified it in the description). Likely a lot will change in the following days, so don't want you to waste your time on corrections that could very well not be included in the final blog post, will let you know once ready ! |
Co-authored-by: Nikita Savelyev <[email protected]>
openvino-vlm.md
Outdated
| | openvino-8bit-woq| 0.247 | 0.016 | 0.482 | 63.928 | | ||
|
|
||
|
|
||
| This benchmark shows that small, optimized multimodal models, like [SmolVLM2-256M](https://huggingface.co/HuggingFaceTB/SmolVLM2-256M-Video-Instruct), can run efficiently on Intel CPUs. Weight-only quantization significantly reduces model size, improving efficiency without majorly impacting throughput. |
There was a problem hiding this comment.
cc @ezelanza would you mind updating once the benchmark is validated on your side
Co-authored-by: Nikita Savelyev <[email protected]>
echarlaix
force-pushed
the
openvino-vlm
branch
from
acdc35e to
60aff81
Compare
Co-authored-by: Eze Lanza (Eze) <[email protected]>
Co-authored-by: Eze Lanza (Eze) <[email protected]>
Co-authored-by: Eze Lanza (Eze) <[email protected]>
echarlaix
requested review from
IlyasMoutawwakil,
ezelanza,
merveenoyan and
pcuenca
| | Configuration |Time To First Token (TTFT)|Time Per Output Token (TPOT)| End-to-End Latency | Decoding Throughput | | ||
| |------------------|--------------------------|----------------------------|-----------------------|-------------------------------| | ||
| | pytorch | 5.150 | 1.385 | 25.927 | 0.722 | | ||
| | openvino | 0.420 | 0.021 | 0.738 | 47.237 | | ||
| | openvino-8bit-woq| 0.247 | 0.016 | 0.482 | 63.928 | |
There was a problem hiding this comment.
@ezelanza are these numbers from the notebook or optimum-benchmark ? cpu vs cpu ? I'm asking because the acceleration x65 seems too good to be true 😅
There was a problem hiding this comment.
would be great to have a reference to the benchmark code
7 participants
from https://huggingface.co/datasets/OpenVINO/documentation/blob/main/blog/openvino_vlm/openvino-vlm.md