Faster generation times

[bluenucleus]

Inference is taking too long. Are there any plans to optimise the processing times? Was anyone successful in bringing the generate times lower? A100 80gb takes upto 8-13 mins for 30 second clip. That's very expensive for a mono model.

[a43992899]

You can increase --stage2_batch_size to speed up the inference, since you have large VRAM.

Try --stage2_batch_size 16.

This issue #8 is also working on quantization.

[alisson-anjos]

I think that if you use the NF4 models you can easily increase the batch size value to 16, I'm going to do this experiment to see if using the NF4, which ends up taking up a smaller amount of VRAM, impacts the possibility of increasing the batch size value, as opposed to using the BF16.

[austin2035]

It is of little significance to optimize only the stage2 , as the reasoning in the stage 1 is very time-consuming and the GPU is not fully utilized at all.

[alisson-anjos]

There was someone who made a fork, added the possibility of using sdpa instead of flash attention and applied a patch to transformers to double the speed.

https://github.com/deepbeepmeep/YuEGP

[jrked]

There was someone who made a fork, added the possibility of using sdpa instead of flash attention and applied a patch to transformers to double the speed.

https://github.com/deepbeepmeep/YuEGP

interesting, thanks for sharing this!

[Mozer]

There's also sage attention (using triton), which is 2 times faster than flash attention. Maybe someone can implement this:
thu-ml/SageAttention#21 (comment)