I5 13600k or RTX 3060 12gb?

[CyberTimon]

Hello Community
I want to build a computer which will run llama.cpp or text generation web ui.
Which should I get? Each config is about the same price.
Should I get the 13600k and no gpu (But I can install one in the future if I have money) or a "bad" cpu and a rtx 3060 12gb?
Which should I get / is faster?
Thank you in advice.

[linouxis9]

RTX 3060 will probably be faster for models that fits in its VRAM, but only llama.cpp will work for models that do not fit in your VRAM.

[CyberTimon]

Does llama 13b 4bit work with a 3060 12gb?

[heroofhyla]

13B 4bit works on a 3060 12 GB for small to moderate context sizes, but it will run out of VRAM if you try to use a full 2048 token context.

On minillm I can get it working if I restrict the context size to 1600.

On text-generation-webui, I haven't found a way to explicitly limit context size, but I can also avoid it running out of VRAM by setting --pre_layer 40, which to my understanding puts 40 of the model's layers on the GPU and the rest on CPU. I don't know how many layers there are total.

Unfortunately the --pre-layer strategy doesn't scale well to bigger models like 30B. The CPU performance on text-generation-webui is much worse than llama.cpp, so much that you lose the benefit of the GPU layers.