Performance improvement through multithreading and better GPU utilization #196
Description
The codebase_rag/embedder.py script does not support multithreading and uses a batch size that is too small for efficient GPU utilization. When indexing/embedding a large codebase, the process takes several days, although with minor changes to this script the performance could likely be improved by at least a factor of 10.