MiniLM can handle like 40 lines of code in one function
so lets a say if a function is bigger , and maybe around 200 lines (assuming that im indexing a big repo) , then the embeddings would be terrible.
so some suggested solutions:
-write a chunker
-Detects functions longer than ~1,200 chars
-Splits them into overlapping 50-line windows (overlap prevents splitting a sentence mid-thought)
-Creates multiple entries for one function — each chunk gets its own vector
-On search, if any chunk matches, you show the full function
MiniLM can handle like 40 lines of code in one function
so lets a say if a function is bigger , and maybe around 200 lines (assuming that im indexing a big repo) , then the embeddings would be terrible.
so some suggested solutions:
-write a chunker
-Detects functions longer than ~1,200 chars
-Splits them into overlapping 50-line windows (overlap prevents splitting a sentence mid-thought)
-Creates multiple entries for one function — each chunk gets its own vector
-On search, if any chunk matches, you show the full function