RFC-104 - Add native Vector Search Index type for Hudi Vector Type

[chrevanthreddy]

This RFC proposes native vector similarity search support in Apache Hudi, enabling approximate nearest neighbor (ANN) queries on embedding columns stored in Hudi tables. The design extends the new VECTOR type in the main data table. A lightweight cluster-routing index in the Hudi metadata table provides file-group pruning, while future hidden columns in the main Parquet files may store RaBitQ binary codes and scalars for fast within-file ANN scanning. Cluster assignment itself stays in MDT.
https://github.com/chrevanthreddy/hudi/blob/f67ef0972f9a8a2928a92cf8b61ad2a81ab3cd72/rfc/rfc-104/rfc-104.md

https://github.com/chrevanthreddy/hudi/pull/1/changes#diff-e8be007ae70221ad49cd11b305f4dbe6d1d09b344549a877555a7eb376d16f2d

Starting off the discussion for Vector Search Index. Numbers regarding the initial testing can be added here to verify effectiveness of adding the index. And also cost of the adding index and maintaining vs brute force vector search

[rahil-c]

@chrevanthreddy Thanks for starting this, will take a look soon!

[chrevanthreddy]

Benchmarking tests on 1Million dataset 1500 dimensions each vector on public dataset :
The ivf-exact path does:

batch_queries crossJoin broadcast(ivf_centroids)
rank centroids per query by squared L2
keep nprobe
join to ivf_vectors by cluster_id
compute exact cosine distance
rank per query by exact distance
keep top-20
Observed 10k-query no-write result:

IVF-only probe rows: 40000
IVF-only probe stage: 29.480s
IVF-only materialize stage: 2042.473s
IVF-only result rows: 200000
IVF-only total runtime (no GCS write): 2071.955s

The ivf-rabitq path does:

batch_queries crossJoin broadcast(ivf_centroids)
rank centroids per query by squared L2
keep nprobe
join to ivf_postings by cluster_id
compute Hamming distance
keep top 1000 candidates per query
join surviving candidates to ivf_vectors
compute exact cosine distance
rank per query by exact distance
keep top-20
Observed 10k-query no-write result:

IVF probes rows: 40000
IVF candidate rows (top-1000 per query): 10000000
Probe stage: 133.598s
Hamming stage: 175.685s
IVF + RaBitQ materialize stage: 178.305s
IVF + RaBitQ result rows: 200000
IVF + RaBitQ total runtime (no GCS write): 487.591s

[vinothchandar]

@chrevanthreddy Can you please open a DRAFT PR in the Hudi repo from the branch and reference here.. It ll be easier to get more community eyes here directly.