switched from ElasticSearch to Weaviate

Author: ramonrails

Gemfile

@@ -16,8 +16,7 @@ gem "puma", ">= 5.0"
 gem "bcrypt", "~> 3.1.7"
 
 # Search & Caching
-gem "elasticsearch-rails", "~> 8.0"
-gem "elasticsearch-model", "~> 8.0"
+gem "faraday", "~> 2.0" # For Weaviate HTTP client
 gem "redis", "~> 5.0"
 gem "redis-namespace", "~> 1.11"
 gem "connection_pool", "~> 2.4"

Gemfile.lock

@@ -96,19 +96,6 @@ GEM
     dotenv (3.1.8)
     drb (2.2.3)
     ed25519 (1.4.0)
-    elastic-transport (8.4.1)
-      faraday (< 3)
-      multi_json
-    elasticsearch (8.19.1)
-      elastic-transport (~> 8.3)
-      elasticsearch-api (= 8.19.1)
-    elasticsearch-api (8.19.1)
-      multi_json
-    elasticsearch-model (8.0.1)
-      activesupport (> 3)
-      elasticsearch (~> 8)
-      hashie
-    elasticsearch-rails (8.0.1)
     erb (5.1.1)
     erubi (1.13.1)
     et-orbi (1.4.0)
@@ -140,7 +127,6 @@ GEM
       raabro (~> 1.4)
     globalid (1.3.0)
       activesupport (>= 6.1)
-    hashie (5.0.0)
     i18n (1.14.7)
       concurrent-ruby (~> 1.0)
     io-console (0.8.1)
@@ -232,7 +218,6 @@ GEM
     mini_portile2 (2.8.9)
     minitest (5.26.0)
     msgpack (1.8.0)
-    multi_json (1.17.0)
     net-http (0.6.0)
       uri
     net-imap (0.5.12)
@@ -469,10 +454,9 @@ DEPENDENCIES
   circuitbox (~> 2.0)
   connection_pool (~> 2.4)
   debug
-  elasticsearch-model (~> 8.0)
-  elasticsearch-rails (~> 8.0)
   factory_bot_rails (~> 6.4)
   faker (~> 3.2)
+  faraday (~> 2.0)
   fast_jsonapi (~> 1.5)
   kamal
   kaminari (~> 1.2)

README.md

@@ -1,28 +1,32 @@
 # DDoc Search
 
-A high-performance, multi-tenant document search API built with Ruby on Rails. This application provides full-text search capabilities powered by Elasticsearch, with support for tenant isolation, rate limiting, caching, and asynchronous document indexing via Kafka.
+A high-performance, multi-tenant document search API built with Ruby on Rails. This application provides full-text search capabilities powered by Weaviate, with support for tenant isolation, rate limiting, caching, and asynchronous document indexing via Kafka.
 
 ## Quick Start
 
 ### Running Locally
 
 1. **Install dependencies**
+
    ```bash
    bundle install
    ```
 
-2. **Start required services** (Elasticsearch, Redis, Kafka)
+2. **Start required services** (Weaviate, Redis, Kafka)
+
    ```bash
    docker compose -f docker-compose.dev.yml up -d
    ```
 
-3. **Setup database and Elasticsearch**
+3. **Setup database and Weaviate**
+
    ```bash
    rails db:drop db:create db:migrate
-   rails runner "Document.__elasticsearch__.create_index! force: true"
+   rails runner "Document.ensure_weaviate_schema!"
    ```
 
 4. **Create a test tenant**
+
    ```bash
    rails runner tmp/create_tenant.rb
    # Save the API key from the output!
@@ -43,9 +47,7 @@ A high-performance, multi-tenant document search API built with Ruby on Rails. T
 
 ### Ready-to-Use curl Commands
 
-```bash
-export TEST_API_KEY="df1a5764855153924486beaae96cebef739f3f54f68e28ebdf0338aea5155ee5"
-```
+Using test API key: `aead1b358e37d400e37bd9f6d031fe3a0fab53f6f6e3839b494740b7373658fe`
 
 **Health Check:**
 
@@ -57,7 +59,7 @@ curl http://localhost:3000/health | jq '.'
 
 ```bash
 curl -X POST http://localhost:3000/v1/documents \
-  -H "X-API-Key: $TEST_API_KEY" \
+  -H "X-API-Key: aead1b358e37d400e37bd9f6d031fe3a0fab53f6f6e3839b494740b7373658fe" \
   -H "Content-Type: application/json" \
   -d '{
     "document": {
@@ -68,25 +70,38 @@ curl -X POST http://localhost:3000/v1/documents \
   }' | jq '.'
 ```
 
+```bash
+curl -X POST http://localhost:3000/v1/documents \
+  -H "X-API-Key: aead1b358e37d400e37bd9f6d031fe3a0fab53f6f6e3839b494740b7373658fe" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "document": {
+      "title": "The Symphony of Earth",
+      "content": "'"$(cat test/fixtures/files/earth.txt | tr '\n' ' ' | sed 's/"/\\"/g')"'",
+      "metadata": {"category": "story", "tags": ["earth", "life", "harmony"]}
+    }
+  }' | jq '.'
+```
+
 **Retrieve Document:**
 
 ```bash
 curl http://localhost:3000/v1/documents/1 \
-  -H "X-API-Key: $TEST_API_KEY" | jq '.'
+  -H "X-API-Key: aead1b358e37d400e37bd9f6d031fe3a0fab53f6f6e3839b494740b7373658fe" | jq '.'
 ```
 
 **Search Documents:**
 
 ```bash
 curl "http://localhost:3000/v1/search?q=car&page=1&per_page=10" \
-  -H "X-API-Key: $TEST_API_KEY" | jq '.'
+  -H "X-API-Key: aead1b358e37d400e37bd9f6d031fe3a0fab53f6f6e3839b494740b7373658fe" | jq '.'
 ```
 
 **Delete Document:**
 
 ```bash
 curl -X DELETE http://localhost:3000/v1/documents/1 \
-  -H "X-API-Key: $TEST_API_KEY" | jq '.'
+  -H "X-API-Key: aead1b358e37d400e37bd9f6d031fe3a0fab53f6f6e3839b494740b7373658fe" | jq '.'
 ```
 
 ### Test Files Available
@@ -102,10 +117,10 @@ The project includes three test files in `test/fixtures/files/` that you can use
 DDoc Search is designed to handle document storage and search for multiple tenants with the following key features:
 
 - **Multi-tenant Architecture**: Complete data isolation per tenant with subdomain-based routing
-- **Full-text Search**: Powered by Elasticsearch with custom analyzers and highlighting
+- **Full-text Search**: Powered by Weaviate with BM25 keyword search
 - **Asynchronous Processing**: Kafka-based message queue for document indexing operations
 - **High Performance**: Redis caching, circuit breakers, and rate limiting
-- **Scalable Design**: Horizontal scaling support with configurable shards and replicas
+- **Scalable Design**: Horizontal scaling support with Weaviate's distributed architecture
 - **RESTful API**: Clean JSON API with comprehensive error handling
 
 ## Architecture
@@ -115,7 +130,7 @@ DDoc Search is designed to handle document storage and search for multiple tenan
 - **Framework**: Ruby on Rails 8.0.3
 - **Ruby Version**: 3.3.0 (3.4.7 for Docker)
 - **Database**: SQLite3 (development/test), with support for multiple databases in production
-- **Search Engine**: Elasticsearch 8.0
+- **Search Engine**: Weaviate 1.26.1
 - **Cache**: Redis 5.0 with connection pooling
 - **Message Queue**: Kafka (via Karafka 2.4)
 - **Background Jobs**: Sidekiq 7.0
@@ -125,7 +140,7 @@ DDoc Search is designed to handle document storage and search for multiple tenan
 ### Key Components
 
 - **Tenant Middleware**: Request-level tenant identification via API keys
-- **Circuit Breaker**: Prevents cascading failures to Elasticsearch
+- **Circuit Breaker**: Prevents cascading failures to Weaviate
 - **Rate Limiter**: Redis-based sliding window rate limiting per tenant
 - **Document Indexing**: Async Kafka-based indexing with automatic retries
 - **Search Analytics**: Background job processing for usage metrics
@@ -164,7 +179,7 @@ Ensure you have the following installed:
 - Ruby 3.3.0 or higher
 - Bundler 2.x
 - PostgreSQL (if migrating from SQLite)
-- Elasticsearch 8.0+
+- Weaviate 1.26+
 - Redis 5.0+
 - Kafka (Apache Kafka or compatible)
 
@@ -191,8 +206,8 @@ Ensure you have the following installed:
    # Database
    DATABASE_URL=sqlite3:storage/development.sqlite3
 
-   # Elasticsearch
-   ELASTICSEARCH_URL=http://localhost:9200
+   # Weaviate
+   WEAVIATE_URL=http://localhost:8080
 
    # Redis
    REDIS_URL=redis://localhost:6379/0
@@ -213,13 +228,13 @@ Ensure you have the following installed:
    rails db:seed  # Optional: creates sample data
    ```
 
-5. **Configure Elasticsearch**
+5. **Configure Weaviate**
 
-   Ensure Elasticsearch is running, then create the index:
+   Ensure Weaviate is running, then create the schema:
 
    ```bash
    rails console
-   > Document.__elasticsearch__.create_index! force: true
+   > Document.ensure_weaviate_schema!
    ```
 
 6. **Start required services**
@@ -272,7 +287,7 @@ docker build -t ddoc_search .
 docker run -d \
   -p 80:80 \
   -e RAILS_MASTER_KEY=<value from config/master.key> \
-  -e ELASTICSEARCH_URL=http://elasticsearch:9200 \
+  -e WEAVIATE_URL=http://weaviate:8080 \
   -e REDIS_URL=redis://redis:6379/0 \
   -e KAFKA_BROKERS=kafka:9092 \
   --name ddoc_search \
@@ -294,7 +309,7 @@ kamal deploy
 
 ### Application Configuration
 
-- [config/initializers/elasticsearch.rb](config/initializers/elasticsearch.rb) - Elasticsearch client configuration
+- [config/initializers/weaviate.rb](config/initializers/weaviate.rb) - Weaviate client configuration
 - [config/initializers/redis.rb](config/initializers/redis.rb) - Redis connection pool setup
 - [config/initializers/karafka.rb](config/initializers/karafka.rb) - Kafka consumer configuration
 - [config/initializers/sidekiq.rb](config/initializers/sidekiq.rb) - Sidekiq background job configuration
@@ -389,10 +404,10 @@ bundle exec brakeman
 ## Performance Features
 
 - **Caching**: Search results cached for 10 minutes, documents cached for 1 hour
-- **Circuit Breaker**: Automatic fallback to SQL search when Elasticsearch is unavailable
+- **Circuit Breaker**: Automatic fallback to SQL search when Weaviate is unavailable
 - **Rate Limiting**: Configurable per-tenant rate limits with Redis-backed sliding window
 - **Connection Pooling**: Redis connection pooling for efficient resource utilization
-- **Elasticsearch Optimization**: 10 shards, 2 replicas, custom analyzers with snowball stemming
+- **Weaviate BM25 Search**: Efficient keyword-based search with relevance scoring
 
 ## Monitoring
 
@@ -401,7 +416,7 @@ The application includes:
 - Health check endpoints at `/health` and `/up`
 - Search analytics tracking (query, results count, response time)
 - Lograge for structured logging
-- Circuit breaker metrics for Elasticsearch availability
+- Circuit breaker metrics for Weaviate availability
 
 ## License
 

app/controllers/health_controller.rb

@@ -27,7 +27,7 @@ def show
   def check_dependencies
     {
       postgresql: check_postgresql,
-      elasticsearch: check_elasticsearch,
+      weaviate: check_weaviate,
       redis: check_redis,
       kafka: check_kafka
     }
@@ -45,15 +45,15 @@ def check_postgresql
     { status: "down", error: e.message }  # Record the error message for debugging purposes.
   end
 
-  # Check Elasticsearch cluster health and response time.
-  def check_elasticsearch
+  # Check Weaviate health and response time.
+  def check_weaviate
     start = Time.current  # Record the current time for latency calculation.
 
-    # Call the Elasticsearch client to retrieve cluster health information.
-    Elasticsearch::Model.client.cluster.health
+    # Call the Weaviate client to retrieve schema information to verify connectivity.
+    WEAVIATE_CLIENT.schema.get
     { status: "up", latency_ms: ((Time.current - start) * 1000).round(2) }  # Successful execution, record latency.
   rescue => e
-    # Handle any errors during Elasticsearch connection or query execution.
+    # Handle any errors during Weaviate connection or query execution.
     { status: "down", error: e.message }  # Record the error message for debugging purposes.
   end
 

app/controllers/search_controller.rb

@@ -22,15 +22,15 @@ def index
     # Records the start time of the search operation for performance monitoring purposes.
     start_time = Time.current
 
-    # Performs the actual search on the documents using the Elasticsearch client.
+    # Performs the actual search on the documents using the Weaviate client.
     results = Document.search_for_tenant(@current_tenant.id, query, page: page, per_page: per_page)
 
     # Calculates the elapsed time since the search began in milliseconds.
     took_ms = ((Time.current - start_time) * 1000).round
 
-    # Handle both Elasticsearch and SQL results
+    # Handle both Weaviate and SQL results
     if results.respond_to?(:total)
-      # Elasticsearch results
+      # Weaviate results
       total = results.total
       documents = results.records
       SearchAnalyticsJob.perform_later(@current_tenant.id, query, total, took_ms)
@@ -66,29 +66,26 @@ def index
   private
 
   def format_search_result(document, search_results)
-    # Handle both Elasticsearch and SQL results
+    # Handle both Weaviate and SQL results
     if search_results.respond_to?(:response)
-      # Elasticsearch results - with highlighting and scores
-      highlight = search_results.response.dig("hits", "hits")
-                    .find { |h| h["_id"] == document.id.to_s }
-                    &.dig("highlight", "content")
-                    &.first
+      # Weaviate results - with scores
+      weaviate_docs = search_results.response.dig("data", "Get", Document.weaviate_class_name) || []
+      weaviate_doc = weaviate_docs.find { |d| d["title"] == document.title }
 
-      score = search_results.response.dig("hits", "hits")
-                .find { |h| h["_id"] == document.id.to_s }
-                &.dig("_score")
+      score = weaviate_doc&.dig("_additional", "score")
+      # Weaviate doesn't provide highlighting by default, so we'll use truncated content
+      snippet = document.content.truncate(200)
     else
       # SQL fallback - no highlighting or scores
-      highlight = nil
+      snippet = document.content.truncate(200)
       score = nil
     end
 
-    # Formats a search result object with the ID, title, snippet (either highlighted content or truncated original content),
-    # score, and created at time of the matched document.
+    # Formats a search result object with the ID, title, snippet, score, and created at time
     {
       id: document.id,
       title: document.title,
-      snippet: highlight || document.content.truncate(200),
+      snippet: snippet,
       score: score,
       created_at: document.created_at
     }

app/jobs/index_document_job.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
-# This class represents a job for indexing documents in Elasticsearch.
+# This class represents a job for indexing documents in Weaviate.
 # It takes care of checking document ownership, attempting to index it,
 # and logging various events throughout the process.
 class IndexDocumentJob < ApplicationJob
@@ -25,12 +25,18 @@ def perform(document_id, tenant_id)
       return
     end
 
+    # Ensure Weaviate schema exists before indexing
+    Document.ensure_weaviate_schema!
+
     # Use a circuit breaker to limit the number of concurrent indexing attempts.
-    CircuitBreaker.call(:elasticsearch) do
-      # Index the document using Elasticsearch's Ruby client. This is
-      # where the actual indexing happens, and it should succeed most
-      # of the time if everything's set up correctly.
-      document.__elasticsearch__.index_document
+    CircuitBreaker.call(:weaviate) do
+      # Index the document using Weaviate's Ruby client.
+      weaviate_object = document.to_weaviate_object
+
+      WEAVIATE_CLIENT.objects.create(
+        class_name: Document.weaviate_class_name,
+        properties: weaviate_object[:properties]
+      )
 
       # Update the document with a timestamp indicating when it was indexed.
       document.update_column(:indexed_at, Time.current)
@@ -42,7 +48,7 @@ def perform(document_id, tenant_id)
     # If the document doesn't exist in the database, log a warning and skip the indexing attempt.
     Rails.logger.warn("Document #{document_id} not found, skipping indexing")
   rescue => e
-    # Catch any other exceptions that might occur during indexing. This includes errors like network connectivity issues or Elasticsearch client errors.
+    # Catch any other exceptions that might occur during indexing. This includes errors like network connectivity issues or Weaviate client errors.
     Rails.logger.error("Failed to index document #{document_id}: #{e.message}")
 
     # If we've reached the maximum retry count (5 attempts), send a job to the dead-letter queue with the exception details.
@@ -61,3 +67,4 @@ def sidekiq_retry_count
     self.class.sidekiq_options_hash["retry_count"] || 0
   end
 end
+