Distributed Log Service (Go)

This repository contains a persistent event-stream / commit log service implemented in Go. The service starts as a single-node log that persists records to disk and evolves into a secure, observable, distributed system that can be deployed to Kubernetes.

The focus is on real-world concerns: storage, networking, security, observability, consensus, and deployment.

---

High-Level Overview

The system is built in stages, starting from a simple core and growing into a distributed service:

1. Single-node HTTP service
   Expose a JSON-over-HTTP API to produce and consume records from a commit log.

2. gRPC-based service
   Introduce Protocol Buffers and gRPC for stronger contracts, streaming support, and better performance.

3. Secure service

   • TLS and mutual TLS (mTLS) between clients and servers
   • Internal Certificate Authority (CA) to issue and rotate certificates
   • Basic authorization with access control lists (ACLs)

4. Observable system

   • Metrics for visibility into traffic and performance
   • Structured logging
   • Traces for request flows (where applicable)

5. Distributed cluster

   • Service discovery (gossip-style membership)
   • Log replication between nodes
   • Raft-based consensus providing a replicated state machine
   • Client-side discovery and load balancing in gRPC

6. Deployment on Kubernetes

   • Local development with kind
   • Packaging and configuration with Helm
   • Deployment to a managed Kubernetes cluster

By the end, this becomes a functioning event-streaming system with the core building blocks expected in modern distributed services.

---

Project Goals

• Build an end-to-end distributed service in Go, not just isolated examples.
• Deepen understanding of:
  • Commit logs and event streaming
  • gRPC and Protocol Buffers
  • TLS/mTLS and internal PKI
  • Service discovery and membership
  • Raft consensus and replicated state machines
  • Client-side load balancing
  • Kubernetes primitives and deployment patterns
• Maintain a codebase that resembles a production service:
  • Clear modular package layout
  • Tests for the log, network layer, and distributed behavior where feasible
  • Simple, repeatable build and run workflows
  • Room for extensions and experiments

---

Repository Structure (Work in Progress)

This is the approximate structure the project is targeting. It will evolve as the implementation grows.

.
├── cmd/
│   └── server/          # Main binary entrypoint for the log service
├── internal/
│   ├── log/             # Commit log implementation and storage mechanics
│   ├── server/          # gRPC/HTTP servers, middleware, and service wiring
│   ├── discovery/       # Service discovery integration (gossip/membership)
│   ├── raft/            # Raft consensus integration and state machine wiring
│   ├── tls/             # TLS/mTLS configuration helpers and CA utilities
│   └── observability/   # Metrics, logging, and tracing setup
├── api/
│   ├── http/            # HTTP handlers and JSON API (early stage)
│   └── proto/           # Protocol Buffer definitions and generated code
├── deploy/
│   ├── k8s/             # Raw Kubernetes manifests (if needed)
│   └── helm/            # Helm chart(s) for deployment
├── scripts/             # Helper scripts (cert generation, local cluster, etc.)
├── go.mod
└── README.md