Skip to content

observability.md

Latest commit

 

History

History
214 lines (164 loc) · 7.31 KB

observability.md

File metadata and controls

214 lines (164 loc) · 7.31 KB
Raw
title Observability
sidebar_position 1

The gateway provides structured logging, Prometheus metrics, and OpenTelemetry distributed tracing for production monitoring.

Structured Logging

Logging uses zap for high-performance structured output. The log format uses variable substitution for access log style output:

logging:
  level: "info"        # debug, info, warn, error
  output: "stdout"     # stdout, stderr, or file path
  format: '$remote_addr - [$time_iso8601] "$request_method $request_uri" $status $body_bytes_sent "$http_user_agent" $response_time'
  rotation:
    max_size: 100      # MB before rotation (default 100)
    max_backups: 3     # old files to keep (default 3)
    max_age: 28        # days to retain (default 28)
    compress: true     # gzip rotated files (default true)
    local_time: false  # local time in filenames (default false)

All variables are available in the format string ($remote_addr, $status, $upstream_response_time, etc.).

Log Levels

Level Description
debug Verbose debugging information
info Normal operational events (default)
warn Potentially harmful situations
error Error conditions

Log Rotation

When output is a file path, the gateway automatically rotates log files using lumberjack. Rotation is configured under logging.rotation:

Field Type Default Description
max_size int 100 Maximum size in megabytes before the log file is rotated
max_backups int 3 Number of old log files to retain
max_age int 28 Number of days to retain old log files
compress bool true Gzip-compress rotated log files
local_time bool false Use local time in rotated file names (UTC by default)

Rotation settings are ignored when output is stdout or stderr.

Prometheus Metrics

Enable the Prometheus metrics endpoint on the admin API:

admin:
  enabled: true
  port: 8081
  metrics:
    enabled: true
    path: "/metrics"     # default

Scrape metrics at http://localhost:8081/metrics:

# Fetch all Prometheus metrics
curl http://localhost:8081/metrics

# Filter for specific metric families
curl -s http://localhost:8081/metrics | grep runway_request

Collected Metrics

The gateway exports metrics for:

  • Request counts and latencies (per route)
  • Backend health and response times
  • Circuit breaker state transitions
  • Cache hit/miss ratios
  • Rate limiter rejections
  • Retry attempts and budget exhaustion
  • WAF blocks and detections
  • Traffic split distribution

Distributed Tracing

OpenTelemetry tracing with OTLP export:

tracing:
  enabled: true
  exporter: "otlp"
  endpoint: "otel-collector:4317"
  service_name: "api-gateway"
  sample_rate: 0.1          # sample 10% of requests
  insecure: true             # use insecure gRPC (for local collectors)
  headers:                   # extra headers for OTLP exporter
    Authorization: "Bearer ${OTEL_TOKEN}"

Trace Propagation

The gateway propagates trace context using W3C Trace Context headers (traceparent, tracestate). Incoming trace headers are forwarded to backends, and new spans are created for each request through the gateway.

Verifying Tracing

Check tracing status via the admin API:

curl http://localhost:8081/tracing

Request ID

Every request gets a unique X-Request-ID header. If the client provides one, it is preserved. Otherwise, the gateway generates a new UUID. The request ID is available as $request_id in log format strings, header transforms, and rule expressions.

# The gateway returns X-Request-ID in responses
curl -v http://localhost:8080/api/test 2>&1 | grep X-Request-ID

# Send a custom request ID
curl -H "X-Request-ID: my-trace-id" http://localhost:8080/api/test

Enhanced Access Logging

Per-route access log overrides allow fine-grained control over what is logged for each route. The global logging middleware remains the single log emission point — per-route settings configure what to capture and when to log.

Per-Route Overrides

routes:
  - id: payments
    path: /api/payments
    backends:
      - url: http://payments:8080
    access_log:
      enabled: true
      format: '$remote_addr $request_method $request_uri $status $response_time'
      headers_include:
        - Content-Type
        - X-Request-Id
        - Authorization
      sensitive_headers:
        - X-Internal-Token
      body:
        enabled: true
        max_size: 4096
        content_types: ["application/json"]
        request: true
        response: true
      conditions:
        status_codes: ["4xx", "5xx"]
        methods: ["POST", "PUT", "DELETE"]
        sample_rate: 0.1

Disabling Logging for a Route

routes:
  - id: health-check
    path: /health
    backends:
      - url: http://app:8080
    access_log:
      enabled: false

Header Logging with Masking

Headers listed in sensitive_headers are logged as ***. The following headers are always masked by default: Authorization, Cookie, Set-Cookie, X-API-Key. User-configured sensitive headers are merged with these defaults.

Use headers_include to log only specific headers, or headers_exclude to log all headers except some. These two options are mutually exclusive.

Body Capture

When body.enabled is true, request and/or response bodies are captured (up to max_size bytes) and included in the log output. Body capture is pass-through — writes are never buffered or delayed.

The content_types filter limits capture to specific MIME types (e.g., only capture JSON bodies, not binary uploads).

Conditional Logging

  • status_codes: Only log responses matching these patterns. Supports single codes ("200"), ranges ("200-299"), and class patterns ("4xx", "5xx").
  • methods: Only log requests with these HTTP methods.
  • sample_rate: Log a random sample of requests (0.0-1.0, where 0 means log all).

Admin API

# View per-route access log configs
curl http://localhost:8081/access-log

Key Config Fields

Field Type Description
logging.level string debug, info, warn, error
logging.output string stdout, stderr, or file path
logging.format string Access log format with $variable substitution
logging.rotation.max_size int Max MB before rotation (default 100)
logging.rotation.max_backups int Old rotated files to keep (default 3)
logging.rotation.max_age int Days to retain old files (default 28)
logging.rotation.compress bool Gzip rotated files (default true)
logging.rotation.local_time bool Local time in filenames (default false)
admin.metrics.enabled bool Enable Prometheus metrics
admin.metrics.path string Metrics endpoint path (default /metrics)
tracing.exporter string otlp
tracing.endpoint string OTLP collector endpoint
tracing.sample_rate float Sampling rate 0.0-1.0
tracing.service_name string Service name in traces

See Configuration Reference for all fields.