Skip to content

Log aggregation overview

Jump to bottom
Deepak Narayana Rao edited this page Oct 16, 2017 · 8 revisions

Tech Stack

  • ELK: Log Aggregation
    • Elasticsearch : Log storage and search APIs
    • Logstash: Transforms logs into structured data to be stored in elsticsearch
    • Kibana: UI to search logs and visualise data
  • Logspout: collecting logs from containers
  • Filebeat: collecting logs from VMs

Please try this simple walkthrough tutorial to get hands on experience on log aggregation https://botleg.com/stories/log-management-of-docker-swarm-with-elk-stack/

Overview

Image : Edit Link

Collecting logs from containers in docker swarm

  • Logspout is run on all worker nodes by running as global replication mode in docker swarm
  • Logspout container on each server communicates to docker engine using docker APIs and fetches logs from all containers running on its host server
  • Logstash is run as a service inside the docker swarm
  • Logstash service is run with syslog input plugin which enables logstash to acts as syslog server
  • Logspout pushes the logs to Logstash using syslog protocol
  • Logstash parses logs to structure the data and pushes structured logs to elasticsearch

Clone this wiki locally