Design Problems/Distributed Logging System (like ELK)

Distributed Logging System (like ELK)

Medium
Log AggregationSearchTime-series

High-Level Architecture

Rendering diagram...

Design a centralized logging system that collects, stores, indexes, and enables searching of logs from thousands of services across a distributed infrastructure. Think ELK Stack (Elasticsearch, Logstash, Kibana) or Splunk. The system must handle massive log volumes while providing fast search capabilities.

Clarifying Questions

  • Log volume? (50TB/day from 10,000 services)
  • Retention? (7 days hot searchable, 30 days warm, 1 year cold/archive)
  • Query latency? (< 5 seconds for most searches within 24h window)
  • Structured vs unstructured? (Both — JSON structured + free-text)
  • Real-time tailing needed? (Yes, for debugging live issues)