Results at a glance
- 1place to search logs from every server and device
- 6+kinds of log sources, parsed and labelled
- Alertson critical log patterns, not just metrics
- Retentionpolicies so storage stays under control
The problem
Troubleshooting meant SSH-ing into server after server and grepping. That's slow during an incident, and once a log rotates away the evidence is gone.
What I built
Grafana + Promtail + Loki
Promtail ships logs from every server to Loki. I wrote parsing and labelling rules for each source:
- Apache2 access and error logs
- syslog and SSH sessions (who logged in, from where)
- network devices
- Kannel / SMS gateway logs
- custom application logs
Grafana dashboards give search, visualisation and analysis in one place, next to the metrics, so you can jump from a spike on a graph to the log lines from the same minute.
Graylog, Filebeat and ELK
For system, application and network logs where full-text search across large volumes matters most, I also run Graylog with Filebeat, Elasticsearch and Kibana.
Retention and alerting
Retention policies keep recent logs searchable without filling the disks. Alert rules fire on critical patterns, for example repeated authentication failures or application errors, so some incidents show up in the logs before anyone notices them in the metrics.
Decisions and trade-offs
- Labels, not everything indexed. Loki indexes only labels, which keeps it cheap to run. The trade-off is that queries need good labels, so I spent the time getting labels right for each source.
- Logs plus metrics. Metrics tell you that something is wrong; logs tell you why. Having both in Grafana is what shortens troubleshooting.
Outcome
Distributed troubleshooting across the production fleet now starts with one search instead of a dozen SSH sessions, and security-relevant events such as SSH logins are recorded centrally where they can't be quietly removed from a single host.
