Optikon: How we saved 95% on logging costs by building an internal platform

In this talk, we will show how we have built our own logging solution that handles terabytes of logs daily, reduces costs to a fraction, and better handles the complex needs of our growing company.

Thorough log ingestion, analysis, and alerting are important to organizations of almost any size. We found that merging multiple companies over the years, all with different tech and tooling stacks, led to high logging costs, disparate systems spread across multiple teams, and in some cases, less-than-ideal log coverage.

In this talk we’ll go over the solution we built as an SRE team to bring all of our engineering and platform teams together into a centralized platform. This solution combines tools running in Kubernetes and cloud-managed services to reduce maintenance costs and increase availability. It has guardrails in place to prevent unforeseen costs from rogue applications suddenly increasing log volume, and implements an abstraction layer to allow us to change any of the underlying components at will.