Gravity Tech: Centralized logging (and more) with Apache Kafka

With numerous large enterprise customers (and several smaller ones), recommendation engines in the Gravity SaaS infrastructure generate around a terabyte of application logs a day, scattered on more than fifty hosts. This counts as moderate in today’s “big data” world, but it already exceeds the size where naive approaches to log aggregation and processing (like copy scripts to a central host or grepping on individual hosts) would work effectively: what you would like to do is see and process all logs at once, near real-time. Generally speaking, this is a distributed system, hence it needs a messaging solution.

(more…)

Read More