All posts

    DATA ENGINEERING

    Real-time Analytics at Petabyte Scale

    2 MIN READ

    The architecture behind handling billions of events per second with sub-millisecond query latency.

    Start with the question

    Real-time is expensive, so first ask how fresh the data truly needs to be. Fraud checks may need an answer within moments, while most business dashboards are fine a minute behind. That single requirement shapes the whole design, and being honest about it can remove a great deal of complexity.

    A durable log at the centre

    Most large systems place an append-only event log, such as Kafka or a compatible service, between producers and consumers. Producers write without knowing who reads. Consumers read at their own pace and can replay history after a bug fix. Partitioning spreads load across machines, and the partition key deserves care because it decides both ordering and hot spots.

    Processing and storage

    A stream processor such as Flink then cleans, joins and aggregates events over time windows. It has to handle events that arrive late or twice, so deduplication and idempotent writes should be designed in from the beginning.

    Results land in a columnar store such as ClickHouse, Apache Pinot or Apache Druid. Columnar layouts read only the fields a query needs and compress well. Pre-computed rollups and careful sort keys are what keep queries fast as data grows. Consistently fast queries at this scale are usually the result of doing the work before the query arrives.

    Keep the bill under control

    Very large systems are as much a cost problem as a technical one. Tiered storage moves older data to cheaper object storage. Retention rules delete what nobody queries, and sampling can be a good answer for high-volume diagnostic data.

    Operate it like a product

    Track consumer lag, freshness and data quality as carefully as you track uptime. A dashboard that is fast and wrong does more damage than one that is slow. Define contracts for your event schemas early, because a quiet change in one producer can corrupt everything downstream.