Kafka
Paper: Kafka Kafka/Distributed Messaging System Goal Design a distributed messaging system that can reliably transfer a high throughput of messages between different entities. Background One common challenge in distributed systems is handling continuous influx of data from multiple sources. E.g. Imagine a log aggregation service that can receive hundreds of log entries per second from different sources. Function of this log aggregation service is to store these logs on a disk at a shared server and build an index on top of these logs so that they can be searched later. Challenges of this service? Distributed Messaging Systems(or Asynchronous processing paradigm) can help. What is a messaging System? System responsible for transferring data amongst various disparate systems like apps, services, processes, servers etc, w/o introducing additional coupling b/w producers and consumers, and by providing asynchronous way of communicating b/w sender and receiver. Two types of Messaging Systems Queue A Particular message can be consumed by one consumer only. Once a message is consumed, it’s removed from the queue. Limits the system as the same messages can’t be read by the multiple consumers. Publish-Subscribe Messaging System In the Pub-Sub model, messages are written into Partitions/Topics. ...