Justin Miller
https://www.confluent.io/kafka-summit-london18/dont-repeat-yourself-introducing-exactly-once-semantics-in-apache-kafka
Matthias J. Sax, Confluent
View Description View Video and Slides
Nowadays users expect instantaneous, real-time interactions with a company’s products and services. Underneath these products and services you will typically find a stream processing architecture with distributed applications and microservices at work to serve all these users. In these modern architectures a key challenge is to ensure smooth and correct operation in spite of machine failures, network outages, and so on. For example, a short outage of a payment gateway should not lead to a payment being lost or sent multiple times.
The most popular streaming platform today is Apache Kafka, which is used by thousands of companies such as Zalando, Netflix, Uber, PayPal, and AirBnB to transform and reshape their industries. Kafka’s rise in popularity has demanded a revisit of its traditional at-least-once processing semantics, where Kafka had thus far guaranteed that, going back to the previous example, a payment would never be lost, but a payment could be processed more than once in the face of failures.
In this talk, we present the recent additions to Apache Kafka 1.0 to achieve what is often referred to as “the holy grail” of stream processing: the support for exactly-once processing semantics. Kafka now guarantees that a payment will be processed exactly once — not more, not less often. We discuss the newly introduced transactional APIs in Kafka, and we will demonstrate at the example of Kafka’s Streams API how these APIs can be leveraged very easily for implementing end-to-end use cases with the new, strong processing guarantees.
great talk