Karthik Ramasamy has more than 20 years of experience including being CPO of Big Data startup that got acquired by Twitter.
Karthik is entrepreneurial, energetic executive and technologist with strong leadership skills. He has extensive background in technology, engineering, product management and business development. Intimately familiar with trends and drivers in Big Data, Cloud Applications, Data Center and Network technologies.
Karthik also has a strong technical background in Big Data, Hadoop, Cassandra, Parallel Database Systems, Network Routing and Switching, AWS. Very deep understanding including Map/Reduce, Real Time Streaming, Interactive Querying, Distributed Query Execution, Query Scheduling and Route Forwarding Algorithms.
Karthik is theauthor of the best selling book "Network Routing - Algorithms, Protocols and Architectures, several publications and ten patents in large scale data processing.
Now he is Head of Streaming at Databricks.
Latency goes sub-second in Apache Spark Structured Streaming.
Apache Spark Structured Streaming is the leading open source stream processing platform. It is also the core technology that powers streaming on the Databricks Lakehouse Platform and provides a unified API for batch and stream processing. As the adoption of streaming is growing rapidly, diverse applications want to take advantage of it for real time decision making. While Spark's design enables high throughput and ease-of-use at a lower cost, it has not been optimized for sub-second latency.
In this talk, we will focus on the improvements we have made around offset management to lower the inherent processing latency of Structured Streaming. These improvements primarily target operational use cases such as real time monitoring and alerting that are simple and stateless. Extensive evaluation of these enhancements indicates that the latency has improved by 68-75% - or as much as 3X - from 700-900 ms to 150-250 ms for throughputs of 100K events/sec, 500K events/sec and 1M events/sec.