Budapest Data 2015

In a world where every "Thing" is producing lots of data, ingesting and processing that large volume of data becomes a big problem. In today's dynamic world, firms have to react to changing conditions very fast, or even better in real time. In this talk we will take on this interesting challenge using latest and greatest tools from Big Data community. We will try to combine awesomeness of Kafka, a resilient pub-sub messaging system, with the powers of Spark streaming for scalable, high-throughput, fault-tolerant stream processing of live data streams. Combining different systems to get even a more powerful system is great, but has its own complexity. With a demo of building a pipeline to ingest and process real time data using these systems, we will explore how the two systems can be intertwined to make the most out of the combined system.