Our use case Here at Fexco, a fundamental part of our architecture requires real time stream processing. Specifically, the fast lane stream processing. To know what I’m talking about, please…
I recently configured a Kafka enabled Event Hub in Azure. I used a Spark Scala cluster to stream these events. The following are my findings. Kafka Enabled Event Hub First…
Background One of the main characteristic of eventual consistency systems is that, we should never update data, just add it. A lot this topic has been discussed in this other…
Hi all, We’ll try to reflect in this post a summary of the main steps to follow when we want to create an ETL process in our Computing Platform. What…
On the last article, we could see an overview about the front end architecture and how we can reduce the amount of requests to Cloud, reducing the network traffic and…
Worried about how to provide a quality SDLC to Databricks based Spark applications? REQUIREMENTS FOR THE READER Please read first this post previously to understand the current topic. Have a…
..or how to work with Spark Applications Abstract This article contains a description of procedures for best practices and Continuous Delivery processes with Spark Applications. Since there is basically three…
A previous post showcased how Apache Spark unifies pipelines with common APIs simplifying the development of data intensive computations. This week we’ll discuss one of the most ubiquitous examples: the…
Last week, the Spark Summit 2017 took place at the Dublin Convention Center. A lot of useful presentations are worth mentioning. Some of the main themes include Data Science, Machine…
Figure 1: Apache Spark architecture overview (courtesy of databricks) Software Engineering and juggling have some interesting similarities. Both activities must be broken down into simple easy to follow tasks, or…