The Future of Streaming Analytics in Financial Services

In capital markets, this can be used for live trade flow monitoring. In payments, continuous algorithmic monitoring of fraud signatures is a common use case. In so many ways, streaming analytics will come to be an essential new digital infrastructure tool to ensure smooth and safe operation.

Streaming Analytics for Prescriptive Uses. The irony of the rising use of digital channels is that human customer service becomes more important, not less. This will take much the same shape as a NASA mission command and control center: Although there is little direct human control of the intricate inner functioning of the spacecraft, there is maximal human control over its guidance and behavior. NASA staff monitor predictive sensor readings and make prescriptive decisions. The concept of prescriptive command and control is coming to digital banking, and this computer-to-human predictive link in an algorithmic banking command and control center will be key in that future model.

Rise of Live Data Marts. Live data marts are an important new innovation that flips traditional query processing on its head. A traditional data mart requires that you send a query to a lake of data, and then pull the result. If you require a refreshed view, you must resubmit the query. In a live data mart, you compose the query and submit it to the server; the server remembers the query and streams real-time data through it. When new events match the query, they are pushed to you. In a sense, a live data mart allows you to query the future by submitting all the queries you’re interested in on a given day, and getting continuous updates as events occur. The business impact of this is broad and stunning: Any report can become a live report that’s constantly updated in real-time as conditions change.

STL Becomes a “Thing.” Extract, Transform, and Load (ETL) is a multi-billion-dollar software market of technologies that load data into data warehouses. Stream, Transform, and Load (STL) is a new data preparation component of real-time systems that consists of aggregating streams (for example, an aggregated view of the FX market in real-time as prices change), binning streams (say, organizing customer support calls in real-time, by product or issue, for routing), or smoothing (as in filtering out noise in market data streams). STL becomes increasingly important as more data sources stream into financial services firms.

New Uses for Historical Data. Data lakes will be exploited in new ways for stream processing. For example, to improve customer experience, financial services firms are capturing terabytes of data from mobile interactions, social media sources, transaction data, and so on. Data scientists and business users can evaluate that historical data and apply predictive models to understand how systems should act in the future to, for example, prevent fraudulent transactions or seize revenue-generation opportunities.

Leveraging Spark. Big data technologies such as Hadoop form data lakes; streaming analytics process data as it flows, before it enters a data lake. These two technologies should be inexorably linked: Data lakes help identify important analytics to evaluate, and streaming analytics close the loop on observations found in the lake. Still, the use of Hadoop in financial services has been limited because it’s way too slow for use in any system that touches real-time data. But now there is Spark, which features real-time, in-memory computing capabilities, and can be bridge the big data gap— capture streaming trade data in the capital markets, for example, and also use that data to discover new trading opportunities.

The future of finance is digital, and a digital financial business must be continuously aware of real-time conditions to predict and act on opportunity and threat in the moment. Streaming analytics power a new generation of algorithmic, predictive, and prescriptive systems, which are driving the digital financial services evolution and proving a source for disruption.