The Future of Streaming Analytics in Financial Services
Mark Palmer, SVP-Engineering & General Manager, TIBCO Software
Mark Palmer, SVP-Engineering & General Manager, TIBCO Software
For example, streaming analytics algorithms can take sliding windows of data and, through just a few programming primitives, constantly pepper those market data streams with queries and conditions: “Trade when the spread between these two stocks deviates by more than 5 percent in any 5 second period” or “This customer hasn’t responded in 5 days to our request for information — send an alert to customer service.” Where traditional analytics look in the rear-view mirror at data lakes to evaluate what has already happened, streaming analytics parse what’s happening in-the-moment and leverage technologies that work on data as its moving.
The pioneering killer app for streaming analytics was algorithmic trading. But today, an even more common use for streaming analytics is continuous computing: continuous risk management; continuous customer engagement; continuous market monitoring. Aite Group, an independent research and advisory firm, recently studied technology in financial services and found that streaming analytics impact the industry in an array of applications. Beyond increased sophistication for electronic trading, streaming analytics implementations are now prevalent in risk management, regulatory compliance, client service, and fraud detection. These use cases aren’t just about automated action. They are about conveying a continuous, real-time, 360-degree view of the financial ecosystem to human users, and providing them with the ability to act with authority on real-time conditions.
So how will this play out in the financial services future?
Streaming Analytics for Algorithmic Operations. A 2015 Bain survey of insurance companies projected that digital channels will significantly replace physical channels in the next 3 to 5 years, with 20-40 percent of physical activities transitioned to digital. One important and rising use case is real-time monitoring, where streaming analytics can absorb streams of log updates, application updates, and transactional updates to create a single business-level dashboard of all digital operations.
The irony of the rising use of digital channels is that human customer service becomes more important, not less
In capital markets, this can be used for live trade flow monitoring. In payments, continuous algorithmic monitoring of fraud signatures is a common use case. In so many ways, streaming analytics will come to be an essential new digital infrastructure tool to ensure smooth and safe operation.
Streaming Analytics for Prescriptive Uses. The irony of the rising use of digital channels is that human customer service becomes more important, not less. This will take much the same shape as a NASA mission command and control center: Although there is little direct human control of the intricate inner functioning of the spacecraft, there is maximal human control over its guidance and behavior. NASA staff monitor predictive sensor readings and make prescriptive decisions. The concept of prescriptive command and control is coming to digital banking, and this computer-to-human predictive link in an algorithmic banking command and control center will be key in that future model.
Rise of Live Data Marts. Live data marts are an important new innovation that flips traditional query processing on its head. A traditional data mart requires that you send a query to a lake of data, and then pull the result. If you require a refreshed view, you must resubmit the query. In a live data mart, you compose the query and submit it to the server; the server remembers the query and streams real-time data through it. When new events match the query, they are pushed to you. In a sense, a live data mart allows you to query the future by submitting all the queries you’re interested in on a given day, and getting continuous updates as events occur. The business impact of this is broad and stunning: Any report can become a live report that’s constantly updated in real-time as conditions change.
STL Becomes a “Thing.” Extract, Transform, and Load (ETL) is a multi-billion-dollar software market of technologies that load data into data warehouses. Stream, Transform, and Load (STL) is a new data preparation component of real-time systems that consists of aggregating streams (for example, an aggregated view of the FX market in real-time as prices change), binning streams (say, organizing customer support calls in real-time, by product or issue, for routing), or smoothing (as in filtering out noise in market data streams). STL becomes increasingly important as more data sources stream into financial services firms.
New Uses for Historical Data. Data lakes will be exploited in new ways for stream processing. For example, to improve customer experience, financial services firms are capturing terabytes of data from mobile interactions, social media sources, transaction data, and so on. Data scientists and business users can evaluate that historical data and apply predictive models to understand how systems should act in the future to, for example, prevent fraudulent transactions or seize revenue-generation opportunities.
Leveraging Spark. Big data technologies such as Hadoop form data lakes; streaming analytics process data as it flows, before it enters a data lake. These two technologies should be inexorably linked: Data lakes help identify important analytics to evaluate, and streaming analytics close the loop on observations found in the lake. Still, the use of Hadoop in financial services has been limited because it’s way too slow for use in any system that touches real-time data. But now there is Spark, which features real-time, in-memory computing capabilities, and can be bridge the big data gap— capture streaming trade data in the capital markets, for example, and also use that data to discover new trading opportunities.
The future of finance is digital, and a digital financial business must be continuously aware of real-time conditions to predict and act on opportunity and threat in the moment. Streaming analytics power a new generation of algorithmic, predictive, and prescriptive systems, which are driving the digital financial services evolution and proving a source for disruption.
Check Out: Top Fintech Solution Companies
