The Perfect Storm is Coming and Our Messaging Infrastructures are at Risk
New Technologies for Market Data Management
Numbers don’t lie. Market data and transactional information volumes continue to grow at an ever-increasing rate. The financial services industry, having built its foundation on software-based data messaging systems, now faces unprecedented risks associated with hitting the limits of legacy infrastructures. New demands for high capacity and low latency are but two of the factors contributing to a larger risk profile for organizations. Too many messaging issues are a harbinger for a perfect storm on the trading floor. And the impact isn’t trivial: lost or late data wreak havoc on algorithmic trading systems – and ultimately on the bottom line.
Are the historical claims of market data messaging systems, including concepts like guaranteed delivery, still viable in today’s market? Are performance issues beginning to surface? Granted, there has been a market response with “new and improved” low-latency messaging systems, but can such point solutions systematically scale to address future data processing requirements of the financial services industry? Solving these problems requires a broader, systematic approach – not an interim, localized solution. There are lessons the market data industry can learn from the evolution of other technologies.
The financial services industry has always been a catalyst for technological evolution. With networking, the industry’s need for connectivity evolved into today’s high-performance communications infrastructure. Initially, software-based routers and bridges were satisfactory, but increasing market data volumes outpaced the capabilities of software running on commodity off-the-shelf hardware. Even though processing power was advancing rapidly, it increasingly and continually lagged behind growing data volumes. Specialized solutions became mandatory for effective operations. Network bridging morphed into silicon-based switching. And, even this wasn’t enough. The need to scale more broadly (without sacrificing performance) forced network routing to take a similar path and yield to hardware.
Does market data messaging need a similar evolution? That can’t be answered until the requirements of the new messaging infrastructure are defined. It’s important to note that unlike previous technological evolutions, there are more dimensions at play here. Yes, high capacity is imperative, but low latency and other aspects are on equal footing. This may exacerbate the challenges, but it validates the premise that there are problems on the horizon.
Seven New Technologies for Market Data Messaging
High Capacity
Message volumes are already exceeding the capacity of certain market data feeds. One example is the Options Price Reporting Authority (OPRA), which is currently modifying its message format to compensate for ominous and debilitating volume ceilings. Add in multiple feeds and inputs and a significant capacity problem emerges, which will adversely impact other areas of the messaging system. In the coming year, we will see new messaging systems that can handle large volumes of data representing millions of messages per second without negatively impacting the messaging infrastructure.
Low-Latency
Low-latency is the holy grail of messaging systems. The reason is simple: market data can be extremely time sensitive and the decision to execute a trade now comes down to a matter of microseconds. With all the banter about low-latency, it’s important to focus on the pragmatic aspects: Are the latency measurements on real-world data or on artificial lab information? How consistent and predictable is latency, both under light volumes and heavy system loads? Next generation messaging systems will deliver consistent low-latency performance independent of volatility and across the capacity spectrum.
Scalability
A messaging system isn’t an island. It’s a transport layer responsible for connecting hundreds and thousands of users, data sources and applications. Systems have to be designed for scalability from the outset. Linking disparate systems will only lead to exponential complexity and stark and disappointing capacity and latency limits. The newest messaging systems will scale to support the largest organizations seamlessly without sacrificing critical operation capabilities.
Reliability
There are two facets to reliability: operational and informational. Operational reliability addresses the capability of the messaging infrastructure to immediately respond and adapt to component failure while minimizing the exposure to those entities which use the transport. Informational reliability ensures that data is delivered in accordance with predefined service levels, regardless of the state of the subsystem. We will begin to see new messaging systems that include both operational and informational reliability.
Interoperability
Massive investments have been made into systems that use legacy messaging infrastructures. It’s simply impractical to change these applications to leverage the capabilities of a new high-performance messaging infrastructure. The transformation must be seamless. The newest data infrastructures will enable even greater connectivity among an increasing number of disparate systems and provide either a native or transformational means to support legacy systems.
Efficiency
A messaging infrastructure sits between the network layer and the application layer. This layering approach, while providing a functional demarcation, often masks the reality that there are operational dependencies as well. For example, the underlying network, whether Ethernet or InfiniBand, is a crucial component to the success of the messaging infrastructure. Insufficiently provisioned network resources can wreak havoc on the upper layers. Conversely, messaging systems can quickly amplify information volumes and compromise both the underlying network infrastructure and overlaying application layer. Vendors will develop new messaging systems that are adaptive and ensure the efficient operation of corresponding networking and application layers.
Visibility
Market data messaging is not an art. Complete visibility into capacity and latency are critical, but operational interdependencies – such as the state of the underlying network – are equally important. Without a clear, meaningful and holistic view of the broader functional system, a scientific approach to messaging tuning and refinement will not be possible. The newest and best messaging systems will provide greater and more accurate insight into system performance.
Avoiding the Storm
With plans for these next generation messaging systems laid out, one fundamental question remains: Can software-based messaging systems running on standard hardware sufficiently tackle the challenges of the evolving financial services market? It's likely that they won't, aside from perhaps a few, small isolated environments. Applying servers to the problem may provide some short-term gain, but will inevitably yield a larger data center footprint and deliver diminishing performance gains.
It’s ironic that messaging systems are the foundation for our most important financial services applications, yet they have not gone through the transformative change similar to what occurred in networking. Are the capacity challenges, latency demands, and other key requirements setting up the conditions for a perfect storm? Definitely.
Existing software-based messaging systems are failing at an alarming rate. The industry needs new, transformative combinations of hardware, software and messaging technologies to fundamentally change how companies address the deluge of messaging while data volumes continue to throttle our legacy systems. New technologies for market data management will be mandatory for avoiding the perfect storm.
Skip navigation
