Unprecedented message volatility needs instant and predictive capacity management.
Steve Moreton, Global Head of Product Management, CJC
The last few months have seen dramatic upheaval and change to our daily lives. Everyone has been affected, regardless of background or circumstance.
In parallel, the capital markets and related technology have also been impacted. Part of our work at CJC is to provide 24x7x365 global support for thousands of servers in hundreds of clients and this gives us perhaps a unique perspective on the extent of that impact, as well as how it can be managed.
Last week, Max Bowie at WatersTechnology wrote about the recent outages on the Nasdaq BX and Cboe Fix market depth feeds, having been hit by an options overload. These were reported by the exchanges as being in a period between Feb 25th and March 13th and CJC confirm that this date range correlates with capacity increases in the platforms we manage.
One of the key market data systems CJC are well known to support is the Refinitiv TREP platform. This platform is central to many financial firms’ data architectures and CJC have many clients who leverage our capabilities on a managed service basis.
As shown above, this client’s normal behaviour is up to 2.5million updates per second (period shown between 10th – 24th Feb). However from 25th Feb, update rates increase almost every day to a peak of 6.6million - representing an eye watering 164% increase in load. Normal behaviour does not return until 16th March.
As a world class provider of managed services, we are consistently held to high standards and have strict SLAs and KPIs in place with some of the biggest names in the capital markets. It is imperative that we know how world events impact client infrastructure technology at the earliest opportunity. The data we derive from mosaicOA provides our managed service teams with indicative behavioural trends - which CJC then leverage to benefit our entire client base.
TREP is not the only system for which CJC provide managed services and tooling – another such system is Solace.
This firm above sends over 50billion messages of capital markets related data around their enterprise every day. From 25th Feb, that figure increased rapidly to over 90billion messages per day – almost a 90% increase in usage.
Although the above views are informative, looking at one month of data from ‘altitude’ is only helpful to an extent. In the capital markets, capacity management can be in the next few hours – examining the data intraday is far more effective. On the morning of 10th March for example, where a peak of almost 94billion was reached, mosaicOA demonstrated that peak was coming before it occurred.
Our mosaicOA tool has powerful ‘busy day’ views, which were built specifically for major world events such as COVID-19. These let end users know if they are above normal capacity and where they will likely be later in the day.
Solace client Busy Day view – showing messages going beyond normal tolerance
Busy day provides a 24-hour window of time and uses a custom historical min, mean and max (last 7 days or last XXX days). Importantly it shows usage of the system live. It is immediately possible to see when a system has started to go beyond normal tolerance levels - earlier than it finally hits an important capacity threshold – and this provides vital early warnings to support teams, who can deliver remediation processes in advance.
World events are still ongoing of course and will continue to impact technology over the weeks and months to come. Although capacity management is a business as usual process, CJC are continuing to support and work with our client base to help keep systems running during this unprecedented time of volatility.