"Where Is My Message?"

The “Where is my message?” question itself may not read scarily, but depending on who is asking, it defines what the message is. For me, the question triggers flashbacks from when I provided FX support with the question coming from traders, who are usually lovely people… until the system went down. An experience that Paul empathised with:

“That's where you see it in two different ways according to whether you're talking about transactional / trading systems – but equally, you also get it in reliable, market data systems. In transactional systems, the missing message represents a trade or treasury transfer that can represent a large amount of money. Given that other funding and balances may depend on this message, time is of the essence to resolve its location.” 

Whenever engineers are asked the question, experience and understanding of how the systems work – from networks, 3^rd parties vendors, straight-through-processing (STP), workflows, and calculations – are crucial. Often, a stop and start of a process in the right part of the infrastructure, will ‘unblock the pipe’ and successfully deliver the message. 

Peter concurred, “In reliable messaging systems, like market data platforms, transparent system behaviour is necessary to enable visibility into where latency, slow subscribers would manifest in the form of buffers. You will know where the message is through that transparency and understanding of the system.” Referring to the A-Team event panel, he adds, “This is where I discussed the scariest question – “where is my message” – taking on a new dimension. Firms moving to the cloud, are expecting the same level of understanding and transparency as their on-premise counterparts, but in short, that transparency is not (currently) there. Equally, platforms situated in the cloud can potentially exacerbate the likelihood of message loss – and the scary question.” 

Paul adds that “ the cloud presents real advantages and introduces new opportunities to do things like automatic load balancing and scaling, but can introduce other areas to worry about. Typically, an on-premise system is scaled and highly optimised for a set maximum throughput and will run with known latencies up to this volume of data. The disadvantage of this approach is that for a lot of the time the network and hardware are underutilised, and the solution cannot be changed quickly. The autoscaling solution allows more flexibility and optimised use of network and hardware resources but may have a similar start-up scenario to a Java program that needs to get optimised as it runs through the code for the first time. When the first messages are sent and hit the load-balancers they find the optimal route and setup which may cause start-up latencies. Equally, when traffic hits certain thresholds again the solution may rebalance, resulting in additional latency spikes. The ‘where is my message’ question, now has an additional 'within timeframe”. 

For every well-known on-premise use case, this presents different challenges in the cloud. The cloud, like on-premise, should not lose the data, the operations support team must know where to look. 

Peter wraps up with what can be done, “although you don’t get the full transparency, we know with a solid ‘cloud focused’ configuration that message loss is minimised. We have found that when the cloud is configured correctly, it has incredible stability and is well worth the move from on-premise. But when it does happen, the focus we have had is on building support processes and dashboards that know where to look, equally showing where the message is not, based on the parts of the system we do have transparency of (see CJC’s monitoring and observability solutions). This normally gives a solid indication of where the message is, which speeds up the meantime to resolution. The cloud providers are continually innovating and focused on the capital markets, providing solutions for better transparency, access, and controls.”