Building on a previous "Multicast in the Cloud" insight article, this technical document has been written to present the findings from the CJC Pub/Sub AWS Transit Gateway Feasibility POC Testing market data via AWS Transit Gateway leveraging PCAP recording and Kafka delivery. It provides high-level background to the tests, a results summary, and a breakdown of the key tests and how they were run.
Amazon MSK (Managed Streaming for Apache Kafka) is a service within Amazon AWS, it is described as a fully managed service in that it manages the installation, upgrade and failure recovery of a Kafka-based solution, based on parameters provided by the administrator. Unlike Google Pub/Sub it does not provide horizontal scaling, or decisions on the capacity i.e., the level of hardware (or partitioning) needed to service the required load or routing of data. At the end of April 2022, Amazon also released a Serverless version of MSK, allowing a system to be run without the administrator managing and scaling the cluster capacity.
The tests in this document all refer to the standard (non-serverless) MSK.
Amazon AWS Transit Gateway
AWS Transit Gateway acts as a cloud router, it can be used to connect on-premises networks and solve peering problems. But an additional feature is that it supports routing multicast traffic between subnets of attached VPCs, and to the individual server instances within a VPC.
As part of this work, an additional test was done to provide an indicative impact on latency if the Transit Gateway was used for multicast.
“CJC has always maintained a unique position of vendor neutrality. We do not provide data nor are we tied to any specific technology or platform. The benefit of the cloud is that we have the same access to these systems as the clients. We test impartially, using real-world scenarios based on our experience and expertise and simply report the results as we observe them. We encourage discussion and collaboration with all capital markets participants and cloud providers to provide transparency and awareness of what can be done”.
Peter Williams, Chief Technology Officer at CJC.
CJC is the leading market data technology consultancy and service provider for global financial markets. CJC provides multi-award-winning consultancy, managed services, cloud solutions, observability, and professional commercial management services for mission-critical market data systems. CJC is ISO 27001 certified, enabling CJC’s partners the freedom to focus on their core business.