Background

Amazon MSK (Managed Streaming for Apache Kafka) is a service within Amazon AWS, it is described as a fully managed service in that it manages the installation, upgrade and failure recovery of a Kafka-based solution, based on parameters provided by the administrator. Unlike Google Pub/Sub it does not provide horizontal scaling, or decisions on the capacity i.e., the level of hardware (or partitioning) needed to service the required load or routing of data. At the end of April 2022, Amazon also released a Serverless version of MSK, allowing a system to be run without the administrator managing and scaling the cluster capacity.

The tests in this document all refer to the standard (non-serverless) MSK.

Amazon AWS Transit Gateway

AWS Transit Gateway acts as a cloud router, it can be used to connect on-premises networks and solve peering problems. But an additional feature is that it supports routing multicast traffic between subnets of attached VPCs, and to the individual server instances within a VPC.

As part of this work, an additional test was done to provide an indicative impact on latency if the Transit Gateway was used for multicast.

What's Inside? 

• Kafka POC Feasibility Tests and Targets.
• Kafka Base Unitary Tests.
• 250 Bytes Target 7000 Packets Over 100-Millisecond Period.
• 250 Bytes Target 7000 Packets Over 1 Second Period.
• PCAP - Target 2000 Packets Over 10-Millisecond Period.
• PCAP - Target 7000 Packets Over 100-Millisecond Period.
• PCAP - Target 16000 Packets Over 1 Second Period.
• PCAP - Target 9.6 Million Packets Published from the Source Over a 10-Minute Period.
• Kafka Typical Dat Test - Replayed from PCAP - 19,268,676 Packets.
• Overview.
• Multicast Latency Test Results - Withing Same Zone.
• Multicast Latency Test Results - Across Zones.

“CJC has always maintained a unique position of vendor neutrality. We do not provide data nor are we tied to any specific technology or platform. The benefit of the cloud is that we have the same access to these systems as the clients. We test impartially, using real-world scenarios based on our experience and expertise and simply report the results as we observe them. We encourage discussion and collaboration with all capital markets participants and cloud providers to provide transparency and awareness of what can be done”.

Peter Williams, Chief Technology Officer at CJC.