Wafer.ai Benchmarks Kimi K3 Open Model on AMD MI355X: Low-Cost Alternative to NVIDIA B300 Despite ROCm Software HurdlesAI acceleration agent service Wafer.ai has published a performance benchmark evaluating Kimi K3 currently the highest-ranked open-weights model available for self-hosting running on AMD’s Instinct MI355X accelerators.
Thanks to the MI355X's massive high-bandwidth memory (HBM) capacity, the entire Kimi K3 model can be deployed on a single 8-GPU node. This offers significant cost savings over NVIDIA's flagship B300 hardware: MI355X cloud instances currently rent for approximately $2.50 per GPU/hour, compared to $6.00 per GPU/hour for the B300.
However, the report highlights notable software optimization friction when deploying Kimi K3 on AMD's ROCm stack. While the model launched out of the box, initial performance was subpar due to key framework incompatibilities:
Speculative Decoding Failure: The spec-decoding module (Kimi-K3-DSpark) initially failed on ROCm. Wafer.ai resolved this by rewriting a single PyTorch function within the pipeline.
Prefill Optimization & MLA Kernel Mismatch: During the prefill stage, ROCm failed to load the AITER Multi-Head Latent Attention (MLA) kernel because Kimi K3 utilizes 12 attention heads a non-standard configuration compared to ROCm’s supported head counts of 4, 8, or 16. The team bypassed this limit by padding the remaining 4 attention head slots with zeros to reach a supported 16-head size, immediately boosting inference throughput by 2x to 3x.
Final benchmark comparisons confirm that NVIDIA’s B300 retains a distinct raw performance lead, achieving a decode speed of 172 tokens per second compared to the MI355X’s 118 tokens per second. Consequently, AMD’s key competitive edge in high-parameter LLM serving remains its compelling price-to-performance ratio rather than peak throughput speed.
Why is single-node serving important? Dividing a large number of high-parameter Mixture-of-Experts (MoE) models across multiple server chassis results in severe network latency over InfiniBand or Ethernet. AMD MI355X's large HBM allocation allows high-parameter models like the Kimi K3 to operate within a single 8-GPU case, preserving in-node connectivity bandwidth (Infinity Fabric) while significantly reducing hosting complexity.
AMD's hardware is competitive, but software optimization requires hands-on engineering. The need to increase the number of zeros on attention heads from 12 to 16 to take advantage of AITER MLA kernels highlights why many organizations continue to pay more for CUDA on NVIDIA B300s. NVIDIA offers readily available performance, while AMD requires engineering teams to fine-tune the kernel to extract maximum hardware performance.
The technique Wafer.ai uses for its tech-savvy audience utilizes GPU Tensor Cores optimized for hardware matrix multiplication arranged in powers of two (4, 8, 16, 32). When the Kimi K3's native 12-head attention mechanism runs on ROCm, it reverts to an unoptimized, slower processing path. By filling the matrix with zeros to emulate the 16-head structure, the GPU can process high-speed vector instructions, sacrificing a small amount of memory for a massive 2-3x speed increase.
Source: Wafer.ai
Wafer.ai Benchmarks Kimi K3 Open Model on AMD MI355X: Low-Cost Alternative to NVIDIA B300 Despite ROCm Software HurdlesAI acceleration agent service Wafer.ai has published a performance benchmark evaluating Kimi K3 currently the highest-ranked open-weights model available for self-hosting running on AMD’s Instinct MI355X accelerators.
Thanks to the MI355X's massive high-bandwidth memory (HBM) capacity, the entire Kimi K3 model can be deployed on a single 8-GPU node. This offers significant cost savings over NVIDIA's flagship B300 hardware: MI355X cloud instances currently rent for approximately $2.50 per GPU/hour, compared to $6.00 per GPU/hour for the B300.
However, the report highlights notable software optimization friction when deploying Kimi K3 on AMD's ROCm stack. While the model launched out of the box, initial performance was subpar due to key framework incompatibilities:
Speculative Decoding Failure: The spec-decoding module (Kimi-K3-DSpark) initially failed on ROCm. Wafer.ai resolved this by rewriting a single PyTorch function within the pipeline.
Prefill Optimization & MLA Kernel Mismatch: During the prefill stage, ROCm failed to load the AITER Multi-Head Latent Attention (MLA) kernel because Kimi K3 utilizes 12 attention heads a non-standard configuration compared to ROCm’s supported head counts of 4, 8, or 16. The team bypassed this limit by padding the remaining 4 attention head slots with zeros to reach a supported 16-head size, immediately boosting inference throughput by 2x to 3x.
Final benchmark comparisons confirm that NVIDIA’s B300 retains a distinct raw performance lead, achieving a decode speed of 172 tokens per second compared to the MI355X’s 118 tokens per second. Consequently, AMD’s key competitive edge in high-parameter LLM serving remains its compelling price-to-performance ratio rather than peak throughput speed.
Why is single-node serving important? Dividing a large number of high-parameter Mixture-of-Experts (MoE) models across multiple server chassis results in severe network latency over InfiniBand or Ethernet. AMD MI355X's large HBM allocation allows high-parameter models like the Kimi K3 to operate within a single 8-GPU case, preserving in-node connectivity bandwidth (Infinity Fabric) while significantly reducing hosting complexity.
AMD's hardware is competitive, but software optimization requires hands-on engineering. The need to increase the number of zeros on attention heads from 12 to 16 to take advantage of AITER MLA kernels highlights why many organizations continue to pay more for CUDA on NVIDIA B300s. NVIDIA offers readily available performance, while AMD requires engineering teams to fine-tune the kernel to extract maximum hardware performance.
The technique Wafer.ai uses for its tech-savvy audience utilizes GPU Tensor Cores optimized for hardware matrix multiplication arranged in powers of two (4, 8, 16, 32). When the Kimi K3's native 12-head attention mechanism runs on ROCm, it reverts to an unoptimized, slower processing path. By filling the matrix with zeros to emulate the 16-head structure, the GPU can process high-speed vector instructions, sacrificing a small amount of memory for a massive 2-3x speed increase.
Source: Wafer.ai
Comments
Post a Comment