Demystifying CTA Wave Scheduling Trade-offs: Row-Major or Column-Major?
TL;DR. The tensor dimensions (M, N, K) and the GPU’s L2 cache capacity determine which tile schedule gives better data reuse. Row-major tends to win at high per-expert concurrency, as in large-batch wide expert parallelism (Wide-EP). Nearby thread blocks reuse the same activations while L2 serves the weights. Column-major tends to win for wide projections with fewer tokens per expert, where keeping the weights close in time saves more memory traffic.
- 1 × NVIDIA GB300
- MegaMoE FP8 Kernel
- 129.25 MiB L2
- 76 CTA pairs
- 128 × 256 tile per pair
What a wave of output tiles shares
In a mixture-of-experts layer, one expert receives M routed tokens. Its up projection consumes K hidden features per token and produces N intermediate features. Matrix multiplication partitions that output into tiles.
How one output tile is built
A cooperative thread array (CTA) is a CUDA thread block. Think of a wave as the set of output tiles that the GPU has room to work on together. Adjacent tasks in a row-major traversal share activation rows. Adjacent tasks in a column-major traversal share weight columns. Each output tile runs the same reduction along K.
Two production schedulers make this choice at different scales. The original Hopper DeepGEMM groups output columns into bands of up to 16 N tiles. It visits the columns of a band for one row tile, then advances through all row tiles before opening the next band. That keeps a band of weights in circulation. Blackwell MegaMoE distributes column groups across scheduling domains. Within a domain, its fastest-changing index picks the column group for the current row tile, bringing consumers of the same activations close together. Our comparison isolates the two traversal extremes within one expert. Hopper scheduler · MegaMoE scheduler
Sixteen consumers can share one trip to HBM
For a 4,096-column gate/up projection, MegaMoE assigns 256 output columns to each CTA pair. That gives one activation row tile 16 consumers. Tracing the native kernel showed that distinct CTA pairs generally loaded it into their own SM shared memory. Where did the repeated loads get their bytes?
With many tokens per expert, the row-major replay fetched roughly one copy of each unique activation tile from HBM. The remaining consumers could load it from L2 into their own shared memory. Column-major separated those consumers by a sweep over the other token rows, and more of their requests reached HBM.
The weight replay tells a different story. At that same token count, both orders read approximately one copy of each expert’s weights from HBM. The individual expert matrix occupied only 16 MiB. Reusing its addresses also mattered in a fused two-GPU test. At 512 tokens per active expert and eight row tiles per expert, native MegaMoE took 0.190 ms with shared weights and 0.405 ms when each row tile addressed a separate, byte-identical copy. The matrix products were the same. Changing the opportunity for cache reuse more than doubled kernel latency.
When should the scheduler switch?
Consider the two shapes in the middle map. Both use K = 4,096 and FP8 operands. With 4,096 tokens and 32,768 output columns, activations occupy 16 MiB and weights occupy 128 MiB. Column-major keeps the large weight operand close in time while revisiting the smaller activation operand. Transposing the output shape swaps those footprints. Now activations occupy 128 MiB, and row-major keeps their consumers together.
How much weight needs to fit? At 32,768 tokens and K = 4,096, a single 256-column group occupies just 1 MiB. In a wider sweep we kept that group size fixed and expanded the full expert weight matrix to 192 MiB. Column-major then cut latency by 11.8% across the paired runs. Estimated whole-device HBM traffic fell from 30.9 to 18.4 GB per projection. The key quantity for row-major is how much data passes through the cache between uses of a group.
Let and be the sizes of the activations and the weights. The element sizes are in bytes. Divide the output into R row tiles and G column groups. Row-major revisits the weights for each successive row tile. Column-major revisits the activations for each successive column group.
Here is the L2 capacity available to the loads. Each e term gives exposed time per revisited byte, accounting for cache misses and how much latency overlaps compute. The boundary is where . Below it on the maps, row-major usually has the smaller reload bill. Increasing output width grows the weight sweep. Increasing tokens per expert grows the activation sweep.
The bend in the black contour follows from this tradeoff. Once most activation revisits already reach HBM under column-major, adding rows can do little more to increase the miss cost per byte. The output width at which column-major pulls ahead can then grow more slowly with token count. The open circles show where the experiment has little power to resolve the precise bend.
We can describe each point on the map by two ratios, activation bytes / usable L2 and weight bytes / usable L2. We checked it by doubling K. Halving M and N at the same time preserves both footprints. Across 112 matched comparisons, the average absolute difference in scheduling advantage was 4.0 percentage points at matched footprints, versus 6.9 points at matched M and N. This was a fixed-GPU test. Scaling the boundary to a different physical L2 capacity remains a prediction. Tile geometry, output traffic and overlap can move it.
On Figure 1, follow the lavender lines for DeepSeek-V4-Flash. Its hidden size is 4,096 and its expert intermediate size is 2,048. Packing gate and up side by side gives an output width of 4,096 in the middle panel. After gating, the down projection consumes 2,048 features and produces 4,096 hidden features in the left panel. At high token counts both shapes enter the row-major region in our FP8 experiment. The released model uses FP4 expert weights, which changes the weight footprint in the model above. Model configuration
How we measured
Figure 1 covers 330 single-expert DeepGEMM FP8 GEMMs with BF16 output on NVIDIA GB300. The scheduling unit spans 128 token rows and 256 output columns across two CTAs. At each shape we changed only the mapping from task index to output tile, checked both outputs bitwise against the stock kernel, and timed 12 counterbalanced pairs of CUDA graph replays. The plotted statistic is the median across pairs of .
Figure 4 uses load traces from the fused kernel’s selected tile sizes. We replayed the two operands separately, counted TMA requests at issue, and estimated device HBM bytes from repeated NVML bandwidth windows. That separation lets us assign traffic to one operand. The fused weight-copy control measures whole-kernel latency across two GPUs with 12 active experts per GPU.