summaryrefslogtreecommitdiff
path: root/volta_kernels/README.md
blob: dd8001882390749832577dbef94ae85203ceacdf (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
# volta_kernels — hand-written sm_70 kernels for Boris-2

Opt-in: `BORIS_VOLTA_KERNELS=1`. Engages only on an sm_70 device (the V100);
eval on the 3060 always runs the stock paths. Unset, nothing changes.

    BORIS_VOLTA_KERNELS=1 bash logs/launch_chain.sh --resume

First use JIT-builds `attn.cu` with nvcc into `build/` (~1 min, then cached).

## What is here

| file | replaces | measured (V100, beside the live run) |
|---|---|---|
| `attn.cu` fwd | CUTLASS mem-efficient SDPA fwd (nested/jagged) | ~2x |
| `attn.cu` bwd | mem-efficient SDPA bwd (was ~4x its own fwd) | ~2x |
| `head.py` | `_FusedLMHead` elementwise middle + per-chunk casts | 1.8x, -25% peak mem |
| `model._pre_attn(out16)` | separate fp32->fp16 casts of q/k/v | fused into the region |

End to end, 8-layer slice of the live config, full micro-step: **1.41x**
contended (stable over 3 repeats), 2.05x in one uncontended window. The
31-layer mb-16 number is only observable on the run itself.

## Why CUDA and not Triton

On sm_70, Triton's `tl.dot` compiles to plain FMA -- no tensor cores (checked:
2,048 `fma.rn` in the PTX, no `mma`). The attention uses raw
`mma.sync.aligned.m8n8k4` (HMMA.884, what CUTLASS sm70 uses). Fragment layouts,
verified empirically on this card:

    A.row: row = j + 4hi, k = 0..3        A.col: k = j, row = 4hi + 0..3
    B.col: n = j + 4hi,   k = 0..3        B.row: k = j, n = 4hi + 0..3
    D f32: row = (lane&1) + (i&2) + 4hi,  col = (i&4) + (lane&2) + (i&1)
    j = lane%4, hi = lane>=16; quad-pair qp = (lane>>2)&3 owns one 8x8 tile

The head's softmax is Triton (elementwise/reduction, no dot), which is fine.

## Precision

Same operand dtypes and fp32 accumulation as what they replace. `verify.py`
`check_volta_kernels` (always runs on sm_70) holds every output to the stock
path's own error against fp64 on the same inputs:

    attention  out 2.22e-4 (= stock)  dq 3.66e-4 (stock 4.23e-4)
               dk 3.51e-4 (stock 4.05e-4)  dv 2.88e-4 (= stock)
    head       dx, dW identical error to production; loss equal to 9 digits;
               ~0.02% of dx elements differ by 1 ulp (lse reduction order)
    model      median grad rel-L2 vs fp32 1.29e-3 (stock 1.30-1.32e-3), at the
               run's 2^15 loss scale. Without a loss scale the two fp16 paths
               differ ~6% on early layers -- subnormal noise, not a bug.

Document isolation and causality are exact (zero drift). dQ uses fp32 atomics,
so attention backward is not bitwise deterministic run to run (fp32 order only).

## Gotchas found on the way

- q/k/v leave `_pre_attn` fp32, not fp16: the Canon masks m1/m2 are fp32 and
  promote. Stock SDPA casts them via autocast; the Volta path does the same cast
  inside the compiled region (`out16`).
- In production the head's `g` is fp32 (x is fp32), and autocast re-casts W to
  fp16 twice per chunk. The rewrite casts once per call.
- `dW.addcmul_(fp16, fp32)` takes torch's slow unvectorized path (0.9 ms/chunk);
  replaced by a Triton FMA. Exact, because `back` is a power of two.
- `__launch_bounds__(128, 3)` makes both kernels slower (spills): 248/216 regs,
  2 CTAs/SM is the best measured point.
- Remaining bwd split (ablation): dQ phase ~26%, tile loads ~6%, atomics ~4%.

## Live result and later experiments (2026-09-27)

Adopted at step 28,239: **35.5k -> 45.2k tok/s, 30.5 -> 38.8 TF, MFU 24.4% ->
31.0%**, ETA 60 -> 47.5 days. `logs/launch_chain.sh` exports
`BORIS_VOLTA_KERNELS=1` by default.

Measured afterwards, so nobody re-tries them blind:

- **Megakernel / launch overhead.** Whole micro-step captured as one CUDA graph
  vs eager, identical kernels: 3.3% faster at micro_batch 2 (8-layer slice),
  i.e. ~0.5% at production mb 16. That is the ceiling for anything whose win is
  removing launches; the GEMMs (~55% of the step) are cuBLAS.
- **Skipping fully-masked MMA blocks on diagonal tiles** (bitwise-identical
  results): fwd 4.5% and bwd 18.6% SLOWER. Warp-dependent loop bounds break the
  unrolled MMA schedule, and the CTA waits at the barrier for its busiest warp
  anyway. Dropped.
- **Adopted, no speed effect:** gd written over the logits buffer (-64 MB);
  shape/device/metadata guards in the extension.
- verify.py now also covers an 8192-token document (stage two; fp32 reference,
  error ratio vs stock 1.000) and a head call with a partial final chunk.
- Test gotcha: `t.to(torch.float16)` on an fp16 tensor returns the same tensor,
  so two gradient runs "on copies" accumulate into one .grad. Clone.
- **Megakernel, stage 1** (`experiments/mega_ffn2.cu`): the whole FFN half of a
  block forward (RMSNorm -> gate/up -> SiLU*up -> down -> residual) in one
  kernel, intermediate never leaving shared memory. Precision identical to stock
  (FFN-delta error vs fp64 6.454e-4 both). Speed at mb 16: 23.7 ms first build,
  16.1 ms with register-prefetched weights, vs **6.5 ms** Inductor+cuBLAS -- 2.5x
  SLOWER. Structural: holding a full 512-wide x tile costs 64 KB of shared
  memory, so a CTA can own only 64 rows, each weight byte is reused by only 64
  rows (cuBLAS reuses across 128x128 tiles), and one CTA per SM leaves nothing
  to hide the barriers. The fusion's best case is avoiding the gate/up write,
  and training's backward needs gate/up saved anyway. Not pursued further.

## Round 2 (2026-09-27 evening) -- reviewed with GPT-6 Sol and Muse Spark

Adopted (dormant until the next relaunch; verify 14/14):
- **Head dW via fp32-output GEMM** (GPT's idea): `torch.addmm(dW, g^T, x16,
  beta=0|1, out_dtype=float32)` selects `volta_s884gemm_fp16` + fp32 split-k
  reduce. dW is never rounded to fp16, so the x-rescale / DW_SHRINK overflow
  guard and the second fp16 operand (gd, 64 MB write per chunk) are gone.
  **Head 12.6% faster; dW error vs fp64 5.24e-4 -> 5.01e-4 (fixture).**
- **Forward K/V register prefetch**: next tile's global loads issued before the
  current tile's MMAs (fwd has register headroom: 216). **fwd 4.5% faster,
  bitwise-identical output.**

Measured and rejected:
- Megakernel FFN at 128 rows (column split, gate/up recomputed): 24.3 ms vs
  6.5 ms stock (and 16.0 ms at 64 rows). 1.67x FLOPs plus spills.
- Blocked two-pass head softmax with online max/sum + evict_first (Muse): no
  gain over the one-pass kernel.
- Rejected on inspection: aliasing P into Ks in fwd (race -- other warps still
  read Ks for QK^T), folding scale*log2e into fp16 Q (changes rounding), Q/dO
  register prefetch in bwd (248 regs, no headroom).
- **Fused Canon + RoPE** (`canon_rope.py`, Triton, fwd + bwd): replaces
  Inductor's zero-padded fp32 copies of qkv, the 3-tap combine, RoPE and their
  backward passes. 1.96x on that region; same error as the region for q/k/v and
  dqkv, weight gradient closer to fp64. Isolation exact at document boundaries.
  Whole 8-layer slice: 1.41x -> **1.538x** vs stock (contended), i.e. ~+9% over
  the kernels live since step 28,239. verify.py: 80/80.
- **Canon + RoPE in CUDA** (`canon_rope_cuda.py`, from a second Claude Code
  reviewer): sliding window down the sequence, 8 channels per thread, 16-byte
  I/O. Bitwise-identical q/k/v/dqkv to the Triton version (checked
  independently), 3.55x the compiled region at mb 16, **+4.1% whole micro-step
  at full size**. The model now uses this; `canon_rope.py` (Triton) is kept as
  the reference. verify.py 80/80.
- **Longest-document-first CTA order** (GPT-6 Sol's research lead): the tile
  table cumsums over documents sorted by tile count and passes the permutation
  to both kernels. Bitwise-identical; idle V100, mb 16: fwd 1.010 -> 0.945 ms
  (-6.5%), bwd 2.712 -> 2.559 ms (-5.6%); ctx 8192 about -6% both. verify 81/81.
- **GEMM algorithm sweep** (TunableOp, idle V100): cuBLAS "Default" won every
  block shape. The GEMMs already run at 54-59% of the 125 TF peak (QKV 73.5 TF,
  gate/up 71, down 74, o_proj 67). There is little left to take from them.
- Attention, by contrast, runs at ~24-27% of peak (fwd ~30 TF, bwd ~27 TF
  counted). That gap is the largest remaining one.
- **Masks only where needed** (after an ablation showed softmax math is ~19%
  of the forward): a CTA-uniform branch skips the per-element integer mask on
  every tile except the diagonal and a document's last tile. Bitwise-identical
  (dQ differs only by its usual atomic-order noise, measured equal to
  old-vs-old). Idle V100: fwd -4.9%, bwd -2.1% at both 2048 and 8192.
  Tried and dropped: skipping the output rescale when no max moved (no gain),
  folding the scale into ex2's FMA (+1.2% more but not bitwise).
- Volta attention literature (2026-09): SparkAttention's backward gains need
  fp16 accumulation (excluded); its register-shuffle C->A layout trick targets
  the P shared-memory round trip, which the ablation shows costs nothing here.
  No public fp32-accumulation varlen training kernel for sm_70 beats this one.
- **Head dx written fp32 from the GEMM** (`out_dtype=float32`): cuBLAS can no
  longer pick split-k kernels that reduce partials in fp16. dx error vs fp64
  5.31e-4 -> 3.43e-4 and 2.9% faster; also makes a larger lm_head_chunk safe
  (chunk 4096: head -11.6%, dx 3.37e-4; with fp16 dx the same chunk raised dx
  error 41%). verify 81/81, model median grad error 1.208e-3 (stock 1.293e-3).
- **Hardware correction:** this PG500-216 has **72 SMs** (not 80) and HBM at
  1107 MHz. fp16 tensor peak = 72 x 8 x 128 x 1.38 GHz = **101.7 TF**, not 125;
  every MFU quoted against 125 understates by 1.23x. Measured device copy
  956 GB/s. So the GEMMs run at ~70-73% of real peak, and the ~680 GB/s
  elementwise kernels have ~30% headroom.
- Tried and reverted: one-thread-per-row bwd_pre_kernel (3.6x SLOWER -- the
  warp-per-row version was already coalesced at ~680 GB/s). The SwiGLU backward
  pairing lead was dropped on reading Inductor's kernel: the "redundant" reads
  hit cache.