From 5d2e4252e695644c34be605b44a14bd80a98a0a6 Mon Sep 17 00:00:00 2001 From: jdj Date: Wed, 18 Nov 2026 14:47:33 -0800 Subject: Boris-2 training code The code, configuration and data filters used to train Boris-2 (144M, 200B tokens, single V100): model, training loop, data preparation, n-gram table, V100 kernels, final stages and the exact run configs. --- volta_kernels/README.md | 165 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 165 insertions(+) create mode 100644 volta_kernels/README.md (limited to 'volta_kernels/README.md') diff --git a/volta_kernels/README.md b/volta_kernels/README.md new file mode 100644 index 0000000..dd80018 --- /dev/null +++ b/volta_kernels/README.md @@ -0,0 +1,165 @@ +# volta_kernels — hand-written sm_70 kernels for Boris-2 + +Opt-in: `BORIS_VOLTA_KERNELS=1`. Engages only on an sm_70 device (the V100); +eval on the 3060 always runs the stock paths. Unset, nothing changes. + + BORIS_VOLTA_KERNELS=1 bash logs/launch_chain.sh --resume + +First use JIT-builds `attn.cu` with nvcc into `build/` (~1 min, then cached). + +## What is here + +| file | replaces | measured (V100, beside the live run) | +|---|---|---| +| `attn.cu` fwd | CUTLASS mem-efficient SDPA fwd (nested/jagged) | ~2x | +| `attn.cu` bwd | mem-efficient SDPA bwd (was ~4x its own fwd) | ~2x | +| `head.py` | `_FusedLMHead` elementwise middle + per-chunk casts | 1.8x, -25% peak mem | +| `model._pre_attn(out16)` | separate fp32->fp16 casts of q/k/v | fused into the region | + +End to end, 8-layer slice of the live config, full micro-step: **1.41x** +contended (stable over 3 repeats), 2.05x in one uncontended window. The +31-layer mb-16 number is only observable on the run itself. + +## Why CUDA and not Triton + +On sm_70, Triton's `tl.dot` compiles to plain FMA -- no tensor cores (checked: +2,048 `fma.rn` in the PTX, no `mma`). The attention uses raw +`mma.sync.aligned.m8n8k4` (HMMA.884, what CUTLASS sm70 uses). Fragment layouts, +verified empirically on this card: + + A.row: row = j + 4hi, k = 0..3 A.col: k = j, row = 4hi + 0..3 + B.col: n = j + 4hi, k = 0..3 B.row: k = j, n = 4hi + 0..3 + D f32: row = (lane&1) + (i&2) + 4hi, col = (i&4) + (lane&2) + (i&1) + j = lane%4, hi = lane>=16; quad-pair qp = (lane>>2)&3 owns one 8x8 tile + +The head's softmax is Triton (elementwise/reduction, no dot), which is fine. + +## Precision + +Same operand dtypes and fp32 accumulation as what they replace. `verify.py` +`check_volta_kernels` (always runs on sm_70) holds every output to the stock +path's own error against fp64 on the same inputs: + + attention out 2.22e-4 (= stock) dq 3.66e-4 (stock 4.23e-4) + dk 3.51e-4 (stock 4.05e-4) dv 2.88e-4 (= stock) + head dx, dW identical error to production; loss equal to 9 digits; + ~0.02% of dx elements differ by 1 ulp (lse reduction order) + model median grad rel-L2 vs fp32 1.29e-3 (stock 1.30-1.32e-3), at the + run's 2^15 loss scale. Without a loss scale the two fp16 paths + differ ~6% on early layers -- subnormal noise, not a bug. + +Document isolation and causality are exact (zero drift). dQ uses fp32 atomics, +so attention backward is not bitwise deterministic run to run (fp32 order only). + +## Gotchas found on the way + +- q/k/v leave `_pre_attn` fp32, not fp16: the Canon masks m1/m2 are fp32 and + promote. Stock SDPA casts them via autocast; the Volta path does the same cast + inside the compiled region (`out16`). +- In production the head's `g` is fp32 (x is fp32), and autocast re-casts W to + fp16 twice per chunk. The rewrite casts once per call. +- `dW.addcmul_(fp16, fp32)` takes torch's slow unvectorized path (0.9 ms/chunk); + replaced by a Triton FMA. Exact, because `back` is a power of two. +- `__launch_bounds__(128, 3)` makes both kernels slower (spills): 248/216 regs, + 2 CTAs/SM is the best measured point. +- Remaining bwd split (ablation): dQ phase ~26%, tile loads ~6%, atomics ~4%. + +## Live result and later experiments (2026-09-27) + +Adopted at step 28,239: **35.5k -> 45.2k tok/s, 30.5 -> 38.8 TF, MFU 24.4% -> +31.0%**, ETA 60 -> 47.5 days. `logs/launch_chain.sh` exports +`BORIS_VOLTA_KERNELS=1` by default. + +Measured afterwards, so nobody re-tries them blind: + +- **Megakernel / launch overhead.** Whole micro-step captured as one CUDA graph + vs eager, identical kernels: 3.3% faster at micro_batch 2 (8-layer slice), + i.e. ~0.5% at production mb 16. That is the ceiling for anything whose win is + removing launches; the GEMMs (~55% of the step) are cuBLAS. +- **Skipping fully-masked MMA blocks on diagonal tiles** (bitwise-identical + results): fwd 4.5% and bwd 18.6% SLOWER. Warp-dependent loop bounds break the + unrolled MMA schedule, and the CTA waits at the barrier for its busiest warp + anyway. Dropped. +- **Adopted, no speed effect:** gd written over the logits buffer (-64 MB); + shape/device/metadata guards in the extension. +- verify.py now also covers an 8192-token document (stage two; fp32 reference, + error ratio vs stock 1.000) and a head call with a partial final chunk. +- Test gotcha: `t.to(torch.float16)` on an fp16 tensor returns the same tensor, + so two gradient runs "on copies" accumulate into one .grad. Clone. +- **Megakernel, stage 1** (`experiments/mega_ffn2.cu`): the whole FFN half of a + block forward (RMSNorm -> gate/up -> SiLU*up -> down -> residual) in one + kernel, intermediate never leaving shared memory. Precision identical to stock + (FFN-delta error vs fp64 6.454e-4 both). Speed at mb 16: 23.7 ms first build, + 16.1 ms with register-prefetched weights, vs **6.5 ms** Inductor+cuBLAS -- 2.5x + SLOWER. Structural: holding a full 512-wide x tile costs 64 KB of shared + memory, so a CTA can own only 64 rows, each weight byte is reused by only 64 + rows (cuBLAS reuses across 128x128 tiles), and one CTA per SM leaves nothing + to hide the barriers. The fusion's best case is avoiding the gate/up write, + and training's backward needs gate/up saved anyway. Not pursued further. + +## Round 2 (2026-09-27 evening) -- reviewed with GPT-6 Sol and Muse Spark + +Adopted (dormant until the next relaunch; verify 14/14): +- **Head dW via fp32-output GEMM** (GPT's idea): `torch.addmm(dW, g^T, x16, + beta=0|1, out_dtype=float32)` selects `volta_s884gemm_fp16` + fp32 split-k + reduce. dW is never rounded to fp16, so the x-rescale / DW_SHRINK overflow + guard and the second fp16 operand (gd, 64 MB write per chunk) are gone. + **Head 12.6% faster; dW error vs fp64 5.24e-4 -> 5.01e-4 (fixture).** +- **Forward K/V register prefetch**: next tile's global loads issued before the + current tile's MMAs (fwd has register headroom: 216). **fwd 4.5% faster, + bitwise-identical output.** + +Measured and rejected: +- Megakernel FFN at 128 rows (column split, gate/up recomputed): 24.3 ms vs + 6.5 ms stock (and 16.0 ms at 64 rows). 1.67x FLOPs plus spills. +- Blocked two-pass head softmax with online max/sum + evict_first (Muse): no + gain over the one-pass kernel. +- Rejected on inspection: aliasing P into Ks in fwd (race -- other warps still + read Ks for QK^T), folding scale*log2e into fp16 Q (changes rounding), Q/dO + register prefetch in bwd (248 regs, no headroom). +- **Fused Canon + RoPE** (`canon_rope.py`, Triton, fwd + bwd): replaces + Inductor's zero-padded fp32 copies of qkv, the 3-tap combine, RoPE and their + backward passes. 1.96x on that region; same error as the region for q/k/v and + dqkv, weight gradient closer to fp64. Isolation exact at document boundaries. + Whole 8-layer slice: 1.41x -> **1.538x** vs stock (contended), i.e. ~+9% over + the kernels live since step 28,239. verify.py: 80/80. +- **Canon + RoPE in CUDA** (`canon_rope_cuda.py`, from a second Claude Code + reviewer): sliding window down the sequence, 8 channels per thread, 16-byte + I/O. Bitwise-identical q/k/v/dqkv to the Triton version (checked + independently), 3.55x the compiled region at mb 16, **+4.1% whole micro-step + at full size**. The model now uses this; `canon_rope.py` (Triton) is kept as + the reference. verify.py 80/80. +- **Longest-document-first CTA order** (GPT-6 Sol's research lead): the tile + table cumsums over documents sorted by tile count and passes the permutation + to both kernels. Bitwise-identical; idle V100, mb 16: fwd 1.010 -> 0.945 ms + (-6.5%), bwd 2.712 -> 2.559 ms (-5.6%); ctx 8192 about -6% both. verify 81/81. +- **GEMM algorithm sweep** (TunableOp, idle V100): cuBLAS "Default" won every + block shape. The GEMMs already run at 54-59% of the 125 TF peak (QKV 73.5 TF, + gate/up 71, down 74, o_proj 67). There is little left to take from them. +- Attention, by contrast, runs at ~24-27% of peak (fwd ~30 TF, bwd ~27 TF + counted). That gap is the largest remaining one. +- **Masks only where needed** (after an ablation showed softmax math is ~19% + of the forward): a CTA-uniform branch skips the per-element integer mask on + every tile except the diagonal and a document's last tile. Bitwise-identical + (dQ differs only by its usual atomic-order noise, measured equal to + old-vs-old). Idle V100: fwd -4.9%, bwd -2.1% at both 2048 and 8192. + Tried and dropped: skipping the output rescale when no max moved (no gain), + folding the scale into ex2's FMA (+1.2% more but not bitwise). +- Volta attention literature (2026-09): SparkAttention's backward gains need + fp16 accumulation (excluded); its register-shuffle C->A layout trick targets + the P shared-memory round trip, which the ablation shows costs nothing here. + No public fp32-accumulation varlen training kernel for sm_70 beats this one. +- **Head dx written fp32 from the GEMM** (`out_dtype=float32`): cuBLAS can no + longer pick split-k kernels that reduce partials in fp16. dx error vs fp64 + 5.31e-4 -> 3.43e-4 and 2.9% faster; also makes a larger lm_head_chunk safe + (chunk 4096: head -11.6%, dx 3.37e-4; with fp16 dx the same chunk raised dx + error 41%). verify 81/81, model median grad error 1.208e-3 (stock 1.293e-3). +- **Hardware correction:** this PG500-216 has **72 SMs** (not 80) and HBM at + 1107 MHz. fp16 tensor peak = 72 x 8 x 128 x 1.38 GHz = **101.7 TF**, not 125; + every MFU quoted against 125 understates by 1.23x. Measured device copy + 956 GB/s. So the GEMMs run at ~70-73% of real peak, and the ~680 GB/s + elementwise kernels have ~30% headroom. +- Tried and reverted: one-thread-per-row bwd_pre_kernel (3.6x SLOWER -- the + warp-per-row version was already coalesced at ~680 GB/s). The SwiGLU backward + pairing lead was dropped on reading Inductor's kernel: the "redundant" reads + hit cache. -- cgit v1.2.3