summaryrefslogtreecommitdiff
path: root/volta_kernels/README.md
diff options
context:
space:
mode:
authorjdj <jdj@kunix.org>2026-11-18 14:47:33 -0800
committerjdj <jdj@kunix.org>2026-11-18 14:47:33 -0800
commit5d2e4252e695644c34be605b44a14bd80a98a0a6 (patch)
tree311a72eda4459af7547b9cd05c0956eafb235f2d /volta_kernels/README.md
downloadboris-2-5d2e4252e695644c34be605b44a14bd80a98a0a6.tar.gz
boris-2-5d2e4252e695644c34be605b44a14bd80a98a0a6.zip
Boris-2 training code
The code, configuration and data filters used to train Boris-2 (144M, 200B tokens, single V100): model, training loop, data preparation, n-gram table, V100 kernels, final stages and the exact run configs.
Diffstat (limited to 'volta_kernels/README.md')
-rw-r--r--volta_kernels/README.md165
1 files changed, 165 insertions, 0 deletions
diff --git a/volta_kernels/README.md b/volta_kernels/README.md
new file mode 100644
index 0000000..dd80018
--- /dev/null
+++ b/volta_kernels/README.md
@@ -0,0 +1,165 @@
+# volta_kernels — hand-written sm_70 kernels for Boris-2
+
+Opt-in: `BORIS_VOLTA_KERNELS=1`. Engages only on an sm_70 device (the V100);
+eval on the 3060 always runs the stock paths. Unset, nothing changes.
+
+ BORIS_VOLTA_KERNELS=1 bash logs/launch_chain.sh --resume
+
+First use JIT-builds `attn.cu` with nvcc into `build/` (~1 min, then cached).
+
+## What is here
+
+| file | replaces | measured (V100, beside the live run) |
+|---|---|---|
+| `attn.cu` fwd | CUTLASS mem-efficient SDPA fwd (nested/jagged) | ~2x |
+| `attn.cu` bwd | mem-efficient SDPA bwd (was ~4x its own fwd) | ~2x |
+| `head.py` | `_FusedLMHead` elementwise middle + per-chunk casts | 1.8x, -25% peak mem |
+| `model._pre_attn(out16)` | separate fp32->fp16 casts of q/k/v | fused into the region |
+
+End to end, 8-layer slice of the live config, full micro-step: **1.41x**
+contended (stable over 3 repeats), 2.05x in one uncontended window. The
+31-layer mb-16 number is only observable on the run itself.
+
+## Why CUDA and not Triton
+
+On sm_70, Triton's `tl.dot` compiles to plain FMA -- no tensor cores (checked:
+2,048 `fma.rn` in the PTX, no `mma`). The attention uses raw
+`mma.sync.aligned.m8n8k4` (HMMA.884, what CUTLASS sm70 uses). Fragment layouts,
+verified empirically on this card:
+
+ A.row: row = j + 4hi, k = 0..3 A.col: k = j, row = 4hi + 0..3
+ B.col: n = j + 4hi, k = 0..3 B.row: k = j, n = 4hi + 0..3
+ D f32: row = (lane&1) + (i&2) + 4hi, col = (i&4) + (lane&2) + (i&1)
+ j = lane%4, hi = lane>=16; quad-pair qp = (lane>>2)&3 owns one 8x8 tile
+
+The head's softmax is Triton (elementwise/reduction, no dot), which is fine.
+
+## Precision
+
+Same operand dtypes and fp32 accumulation as what they replace. `verify.py`
+`check_volta_kernels` (always runs on sm_70) holds every output to the stock
+path's own error against fp64 on the same inputs:
+
+ attention out 2.22e-4 (= stock) dq 3.66e-4 (stock 4.23e-4)
+ dk 3.51e-4 (stock 4.05e-4) dv 2.88e-4 (= stock)
+ head dx, dW identical error to production; loss equal to 9 digits;
+ ~0.02% of dx elements differ by 1 ulp (lse reduction order)
+ model median grad rel-L2 vs fp32 1.29e-3 (stock 1.30-1.32e-3), at the
+ run's 2^15 loss scale. Without a loss scale the two fp16 paths
+ differ ~6% on early layers -- subnormal noise, not a bug.
+
+Document isolation and causality are exact (zero drift). dQ uses fp32 atomics,
+so attention backward is not bitwise deterministic run to run (fp32 order only).
+
+## Gotchas found on the way
+
+- q/k/v leave `_pre_attn` fp32, not fp16: the Canon masks m1/m2 are fp32 and
+ promote. Stock SDPA casts them via autocast; the Volta path does the same cast
+ inside the compiled region (`out16`).
+- In production the head's `g` is fp32 (x is fp32), and autocast re-casts W to
+ fp16 twice per chunk. The rewrite casts once per call.
+- `dW.addcmul_(fp16, fp32)` takes torch's slow unvectorized path (0.9 ms/chunk);
+ replaced by a Triton FMA. Exact, because `back` is a power of two.
+- `__launch_bounds__(128, 3)` makes both kernels slower (spills): 248/216 regs,
+ 2 CTAs/SM is the best measured point.
+- Remaining bwd split (ablation): dQ phase ~26%, tile loads ~6%, atomics ~4%.
+
+## Live result and later experiments (2026-09-27)
+
+Adopted at step 28,239: **35.5k -> 45.2k tok/s, 30.5 -> 38.8 TF, MFU 24.4% ->
+31.0%**, ETA 60 -> 47.5 days. `logs/launch_chain.sh` exports
+`BORIS_VOLTA_KERNELS=1` by default.
+
+Measured afterwards, so nobody re-tries them blind:
+
+- **Megakernel / launch overhead.** Whole micro-step captured as one CUDA graph
+ vs eager, identical kernels: 3.3% faster at micro_batch 2 (8-layer slice),
+ i.e. ~0.5% at production mb 16. That is the ceiling for anything whose win is
+ removing launches; the GEMMs (~55% of the step) are cuBLAS.
+- **Skipping fully-masked MMA blocks on diagonal tiles** (bitwise-identical
+ results): fwd 4.5% and bwd 18.6% SLOWER. Warp-dependent loop bounds break the
+ unrolled MMA schedule, and the CTA waits at the barrier for its busiest warp
+ anyway. Dropped.
+- **Adopted, no speed effect:** gd written over the logits buffer (-64 MB);
+ shape/device/metadata guards in the extension.
+- verify.py now also covers an 8192-token document (stage two; fp32 reference,
+ error ratio vs stock 1.000) and a head call with a partial final chunk.
+- Test gotcha: `t.to(torch.float16)` on an fp16 tensor returns the same tensor,
+ so two gradient runs "on copies" accumulate into one .grad. Clone.
+- **Megakernel, stage 1** (`experiments/mega_ffn2.cu`): the whole FFN half of a
+ block forward (RMSNorm -> gate/up -> SiLU*up -> down -> residual) in one
+ kernel, intermediate never leaving shared memory. Precision identical to stock
+ (FFN-delta error vs fp64 6.454e-4 both). Speed at mb 16: 23.7 ms first build,
+ 16.1 ms with register-prefetched weights, vs **6.5 ms** Inductor+cuBLAS -- 2.5x
+ SLOWER. Structural: holding a full 512-wide x tile costs 64 KB of shared
+ memory, so a CTA can own only 64 rows, each weight byte is reused by only 64
+ rows (cuBLAS reuses across 128x128 tiles), and one CTA per SM leaves nothing
+ to hide the barriers. The fusion's best case is avoiding the gate/up write,
+ and training's backward needs gate/up saved anyway. Not pursued further.
+
+## Round 2 (2026-09-27 evening) -- reviewed with GPT-6 Sol and Muse Spark
+
+Adopted (dormant until the next relaunch; verify 14/14):
+- **Head dW via fp32-output GEMM** (GPT's idea): `torch.addmm(dW, g^T, x16,
+ beta=0|1, out_dtype=float32)` selects `volta_s884gemm_fp16` + fp32 split-k
+ reduce. dW is never rounded to fp16, so the x-rescale / DW_SHRINK overflow
+ guard and the second fp16 operand (gd, 64 MB write per chunk) are gone.
+ **Head 12.6% faster; dW error vs fp64 5.24e-4 -> 5.01e-4 (fixture).**
+- **Forward K/V register prefetch**: next tile's global loads issued before the
+ current tile's MMAs (fwd has register headroom: 216). **fwd 4.5% faster,
+ bitwise-identical output.**
+
+Measured and rejected:
+- Megakernel FFN at 128 rows (column split, gate/up recomputed): 24.3 ms vs
+ 6.5 ms stock (and 16.0 ms at 64 rows). 1.67x FLOPs plus spills.
+- Blocked two-pass head softmax with online max/sum + evict_first (Muse): no
+ gain over the one-pass kernel.
+- Rejected on inspection: aliasing P into Ks in fwd (race -- other warps still
+ read Ks for QK^T), folding scale*log2e into fp16 Q (changes rounding), Q/dO
+ register prefetch in bwd (248 regs, no headroom).
+- **Fused Canon + RoPE** (`canon_rope.py`, Triton, fwd + bwd): replaces
+ Inductor's zero-padded fp32 copies of qkv, the 3-tap combine, RoPE and their
+ backward passes. 1.96x on that region; same error as the region for q/k/v and
+ dqkv, weight gradient closer to fp64. Isolation exact at document boundaries.
+ Whole 8-layer slice: 1.41x -> **1.538x** vs stock (contended), i.e. ~+9% over
+ the kernels live since step 28,239. verify.py: 80/80.
+- **Canon + RoPE in CUDA** (`canon_rope_cuda.py`, from a second Claude Code
+ reviewer): sliding window down the sequence, 8 channels per thread, 16-byte
+ I/O. Bitwise-identical q/k/v/dqkv to the Triton version (checked
+ independently), 3.55x the compiled region at mb 16, **+4.1% whole micro-step
+ at full size**. The model now uses this; `canon_rope.py` (Triton) is kept as
+ the reference. verify.py 80/80.
+- **Longest-document-first CTA order** (GPT-6 Sol's research lead): the tile
+ table cumsums over documents sorted by tile count and passes the permutation
+ to both kernels. Bitwise-identical; idle V100, mb 16: fwd 1.010 -> 0.945 ms
+ (-6.5%), bwd 2.712 -> 2.559 ms (-5.6%); ctx 8192 about -6% both. verify 81/81.
+- **GEMM algorithm sweep** (TunableOp, idle V100): cuBLAS "Default" won every
+ block shape. The GEMMs already run at 54-59% of the 125 TF peak (QKV 73.5 TF,
+ gate/up 71, down 74, o_proj 67). There is little left to take from them.
+- Attention, by contrast, runs at ~24-27% of peak (fwd ~30 TF, bwd ~27 TF
+ counted). That gap is the largest remaining one.
+- **Masks only where needed** (after an ablation showed softmax math is ~19%
+ of the forward): a CTA-uniform branch skips the per-element integer mask on
+ every tile except the diagonal and a document's last tile. Bitwise-identical
+ (dQ differs only by its usual atomic-order noise, measured equal to
+ old-vs-old). Idle V100: fwd -4.9%, bwd -2.1% at both 2048 and 8192.
+ Tried and dropped: skipping the output rescale when no max moved (no gain),
+ folding the scale into ex2's FMA (+1.2% more but not bitwise).
+- Volta attention literature (2026-09): SparkAttention's backward gains need
+ fp16 accumulation (excluded); its register-shuffle C->A layout trick targets
+ the P shared-memory round trip, which the ablation shows costs nothing here.
+ No public fp32-accumulation varlen training kernel for sm_70 beats this one.
+- **Head dx written fp32 from the GEMM** (`out_dtype=float32`): cuBLAS can no
+ longer pick split-k kernels that reduce partials in fp16. dx error vs fp64
+ 5.31e-4 -> 3.43e-4 and 2.9% faster; also makes a larger lm_head_chunk safe
+ (chunk 4096: head -11.6%, dx 3.37e-4; with fp16 dx the same chunk raised dx
+ error 41%). verify 81/81, model median grad error 1.208e-3 (stock 1.293e-3).
+- **Hardware correction:** this PG500-216 has **72 SMs** (not 80) and HBM at
+ 1107 MHz. fp16 tensor peak = 72 x 8 x 128 x 1.38 GHz = **101.7 TF**, not 125;
+ every MFU quoted against 125 understates by 1.23x. Measured device copy
+ 956 GB/s. So the GEMMs run at ~70-73% of real peak, and the ~680 GB/s
+ elementwise kernels have ~30% headroom.
+- Tried and reverted: one-thread-per-row bwd_pre_kernel (3.6x SLOWER -- the
+ warp-per-row version was already coalesced at ~680 GB/s). The SwiGLU backward
+ pairing lead was dropped on reading Inductor's kernel: the "redundant" reads
+ hit cache.