From 5d2e4252e695644c34be605b44a14bd80a98a0a6 Mon Sep 17 00:00:00 2001 From: jdj Date: Wed, 18 Nov 2026 14:47:33 -0800 Subject: Boris-2 training code The code, configuration and data filters used to train Boris-2 (144M, 200B tokens, single V100): model, training loop, data preparation, n-gram table, V100 kernels, final stages and the exact run configs. --- docs/design-notes-0920.md | 218 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 218 insertions(+) create mode 100644 docs/design-notes-0920.md (limited to 'docs/design-notes-0920.md') diff --git a/docs/design-notes-0920.md b/docs/design-notes-0920.md new file mode 100644 index 0000000..df4ddf6 --- /dev/null +++ b/docs/design-notes-0920.md @@ -0,0 +1,218 @@ +# Boris-2-0920 + +A 143.7M-parameter decoder-only language model trained from scratch on 200B +tokens on a single V100 32GB. Deep-narrow trunk (d512 x 31L) with fused Canon +layers, a 15M-parameter n-gram lookup table, Muon on the matmul weights, a WSD +schedule, and a context length that extends from 2048 to 8192 at 80% of the run. + + GPU 0 Tesla V100 32GB (sm_70) trains. fp16 only -- no bf16 path exists. + GPU 1 RTX 3060 12GB (sm_86) runs eval.py, in its own process, never + blocking or pausing the training loop. + +## Parameters + +| | | +|---|---| +| attention (31 x 1,048,576) | 32,505,856 | +| SwiGLU FFN, hidden 1664 | 79,233,024 | +| Canon depthwise kernels | 142,848 | +| RMSNorms | 32,256 | +| tied embedding / LM head | 16,777,216 | +| n-gram table (234,376 x 64) + projection | 15,032,832 | +| **total** | **143,724,032** | + +Muon holds 111,771,648 parameters; AdamW holds the remaining trainable tensors. +`python config.py` prints this and asserts it; `train.py` prints the +per-parameter assignment and refuses to start if it disagrees with the budget. + +The 0917 rebalance moved capacity from the FFN into full multi-head attention +while preserving the same overall size class. The FFN width is 1664 (3.25x). + +## Layout + + config.py every judgement call, serialised into every checkpoint + tokenizer.py our own 32,768-token byte-level BPE + prepare.py stream -> decontaminate -> tokenize -> uint16 shards + ngram.py the n-gram table: file format, builder, module + model.py the model; fused Canon, varlen attention, fused LM head + optim.py Muon (shape-batched Newton-Schulz), AdamW, the split + schedule.py WSD, the mixture curriculum, the context stages + data.py varlen packer + prefetcher + checkpoint.py rolling / full / export, atomic + train.py the loop + eval.py standalone, on GPU 1, from a weights-only export + verify.py 31 correctness checks + preflight.py throughput, micro-batch sweep, ablations + +`data/` and `checkpoints/` are symlinks onto `/media/joseph/Media` -- `/home` has +20GB free and 200B tokens is 400GB. + +## Order of operations + + python prepare.py plan # what the curriculum will draw + python prepare.py sample --mb 2000 + python tokenizer.py train --sample data/tokenizer_sample.txt + python prepare.py source --name ultra_fineweb --tokens 106e9 + python prepare.py source --name cosmopedia --tokens 33e9 + python prepare.py source --name finephrase --tokens 34e9 + python prepare.py source --name fineweb_edu --tokens 19e9 + python prepare.py source --name dclm --tokens 10e9 + python prepare.py val --tokens 2e7 + python ngram.py build --shards data --out data/ngram_vocab.bin + + python verify.py # 31 checks, all must pass + python preflight.py --sweep 4,8,16 # set micro_batch from measurement + + python train.py # or --resume + python eval.py --watch # on the other GPU, separately + +`prepare.py` is resumable and writes incrementally, so training can start on the +first sources while the rest are still arriving. + +## Measured throughput (V100, 2026-09-06) + + ctx 2048, micro_batch 16 40.1k tok/s 33.7 TFLOPS 27.0% MFU 23.9 GB + ctx 8192, micro_batch 4 32.6k tok/s 28.5 TFLOPS 22.8% MFU 24.0 GB + +Worst case (one full-block document per row) measures 31.2k / 23.8 GB at ctx +2048 and 15.8k / 23.8 GB at ctx 8192 -- memory is unchanged, because the +memory-efficient attention kernel is O(N), so the chosen micro-batches are safe. +Throughput falls because attention costs sum(len_i^2). + +Two-stage projection for 200B: roughly 51 days at ctx 2048 for the first 160B, +plus roughly 21 days at ctx 8192 for the last 40B -- call it **~72 days**. The +single-stage "200B in" column that preflight prints is optimistic by exactly +this amount. + +What each optimisation is worth, measured by removing it: + +| removed | throughput | vs. all | +|---|---|---| +| nothing (all optimisations, mb 16) | 37.6k tok/s | — | +| compiled regions (at mb 8, since mb 16 OOMs without them) | 21.4k vs 29.1k | **-26%** and +34% memory | +| fp16 residual stream | 35.0k | -6.8% | +| batched Newton-Schulz | 35.6k | -5.1% | +| fp16 reduced-precision GEMM reduction | 34.9k | -6.9% | +| fused Canon (i.e. what Canon costs) | 38.6k | +2.6% | + +The starting point before this round of work was 31.3k tok/s at 21.0% MFU. The +compiled regions, the block-fused projection weights and the compiled LM-head +chain took it to 37.6k at 25.3%; removing the host syncs took it to 40.1k at +27.0%. + +**Host syncs were worth 6.7%.** The nested-tensor SDPA path derives min and max +sequence length from the offsets with a `.item()` on every call -- 131 host syncs +per forward, about 2,100 per optimizer step, each one draining CPU run-ahead. The +packer knows both while it is laying documents down, so they are passed to +`nested_tensor_from_jagged` explicitly and forward+backward now performs exactly +one sync. Accumulating the step loss as a GPU tensor rather than calling +`float()` per micro-step removes the last one from the inner loop. +`probe/syncs.py` counts them; run it after touching the attention path. + +Canon costing only 2.6% is the point of fusing it: the same layers as three +separate convolutions per block measured 4x that. + +## What is not the textbook spelling, and why + +Every one of these is a measured win on this card, not a preference. The +measurements are in the docstrings of the modules they belong to. + +**Only fixed-shape regions are compiled, and they are wide.** Each block is two +compiled regions -- norm+QKV+Canon+RoPE, then o_proj+residual+norm+SwiGLU+ +residual -- with only attention between them. Written as separate eager modules +the RMSNorms upcast to fp32 and materialise a (B,T,512) fp32 tensor 53 times; +inside the regions Inductor keeps the reduction in fp32 without ever writing it. +This is worth 36% and a quarter of the activation memory. Whole-model compile is +still not used: document counts vary, `cu_seqlens` changes length every step, and +Inductor recompiles mid-run. + +**Q/K/V is one Parameter, and so is gate/up.** They carry a `_boris_blocks` tag +and Muon orthogonalises each row range independently -- identical to treating +them as separate Parameters, since the quintic iteration acts on one matrix at a +time, and `verify.py` asserts the two spellings give bit-identical updates. This +removes a `torch.cat` per layer per micro-step from the forward and its backward +slice. + +**Canon is one kernel over the fused Q|K|V block.** Depthwise convolution is +per-channel, so one 1536-channel convolution equals three separate ones on +Q/K/V channel ranges -- 31 kernels per forward, not 93. It is spelled as three +shifted multiply-adds on the native (B,T,C) layout inside one compiled region +rather than `F.conv1d(groups=C)`, which needs a transpose whose following +reshape materialises a copy. The older grouped-convolution spelling was +substantially slower on the V100. + +**Attention is varlen over a jagged nested tensor.** One SDPA launch for the +whole batch and cost `sum(len_i^2)` instead of `T^2`. At ctx 2048 with realistic +document lengths, fwd+bwd: nested 11.1 ms, dense block-diagonal mask 27.0 ms, +per-document loop 45.8 ms -- and nested is bit-identical to the loop. + +**The LM head never materialises (N, V) logits.** dx and dW are computed during +the forward walk over vocab chunks, so there is no recomputation and the total +GEMM work equals the unfused head. 4.2x less memory, and *more* accurate than +the unfused fp16 path: dx 8.4e-04 vs 1.3e-02, dW 2.0e-04 vs 3.9e-03, both +against fp32. + +**Newton-Schulz is batched by shape.** 218 logical matrices in a few distinct shapes +become four bmm chains -- about 60 kernel launches per optimizer step instead of +2,300. In fp32: five steps amplify rounding well past bf16 epsilon, and sm_70 +has no bf16 tensor cores to gain from. + +**The residual stream is fp32.** The live V100 run measured activations above +fp16's finite range. Compiled regions still cast matmul inputs under autocast, +so tensor-core GEMMs remain fp16 while residual additions stay finite. + +**The packer emits document boundaries directly.** It knows them while laying +documents down; deriving them later costs a GPU scan and a host sync per forward. + +## Document isolation + +Rows hold several packed documents and three separate mechanisms keep them +apart: varlen attention, the Canon boundary mask, and the n-gram history mask. +Kernel-3 Canon stacked 31 deep reaches 62 tokens back, so without the mask every +block pulls the previous document's tail across the boundary. + +`verify.py` de-zeroes the Canon kernels before probing for a leak. They are +zero-initialised, so at init the convolution contributes exactly nothing and an +isolation test passes no matter how badly the model leaks -- which is how the +75M run shipped that bug behind a green test for weeks. + +## Interruption + +The run is expected to be interrupted. A rolling checkpoint every 45 minutes +(two kept, atomic) carries the weights, both optimizer states, the loss scale, +the packer's RNG and its carried document tail, so a resume replays the identical +token stream. The 50B full checkpoints are archival milestones, not the resume +point. `stable.pt`, written at the end of the stable phase, is permanent and +never rotated: variants branch from it as a tree, and re-warming from a fully +decayed model costs a loss spike and a slow recovery. + +A resume refuses to load a checkpoint whose config fingerprint or n-gram vocab +fingerprint disagrees with the running configuration. `--force-config` overrides. + +## Eval + +`eval.py --watch` polls the export manifest and appends one row per checkpoint to +`eval_out/results.jsonl`, keyed by `tokens_seen`, so the run plots directly +against training tokens. All five benchmarks are multiple-choice log-likelihood, +ArithMark-3 included; both raw and length-normalised accuracy are recorded, plus +held-out loss and perplexity on a fixed decontaminated split. + +## September 22 repair and active run + +Boris-2-0920 was restarted from initialization on September 22 at 18:19 PDT, +with the same 143,724,032 parameters and 200B-token target. The previous attempt +is preserved under `/media/joseph/Media/Boris-2-0920/failed_20260922`. + +The confirmed fused-head overflow was fixed by widening the dW GEMM output +before undoing its protective scaling. Validation now uses the retokenized +`climbmix_val` source. Regression checks call the production head under autocast; +64 verification checks passed, followed by 20 finite V100 updates from the failed +checkpoint with zero skips. Details and source snapshots are under +`/media/joseph/Media/Boris-2-0920/repair_20260922/`. + +Current status is `/media/joseph/Media/Boris-2-0920/process_status.json` and the +live console log is `/media/joseph/Media/Boris-2-0920/train.log`. +Resume with `bash logs/launch_chain.sh --resume`. The launcher uses +`supervise_run.py`; duplicate supervisors are refused by an exclusive lock. +Missing resume checkpoints and accidental fresh launches into a nonempty +checkpoint directory are errors. Do not run the old `finalize_0920.sh` to resume. -- cgit v1.2.3