# Boris-2-0920 A 143.7M-parameter decoder-only language model trained from scratch on 200B tokens on a single V100 32GB. Deep-narrow trunk (d512 x 31L) with fused Canon layers, a 15M-parameter n-gram lookup table, Muon on the matmul weights, a WSD schedule, and a context length that extends from 2048 to 8192 at 80% of the run. GPU 0 Tesla V100 32GB (sm_70) trains. fp16 only -- no bf16 path exists. GPU 1 RTX 3060 12GB (sm_86) runs eval.py, in its own process, never blocking or pausing the training loop. ## Parameters | | | |---|---| | attention (31 x 1,048,576) | 32,505,856 | | SwiGLU FFN, hidden 1664 | 79,233,024 | | Canon depthwise kernels | 142,848 | | RMSNorms | 32,256 | | tied embedding / LM head | 16,777,216 | | n-gram table (234,376 x 64) + projection | 15,032,832 | | **total** | **143,724,032** | Muon holds 111,771,648 parameters; AdamW holds the remaining trainable tensors. `python config.py` prints this and asserts it; `train.py` prints the per-parameter assignment and refuses to start if it disagrees with the budget. The 0917 rebalance moved capacity from the FFN into full multi-head attention while preserving the same overall size class. The FFN width is 1664 (3.25x). ## Layout config.py every judgement call, serialised into every checkpoint tokenizer.py our own 32,768-token byte-level BPE prepare.py stream -> decontaminate -> tokenize -> uint16 shards ngram.py the n-gram table: file format, builder, module model.py the model; fused Canon, varlen attention, fused LM head optim.py Muon (shape-batched Newton-Schulz), AdamW, the split schedule.py WSD, the mixture curriculum, the context stages data.py varlen packer + prefetcher checkpoint.py rolling / full / export, atomic train.py the loop eval.py standalone, on GPU 1, from a weights-only export verify.py 31 correctness checks preflight.py throughput, micro-batch sweep, ablations `data/` and `checkpoints/` are symlinks onto `/media/joseph/Media` -- `/home` has 20GB free and 200B tokens is 400GB. ## Order of operations python prepare.py plan # what the curriculum will draw python prepare.py sample --mb 2000 python tokenizer.py train --sample data/tokenizer_sample.txt python prepare.py source --name ultra_fineweb --tokens 106e9 python prepare.py source --name cosmopedia --tokens 33e9 python prepare.py source --name finephrase --tokens 34e9 python prepare.py source --name fineweb_edu --tokens 19e9 python prepare.py source --name dclm --tokens 10e9 python prepare.py val --tokens 2e7 python ngram.py build --shards data --out data/ngram_vocab.bin python verify.py # 31 checks, all must pass python preflight.py --sweep 4,8,16 # set micro_batch from measurement python train.py # or --resume python eval.py --watch # on the other GPU, separately `prepare.py` is resumable and writes incrementally, so training can start on the first sources while the rest are still arriving. ## Measured throughput (V100, 2026-09-06) ctx 2048, micro_batch 16 40.1k tok/s 33.7 TFLOPS 27.0% MFU 23.9 GB ctx 8192, micro_batch 4 32.6k tok/s 28.5 TFLOPS 22.8% MFU 24.0 GB Worst case (one full-block document per row) measures 31.2k / 23.8 GB at ctx 2048 and 15.8k / 23.8 GB at ctx 8192 -- memory is unchanged, because the memory-efficient attention kernel is O(N), so the chosen micro-batches are safe. Throughput falls because attention costs sum(len_i^2). Two-stage projection for 200B: roughly 51 days at ctx 2048 for the first 160B, plus roughly 21 days at ctx 8192 for the last 40B -- call it **~72 days**. The single-stage "200B in" column that preflight prints is optimistic by exactly this amount. What each optimisation is worth, measured by removing it: | removed | throughput | vs. all | |---|---|---| | nothing (all optimisations, mb 16) | 37.6k tok/s | — | | compiled regions (at mb 8, since mb 16 OOMs without them) | 21.4k vs 29.1k | **-26%** and +34% memory | | fp16 residual stream | 35.0k | -6.8% | | batched Newton-Schulz | 35.6k | -5.1% | | fp16 reduced-precision GEMM reduction | 34.9k | -6.9% | | fused Canon (i.e. what Canon costs) | 38.6k | +2.6% | The starting point before this round of work was 31.3k tok/s at 21.0% MFU. The compiled regions, the block-fused projection weights and the compiled LM-head chain took it to 37.6k at 25.3%; removing the host syncs took it to 40.1k at 27.0%. **Host syncs were worth 6.7%.** The nested-tensor SDPA path derives min and max sequence length from the offsets with a `.item()` on every call -- 131 host syncs per forward, about 2,100 per optimizer step, each one draining CPU run-ahead. The packer knows both while it is laying documents down, so they are passed to `nested_tensor_from_jagged` explicitly and forward+backward now performs exactly one sync. Accumulating the step loss as a GPU tensor rather than calling `float()` per micro-step removes the last one from the inner loop. `probe/syncs.py` counts them; run it after touching the attention path. Canon costing only 2.6% is the point of fusing it: the same layers as three separate convolutions per block measured 4x that. ## What is not the textbook spelling, and why Every one of these is a measured win on this card, not a preference. The measurements are in the docstrings of the modules they belong to. **Only fixed-shape regions are compiled, and they are wide.** Each block is two compiled regions -- norm+QKV+Canon+RoPE, then o_proj+residual+norm+SwiGLU+ residual -- with only attention between them. Written as separate eager modules the RMSNorms upcast to fp32 and materialise a (B,T,512) fp32 tensor 53 times; inside the regions Inductor keeps the reduction in fp32 without ever writing it. This is worth 36% and a quarter of the activation memory. Whole-model compile is still not used: document counts vary, `cu_seqlens` changes length every step, and Inductor recompiles mid-run. **Q/K/V is one Parameter, and so is gate/up.** They carry a `_boris_blocks` tag and Muon orthogonalises each row range independently -- identical to treating them as separate Parameters, since the quintic iteration acts on one matrix at a time, and `verify.py` asserts the two spellings give bit-identical updates. This removes a `torch.cat` per layer per micro-step from the forward and its backward slice. **Canon is one kernel over the fused Q|K|V block.** Depthwise convolution is per-channel, so one 1536-channel convolution equals three separate ones on Q/K/V channel ranges -- 31 kernels per forward, not 93. It is spelled as three shifted multiply-adds on the native (B,T,C) layout inside one compiled region rather than `F.conv1d(groups=C)`, which needs a transpose whose following reshape materialises a copy. The older grouped-convolution spelling was substantially slower on the V100. **Attention is varlen over a jagged nested tensor.** One SDPA launch for the whole batch and cost `sum(len_i^2)` instead of `T^2`. At ctx 2048 with realistic document lengths, fwd+bwd: nested 11.1 ms, dense block-diagonal mask 27.0 ms, per-document loop 45.8 ms -- and nested is bit-identical to the loop. **The LM head never materialises (N, V) logits.** dx and dW are computed during the forward walk over vocab chunks, so there is no recomputation and the total GEMM work equals the unfused head. 4.2x less memory, and *more* accurate than the unfused fp16 path: dx 8.4e-04 vs 1.3e-02, dW 2.0e-04 vs 3.9e-03, both against fp32. **Newton-Schulz is batched by shape.** 218 logical matrices in a few distinct shapes become four bmm chains -- about 60 kernel launches per optimizer step instead of 2,300. In fp32: five steps amplify rounding well past bf16 epsilon, and sm_70 has no bf16 tensor cores to gain from. **The residual stream is fp32.** The live V100 run measured activations above fp16's finite range. Compiled regions still cast matmul inputs under autocast, so tensor-core GEMMs remain fp16 while residual additions stay finite. **The packer emits document boundaries directly.** It knows them while laying documents down; deriving them later costs a GPU scan and a host sync per forward. ## Document isolation Rows hold several packed documents and three separate mechanisms keep them apart: varlen attention, the Canon boundary mask, and the n-gram history mask. Kernel-3 Canon stacked 31 deep reaches 62 tokens back, so without the mask every block pulls the previous document's tail across the boundary. `verify.py` de-zeroes the Canon kernels before probing for a leak. They are zero-initialised, so at init the convolution contributes exactly nothing and an isolation test passes no matter how badly the model leaks -- which is how the 75M run shipped that bug behind a green test for weeks. ## Interruption The run is expected to be interrupted. A rolling checkpoint every 45 minutes (two kept, atomic) carries the weights, both optimizer states, the loss scale, the packer's RNG and its carried document tail, so a resume replays the identical token stream. The 50B full checkpoints are archival milestones, not the resume point. `stable.pt`, written at the end of the stable phase, is permanent and never rotated: variants branch from it as a tree, and re-warming from a fully decayed model costs a loss spike and a slow recovery. A resume refuses to load a checkpoint whose config fingerprint or n-gram vocab fingerprint disagrees with the running configuration. `--force-config` overrides. ## Eval `eval.py --watch` polls the export manifest and appends one row per checkpoint to `eval_out/results.jsonl`, keyed by `tokens_seen`, so the run plots directly against training tokens. All five benchmarks are multiple-choice log-likelihood, ArithMark-3 included; both raw and length-normalised accuracy are recorded, plus held-out loss and perplexity on a fixed decontaminated split. ## September 22 repair and active run Boris-2-0920 was restarted from initialization on September 22 at 18:19 PDT, with the same 143,724,032 parameters and 200B-token target. The previous attempt is preserved under `/media/joseph/Media/Boris-2-0920/failed_20260922`. The confirmed fused-head overflow was fixed by widening the dW GEMM output before undoing its protective scaling. Validation now uses the retokenized `climbmix_val` source. Regression checks call the production head under autocast; 64 verification checks passed, followed by 20 finite V100 updates from the failed checkpoint with zero skips. Details and source snapshots are under `/media/joseph/Media/Boris-2-0920/repair_20260922/`. Current status is `/media/joseph/Media/Boris-2-0920/process_status.json` and the live console log is `/media/joseph/Media/Boris-2-0920/train.log`. Resume with `bash logs/launch_chain.sh --resume`. The launcher uses `supervise_run.py`; duplicate supervisors are refused by an exclusive lock. Missing resume checkpoints and accidental fresh launches into a nonempty checkpoint directory are errors. Do not run the old `finalize_0920.sh` to resume.