summaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
authorjdj <jdj@kunix.org>2026-11-18 14:47:33 -0800
committerjdj <jdj@kunix.org>2026-11-18 14:47:33 -0800
commit5d2e4252e695644c34be605b44a14bd80a98a0a6 (patch)
tree311a72eda4459af7547b9cd05c0956eafb235f2d /docs
downloadboris-2-5d2e4252e695644c34be605b44a14bd80a98a0a6.tar.gz
boris-2-5d2e4252e695644c34be605b44a14bd80a98a0a6.zip
Boris-2 training code
The code, configuration and data filters used to train Boris-2 (144M, 200B tokens, single V100): model, training loop, data preparation, n-gram table, V100 kernels, final stages and the exact run configs.
Diffstat (limited to 'docs')
-rw-r--r--docs/design-notes-0920.md218
1 files changed, 218 insertions, 0 deletions
diff --git a/docs/design-notes-0920.md b/docs/design-notes-0920.md
new file mode 100644
index 0000000..df4ddf6
--- /dev/null
+++ b/docs/design-notes-0920.md
@@ -0,0 +1,218 @@
+# Boris-2-0920
+
+A 143.7M-parameter decoder-only language model trained from scratch on 200B
+tokens on a single V100 32GB. Deep-narrow trunk (d512 x 31L) with fused Canon
+layers, a 15M-parameter n-gram lookup table, Muon on the matmul weights, a WSD
+schedule, and a context length that extends from 2048 to 8192 at 80% of the run.
+
+ GPU 0 Tesla V100 32GB (sm_70) trains. fp16 only -- no bf16 path exists.
+ GPU 1 RTX 3060 12GB (sm_86) runs eval.py, in its own process, never
+ blocking or pausing the training loop.
+
+## Parameters
+
+| | |
+|---|---|
+| attention (31 x 1,048,576) | 32,505,856 |
+| SwiGLU FFN, hidden 1664 | 79,233,024 |
+| Canon depthwise kernels | 142,848 |
+| RMSNorms | 32,256 |
+| tied embedding / LM head | 16,777,216 |
+| n-gram table (234,376 x 64) + projection | 15,032,832 |
+| **total** | **143,724,032** |
+
+Muon holds 111,771,648 parameters; AdamW holds the remaining trainable tensors.
+`python config.py` prints this and asserts it; `train.py` prints the
+per-parameter assignment and refuses to start if it disagrees with the budget.
+
+The 0917 rebalance moved capacity from the FFN into full multi-head attention
+while preserving the same overall size class. The FFN width is 1664 (3.25x).
+
+## Layout
+
+ config.py every judgement call, serialised into every checkpoint
+ tokenizer.py our own 32,768-token byte-level BPE
+ prepare.py stream -> decontaminate -> tokenize -> uint16 shards
+ ngram.py the n-gram table: file format, builder, module
+ model.py the model; fused Canon, varlen attention, fused LM head
+ optim.py Muon (shape-batched Newton-Schulz), AdamW, the split
+ schedule.py WSD, the mixture curriculum, the context stages
+ data.py varlen packer + prefetcher
+ checkpoint.py rolling / full / export, atomic
+ train.py the loop
+ eval.py standalone, on GPU 1, from a weights-only export
+ verify.py 31 correctness checks
+ preflight.py throughput, micro-batch sweep, ablations
+
+`data/` and `checkpoints/` are symlinks onto `/media/joseph/Media` -- `/home` has
+20GB free and 200B tokens is 400GB.
+
+## Order of operations
+
+ python prepare.py plan # what the curriculum will draw
+ python prepare.py sample --mb 2000
+ python tokenizer.py train --sample data/tokenizer_sample.txt
+ python prepare.py source --name ultra_fineweb --tokens 106e9
+ python prepare.py source --name cosmopedia --tokens 33e9
+ python prepare.py source --name finephrase --tokens 34e9
+ python prepare.py source --name fineweb_edu --tokens 19e9
+ python prepare.py source --name dclm --tokens 10e9
+ python prepare.py val --tokens 2e7
+ python ngram.py build --shards data --out data/ngram_vocab.bin
+
+ python verify.py # 31 checks, all must pass
+ python preflight.py --sweep 4,8,16 # set micro_batch from measurement
+
+ python train.py # or --resume
+ python eval.py --watch # on the other GPU, separately
+
+`prepare.py` is resumable and writes incrementally, so training can start on the
+first sources while the rest are still arriving.
+
+## Measured throughput (V100, 2026-09-06)
+
+ ctx 2048, micro_batch 16 40.1k tok/s 33.7 TFLOPS 27.0% MFU 23.9 GB
+ ctx 8192, micro_batch 4 32.6k tok/s 28.5 TFLOPS 22.8% MFU 24.0 GB
+
+Worst case (one full-block document per row) measures 31.2k / 23.8 GB at ctx
+2048 and 15.8k / 23.8 GB at ctx 8192 -- memory is unchanged, because the
+memory-efficient attention kernel is O(N), so the chosen micro-batches are safe.
+Throughput falls because attention costs sum(len_i^2).
+
+Two-stage projection for 200B: roughly 51 days at ctx 2048 for the first 160B,
+plus roughly 21 days at ctx 8192 for the last 40B -- call it **~72 days**. The
+single-stage "200B in" column that preflight prints is optimistic by exactly
+this amount.
+
+What each optimisation is worth, measured by removing it:
+
+| removed | throughput | vs. all |
+|---|---|---|
+| nothing (all optimisations, mb 16) | 37.6k tok/s | — |
+| compiled regions (at mb 8, since mb 16 OOMs without them) | 21.4k vs 29.1k | **-26%** and +34% memory |
+| fp16 residual stream | 35.0k | -6.8% |
+| batched Newton-Schulz | 35.6k | -5.1% |
+| fp16 reduced-precision GEMM reduction | 34.9k | -6.9% |
+| fused Canon (i.e. what Canon costs) | 38.6k | +2.6% |
+
+The starting point before this round of work was 31.3k tok/s at 21.0% MFU. The
+compiled regions, the block-fused projection weights and the compiled LM-head
+chain took it to 37.6k at 25.3%; removing the host syncs took it to 40.1k at
+27.0%.
+
+**Host syncs were worth 6.7%.** The nested-tensor SDPA path derives min and max
+sequence length from the offsets with a `.item()` on every call -- 131 host syncs
+per forward, about 2,100 per optimizer step, each one draining CPU run-ahead. The
+packer knows both while it is laying documents down, so they are passed to
+`nested_tensor_from_jagged` explicitly and forward+backward now performs exactly
+one sync. Accumulating the step loss as a GPU tensor rather than calling
+`float()` per micro-step removes the last one from the inner loop.
+`probe/syncs.py` counts them; run it after touching the attention path.
+
+Canon costing only 2.6% is the point of fusing it: the same layers as three
+separate convolutions per block measured 4x that.
+
+## What is not the textbook spelling, and why
+
+Every one of these is a measured win on this card, not a preference. The
+measurements are in the docstrings of the modules they belong to.
+
+**Only fixed-shape regions are compiled, and they are wide.** Each block is two
+compiled regions -- norm+QKV+Canon+RoPE, then o_proj+residual+norm+SwiGLU+
+residual -- with only attention between them. Written as separate eager modules
+the RMSNorms upcast to fp32 and materialise a (B,T,512) fp32 tensor 53 times;
+inside the regions Inductor keeps the reduction in fp32 without ever writing it.
+This is worth 36% and a quarter of the activation memory. Whole-model compile is
+still not used: document counts vary, `cu_seqlens` changes length every step, and
+Inductor recompiles mid-run.
+
+**Q/K/V is one Parameter, and so is gate/up.** They carry a `_boris_blocks` tag
+and Muon orthogonalises each row range independently -- identical to treating
+them as separate Parameters, since the quintic iteration acts on one matrix at a
+time, and `verify.py` asserts the two spellings give bit-identical updates. This
+removes a `torch.cat` per layer per micro-step from the forward and its backward
+slice.
+
+**Canon is one kernel over the fused Q|K|V block.** Depthwise convolution is
+per-channel, so one 1536-channel convolution equals three separate ones on
+Q/K/V channel ranges -- 31 kernels per forward, not 93. It is spelled as three
+shifted multiply-adds on the native (B,T,C) layout inside one compiled region
+rather than `F.conv1d(groups=C)`, which needs a transpose whose following
+reshape materialises a copy. The older grouped-convolution spelling was
+substantially slower on the V100.
+
+**Attention is varlen over a jagged nested tensor.** One SDPA launch for the
+whole batch and cost `sum(len_i^2)` instead of `T^2`. At ctx 2048 with realistic
+document lengths, fwd+bwd: nested 11.1 ms, dense block-diagonal mask 27.0 ms,
+per-document loop 45.8 ms -- and nested is bit-identical to the loop.
+
+**The LM head never materialises (N, V) logits.** dx and dW are computed during
+the forward walk over vocab chunks, so there is no recomputation and the total
+GEMM work equals the unfused head. 4.2x less memory, and *more* accurate than
+the unfused fp16 path: dx 8.4e-04 vs 1.3e-02, dW 2.0e-04 vs 3.9e-03, both
+against fp32.
+
+**Newton-Schulz is batched by shape.** 218 logical matrices in a few distinct shapes
+become four bmm chains -- about 60 kernel launches per optimizer step instead of
+2,300. In fp32: five steps amplify rounding well past bf16 epsilon, and sm_70
+has no bf16 tensor cores to gain from.
+
+**The residual stream is fp32.** The live V100 run measured activations above
+fp16's finite range. Compiled regions still cast matmul inputs under autocast,
+so tensor-core GEMMs remain fp16 while residual additions stay finite.
+
+**The packer emits document boundaries directly.** It knows them while laying
+documents down; deriving them later costs a GPU scan and a host sync per forward.
+
+## Document isolation
+
+Rows hold several packed documents and three separate mechanisms keep them
+apart: varlen attention, the Canon boundary mask, and the n-gram history mask.
+Kernel-3 Canon stacked 31 deep reaches 62 tokens back, so without the mask every
+block pulls the previous document's tail across the boundary.
+
+`verify.py` de-zeroes the Canon kernels before probing for a leak. They are
+zero-initialised, so at init the convolution contributes exactly nothing and an
+isolation test passes no matter how badly the model leaks -- which is how the
+75M run shipped that bug behind a green test for weeks.
+
+## Interruption
+
+The run is expected to be interrupted. A rolling checkpoint every 45 minutes
+(two kept, atomic) carries the weights, both optimizer states, the loss scale,
+the packer's RNG and its carried document tail, so a resume replays the identical
+token stream. The 50B full checkpoints are archival milestones, not the resume
+point. `stable.pt`, written at the end of the stable phase, is permanent and
+never rotated: variants branch from it as a tree, and re-warming from a fully
+decayed model costs a loss spike and a slow recovery.
+
+A resume refuses to load a checkpoint whose config fingerprint or n-gram vocab
+fingerprint disagrees with the running configuration. `--force-config` overrides.
+
+## Eval
+
+`eval.py --watch` polls the export manifest and appends one row per checkpoint to
+`eval_out/results.jsonl`, keyed by `tokens_seen`, so the run plots directly
+against training tokens. All five benchmarks are multiple-choice log-likelihood,
+ArithMark-3 included; both raw and length-normalised accuracy are recorded, plus
+held-out loss and perplexity on a fixed decontaminated split.
+
+## September 22 repair and active run
+
+Boris-2-0920 was restarted from initialization on September 22 at 18:19 PDT,
+with the same 143,724,032 parameters and 200B-token target. The previous attempt
+is preserved under `/media/joseph/Media/Boris-2-0920/failed_20260922`.
+
+The confirmed fused-head overflow was fixed by widening the dW GEMM output
+before undoing its protective scaling. Validation now uses the retokenized
+`climbmix_val` source. Regression checks call the production head under autocast;
+64 verification checks passed, followed by 20 finite V100 updates from the failed
+checkpoint with zero skips. Details and source snapshots are under
+`/media/joseph/Media/Boris-2-0920/repair_20260922/`.
+
+Current status is `/media/joseph/Media/Boris-2-0920/process_status.json` and the
+live console log is `/media/joseph/Media/Boris-2-0920/train.log`.
+Resume with `bash logs/launch_chain.sh --resume`. The launcher uses
+`supervise_run.py`; duplicate supervisors are refused by an exclusive lock.
+Missing resume checkpoints and accidental fresh launches into a nonempty
+checkpoint directory are errors. Do not run the old `finalize_0920.sh` to resume.