1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
|
# Boris-2-0920
A 143.7M-parameter decoder-only language model trained from scratch on 200B
tokens on a single V100 32GB. Deep-narrow trunk (d512 x 31L) with fused Canon
layers, a 15M-parameter n-gram lookup table, Muon on the matmul weights, a WSD
schedule, and a context length that extends from 2048 to 8192 at 80% of the run.
GPU 0 Tesla V100 32GB (sm_70) trains. fp16 only -- no bf16 path exists.
GPU 1 RTX 3060 12GB (sm_86) runs eval.py, in its own process, never
blocking or pausing the training loop.
## Parameters
| | |
|---|---|
| attention (31 x 1,048,576) | 32,505,856 |
| SwiGLU FFN, hidden 1664 | 79,233,024 |
| Canon depthwise kernels | 142,848 |
| RMSNorms | 32,256 |
| tied embedding / LM head | 16,777,216 |
| n-gram table (234,376 x 64) + projection | 15,032,832 |
| **total** | **143,724,032** |
Muon holds 111,771,648 parameters; AdamW holds the remaining trainable tensors.
`python config.py` prints this and asserts it; `train.py` prints the
per-parameter assignment and refuses to start if it disagrees with the budget.
The 0917 rebalance moved capacity from the FFN into full multi-head attention
while preserving the same overall size class. The FFN width is 1664 (3.25x).
## Layout
config.py every judgement call, serialised into every checkpoint
tokenizer.py our own 32,768-token byte-level BPE
prepare.py stream -> decontaminate -> tokenize -> uint16 shards
ngram.py the n-gram table: file format, builder, module
model.py the model; fused Canon, varlen attention, fused LM head
optim.py Muon (shape-batched Newton-Schulz), AdamW, the split
schedule.py WSD, the mixture curriculum, the context stages
data.py varlen packer + prefetcher
checkpoint.py rolling / full / export, atomic
train.py the loop
eval.py standalone, on GPU 1, from a weights-only export
verify.py 31 correctness checks
preflight.py throughput, micro-batch sweep, ablations
`data/` and `checkpoints/` are symlinks onto `/media/joseph/Media` -- `/home` has
20GB free and 200B tokens is 400GB.
## Order of operations
python prepare.py plan # what the curriculum will draw
python prepare.py sample --mb 2000
python tokenizer.py train --sample data/tokenizer_sample.txt
python prepare.py source --name ultra_fineweb --tokens 106e9
python prepare.py source --name cosmopedia --tokens 33e9
python prepare.py source --name finephrase --tokens 34e9
python prepare.py source --name fineweb_edu --tokens 19e9
python prepare.py source --name dclm --tokens 10e9
python prepare.py val --tokens 2e7
python ngram.py build --shards data --out data/ngram_vocab.bin
python verify.py # 31 checks, all must pass
python preflight.py --sweep 4,8,16 # set micro_batch from measurement
python train.py # or --resume
python eval.py --watch # on the other GPU, separately
`prepare.py` is resumable and writes incrementally, so training can start on the
first sources while the rest are still arriving.
## Measured throughput (V100, 2026-09-06)
ctx 2048, micro_batch 16 40.1k tok/s 33.7 TFLOPS 27.0% MFU 23.9 GB
ctx 8192, micro_batch 4 32.6k tok/s 28.5 TFLOPS 22.8% MFU 24.0 GB
Worst case (one full-block document per row) measures 31.2k / 23.8 GB at ctx
2048 and 15.8k / 23.8 GB at ctx 8192 -- memory is unchanged, because the
memory-efficient attention kernel is O(N), so the chosen micro-batches are safe.
Throughput falls because attention costs sum(len_i^2).
Two-stage projection for 200B: roughly 51 days at ctx 2048 for the first 160B,
plus roughly 21 days at ctx 8192 for the last 40B -- call it **~72 days**. The
single-stage "200B in" column that preflight prints is optimistic by exactly
this amount.
What each optimisation is worth, measured by removing it:
| removed | throughput | vs. all |
|---|---|---|
| nothing (all optimisations, mb 16) | 37.6k tok/s | — |
| compiled regions (at mb 8, since mb 16 OOMs without them) | 21.4k vs 29.1k | **-26%** and +34% memory |
| fp16 residual stream | 35.0k | -6.8% |
| batched Newton-Schulz | 35.6k | -5.1% |
| fp16 reduced-precision GEMM reduction | 34.9k | -6.9% |
| fused Canon (i.e. what Canon costs) | 38.6k | +2.6% |
The starting point before this round of work was 31.3k tok/s at 21.0% MFU. The
compiled regions, the block-fused projection weights and the compiled LM-head
chain took it to 37.6k at 25.3%; removing the host syncs took it to 40.1k at
27.0%.
**Host syncs were worth 6.7%.** The nested-tensor SDPA path derives min and max
sequence length from the offsets with a `.item()` on every call -- 131 host syncs
per forward, about 2,100 per optimizer step, each one draining CPU run-ahead. The
packer knows both while it is laying documents down, so they are passed to
`nested_tensor_from_jagged` explicitly and forward+backward now performs exactly
one sync. Accumulating the step loss as a GPU tensor rather than calling
`float()` per micro-step removes the last one from the inner loop.
`probe/syncs.py` counts them; run it after touching the attention path.
Canon costing only 2.6% is the point of fusing it: the same layers as three
separate convolutions per block measured 4x that.
## What is not the textbook spelling, and why
Every one of these is a measured win on this card, not a preference. The
measurements are in the docstrings of the modules they belong to.
**Only fixed-shape regions are compiled, and they are wide.** Each block is two
compiled regions -- norm+QKV+Canon+RoPE, then o_proj+residual+norm+SwiGLU+
residual -- with only attention between them. Written as separate eager modules
the RMSNorms upcast to fp32 and materialise a (B,T,512) fp32 tensor 53 times;
inside the regions Inductor keeps the reduction in fp32 without ever writing it.
This is worth 36% and a quarter of the activation memory. Whole-model compile is
still not used: document counts vary, `cu_seqlens` changes length every step, and
Inductor recompiles mid-run.
**Q/K/V is one Parameter, and so is gate/up.** They carry a `_boris_blocks` tag
and Muon orthogonalises each row range independently -- identical to treating
them as separate Parameters, since the quintic iteration acts on one matrix at a
time, and `verify.py` asserts the two spellings give bit-identical updates. This
removes a `torch.cat` per layer per micro-step from the forward and its backward
slice.
**Canon is one kernel over the fused Q|K|V block.** Depthwise convolution is
per-channel, so one 1536-channel convolution equals three separate ones on
Q/K/V channel ranges -- 31 kernels per forward, not 93. It is spelled as three
shifted multiply-adds on the native (B,T,C) layout inside one compiled region
rather than `F.conv1d(groups=C)`, which needs a transpose whose following
reshape materialises a copy. The older grouped-convolution spelling was
substantially slower on the V100.
**Attention is varlen over a jagged nested tensor.** One SDPA launch for the
whole batch and cost `sum(len_i^2)` instead of `T^2`. At ctx 2048 with realistic
document lengths, fwd+bwd: nested 11.1 ms, dense block-diagonal mask 27.0 ms,
per-document loop 45.8 ms -- and nested is bit-identical to the loop.
**The LM head never materialises (N, V) logits.** dx and dW are computed during
the forward walk over vocab chunks, so there is no recomputation and the total
GEMM work equals the unfused head. 4.2x less memory, and *more* accurate than
the unfused fp16 path: dx 8.4e-04 vs 1.3e-02, dW 2.0e-04 vs 3.9e-03, both
against fp32.
**Newton-Schulz is batched by shape.** 218 logical matrices in a few distinct shapes
become four bmm chains -- about 60 kernel launches per optimizer step instead of
2,300. In fp32: five steps amplify rounding well past bf16 epsilon, and sm_70
has no bf16 tensor cores to gain from.
**The residual stream is fp32.** The live V100 run measured activations above
fp16's finite range. Compiled regions still cast matmul inputs under autocast,
so tensor-core GEMMs remain fp16 while residual additions stay finite.
**The packer emits document boundaries directly.** It knows them while laying
documents down; deriving them later costs a GPU scan and a host sync per forward.
## Document isolation
Rows hold several packed documents and three separate mechanisms keep them
apart: varlen attention, the Canon boundary mask, and the n-gram history mask.
Kernel-3 Canon stacked 31 deep reaches 62 tokens back, so without the mask every
block pulls the previous document's tail across the boundary.
`verify.py` de-zeroes the Canon kernels before probing for a leak. They are
zero-initialised, so at init the convolution contributes exactly nothing and an
isolation test passes no matter how badly the model leaks -- which is how the
75M run shipped that bug behind a green test for weeks.
## Interruption
The run is expected to be interrupted. A rolling checkpoint every 45 minutes
(two kept, atomic) carries the weights, both optimizer states, the loss scale,
the packer's RNG and its carried document tail, so a resume replays the identical
token stream. The 50B full checkpoints are archival milestones, not the resume
point. `stable.pt`, written at the end of the stable phase, is permanent and
never rotated: variants branch from it as a tree, and re-warming from a fully
decayed model costs a loss spike and a slow recovery.
A resume refuses to load a checkpoint whose config fingerprint or n-gram vocab
fingerprint disagrees with the running configuration. `--force-config` overrides.
## Eval
`eval.py --watch` polls the export manifest and appends one row per checkpoint to
`eval_out/results.jsonl`, keyed by `tokens_seen`, so the run plots directly
against training tokens. All five benchmarks are multiple-choice log-likelihood,
ArithMark-3 included; both raw and length-normalised accuracy are recorded, plus
held-out loss and perplexity on a fixed decontaminated split.
## September 22 repair and active run
Boris-2-0920 was restarted from initialization on September 22 at 18:19 PDT,
with the same 143,724,032 parameters and 200B-token target. The previous attempt
is preserved under `/media/joseph/Media/Boris-2-0920/failed_20260922`.
The confirmed fused-head overflow was fixed by widening the dW GEMM output
before undoing its protective scaling. Validation now uses the retokenized
`climbmix_val` source. Regression checks call the production head under autocast;
64 verification checks passed, followed by 20 finite V100 updates from the failed
checkpoint with zero skips. Details and source snapshots are under
`/media/joseph/Media/Boris-2-0920/repair_20260922/`.
Current status is `/media/joseph/Media/Boris-2-0920/process_status.json` and the
live console log is `/media/joseph/Media/Boris-2-0920/train.log`.
Resume with `bash logs/launch_chain.sh --resume`. The launcher uses
`supervise_run.py`; duplicate supervisors are refused by an exclusive lock.
Missing resume checkpoints and accidental fresh launches into a nonempty
checkpoint directory are errors. Do not run the old `finalize_0920.sh` to resume.
|