summaryrefslogtreecommitdiff
path: root/README.md
blob: cd9e4c5a25a2c542008e7bd4c53eab4127298033 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
# Boris-2: training code

The complete code used to train [Boris-2](https://huggingface.co/opencerebral/Boris-2), a 144M-parameter language model trained from scratch on 200B tokens on a single NVIDIA V100 32GB, and its post-trained variants (Instruct, Think, Think-Instruct).

Weights, tokenizer and model card: <https://huggingface.co/opencerebral/Boris-2>.

## What is here

| Path | Contents |
|---|---|
| `config.py` | Every setting of the run: model shape, n-gram table, optimizer, schedule, data mixture phases. `python config.py` prints the parameter budget and asserts it. |
| `model.py` | The model: decoder with fused Canon layers, a 15M-parameter n-gram lookup table and a float32 residual stream. |
| `train.py` | The training loop: Muon + AdamW, fp16 autocast with GradScaler, rolling checkpoints, resume. |
| `data.py` | Variable-length packing; attention, Canon and the n-gram lookup are masked at document boundaries. |
| `ngram.py` | Builds the bigram/trigram table vocabulary from the training data. |
| `optim.py`, `schedule.py`, `checkpoint.py` | Optimizers, warmup-stable-decay schedule with phase-interpolated mixtures, checkpoint format. |
| `prepare.py`, `prepare_0920.sh`, `retokenize.py`, `tokenizer.py` | Downloading, filtering and tokenizing every source into shards. |
| `eval.py`, `verify.py`, `preflight.py` | In-run evaluation, numerical verification of the fused paths against references, pre-launch checks. |
| `volta_kernels/` | Hand-written CUDA/Triton kernels for the V100 (sm_70): varlen attention with fused Canon and RoPE, fused LM head. Used only when `BORIS_VOLTA_KERNELS=1` on an sm_70 GPU; every other GPU uses the stock PyTorch path. |
| `sft_chat.py` | Supervised fine-tuning for the chat and reasoning variants. |
| `scripts/` | The final training stages and the data filters used for the decay sources (see below). |
| `libremath/` | The generator behind the LibreMath v1 dataset. |
| `mix.md` | The complete data mixture: sources, filters, and the draw-weight schedule of every phase. |
| `configs/boris-2-0929/` | The exact configuration of the released run, including the configuration before each mixture change, and the final decay draw weights. |
| `docs/design-notes-0920.md` | Design notes written during the run (historical; some plans described there changed later). |

## Reproducing Boris-2

1. **Data.** `prepare_0920.sh` downloads and tokenizes every source with `prepare.py`. The sources, their share of each phase and the filters applied are in `configs/boris-2-0929/config.json` and the model card. Decay-only sources use the scripts in `scripts/`: `prepare_wordprob.py` builds the Cosmopedia tutorial sources (`cosmo2_how`, `cosmo_how_v1_tf`), dropping any article whose title matches a HellaSwag WikiHow item, plus the word-problem, MetaMathQA and LibreMath sources; `prepare_orca.py` builds Orca-Math. `hellaswag_title_split.py` and `scan_decay_contam.py` are the checks run on the result. `data/` is expected to point at the prepared shards.
2. **n-gram table.** `python ngram.py build` samples the training mixture and writes the table vocabulary.
3. **Main run.** `BORIS_RUN_DIR=<run dir> BORIS_VOLTA_KERNELS=1 python train.py --config configs/boris-2-0929/config.json`. Resume with `--resume`.
4. **Final stages** (from the 200B checkpoint), with `scripts/pilot_train.py`:
   - 30M tokens at 0.1x peak learning rate on the "nudge" mixture (`--peak 0.1`),
   - 10M tokens of LibreMath v1 at 0.02x peak learning rate (`--peak 0.02`).
   The exact mixtures and seeds are in `configs/boris-2-0929/RECIPE.md`.

Paths such as `/media/joseph/Media/...` are the defaults of the original machine. Set `BORIS_RUN_DIR`, point `data/` at your shards and edit the remaining paths in the configs for your own setup.

## Data licenses

The training data comes from public datasets with their own licenses. Most are permissive (Apache-2.0, MIT, ODC-By, CC-BY-4.0). The largest source, ClimbMix, is published by NVIDIA under CC-BY-NC-4.0 (non-commercial); Boris-2 was trained on the `karpathy/climbmix-400b-shuffle` mirror of it. Check the license of each source before reusing the data.

## Copyright & License

*Copyright 2026 OpenCerebral*

Licensed under the Apache License, Version 2.0 (the "License"); you may not use these files except in compliance with the License. You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.