From 79aff457f6b3a9cf2dd797d1355cb7024b01117e Mon Sep 17 00:00:00 2001 From: jdj Date: Wed, 18 Nov 2026 14:52:51 -0800 Subject: Add mix.md: the complete data mixture Every source with its dataset, subsets, filters and license; the draw-weight schedule of every phase; batch, context and final stages. --- mix.md | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 mix.md (limited to 'mix.md') diff --git a/mix.md b/mix.md new file mode 100644 index 0000000..d8f87a9 --- /dev/null +++ b/mix.md @@ -0,0 +1,74 @@ +# Boris-2 data mixture + +The complete data mixture of the released Boris-2 run: every source, how it was built, and the schedule of draw weights over the 200B-token run and the two final stages. The machine-readable version is `configs/boris-2-0929/config.json` (`data.sources`, `curriculum.phases`); the files `config.before_*.json` are the configuration as it stood before each change made during the run. + +## Sources + +| Name in the configs | Dataset | What was used | License | +|---|---|---|---| +| `climbmix` | [karpathy/climbmix-400b-shuffle](https://huggingface.co/datasets/karpathy/climbmix-400b-shuffle), a pre-shuffled mirror of [nvidia/ClimbMix](https://huggingface.co/datasets/nvidia/ClimbMix) | the first ~200B tokens (the mirror is shuffled, so a prefix is a representative sample) | CC-BY-NC-4.0 (upstream); the mirror is tagged MIT | +| `finemath` | [HuggingFaceTB/finemath](https://huggingface.co/datasets/HuggingFaceTB/finemath) | `finemath-4plus` first, then `finemath-3plus` | ODC-By | +| `ultra_math_l3` | [openbmb/UltraData-Math](https://huggingface.co/datasets/openbmb/UltraData-Math) | the four L3 subsets: QA, Textbook-Exercise, Multi-Style and Conversation (synthetic rewrites) | Apache-2.0 | +| `orca_math` | [microsoft/orca-math-word-problems-200k](https://huggingface.co/datasets/microsoft/orca-math-word-problems-200k) | question and answer as one document | MIT | +| `cosmo2_how` | Cosmopedia v2 in [HuggingFaceTB/smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | the `wikihow`-format articles only, minus every article whose title matches a HellaSwag WikiHow item (validation or test) | ODC-By | +| `cosmo_how_v1_tf` | [HuggingFaceTB/cosmopedia](https://huggingface.co/datasets/HuggingFaceTB/cosmopedia) | the `wikihow` subset, with the same HellaSwag-title filter | Apache-2.0 | +| `metamath_gsm` | [meta-math/MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA) | the GSM8K-derived rows only (`type` starting with `GSM`); the MATH-derived rows are left out | MIT | +| `elem_wordprob` | [MU-NLPC/Calc-ape210k](https://huggingface.co/datasets/MU-NLPC/Calc-ape210k) and [MU-NLPC/Calc-mawps](https://huggingface.co/datasets/MU-NLPC/Calc-mawps) | training splits only | MIT | +| `dclm_r0920` | [mlfoundations/dclm-baseline-1.0-parquet](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0-parquet) | a sample | CC-BY-4.0 | +| `libremath_v1` | [opencerebral/LibreMath-v1](https://huggingface.co/datasets/opencerebral/LibreMath-v1) | the whole set (final stage only) | Apache-2.0 | + +Build code: `prepare.py` for the web, FineMath, UltraData-Math and DCLM sources; `scripts/prepare_wordprob.py` for `cosmo2_how`, `cosmo_how_v1_tf`, `metamath_gsm`, `elem_wordprob` and `libremath_v1`; `scripts/prepare_orca.py` for `orca_math`. + +### Decontamination + +Every source passes the same inline filter in `prepare.py`: a document is dropped if it shares a 13-word span with any item of the evaluation sets in `eval.py` (questions and answers). Every overlapping 13-gram of the evaluation side is banned, so any shared run of 25 words or more is always caught. The two Cosmopedia sources additionally drop any article whose title matches a HellaSwag WikiHow title, because some of those articles were generated from real WikiHow titles. + +## Schedule + +The weights below are **per-document draw probabilities**: for each document the packer picks a source with these probabilities, then a random document from it. Sources differ in average document length, so the share of tokens differs from the share of draws (FineMath documents are about 2.4x longer than ClimbMix ones, for example). Between two phases the weights are interpolated linearly in token progress. + +| Tokens | Phase | climbmix | finemath | ultra_math_l3 | orca_math | cosmo2_how | cosmo_how_v1_tf | metamath_gsm | elem_wordprob | dclm_r0920 | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| 0 – 55B | early | 0.95 | 0.05 | | | | | | | | +| 55B – 150B | flat | 0.932109 | | 0.064783 | 0.003108 | | | | | | +| 150B – 174B | ramp to `pre_decay_ramp` | 0.821333 | | 0.081144 | 0.003244 | 0.085976 | 0.008302 | | | | +| 174B – 175B | ramp to `decay` | 0.775534 | | 0.078620 | 0.003143 | 0.083301 | 0.008044 | 0.025126 | 0.020974 | 0.005257 | +| 175B – 196.5B | decay | 0.775534 | | 0.078620 | 0.003143 | 0.083301 | 0.008044 | 0.025126 | 0.020974 | 0.005257 | +| 196.5B – 197.5B | ramp to tail | 0.580561 | | 0.053042 | 0.002545 | 0.067493 | 0.006468 | 0.135615 | 0.141507 | 0.012769 | +| 197.5B – 200B | tail | 0.580561 | | 0.053042 | 0.002545 | 0.067493 | 0.006468 | 0.135615 | 0.141507 | 0.012769 | + +In tokens, the early phase is about 89% ClimbMix and 11% FineMath; the flat phase about 90% ClimbMix, 10% UltraData-Math L3 and 0.15% Orca-Math. The decay mix, by tokens: + +| Source | Share of decay tokens | +|---|---:| +| ClimbMix | 73.92% | +| UltraData-Math L3 | 12% | +| Cosmopedia v2 WikiHow-style tutorials | 10.9% | +| Cosmopedia v1 tutorials (title-filtered) | 1.1% | +| DCLM-Baseline | 1% | +| MetaMathQA (GSM8K-derived rows) | 0.75% | +| Calc-ape210k / Calc-mawps | 0.18% | +| Orca-Math | 0.15% | + +Some of this schedule was changed while the run was in progress, including the FineMath-to-UltraData-Math switch at 55B, the decay mixture and the final tail; the `config.before_*.json` files record the configuration before each change. + +### Learning rate, batch and context + +- Learning rate: 1% warmup, constant to 174B (87% of the run), then 1-sqrt decay to zero over the last 26B tokens. +- Batch: 524,288 tokens per step to 170B, then 1,048,576. +- Context: 2,048 tokens for the whole run. Documents are packed into rows with attention, Canon and the n-gram lookup masked at document boundaries; documents longer than a row continue in the next row instead of being cut. +- Long-document preference: off. `curriculum.long_doc_floor_tokens` and `long_doc_target_frac` only take effect when the context grows beyond its first stage, and the planned 8,192-token stage was removed before it started. + +## Final stages + +Both use `scripts/pilot_train.py`, starting from the 200B checkpoint; the LibreMath stage starts with fresh optimizer state. + +1. **Nudge**: 30M tokens at 0.1x the peak learning rate, 1-sqrt to zero, seed 1234. Per-document draw weights: + + | climbmix | cosmo2_how | cosmo_how_v1_tf | ultra_math_l3 | metamath_gsm | orca_math | + |---:|---:|---:|---:|---:|---:| + | 0.55 | 0.30 | 0.05 | 0.07 | 0.02 | 0.01 | + +2. **LibreMath**: 10M tokens of `libremath_v1` only, at 0.02x the peak learning rate, 1-sqrt to zero. + +Exact commands, seeds and the selection of the released checkpoint are in `configs/boris-2-0929/RECIPE.md`. -- cgit v1.2.3