From 0ebd25864322ac8706fbaf1e0acb6124b36689d8 Mon Sep 17 00:00:00 2001 From: jdj Date: Sat, 21 Nov 2026 09:36:58 -0800 Subject: mix.md: ClimbMix was not decontaminated The decontamination section said every source passes the 13-gram filter. ClimbMix was tokenized in bulk shards outside prepare.py and was not filtered, as the model card already states. Add the per-source dropped counts for the filtered sources. --- mix.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mix.md b/mix.md index d8f87a9..4007ec4 100644 --- a/mix.md +++ b/mix.md @@ -17,11 +17,15 @@ The complete data mixture of the released Boris-2 run: every source, how it was | `dclm_r0920` | [mlfoundations/dclm-baseline-1.0-parquet](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0-parquet) | a sample | CC-BY-4.0 | | `libremath_v1` | [opencerebral/LibreMath-v1](https://huggingface.co/datasets/opencerebral/LibreMath-v1) | the whole set (final stage only) | Apache-2.0 | -Build code: `prepare.py` for the web, FineMath, UltraData-Math and DCLM sources; `scripts/prepare_wordprob.py` for `cosmo2_how`, `cosmo_how_v1_tf`, `metamath_gsm`, `elem_wordprob` and `libremath_v1`; `scripts/prepare_orca.py` for `orca_math`. +Build code: `prepare.py` for the FineMath, UltraData-Math and DCLM sources (ClimbMix was tokenized separately, in bulk shards, without the filter; see Decontamination); `scripts/prepare_wordprob.py` for `cosmo2_how`, `cosmo_how_v1_tf`, `metamath_gsm`, `elem_wordprob` and `libremath_v1`; `scripts/prepare_orca.py` for `orca_math`. ### Decontamination -Every source passes the same inline filter in `prepare.py`: a document is dropped if it shares a 13-word span with any item of the evaluation sets in `eval.py` (questions and answers). Every overlapping 13-gram of the evaluation side is banned, so any shared run of 25 words or more is always caught. The two Cosmopedia sources additionally drop any article whose title matches a HellaSwag WikiHow title, because some of those articles were generated from real WikiHow titles. +**ClimbMix was not filtered.** It was tokenized separately from `prepare.py`, in bulk shards, so none of the checks below were applied to it, and it is the large majority of the training data. This matches the model card. An earlier version of this file wrongly said every source was filtered. + +Every other source passes the same inline filter in `prepare.py` (or the same check in `scripts/prepare_wordprob.py` and `scripts/prepare_orca.py`): a document is dropped if it shares a 13-word span with any item of the evaluation sets in `eval.py` (questions and answers). Every overlapping 13-gram of the evaluation side is banned, so any shared run of 25 words or more is always caught. The two Cosmopedia sources additionally drop any article whose title matches a HellaSwag WikiHow title, because some of those articles were generated from real WikiHow titles. + +Documents dropped by the 13-gram filter: FineMath 171 + 190, FineWeb-Edu 23 + 32, UltraData-Math L3 1 + 6 + 26 + 44 (per shard group). ## Schedule -- cgit v1.2.3