diff options
| author | jdj <jdj@kunix.org> | 2026-11-21 09:36:58 -0800 |
|---|---|---|
| committer | jdj <jdj@kunix.org> | 2026-11-21 09:36:58 -0800 |
| commit | 0ebd25864322ac8706fbaf1e0acb6124b36689d8 (patch) | |
| tree | e37b2e887be2d485ea51cd8073f4af3275b87c46 /mix.md | |
| parent | 79aff457f6b3a9cf2dd797d1355cb7024b01117e (diff) | |
| download | boris-2-main.tar.gz boris-2-main.zip | |
The decontamination section said every source passes the 13-gram filter.
ClimbMix was tokenized in bulk shards outside prepare.py and was not
filtered, as the model card already states. Add the per-source dropped
counts for the filtered sources.
Diffstat (limited to 'mix.md')
| -rw-r--r-- | mix.md | 8 |
1 files changed, 6 insertions, 2 deletions
@@ -17,11 +17,15 @@ The complete data mixture of the released Boris-2 run: every source, how it was | `dclm_r0920` | [mlfoundations/dclm-baseline-1.0-parquet](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0-parquet) | a sample | CC-BY-4.0 | | `libremath_v1` | [opencerebral/LibreMath-v1](https://huggingface.co/datasets/opencerebral/LibreMath-v1) | the whole set (final stage only) | Apache-2.0 | -Build code: `prepare.py` for the web, FineMath, UltraData-Math and DCLM sources; `scripts/prepare_wordprob.py` for `cosmo2_how`, `cosmo_how_v1_tf`, `metamath_gsm`, `elem_wordprob` and `libremath_v1`; `scripts/prepare_orca.py` for `orca_math`. +Build code: `prepare.py` for the FineMath, UltraData-Math and DCLM sources (ClimbMix was tokenized separately, in bulk shards, without the filter; see Decontamination); `scripts/prepare_wordprob.py` for `cosmo2_how`, `cosmo_how_v1_tf`, `metamath_gsm`, `elem_wordprob` and `libremath_v1`; `scripts/prepare_orca.py` for `orca_math`. ### Decontamination -Every source passes the same inline filter in `prepare.py`: a document is dropped if it shares a 13-word span with any item of the evaluation sets in `eval.py` (questions and answers). Every overlapping 13-gram of the evaluation side is banned, so any shared run of 25 words or more is always caught. The two Cosmopedia sources additionally drop any article whose title matches a HellaSwag WikiHow title, because some of those articles were generated from real WikiHow titles. +**ClimbMix was not filtered.** It was tokenized separately from `prepare.py`, in bulk shards, so none of the checks below were applied to it, and it is the large majority of the training data. This matches the model card. An earlier version of this file wrongly said every source was filtered. + +Every other source passes the same inline filter in `prepare.py` (or the same check in `scripts/prepare_wordprob.py` and `scripts/prepare_orca.py`): a document is dropped if it shares a 13-word span with any item of the evaluation sets in `eval.py` (questions and answers). Every overlapping 13-gram of the evaluation side is banned, so any shared run of 25 words or more is always caught. The two Cosmopedia sources additionally drop any article whose title matches a HellaSwag WikiHow title, because some of those articles were generated from real WikiHow titles. + +Documents dropped by the 13-gram filter: FineMath 171 + 190, FineWeb-Edu 23 + 32, UltraData-Math L3 1 + 6 + 26 + 44 (per shard group). ## Schedule |
