summaryrefslogtreecommitdiff
path: root/libremath/tgen.py
diff options
context:
space:
mode:
authorjdj <jdj@kunix.org>2026-11-21 09:36:58 -0800
committerjdj <jdj@kunix.org>2026-11-21 09:36:58 -0800
commit0ebd25864322ac8706fbaf1e0acb6124b36689d8 (patch)
treee37b2e887be2d485ea51cd8073f4af3275b87c46 /libremath/tgen.py
parent79aff457f6b3a9cf2dd797d1355cb7024b01117e (diff)
downloadboris-2-0ebd25864322ac8706fbaf1e0acb6124b36689d8.tar.gz
boris-2-0ebd25864322ac8706fbaf1e0acb6124b36689d8.zip
mix.md: ClimbMix was not decontaminatedHEADmain
The decontamination section said every source passes the 13-gram filter. ClimbMix was tokenized in bulk shards outside prepare.py and was not filtered, as the model card already states. Add the per-source dropped counts for the filtered sources.
Diffstat (limited to 'libremath/tgen.py')
0 files changed, 0 insertions, 0 deletions