Curating 25 Trillion Tokens for LLM Pretraining

Pretraining a large language model requires many trillions of tokens of text and code from a wide range of sources and topics. Some documents demonstrate key skills and knowledge and are worth repeating many times; others are relatively low-quality but still useful, as much of what the model encounters in the world will be messy. To train a good model, we need to choose what it sees and how often with some care.

Datakit is the Marin team’s pipeline for turning more than a hundred open datasets into such a pretraining mixture. It catalogues all the data used in the Marin project, globally deduplicates and decontaminates it, groups it into semantic categories, and then lets us learn empirically how much data to sample from each category.

For our most recent large-scale run, Datakit began with 18.71 billion raw documents containing 25.25 trillion Llama 3 tokens. We globally deduplicated and decontaminated those documents, categorized them by both quality and topic, and then ran just over 1,000 small-scale training experiments to fit a regression that estimates benchmark performance from the sampling weights over those topic and quality buckets. Our source registry, processing code, and proxy-run dataset are all open source. Future posts will explore parts of this pipeline in more depth.

Summary

• Datakit starts with 152 open datasets containing 25.25 trillion tokens, although most of that volume comes from just a handful of open-source labs.
• Global deduplication removes 2.33 billion documents—2.13 trillion tokens.
• Decontamination—removing benchmark answers from our training data—using exact 13-word n-gram matching, removes every flagged document, another 13.66 billion tokens under the current policy.
• Each document is assigned to one of 40 topics using an unsupervised embedding model, while a supervised quality scorer places it in one of five quality bands, with quality thresholds calibrated separately by content type, such as code, prose, and mathematics. Together these form 200 sampling buckets.
• More than 1,000 small proxy runs fit a regression that predicts benchmark performance from candidate weights over those buckets.

Beauty and Chaos in Open Data

Training a capable model requires not only good data but an enormous amount of it: recent open-weight models train on anywhere from 15 to 40 trillion tokens. Assembling that much data while keeping it high-quality, well licensed, and openly accessible is difficult. Thankfully, open-source work is a community effort. Many other big labs, academic groups, and individual developers have put enormous effort into building open datasets, most of them easily reached on the HuggingFace Hub. The attractive course, whenever possible, is thus to stand on the shoulders of these giants and use the open data that already exists.

But what does “using existing data” actually mean? It turns out that many datasets are largely duplicates, often derived from the same sources (the Common Crawl, for example). Some are tiny and exceptionally high-quality, while others are enormous but have relatively little information per example. A single dataset may cover topics as broad as the whole web or as narrow as a set of LeetCode interview questions. And even the labels can be misleading: a split called train may contain the test set of an existing benchmark.

Even if all you want to do is reuse existing data, you still need tools for curating it. For this blog, I’ll divide the pipeline into three stages. First, we prepare the data so documents from different sources can be used together. Then we organize those documents into buckets by topic and quality. Finally, we learn a mixture by testing how often to sample from each bucket.

1. Prepare the data

Datakit first rewrites every source into a standard data format, and then applies the same corpus-wide cleanup for duplication and contamination. Downstream experiments can therefore reasonably assume that the data has passed the same checks, without source-specific special-casing.

Download and normalize

A good curator begins by writing down what there is to work with. In Datakit, we catalogue our data in code, tracking the provenance of 152 data sources as of the time this post is written. Most of the data comes from a few datasets produced by major open-source modeling organizations: AI2, NVIDIA, HuggingFace, and the EU's High-Performance Language Technologies project. These major sources are derived from general purpose public crawls such as the Internet Archive, the Software Heritage Archive, and the Common Crawl. But there is also a long tail of high-quality, targeted data from smaller, one-off sources: papers from Eleuther AI, books from the Institutional Data Initiative, mathematics problems and solutions from Project Numina, agent traces from Kevin Xiang Li¹, and SVG illustrations from SVGFind.

Each entry in the catalogue records where Marin obtained the data and as much version info as is available, usually a HuggingFace repository pinned to a particular commit. Many datasets require extra formatting or processing, which is recorded as part of the download step within Marin. We ingest datasets which either have broadly permissive licensing from the creator or explicitly permit model training. You can view the datasets, along with their licenses, provenance, and Llama 3 token count on our token count viewer. The goal here is to have clean pipelines which allow anyone to replicate Marin’s ingestion process from the original sources we use. Notably, we do not republish duplicates of upstreamed data and do not plan to! While this creates some work when setting up a new cluster, it preserves attribution to the original creators and leaves them, rather than us, in control of their data. If a creator takes down all or part of a dataset, that decision carries through to Marin since our pipeline reads the data they maintain.

This choice does have an obvious downside compared with rehosting the data directly: some external reproducibility is lost! While pinning commits prevents minor or methodological drift, data that is taken down is unfortunately inherently unreproducible. Consider the Stack v3 from HuggingFace, which we use and which is regularly updated to respect opt-out requests. Each time you use Datakit to download and normalize this source, a small fraction of the data may therefore have been removed. This makes the overall dataset not exactly reproducible, but this tradeoff is necessary to avoid hosting copies which create a sort of whack-a-mole for dataset creators or their underlying contributors.

Once a dataset is catalogued, we normalize its records and metadata into a common representation. Datakit writes the text to Parquet, together with a stable hash of the text, a source_id, and catch-all source-specific metadata. Along the way it removes empty records, compacts excess whitespace,² enforces some reasonable file/row-group sizes, and puts the records in a stable order. Ideally, normalization is uneventful: its only purpose is to let the corpus-wide operations that follow proceed efficiently and without any special casing!

Deduplicate across sources

When training language models, epochs—the number of times data is repeated—can be an important factor in whether your model generalizes or memorizes. So in downstream experiments we want epoching behaviour to be a cleanly controlled hyperparameter. Unfortunately, individual data points are frequently duplicated, both within and especially across datasets! While natural duplication can carry information about document quality,³ that is certainly not the case when repetition occurs instead because we have ingested two sources that parsed the same webpage, or for other non-natural reasons. Thus it's important that we have reasonable confidence that every document in our corpus occurs only once.

Datakit therefore deduplicates globally. It first catches exact duplicates using the hash produced during normalization. This amounts to around 500 million exact duplicates within sources with another 5 million or so exact duplicates across sources. But the same document can vary slightly because of parsing, boilerplate removal, or other minor formatting differences that leave its fundamental information unchanged. To find these fuzzy duplicates without comparing every pair of documents, Datakit uses locality-sensitive hashing (LSH) to retrieve likely near-copies from MinHash signatures computed over word 5-grams.⁴

Unfortunately, upon inspection, we found that LSH deduplication collapsed some datasets which were not highly duplicated semantically, such as our genetic sequence and tool-calling datasets, into only a few examples. Why? We tune LSH to tolerate small differences deliberately so it can catch near duplicates. For data in which small differences are meaningful—such as DNA, where a single base pair edit can change a gene’s function—that same tolerance causes false positives. To account for this, we apply additional heuristics that would be too expensive across the entire corpus but become practical once LSH has narrowed the search, to reduce false positives. We’ll explore them in a later post; for now, see the pipeline implementation.

In our latest large-scale run, deduplication removed 2.33 billion documents—12.5% of the raw data pool—and 2.13 trillion tokens, or 8.4%. Most of these duplicates come from datasets that explicitly share source data. The largest source, for example, is the overlap between Nemotron’s processing of Common Crawl and its synthetic rephrasings of the same documents: many of the synthetic documents still overlap significantly with the real seed documents from which they were generated.

Remove benchmark contamination

Despite entertaining papers to the contrary, training on the test set is widely considered bad form. The most obvious reason is that contamination can dishonestly inflate benchmark scores.⁵ More subtly, it can bias architecture research toward models with greater memorization capacity but limited generalization—leaving lasting harms even after the contamination is removed. Rigorous decontamination is therefore important both for internal development sets and for benchmarks that will be reported as test sets upon model release.

Fortunately decontamination is smaller scale than deduplication which makes costly, high-precision filtering tractable.⁶ Datakit searches exactly for 13-word n-gram overlaps with the Artificial Analysis Intelligence Index and Marin’s pretraining evaluation sets. For each paragraph in the pretraining data, we ask what fraction of its eligible 13-gram occurrences also appear in the evaluation material (if an n-gram is repeated, we count every occurrence). Our threshold is 50%: if at least half of the eligible 13-gram occurrences in any one paragraph match, we flag the document.⁷ This rule favours precision—it catches obvious copies, but misses paraphrases and some embedded examples. Even so, the figure below finds possible contamination in many of the large datasets we train on.

2. Organize the data into buckets

Once we've normalized, deduplicated, and decontaminated each source, we can actually start training on this data. A simple sampling policy is to draw from each source in proportion to the number of tokens it contains. This approach, called proportional sampling, is a strong baseline!⁸ Still, we might expect some of our data to be more useful than others, depending on our goals.

Let's say we want to make a model dedicated to solving math problems. We might reasonably want to upsample high-quality data related to mathematics. For some sources, the dataset description tells us that's what they contain. But math data also appears in larger, heterogeneous datasets like web crawls, jumbled up with recipes, advertisements, and much else.⁹ A sampling policy expressed purely in terms of sources doesn't let us upweight these documents alongside the dedicated math data.

To enable policies that upsample data in this more intuitive, semantic fashion, we label each document with a quality score and a topic tag. These annotations route documents into buckets, grouping together data of similar topics and quality. One source can contribute to many buckets, and one bucket can contain documents from many sources.

A data mixture is then expressed as sampling weights over these buckets, with each weight specifying the fraction of training tokens its bucket supplies.¹⁰ Whether that uses only part of the bucket or repeats its documents several times depends on how much data the bucket contains and how long we train. To learn this mixture, balancing tradeoffs across domains and making better use of our data, we train many small models on different mixtures, then fit a regression to estimate how their sampling weights affect performance.

Embed the documents

Our first order of business is to categorize the data. But how? Should we assign each document to one of a list of topics, as Organize the Web does? Should we annotate separate properties, as Propella does? Or should we take inspiration from librarians and use a Dewey-inspired system, as Essential AI does?

All three are reasonable, and all draw distinctions that make sense when you look at the data. But it doesn't necessarily follow that they draw the best distinctions for training a model, especially as data distributions shift over time. Should code be a single bucket, or should we, for example, mix software repositories separately from agentic traces? Ideally, we could try different taxonomies flexibly and at relatively low compute cost.

To try and enable this, Datakit separates out the (expensive) work of mapping a document to features using a neural network from the (relatively cheaper) work of using those features to map a document to a human-interpretable class. We embed every document with Harrier, an open-source model from Microsoft. Harrier is a fine-tuned version of Qwen designed for prompt-guided retrieval, and its large, 1,024-dimensional embeddings precompute multifaceted aspects of a document for downstream use.

These embeddings themselves are not a taxonomy. They aren't human-interpretable, which means using them alone wouldn't allow us to sanity-check the reasonableness of our mixture. Nor are they magical: there are certainly some categorizations that they are unlikely to make separable in embedding space.¹¹ However, their precomputed representations allow us to iterate on different types and granularities of taxonomy relatively quickly using lightweight models conditioned on this more costly, relatively general-purpose representation.

Cluster by topic

Data mixtures are best understood as upsampling or downsampling different topics compared to their natural frequency. However, since repetition is eventually harmful, we are limited in how much we can upsample any dataset. A topic with very few tokens therefore has limited influence over the overall mixture, even if it is very high quality. The effects of such small changes can be hard to measure, leaving the relative sizes of topics in the taxonomy to dictate much of the sampling policy. We therefore aim for evenly sized buckets, giving each weight comparable influence over what the model sees. With a fixed taxonomy, though, we have little reason to expect different topics to be equally represented in our data. Instead, we use an unsupervised clustering process in the Harrier embedding space to determine document topics, with the only constraint being to keep cluster size relatively uniform.

We first use spherical k-means to divide a sample of the corpus into 5,000 compact regions, then assign a much larger sample to those regions to estimate their sizes across the whole corpus. These fine-grained clusters are then agglomeratively merged into 40 larger topics, each containing at least 1% of the total data.

To iterate and measure progress on our clustering process, we adopt Chang et al.'s randomized intruder study with an LLM judge. We show the judge several documents from the same cluster and a single document from another cluster: the intruder. The judge then tries to identify the intruder. In small intruder tests, the judge identified the odd document out in 84.4% of cases for the unagglomerated 1,000 clusters, and 62.5% for the margin-refined 40-topic taxonomy. Both exceed the 20% chance baseline, with p < 0.001 under one-sided exact binomial tests. This does not prove that our taxonomy is unique or optimal, but it does give us a lightweight reliable measure to hill climb for the semantic coherence of our clusters.

Score quality

While upsampling relevant topics is great, upsampling bad data from relevant topics is not so great. As such, every document receives a second label which ultimately places it into one of five quality bands, with thresholds calibrated by content type. We combine this band with the document’s topic to select its sampling bucket. Our classifier follows the pattern established by FineWeb-Edu, which first labels documents using a large model given a rubric for quality, then distills this into a lightweight classifier which can be used to label the entire corpus with scores.

Notably, we use these scores to group data rather than filter it in advance. Similar to our use of embedding, this is to avoid the need for reprocessing if our requirements change. In particular, data filtering has been repeatedly shown to be conditional on budget.¹² With a small amount of compute, data needs to be heavily filtered as flops should be spent on your best data. However, over a larger run, diversity can be worth decreased quality tradeoffs to avoid overfitting to a small number of documents. By scoring rather than filtering, Datakit allows downstream usage to be more easily re-calibrated depending on budget.

3. Learn the data mixture

Crossing 40 topics with five quality bands gives us 200 independently weighted buckets. How do we choose among possible weights for each bucket to create a mixture which produces a strong model?

Trying to brute force the candidates at full scale would be ruinously expensive, so we train a swarm of small proxy models on different mixtures.¹³ Each proxy gets a training budget proportional to its active parameter count, along with a proportionally scaled-down data pool for every bucket.¹⁴ Under a given mixture, the proxy therefore repeats each bucket roughly as many times as the hero would: a short-run simulation of the hero’s epoching. This lets us explore the tradeoff between repeating higher-quality data and seeing more unique data that may be lower quality.

The pipeline described here is the one we used to prepare the data and learn the mixture for our most recent hero run. The proxy dataset now contains 872 observations from our smallest possible MoE setup. We fit a regression to the proxy evaluation results, then use it to predict and optimize the data mixture, using a functional form inspired by Calvin Xu's work on the topic in Marin.¹⁵ This procedure is increasingly common, though we plan to spend a future post in this series diving deeper into our approach and experiments.

Before launching the hero run, we compare the optimized mixture with proportional sampling—the naive baseline—across a ladder of larger models, asking whether its advantage survives scale.

What's next

Among the follow-ups, we’ll look more closely at deduplication and decontamination; embeddings, topic clustering, and quality scoring; mixture experiment design, including surrogate modeling and optimization, validation, and proxy-to-target transfer; and the datasets Marin built itself for our most recent hero run.

As with all of Marin’s work, all software used is released openly:

• Source registry: the upstream repository, revision, and license for every source, recorded in a readable sources.py catalogue
• Processing code: normalization, deduplication, and decontamination in the Datakit module
• Proxy-run dataset: the mixture weights and evaluation results behind the opening figure, at marin-community/grug-moe-mix-swarm
• Token counts and licenses: per-source totals in the token count viewer

For the reasons above, we leave the upstream datasets with the people who made them. But if there’s a Datakit artifact you’d like us to release that we haven’t, let us know on Discord.

Contributions and acknowledgments

• Rafal Wojdyla: Implemented Datakit’s data-processing pipeline, including normalization, exact and fuzzy deduplication, decontamination, and the core infrastructure for quality classification.
• Will Held: Integrated data sources, implemented the Harrier embedding pipeline and mixture-swarm tooling, and developed topic clustering for the hero run. Prepared this post and its figures.
• Mark Muchane: Developed the quality classifier used for the hero run.
• Russell Power and Will Moss: Built and improved the execution, storage, and resource-management infrastructure used to process the corpus at scale.
• Calvin Xu: Developed earlier mixture-learning experiments and regression methods that informed this approach.

We thank the dataset contributors who made this work possible: AI2, NVIDIA, Hugging Face, BigCode, EleutherAI, the Common Pile contributors, HPLT, the Institutional Data Initiative, Project Numina, Essential AI, Amazon Science, Alibaba Apsara, GAIR, Locus Lab, Nebius, Open Thoughts, PleIAs, Prime Intellect, SuperDoc, Teraflop AI, Together AI, Trillion Labs, Common Crawl, the National Science Foundation, the GHALogs contributors, Ben Feuer, Kevin Xiang Li, Tim O’Donnell, Andy Arditi, Kanta Hayashi, nyuuzyou, and the Marin community.

We thank Isaac Hodes, Percy Liang, David Hall, and Rafal Wojdyla for feedback on this post.

@misc{held2026_marin_data_pipeline_overview,
  author = {Held, Will},
  title = {Curating 25 Trillion Tokens for LLM Pretraining},
  year = {2026},
  month = {sep},
  howpublished = {\url{https://www.openathena.ai/blog/marin-data-pipeline-overview/}},
  note = {Open Athena Blog}
}