A 1B standard Transformer rivals Evo 2 40B on variant effect prediction

MarinDNA applies the tools and open-development approach of Marin to genomic language modeling. This post summarizes how data curation, hyperparameter transfer, scaling laws, and data-mixture experiments produced a 1B GPT-style model competitive with Evo 2 40B, while using ~1,980× fewer training FLOPs and scoring variants ~2,330× faster.

Summary

• Balanced data mixtures produce more even VEP performance across coding and non-coding regions. Uniform weighting prevents the larger CDS dataset from dominating training. Adding ncRNA and enhancer data improves performance on ncRNA and distal variants.
• Optimization transfers, and loss scales predictably. Hyperparameters tuned on ~25M-parameter reference models predicted the optimal learning rate for 255M, 476M, and 1B models; using that recipe, validation loss followed a clean scaling law through 4B parameters.
• Zero-shot VEP can regress with scale even as linear-probe VEP improves. Zero-shot LLR improves for most variant classes, but deteriorates for Mendelian missense as model scale and validation log-likelihood increase.
• The resulting 1B model rivals Evo 2 40B on Mendelian VEP. MarinDNA m5.1 slightly leads Evo 2 40B in zero-shot macro-average AUPRC while using ~1,980× fewer training FLOPs and scoring variants ~2,330× faster, although alignment-based and supervised models remain stronger overall.

A note on open development. This post turns a branching research process into a linear narrative; the MarinDNA repository preserves the underlying experiments, including unsuccessful and inconclusive directions.

Introduction

Optimization of genomic language models (gLMs) has historically involved a lot of focus on specialized architectures.¹ We take a different approach: keep the architecture within the standard GPT family and focus on data curation and mixtures, training hyperparameters, and model scale. The experiments below test how far that simpler, standardized recipe can go.

Why alignment-free gLMs?

Many of the strongest genomic sequence models rely on whole-genome alignments, as in GPN-Star,² or functional-genomics measurements, as in AlphaGenome—resources concentrated in a small number of well-studied species because they are costly to produce and difficult to construct comprehensively. Unlabeled DNA sequence, by contrast, is available for a rapidly growing number of species. Alignment-free gLMs—also called single-sequence gLMs—can be trained directly on this growing collection of genomes and applied to evolutionary constraint prediction, sequence design, and transfer learning.

For humans and other well-studied species, single-sequence gLMs are still far from replacing alignment-based models or models supervised with functional-genomics data. Our near-term goal is to build useful models for species that lack high-quality whole-genome alignments and functional-genomics data. As a concrete example, we aim to enable accurate, cost-effective genome-wide mapping of evolutionary constraint for any mammalian species.³ Longer term, sequence-only models may learn from sequence in ways that complement alignments, conservation scores, and functional-genomics models even in data-rich species (research question #396).

Why data curation?

Many early gLMs were trained on the human genome alone, with little to no filtering. Subsequent work made it clear that two factors were key drivers of model performance: including multiple species, and enriching for functional regions rather than sampling uniformly from the majority-neutral background of mammalian genomes.⁴

In this work, we follow up on two findings from TraitGym. First, 152M-parameter GPN-Promoter, trained only on animal promoters, performed comparably to Evo 2 40B on human promoter variants. Second, Evo 2 improved substantially with scale overall but still struggled on distal variants. Enhancers were the only region type not actively curated into its training data and were sparse among the intergenic regions it saw.

MarinDNA therefore treats dataset construction as a primary modeling lever: which species and evolutionary timescales to include (research question #394), which functional regions to sample and how to identify them (research question #395), how to weigh them, and when during training to introduce them.

We began with annotation-derived datasets for coding, upstream, and downstream sequences.⁵ Standard genome annotations make these regions relatively easy to identify and extract consistently across many species.

Later, we added ncRNA⁶ and enhancers⁷ built by alignment projection. Because comparable annotations were not directly available across the target species, we projected human annotations through whole-genome alignments. The broader idea of using whole-genome alignments to expand reference-species enhancer data across species has also appeared recently in chromatin-accessibility prediction and enhancer-generation work.⁸

Why GPT-style architecture?

By GPT-style, we mean the deliberately simple approach of treating DNA as text and training a stock causal, autoregressive, decoder-only language model on it. In these experiments, that architecture is literally Qwen3 rather than a genomics-specific design. This approach is not new; a substantial line of prior gLM work has used causal language modeling with GPT- or Llama-like architectures.⁹ What is new here is the quality target. Even recent models in this family, such as Carbon, generally aim for non-inferiority to smaller Evo 2 checkpoints and still underperform Evo 2 40B on the broad zero-shot VEP setting we care about.¹⁰ If the quality gap can be closed, GPT-style models have obvious advantages for deployment. They run through familiar training and inference stacks, move cleanly across hardware, and avoid model-specific kernels or bespoke architecture code, which matters a lot for cost, flexibility, and usability.

MarinDNA's first experiment compared three training objectives on promoter sequences: causal language modeling, masked language modeling, and masked diffusion (experiment #3). Causal language modeling looked most promising in the initial training steps. This was not a definitive matched-compute comparison of objectives, but it provided enough direction-setting evidence to pursue a simple causal architecture. That choice also need not permanently constrain the model to left-to-right representations: decoder-only language models can be adapted into bidirectional encoders with further training (research question #393).

Our initial goal was to model individual functional elements of the genome—such as an exon or an enhancer—well. We therefore deliberately used a short 255-bp context: it was sufficient for the functional-element tasks we tested and made training and evaluation much faster. Centering each example on an individual functional element also made the training data easier to construct, filter, audit, and interpret. How best to extend context—either through additional long-context next-token pretraining or during downstream-task fine-tuning—remains future work (research question #392).

Why VEP evaluation?

Variant effect prediction (VEP) is one of the most important applications of gLMs. A useful VEP model can help scale clinical interpretation for rare disease, hereditary cancer, and variants of uncertain significance.¹¹ It can also help connect genetic association signals to disease mechanisms, target selection, and causal-variant prioritization in GWAS fine-mapping.¹² The same kind of evidence is relevant to clinical trial design when genetics can inform patient stratification, enrollment criteria, or mechanism-based cohort definition. Together, these are commonly used levers for improving the efficiency of pharmaceutical development, and it is uncommon for other gLM evaluations to have such a direct connection to commercially relevant research tasks. VEP is also one of the few evaluations backed by decades of costly clinical genetics curation, with resources such as ClinVar and OMIM providing a level of human variant evidence that has no real analogue in other species.¹³ That combination makes it a substantive test of whether a gLM has learned sequence constraints that actually matter for human biology. If a model has learned useful sequence-level constraints from DNA alone, it should help rank variants in places where direct experimental evidence is weak or nonexistent.

This creates a deliberate mismatch between evaluation and intended use: although we expect single-sequence gLMs to be especially valuable for non-model organisms in the near term, we evaluate them in humans because comparable variant-effect data are not yet available across species. Human VEP is therefore the most rigorous available test of learned functional constraint, but it does not by itself establish transfer to other organisms; evaluating that transfer will require broader population-genetic or experimental datasets.

In this work, we focus on predicting deleteriousness, pathogenicity, or, more generally, functional constraint. This task is the one most directly connected to the language-modeling training objective and is therefore easy to evaluate with zero-shot or linear-probing protocols. We leave the prediction of changes in gene expression—the main application of sequence-to-function models—to follow-up work, as it requires more complex fine-tuning protocols and much larger context sizes.¹⁴

We use two complementary sources of evidence: clinically curated Mendelian variants¹⁵ and saturation genome-editing (SGE) measurements. The Mendelian benchmark compares pathogenic and putatively benign variants across broad coding and non-coding consequence types. The SGE benchmark uses experimentally measured variant effects from a few genes in MaveDB, currently covering missense and splicing variants.

We evaluate each frozen gLM with two readouts: a zero-shot sequence log-likelihood ratio and a linear probe trained on paired reference/alternate embeddings. The zero-shot score tests whether the model's learned sequence likelihood reflects functional constraint: deleterious alternate alleles should incur larger likelihood penalties relative to the reference allele. The probe instead asks what variant-relevant information is encoded in the model's learned representation, including information that may not be directly reflected in its sequence likelihoods.

Why compare with Evo 2 40B?

Evo 2 40B remains the strongest broadly evaluated baseline among single-sequence DNA models. Despite substantial subsequent work on gLMs, we are not aware of a model in this setting that has clearly surpassed it across diverse genomic regions; many newer models do not attempt a direct comparison with Evo 2 40B. The other reason Evo 2 40B matters is its training budget. Its reported 2.25e24 training FLOPs are unrivaled among gLMs, corresponding to roughly $2.5M of H100 time.¹⁶ That budget is unusual in biology and comparable to major open-weight LLM training runs from recent model generations,¹⁷ e.g. just above Qwen2.5-14B and below Qwen2.5-32B, and roughly between DeepSeek-V2 and DeepSeek-V3. Evo 2 40B is therefore the right baseline for asking whether a much simpler single-sequence gLM can be competitive.

Results

Early mixture experiments

We first trained a 1.7B upstream-region specialist, trying to replicate the success of GPN-Promoter (experiment #21).¹⁸ Although we used reasonable defaults rather than the systematic hyperparameter-transfer recipe developed later, performance was broadly comparable to Evo 2 40B. We saw a similar pattern when training a CDS specialist (experiment #27). Overall, however, GPN-Star remained stronger.

After testing upstream and CDS specialists independently, the next experiment asked whether one model could retain both capabilities (experiment #13). Sampling in proportion to dataset size—10% upstream and 90% CDS—is the naive default when the two datasets are simply pooled without reweighting. In practice, it behaved similarly to CDS-only training. Equal 50/50 upstream/CDS sampling produced balanced performance across both regions. This made explicit mixture control a central axis of investigation. Even the 50/50 mixture may not be optimal: regions can differ both in size and in the density of learnable biological signal.

Hyperparameter transfer

Our first attempt to scale manually lowered the learning rate as the model grew from 0.6B to 1.7B and 4B parameters (experiment #57). The larger models did not improve over the 0.6B model, and the early 4B run became unstable. That failure made systematic hyperparameter transfer a prerequisite: without a trustworthy optimization recipe, a model-size comparison would confound scale with tuning quality.

The annotation-derived DNA pool available at the time contained ~85B tokens. Fig. 7 shows its proportional CDS, upstream, and downstream composition.

That pool is large by genomics standards but small relative to modern accelerator-era training corpora. This makes the project data-constrained in principle even though our practical constraints are messier. We train on preemptible Google TPU Research Cloud resources, do not have consistent access to slices much larger than roughly 32 H100s worth of peak FLOPs, and want the recipe to remain reproducible at academic compute scale. O(100B) tokens therefore lands in an awkward middle ground where compute-constrained methods are still relevant, even though modest epoching is possible and likely breaks their assumptions at some unknown rate.

We started with hyperparameter transfer for that reason. If a proven data-constrained transfer framework existed, we would use it. We do not know of one, so we followed the same basic pattern as Delphi, fitting a small reference sweep with the Vizier Bayesian optimization framework and then scaling the result using a Complete(d)-inspired AdamH heuristic.¹⁹

Fig. 8 separates reference calibration from target application. Two heuristics sit behind that workflow and are fixed before tuning: an inherited rule maps hidden width D to model geometry,²⁰ and a second rule maps reference optimizer settings to a new batch size B and token horizon T. The reference sweep tunes initialization scale, the two learning rates, β₁, β₂, ε, gradient clipping, and z-loss. Here, tuning a learning rate means tuning its peak value under a fixed fractional schedule; the target run reuses that schedule shape, so warmup and decay scale with the run length rather than keeping fixed absolute step counts.²¹ At the target, the two learning rates, β₂, and ε are transformed with B and T, while initialization scale, β₁, gradient clipping, and z-loss are reused unchanged.²²

The reference sweep used ~25M-parameter models trained for 2.5B tokens with a 16k-token batch, or roughly 4e17 FLOPs per run. We then validated the transferred hyperparameters across 255M–1B-parameter models, with 4x as many tokens, 1/4x the batch size, and roughly 170x the FLOPs per run. The first test was whether the learning-rate prediction survived that regime. Fig. 9 shows that the transferred prediction lands exactly on the best observed learning-rate setting at all three validation scales, outperforming both the unchanged reference optimum and every other target-scale sweep setting; the less sensitive optimizer hyperparameters are shown separately in Fig. 10.

That validation is a fairly unforgiving test. If the transferred learning rate were merely close by accident, it would be surprising for it to land correctly across all three validation scales, but the prediction remains well centered at each one. For DNA, that is a pretty cool result. Prior biology foundation-model work has used μP-style transfer, but we are not aware of a DNA result showing that a more inclusive framework like Complete(d) works across token horizon and batch size, which are the axes we keep leaning on later in ad hoc runs across epochs. The same is mostly true for the other optimizer hyperparameters too, although Adam β₂ shows some signs of being a bit aggressive at the largest scale. Fig. 11 makes the same point across CDS, upstream, and downstream sequence, with no qualitative difference in transfer behavior across region types. That gives us enough confidence that the following parameter-scaling runs are at least close to optimally configured.

Fig. 11: Transfer validation by region

Parameter scaling

Before asking whether better validation loss²³ translates into better VEP performance, we first needed to check whether validation loss scaled the way it should. The parameter sweep uses the same training recipe at each model size, with all hyperparameters set by the transfer heuristic above, and then asks whether the resulting losses fit a Kaplan-style scaling law well (they do).²⁴ Despite this being a simple experiment conceptually, actually getting there took months — fitting the hyperparameter transfer heuristic, running the validation experiments, and training the 4B model, which alone took about three weeks to finish. The final sweep spans 8 model sizes from 46M to 4B parameters, each trained on ~84B tokens, for ~4.3e21 FLOPs across the sweep. That puts it on par with canonical scaling-law studies in language modeling, e.g. its ~2.1e21 FLOP 4B run matches the compute Hugging Face used at that exact model scale in their data-constrained scaling work.²⁵

The result is about as tidy as we could hope for. Training is stable at every scale, and both training and validation loss decrease monotonically and predictably, as shown in Fig. 12. We use WSD learning-rate schedules with 10% warmup and 20% decay, which causes the visible drop in both losses over the final 20% of tokens. Most importantly, the sweep gives a high-quality Kaplan scaling-law fit (R²=0.999), which makes the next question much better posed. Does lower validation loss actually correlate with better downstream VEP performance?

Downstream performance

The final sweep shows a mostly consistent relationship between parameter count and downstream VEP performance. When zero-shot LLR and linear probes are evaluated on identical variants, performance improves with scale for most variant types. The clearest exception is Mendelian missense: zero-shot LLR peaks at 128M parameters and then deteriorates, even as linear-probe performance continues to improve.²⁶ This is not a general failure on missense variants—the SGE missense benchmark improves with scale under both scoring protocols.

Plotting the same results against matched-region validation log-likelihood gives the same picture. For all other combinations of variant type and scoring protocol, better validation log-likelihood is associated with better downstream performance. Zero-shot Mendelian missense again points in the opposite direction: performance declines even as validation log-likelihood improves.

This divergence is not unique to MarinDNA. On the same Mendelian missense benchmark, Evo 2 also shows improving linear-probe performance alongside declining zero-shot LLR performance as model size increases. For now, this should be interpreted as a recurring pattern for Mendelian missense across these two model families—not as evidence that zero-shot readouts generally deteriorate with scale. It also cautions against using zero-shot LLR alone to judge whether scaling has improved the learned representations for this task.

Later mixture experiments

At this point we move away from theoretically grounded, compute-constrained methods. The later experiments still rely on the transfer heuristics above, since we need learning rates and other hyperparameters for runs with very different token horizons. But the actual optimization problem becomes much more ad hoc — we start changing mixture constituents, epoch them freely, and see whether in-flight changes can compensate for observed performance gaps.

We standardized these experiments on a 1B-parameter model.²⁷

Our starting point was a uniform mixture of CDS, upstream, and downstream sequence. These were the three region datasets we could initially construct consistently across many species from standard genome annotations. The scaling study had instead sampled these regions in proportion to dataset size. Echoing the earlier mixture experiments, that recipe produced good CDS performance but left a large gap on the less abundant upstream tasks. We therefore switched to uniform weighting so that each functional region received meaningful exposure.

We use the internal labels m5.1, m1.3, and m3.3 for the three model-mixture lineages compared below. In this naming scheme, m denotes a mixture lineage, the leading number identifies the mixture strategy, and the suffix identifies a continuation within that lineage. m5.1 is the staged lineage: we first trained its 1B model for ~104B tokens on the uniform three-region mixture. Alignment projection then made it possible to turn human ncRNA and enhancer annotations into comparable multi-species training datasets. Once those data became available, we added them to form a uniform five-region mixture and continued training the same model for another ~62B tokens. We compare this staged history with two controls, m1.3 and m3.3, trained on five-region mixtures from the beginning.

For m5.1, the first evaluated checkpoint after adding ncRNA and enhancer data shows gains in variant-effect performance in the corresponding ncRNA and distal subsets. Across all eight subsets, m5.1 ultimately finishes with the highest macro-average AUPRC under both zero-shot LLR and the linear probe, although the linear-probe trajectories are visibly noisier.²⁸ Its advantage is broad but not universal: the lineages trained on five regions from the beginning retain stronger endpoints for some distal and ncRNA subsets. m5.1's strong endpoint raises the possibility that exposure order matters: learning first from the three-region mixture and introducing ncRNA and enhancer data later may be more effective than training on all five regions from the beginning, though this remains uncertain and requires further investigation.

Show the linear-probe view

Leaderboard scores

The result of the previous mixture experiments is m5.1, a 1B GPT-style model evaluated alongside other models on our live Mendelian VEP leaderboard, where we continue to add experimental runs and baselines. In the zero-shot snapshot shown here, m5.1 comes out slightly ahead of Evo 2 40B. Its advantage is considerably larger under linear probing.

m5.1 was trained on 166B tokens (~1.1e21 FLOPs), compared with 9.3T tokens (~2.25e24 FLOPs) for Evo 2 40B. At their native context lengths on the same GH200, m5.1 scores one million variants in about 41 minutes, compared with roughly 66 days for Evo 2 40B—a roughly 2,330× throughput advantage.²⁹

Most notably, m5.1 closes Evo 2's main gap on distal variants, outperforming Evo 2 40B there under both readouts. The improvement is not uniform, however: Evo 2 40B remains ahead on splicing under both readouts, as well as on promoter and synonymous variants in the zero-shot evaluation and missense variants under linear probing.

The current leaderboard also suggests there is considerable headroom: although m5.1 leads on Macro Avg among MarinDNA models, another MarinDNA run has a higher point estimate in seven of the eight displayed subsets, with m5.1 leading only on ncRNA. Several of these winners are region specialists, suggesting that further mixture refinement could recover more of their complementary strengths.

On the broader zero-shot leaderboard, m5.1 remains slightly behind AlphaGenome and substantially behind GPN-Star.³⁰ These are different model families: AlphaGenome learns from functional-genomics supervision, while GPN-Star uses whole-genome alignments. Further improvements to the alignment-free recipe may narrow the gap to GPN-Star, but it is not clear that they will eliminate it. Some of the remaining difference may reflect information that an alignment-free model cannot recover from unaligned sequence alone (research question #397).

Conclusion

These experiments show how far a simple and standardized genomic modeling recipe can go. By keeping the architecture within the GPT family and iterating on data curation, training hyperparameters, model scale, and data mixtures, MarinDNA produced an alignment-free 1B model that is competitive with Evo 2 40B on Mendelian variant effect prediction while requiring far less training compute and offering much faster inference. In terms of capabilities, we next want to improve our performance on complex-trait variant effect prediction (research question #391).

On the training side, neither scaling nor optimization appears exhausted. Performance continued to improve through the largest model we tested, with no clear sign of saturation, making further scaling a natural next step. Regularization also remains a largely unexplored part of the training recipe and may be especially valuable in the awkward regime between data- and compute-constrained training. The narrowed recipe and better infrastructure for using the Google TPU Research Cloud compute donated for this work should make both larger-scale runs and more systematic optimization easier to pursue.

There is plenty of work left to do, but we think these results clearly show the potential value of a general-purpose training platform like Marin for accelerating scientific foundation model development.

Resources

• 🧪 MarinDNA repository (source code and experiments tracked through GitHub issues)
• 🤗 Hugging Face collection (training datasets, benchmarks, and model)
• 🏆 MarinDNA leaderboard
• 🧬 Interactive sequence explorer
• 💻 Model inference and BRCA1 variant effect prediction notebook

Acknowledgements

We thank Isaac Hodes, Yael Elmatad, David Hall, Will Held, and Jeff Hammerbacher, as well as others across the Open Athena and Marin communities, for thoughtful discussions. We also thank the Google TPU Research Cloud (TRC) program for providing compute resources.

@misc{benegas2026_marin_dna,
  author = {Benegas, Gonzalo and Czech, Eric},
  title = {A 1B standard Transformer rivals Evo 2 40B on variant effect prediction},
  year = {2026},
  month = {aug},
  howpublished = {\url{https://www.openathena.ai/blog/marin-dna/}},
  note = {Open Athena Blog}
}