A 1B standard Transformer rivals Evo 2 40B on variant effect prediction
How MarinDNA combined data curation, hyperparameter transfer, scaling, and data-mixture experiments to build an efficient 1B alignment-free genomic language model with balanced performance across genomic regions.
Scaling Samudra: Our journey towards better ocean emulators
We scaled Samudra, our neural ocean emulator, to train on 16x more data on the same hardware — cutting training time from four days to four hours. We found out systems performance tuning is a humbling process.
I argued with the father of open source for 2 years. Now the AI fight is the same — only bigger.
David Siegel, founder and chairman of Open Athena, writes in Fortune about the importance of open source in AI.
Research Software Engineering in the Age of Agentic Tooling
With the rise of AI coding agents, the role of research software engineers is changing. Instead of building libraries and using them to write bespoke code, RSEs must shift their focus both upstream and downstream of the software creation process.
Meet our team: Betsy Cannon
In this Q&A, Betsy Cannon explains what drew her to this work, her cookie-related introduction to LLMs, and what a project on electron density can learn from one about ocean modeling.
Open Development of Frontier AI
Open development is key to making frontier-level artificial intelligence legible, reusable, and public.
Cluster Scheduling with Iris
Training a frontier-level LLM requires significant resources across a variety of providers and accelerator types. Our Iris scheduling system has allowed us to effectively make use of these precious resources using a custom global scheduler; in the months since rollout, our sustained concurrent TPU usage has roughly doubled.
Preparing for the AI Future with Ethics in Mind
At a recent panel discussion, our COO and CSO Jared Crooks explained why openness is key to understanding AI, and dug into the importance of embedding ethics in this new technology.
Improving our LLM Pretraining Efficiency
How Marin pretraining became more efficient through Mixture of Experts, higher expert sparsity, MuonH, PKO, and routed expert normalization.
Scaling Laws That Extrapolate 300× Past the Fit
Delphi is an open scaling suite ranging from 3e18 to 1e23 FLOPs. A pre-registered forecast from its scaling law predicted the loss of the largest run within 0.2%, extrapolating 300× past the largest run used in the fit.
Mixture of Experts Quantile Balancing: Validated at 32B-A5B (1e22 FLOPs) Scale
Quantile Balancing (QB) is a hyperparameter-free load balancer for Mixture of Experts models, introduced by Jianlin Su. We validated it on a 32B-A5B (1e22 FLOPs) Marin run over 326B tokens: zero hyperparameters, zero loss spikes, and no need for leading dense layers, auxiliary losses, or capacity overload factors.
Problems with Chinchilla Approach 2
The Chinchilla paper's "Approach 2," fitting parabolas to IsoFLOP curves, turns out to have some subtle biases that can add up. We show these can lead to non-trivial errors in compute-optimal allocation (around 6.5% of total compute for Llama 3, worth over $1M in GPU time), especially when IsoFLOP grids aren't perfectly centered or symmetric. The paper also proposes a reparameterization of "Approach 3" that makes direct parametric fitting simple and stable. You can even run it in 70 lines of JavaScript.
