New Science Blog: Why has AI advanced faster in coding than in biology?
To agents, bio databases are like cities built before cars—maddening to drive in because they're designed for different traffic.
How do we build infrastructure agents can use?
anthropic.com/research/agent…
nanochat can now train GPT-2 grade LLM for <<$100 (~$73, 3 hours on a single 8XH100 node).
GPT-2 is just my favorite LLM because it's the first time the LLM stack comes together in a recognizably modern form. So it has become a bit of a weird & lasting obsession of mine to train a model to GPT-2 capability but for much cheaper, with the benefit of ~7 years of progress. In particular, I suspected it should be possible today to train one for <<$100.
Originally in 2019, GPT-2 was trained by OpenAI on 32 TPU v3 chips for 168 hours (7 days), with $8/hour/TPUv3 back then, for a total cost of approx. $43K. It achieves 0.256525 CORE score, which is an ensemble metric introduced in the DCLM paper over 22 evaluations like ARC/MMLU/etc.
As of the last few improvements merged into nanochat (many of them originating in modded-nanogpt repo), I can now reach a higher CORE score in 3.04 hours (~$73) on a single 8XH100 node. This is a 600X cost reduction over 7 years, i.e. the cost to train GPT-2 is falling approximately 2.5X every year. I think this is likely an underestimate because I am still finding more improvements relatively regularly and I have a backlog of more ideas to try.
A longer post with a lot of the detail of the optimizations involved and pointers on how to reproduce are here:
github.com/karpathy/nanoc…
Inspired by modded-nanogpt, I also created a leaderboard for "time to GPT-2", where this first "Jan29" model is entry #1 at 3.04 hours. It will be fun to iterate on this further and I welcome help! My hope is that nanochat can grow to become a very nice/clean and tuned experimental LLM harness for prototyping ideas, for having fun, and ofc for learning.
The biggest improvements of things that worked out of the box and simply produced gains right away were 1) Flash Attention 3 kernels (faster, and allows window_size kwarg to get alternating attention patterns), Muon optimizer (I tried for ~1 day to delete it and only use AdamW and I couldn't), residual pathways and skip connections gated by learnable scalars, and value embeddings. There were many other smaller things that stack up.
Image: semi-related eye candy of deriving the scaling laws for the current nanochat model miniseries, pretty and satisfying!
35K Followers 401 FollowingMy random thoughts on EVs, clean energy, chips, aerospace and other tech. Find more extended pieces at substack https://t.co/Jmo8iyjHrn
32K Followers 300 FollowingGlobal Lithium™ provides advisory services to the lithium-ion battery supply chain with clients on 5 continents. Host of The Global Lithium Podcast and author.
193K Followers 370 FollowingSpeculation is the search for truth in price and time. Not investment advice - just personal views. Blog at https://t.co/DD2iUKbdLz
156K Followers 2K FollowingChairman of Chan Soon-Shiong Family Foundation, Exec Chairman ImmunityBio, Chairman and Chief Executive Officer of Los Angeles Times Media Group (LATMG)
10K Followers 29 FollowingWe empower visionary, high-leverage science and technology projects with the capacity to create transformative progress for human civilization.
1K Followers 12 FollowingTo understand human aging and craft a means of intervention in it is the greatest calling of mankind. How we meet that challenge will define us as a species.
9K Followers 1K FollowingCommitted to striving towards the vision of being the most reliable biotech company in the world to make humans and nature healthier through biotechnology.
1.4M Followers 2 FollowingWe're an AI safety and research company that builds reliable, interpretable, and steerable AI systems. Talk to our AI assistant @claudeai on https://t.co/FhDI3KQh0n.
2K Followers 3K FollowingCEO/founder of Virgo. Building frontier AI to solve gut-mediated diseases like colorectal cancer, pancreatic cancer, IBD, and more
135 Followers 314 FollowingWe provide medical & scientific updates from Legend Biotech. Intended for US HCPs only. See pinned post for inquiries and community guidelines.
1.2M Followers 787 FollowingProfessor at NYU & Executive Chairman at AMI Labs.
Ex-Chief AI Scientist at Meta.
Researcher in AI, Machine Learning, Robotics, etc.
ACM Turing Award Laureate.