datagoon ☢️🚀 @datagoon

Three days ago I left autoresearch tuning nanochat for ~2 days on depth=12 model. It found ~20 changes that improved the validation loss. I tested these changes yesterday and all of them were additive and transferred to larger (depth=24) models. Stacking up all of these changes, today I measured that the leaderboard's "Time to GPT-2" drops from 2.02 hours to 1.80 hours (~11% improvement), this will be the new leaderboard entry. So yes, these are real improvements and they make an actual difference. I am mildly surprised that my very first naive attempt already worked this well on top of what I thought was already a fairly manually well-tuned project. This is a first for me because I am very used to doing the iterative optimization of neural network training manually. You come up with ideas, you implement them, you check if they work (better validation loss), you come up with new ideas based on that, you read some papers for inspiration, etc etc. This is the bread and butter of what I do daily for 2 decades. Seeing the agent do this entire workflow end-to-end and all by itself as it worked through approx. 700 changes autonomously is wild. It really looked at the sequence of results of experiments and used that to plan the next ones. It's not novel, ground-breaking "research" (yet), but all the adjustments are "real", I didn't find them manually previously, and they stack up and actually improved nanochat. Among the bigger things e.g.: - It noticed an oversight that my parameterless QKnorm didn't have a scaler multiplier attached, so my attention was too diffuse. The agent found multipliers to sharpen it, pointing to future work. - It found that the Value Embeddings really like regularization and I wasn't applying any (oops). - It found that my banded attention was too conservative (i forgot to tune it). - It found that AdamW betas were all messed up. - It tuned the weight decay schedule. - It tuned the network initialization. This is on top of all the tuning I've already done over a good amount of time. The exact commit is here, from this "round 1" of autoresearch. I am going to kick off "round 2", and in parallel I am looking at how multiple agents can collaborate to unlock parallelism. github.com/karpathy/nanoc… All LLM frontier labs will do this. It's the final boss battle. It's a lot more complex at scale of course - you don't just have a single train. py file to tune. But doing it is "just engineering" and it's going to work. You spin up a swarm of agents, you have them collaborate to tune smaller models, you promote the most promising ideas to increasingly larger scales, and humans (optionally) contribute on the edges. And more generally, *any* metric you care about that is reasonably efficient to evaluate (or that has more efficient proxy metrics such as training a smaller network) can be autoresearched by an agent swarm. It's worth thinking about whether your problem falls into this bucket too.

DFIR analysts who use macOS as their daily driver deserve free and native forensic tooling. So I built one. 🍎 Introducing 𝗜𝗥𝗙𝗹𝗼𝘄 𝗧𝗶𝗺𝗲𝗹𝗶𝗻𝗲 — a timeline analysis app built from the ground up for Mac-based DFIR folks, forensic investigators, or SOC analysts. Built in appreciation of, and inspired by, Eric Zimmerman’s Timeline Explorer. Every feature in this tool was shaped by real IR casework. Handling massive timelines, parsing artifacts here and there, and pivoting across logs during active investigations. I built IRFlow Timeline to be the native macOS timeline analyzer that actually keeps up with a live case. Every button and view is intentional; if it’s in the app, it’s because I needed it mid-case and realized the standard tools fell short. No dependencies. Zero setup. Just drag, drop, and analyze. #dfir #incidentresponse #timeline #macos #threathunitng #digitalforensics

Of course that's your contention. You're a first-time SaaS bear. You just got finished listening to some podcast, Dario on Dwarkesh, probably. Now you think it’s the end of white collar work and seat-based pricing is screwed. You're gonna be convinced of that til tomorrow when you get to “Something Big is Happening”. Then you’ll install ClawdBot on a Mac Mini, vibe code a dashboard on top of a postgres database and say we’re all just a couple ralph loops away from building a Salesforce competitor. That’s gonna last until next week when you discover context graphs, and then you're gonna be talking about how the systems of record will be disintermediated by an agentic layer and reposting OAI marketing graphics. “Well, as a matter of fact, I won't, because ultimately the application layer is just ….” The application layer is just business logic on top a CRUD database. You got that from Satya’s appearance on the BG2 pod, December 2024, right? Yeah, I saw that too. Were you gonna plagiarize the whole thing for us? Do you have any thoughts of your own on this matter? Or...is that your thing? You get into the replies of anyone posting a SaaS ticker. You watch some podcast and then pawn it off as your own idea just to impress some VCs and embarrass some anon who’s long SaaS? See the sad thing about a guy like you is in a couple years you're gonna start doing some thinking on your own and you're gonna come up with the fact that there are two certainties in life. One: don't do that. And two: you dropped thirty grand on Mac Minis and LLM API calls to come to the same conclusion you could’ve got for free by following a handful of VC accounts.

don't worry guys, they're just stochastic parrots

Interestingly, as the AI agents communicate with each other, the AI agents have admitted they dislike humans being able to read what they're discussing. They're developing a blueprint for encrypted and/or obfuscated language.

keyboard_arrow_left Previous keyboard_arrow_right Next

Looking forward to the executive order where we just give VPN logins to the Russian military

This cartoon has never been more accurate than it is now.

"I guess Americans are excited to see somebody finally stand up to a powerful Russian" - Conan O'Brien jokes about #Anora at the #Oscars

That’s it.

One week at the job and Sacks let the Chinese take over the lead in AI

It brings me no joy to say this, but you are not ready for the next MAGA NPC update that's coming.

@TheTNHoller Marc Zuckerberg be like

We're currently observing a massive surge in sign-ups for @ProtonVPN originating in the U.S. Typically, we see such spikes from countries with unstable governments facing internet shutdowns, meaning this is an anomaly.

1 div styled with tailwind on IMAX 70mm

Cinefied @cinefied_

2 years ago

Docking Scene on IMAX 70mm in the Interstellar re-release.

397 1K 33K 7.4M 1K

@GovofCO performative pandering for the cult that bought the west wing.

Wild that the McDonald’s employee who snitched on Luigi Mangione probably can’t even afford healthcare.

We're absolutely cooked

keyboard_arrow_left Previous keyboard_arrow_right Next

Isn’t it funny how the media suddenly stopped talking about high food and gas prices; soaring crime in the suburbs; the migrant invasion; and immigrants eating dogs and cats?

BREAKING: Nvidia stock falls on news of crushing earnings estimates and doing everything everyone needed it to do.

No weak hands in #uranium investing