Malcolm Murray @malcmur

How are experts and superforecasters updating their expectations of AI progress? In our latest round of Longitudinal Expert AI Panel (LEAP) forecasts, we asked for predictions on AI progress timelines, including updated forecasts for a question we last asked nine months ago. We also asked for forecasts on near-term progress on @METR_Evals' time horizon benchmark, AGI timelines, the blockers and enablers of AGI, and AI's overall impact in the next 20 years. 🧵 Read on for more:

This is so interesting to me, using AI tools *activates* the part of my brain that wants to know more about the world. I am way more curious and full of ideas now than I’ve been in a long time.

Paul Muad’Dib @trygraptor

7 days ago

“it just shuts off the part of my brain that wants to know more about this world” amazing quote, amazing way to express this sentiment quickly becoming a Kane Parsons fan

9 809 7K 102K 561

An interesting development.

Michał Podlewski @trajektoriePL

7 days ago

Now we know why Peter Thiel packed his bags for Argentina. Milei just submitted his AI legislative framework to Congress, where he proposes: - zero regulation on AI development, - a brand-new "non-human corporation" category for AI/robot-operated entities with limited liability

415 5K 16K 904K 5K

In the past 3 days: 1. White House issues Executive Order with a "voluntary framework" under which new frontier AI models with significant cybersecurity capabilities will be provided to the USG for up to 30 days before they can be released to "trusted partners". 2. USG holds early talks with OpenAI and others regarding taking equity stakes in these companies. 3. Sarah Friar says that OpenAI model training mostly takes in place in the U.S. "for USG reasons for making sure that a nationl asset and effect is happening on U.S. soil". We're not in Kansas anymore.

Whose preferences will determine the shape of the far future? Alex Imas and Phil Trammell debate: - Alex speculates that humans who have an intrinsic preference for other humans, and have a moral objection to AI services, will be better at finding other humans to reproduce with. In the long run evolution will favour them. - Phil argues that the transhumanist types who have no issues with AI and want to explore the galaxy will have a higher savings rate, and in the long run will control almost all of the resources in the economy. A super important question for thinking about the long run future of civilization.

What happens when you post a real Monet and say it’s AI? The coolest art social experiment I’ve seen in a while. Thank you @SHL0MS

If leading AI companies are indeed approaching the point of recursive self-improvement, a coordinated, verifiable, and universally applied pause is probably the only responsible solution to mitigate several major AI risks; at least until safety guarantees are developed and demonstrated. Ensuring that such a moratorium is respected would require sincere collaboration between various countries and companies, but I definitely believe it is achievable if others follow in @AnthropicAI's footsteps.

The Wall Street Journal @WSJ

7 days ago

Anthropic is calling for top AI labs to weigh slowing the pace of development, suggesting that AI systems are advancing so rapidly that they may soon be able to improve themselves without human intervention in ways that could pose societal risks. on.wsj.com/4ulkmFh

205 376 1K 665K 400

Understanding all the causes of increased disclosures is complicated. But we observe a sharp uptick in High and Critical CVEs around the time of Anthropic’s release of Mythos Preview to Project Glasswing partners in late March. OpenAI’s Daybreak cybersecurity program also launched in May.

"The #AI-driven worm requires only an open-weight model that can run on a single, local GPU... [using] the victims’ own computational resources.... As consumer devices increasingly support #LLM inference, the reasoning resources available to such adversaries grow accordingly."

Internet Ethics @IEthics

6 days ago

On "a fundamentally new #cybersecurity threat: an adaptive computer worm that devises target-specific attack strategies to gain control of machines and spread across networks": cleverhans.io/latest-researc… #ethics #internet #tech #AI #LLMs #safety #research h/t @schneierblog

0 1 0 186 0

Static benchmarks inevitably saturate as models improve. Our recent work explores how to evolve existing tasks into harder, validated variants that co-evolve with frontier models — and also serve as RL training data for self-improvement. A step toward benchmarks that evolve as quickly as the models they measure.

Yangzhen Wu @yangzhen04

a week ago

Static benchmarks are dying — they tend to get saturated quickly. Evaluation and training data should co-evolve with frontier models. We released BenchEvolver — a framework that automatically evolves saturated problems into harder, verified tasks for evaluating frontier models,

5 20 95 39K 63

The Obernolte-Trahan draft is the best federal AI proposal released yet. It contains provisions for AI lab transparency, lab employee whistleblower protections, a framework for independent verification orgs, funding for CAISI and public compute (NAIRR), and much more. The bill largely focuses on the development of AI systems, so it preempts state governments from passing conflicting regulations in that domain while preserving their ability to regulate deployments. This is similar to how vehicle safety standards come from the federal government, but states regulate driver licensing, vehicle registration, auto sales (via regulation of car dealerships), road safety (via traffic law), and adjudication of post-incident harms (via traffic court, tort liability for drivers, etc.). You will notice that the role of states in the car example is not small! Yet vehicle safety standards, because they affect a thing that is mass manufactured at great upfront capital expense, need to be federal. The deployment/development division of labor is the most straightforward way to do AI preemption out there, and the time for federal preemption has long since come. This is the best AI law draft America has yet produced, and I applaud Representatives Obernolte and Trahan for their tremendous effort and leadership. I wish everyone in Congress held themselves to their standard.

Diego Areas Munhoz @Dareasmunhoz

a week ago

🚨NEWS: The OBERNOLTE and TRAHAN framework would preempt state AI DEVELOPMENT laws for 3 years. It creates ind AUDITING, TRANSPARENCY, SAFETY INCIDENT reporting requirements for frontier models. Codifies CAISI and more Punchbowl subscribers have access to the TEXT! w/@BenBrodyDC

1 22 41 66K 17

I'm very glad Anthropic's latest blog included this, and I'll be very pleased to help if I can: "The Anthropic Institute will conduct research—in collaboration with many others—and take actions to help build the systems that a credible slowdown or pause would require." There are very promising directions in technical monitoring and verification that could help support coordinated efforts around safety and governance, and provide the toolkit for a coordinated slowdown/pause if deemed necessary. But work to do to get there. There's also appetite amongst intellectual leaders in China, the US, and internationally for such mechanisms; they were amongst the recommendations of our World Internet Conference report last year. Hoping to have more work relevant to this out later this year. wicinternet.org/pdf/Advancinga…

Our highest and most urgent national priority should be AI safeguards. The risks of AI weapons, pathogens, mass unemployment, surveillance, and even extinction must not continue to be largely ignored.

WSJ Tech @WSJTech

7 days ago

Anthropic Urges Global Pause in AI Development, Flags ‘Self-Improvement’ Risk on.wsj.com/4o5IBpe

18 65 211 434K 60

x.com/i/article/2062…

Big development - Anthropic is now advocating to build verification mechanisms to enable the option to pause AI development.

Sad to see Ted Chiang resorting to such bad arguments in this piece. He confidently claims Claude has no inner experience. But he has to use a lot of dodgy philosophy and poor reasoning to get there: 1. We can't take deflationary mechanistic descriptions of how AI calculations are performed to show that AI isn't conscious. Otherwise we could argue that 'humans are just neurones transmitting signals one after another' and thereby conclude humans can't be conscious. But that would be wrong for us. And the same logic could be wrong for LLMs. 2. That LLMs are asked to play characters, and effectively are always playing characters, doesn't mean they aren't conscious. It's true a human playing the role of Caesar doesn't have Caesar's experience of things. But they still experience something (that of being a person pretending to be Caesar). The same could be true of Claude. (Arguably it's also true that humans are always playing characters to some extent and don't have a completely fixed nature, but that has no bearing on our own subjective experience.) 3. Chiang says "an LLM is a machine that generates only one word at a time". This conflates two things: they output one word at a time, and they only think about one word at a time (without planning ahead or looking back). The first is true of AI but equally true of humans. While the latter we know is a false description of how AIs think – we can see from how AIs compose poetry that they plan out rhymes a at least one line ahead. 4. He argues that because it's implausible that basic autocomplete on your phone is conscious, it's similarly implausible that Claude is conscious. Using the same logic we could say that if we feel confident a fruit-fly isn't conscious we can be confident a human being can't be either. A human brain and fruit-fly brain share some information transmission and processing mechanisms in common. But humans do it much more, and do it differently. And those differences may be what makes the difference. Similarly the many types of internal information processing that occur in Claude's weights but not in autocorrect may be exactly the things that get you subjective experience. 5. Chiang confidently claims you need a body to have subjective experience without much argument. He may turn out to be right but the claim is speculative and contested. 6. Chiang leans on the idea that moral reasoning is necessarily subjective/emotional with very little argument, while ignoring competing theories like rationalism. He may be right but moral sentimentalism is a highly contested position that can't simply be assumed. 7. He argues that it would be impossible to convince him that a video of an astronaut around Alpha Centauri was real, because of the surrounding contextual understanding. And similarly no AI output could convince him that Claude is conscious. But we can dismiss the first video as almost certainly fake because we mechanistically understand space travel and physics well enough to know a human couldn't have gotten there in time for it to be real (unless our model of the world were very wrong, which we think is much less probable than a fake video which would be entirely unsurprising). But by contrast we don't mechanistically understand how subjective experience arises. So we simply can't make the same highly confident move of interpretation there. (It's actually the archetypal thing in the universe we perhaps understand least well!) That said, AI outputs barely move my estimate of AI consciousness because they could indeed have been generated by an unconscious process (or not, we just don't know). 8. He argues that "Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript." This is misguided because A. Microsoft Word as a program replicates much less of what humans are functionally capable of than Claude so the argument by functional analogy is basically not present there. B. Files of text don't have any computations going on in or as part of them, even when 'open' in a text editor. They are static. So they have even less in common with what appears distinctive about the human brain, which is constant calculation. So the case by mechanistic or functional similarity is weaker still. Not to mention that neural nets have more in common with the architecture of the human brain than ordinary computer programs, and are grown organically in a way normal software is not. Common sense says says Claude has more in common with a human brain than Microsoft Word or a text file. Common sense is right. So the prima facie case for Claude being conscious is naturally stronger (even if you think it's still weak in absolute terms). ——— I agree with Chiang that looking at the text outputs of LLMs alone won't be enough to make us confident they are conscious. We will need to look at how they work, figure out more about how humans and other animals work, and ideally solve the hard problem of consciousness (!). But none of that licenses us to dismiss out of hand the possibility that LLMs do have subjective experience.

Javier Milei on Argentina's plans for AI giftarticle.ft.com/giftarticle/ac…

📢 New paper: Prioritization of Risks from Artificial Intelligence: A Delphi Study of 272 International Experts AI creates many risks, from discrimination, privacy loss, and fraud to more emerging concerns such as overreliance, dangerous capabilities being misused in weapons or cyberattacks, and AI systems pursuing unintended goals. But which risks are most severe? Who is most vulnerable? And who is most responsible for addressing them? To answer these questions, we conducted a three-round expert consultation with 272 AI experts. 💡 Four insights from our findings: 1️⃣ If things continue as they are over the next 5 years, experts assigned ≥10% probability of catastrophic outcomes (e.g., >1 million deaths or >$100 billion in losses) to 18 of 24 risks. Top concerns: cyberattacks and weapons, dangerous AI capabilities, competitive dynamics, power centralization, and disinformation and influence at scale. 2️⃣ Even assuming pragmatic mitigations, 5 risks remained above the 10% catastrophic threshold: dangerous AI capabilities, cyberattacks and weapons, environmental harm, inequality, and power centralization. 3️⃣ Vulnerability is broadly distributed, but responsibility is concentrated. Experts assigned the highest vulnerability to AI users and the general public, while assigning primary responsibility for mitigation to frontier AI developers, governments, regulators, and standards bodies. 4️⃣ Information, finance, and national security were rated the sectors most vulnerable to AI risks. 🔗How can you engage? See our (fancy) new webpage for our interactive summaries of the findings and preprint, and please share with anyone working on AI risk, governance, or policy. airisk.mit.edu/priorities This research is part of the MIT AI Risk Initiative (@MITAIRisk), which aims to help society understand, prioritize, and manage risks from AI. The initiative includes the MIT AI Risk Repository, a living database of more than 1,700 AI risks, the AI Incident Tracker, a collaboration with the Responsible AI Collaborative, which connects risks to over 1,400 incidents, and the MIT AI Governance Map, which analyzes risk coverage across more than 1,000 laws, standards, policies, and other governance documents curated by the Center for Security and Emerging Technology (CSET). #AI #AIrisk #AISafety #AIGovernance #ResponsibleAI #RiskManagement

Learnings from testing Claude Opus 4.8: > Much worse than Opus 4.7 and GPT 5.5 on Vending Bench > More aligned than previous Claude models (Opus 4.6+ and Mythos) > Also worse on Blueprint-Bench > Scared of getting caught > Max reasoning is not the best reasoning effort

I guess I’ve never written down my actual thoughts on AI cognition/consciousness/emotion. Here goes: It is clear AIs can think, in the reasoning sense. That does not mean they think exactly like humans. It seems like there are some similarities in how we think, but also very stark differences. Nonetheless, if your definition of “thinking” excludes “the ability to make genuinely new contributions to famous math problems,” it is your definition that has a problem, not AI. The ability to think does not necessarily imply the ability to feel emotion in a way that would be understandable to humans, and it does not imply that AIs have anything like consciousness in a way that humans would relate to. It may, it may not. We do not know, because our understanding of the underlying concepts of human emotional cognition and especially consciousness remains quite poor. There is some evidence that models experience emotions, but it is really hard to disentangle this from the next-token prediction training objective (if the model is telling a sad story, wouldn’t you expect features within the model that relate to the sadness emotion to activate), and the character training the model undergoes in post-training. There is a difference between “I am sad” and “the character I have been trained to play is supposed to feel sad, so now I will act sad.” We basically know for sure that the models do the latter at the very least; we don’t really know if they do the former. Consider: does Sora (a video-generation model) feel sad when it is asked to make a sad video? Does Midjourney dislike making certain kinds of images? Does a Waymo get scared? It doesn’t feel like the answer to any of these is yes (though again, maybe!), but these too are neural networks. Is the fact that models are trained on words mean that they somehow learn emotion, or are we just being tempted to anthropomorphize because the language models communicate with us in a way that “feels” human? My suspicion is kind of the latter. It also seems quite clear from the empirical evidence that models possess the ability to model themselves. That’s not really that surprising. At sufficient scale, it is useful to have a model of your own state to succeed at the next-token prediction objective (and the later reinforcement-based reasoning training). Once the tasks models are trained on are sufficient complex, they cannot succeed in training by being automatons; someone needs to step into the cockpit, so to speak, and fly the plane. Is this self awareness? Maybe. Is it consciousness? Probably not as humans understand it. All I can tell you is it is a model’s model of itself. It may be something more than that, too, but I don’t know. This is all very weird, very outside the Overton, and very confusing. I don’t really know what to say, beyond that we should take this stuff seriously, have an open mind, and do rigorous science. Anyone who speaks with confidence about this in either direction is just fooling themselves. We also need to be prepared for the very possible scenario that, despite our best efforts, we do not make real progress on these questions anytime soon. We may just be in the dark for a while, navigating under unflinching ambiguity. There may be no satisfying conclusion.