Matt Lehman @obsecurus

Principal Investigator | AIxCC Lead Architect (views my own) aicyberchallenge.com Columbus, Ohio Joined May 2010

Tweets

980
Followers

797
Following

1K
Likes

2K

Matt Lehman @obsecurus

a hour ago

@DavidSacks Similar to the $520k/yr salary as signal from all-in. If we’re truly there why do they have so many dependencies including cyber products? The models should be solving it all. There are indicators beyond benchmarks at providers: x.com/obsecurus/stat…

Matt Lehman @obsecurus

a day ago

x.com/i/article/2064…

1 1 5 33K 0

0 0 0 159 0

View Details

Matt Lehman @obsecurus

7 hours ago

@mitchellh The speed delta is painful but been able to generate interesting results simulated thinking, and iterative innovation which is interesting.

0 0 0 83 0

View Details

Matt Lehman @obsecurus

9 hours ago

@chamath now that this is confirmed to be happening, how do we distinguish between real ML vs. a frontier lab acquiring a unique company and slapping an OpenAI compliant translator behind the product API and claiming its ML? This would offer huge potential offloads from GPU use with no means of verification at which point it becomes the same sort of Ponzi scheme that caused you to exit VC for a while and drives monopolies instead of actually driving innovation. Without verification and better trust systems built into the inference layer how would anyone know?

0 0 0 65 0

View Details

Matt Lehman @obsecurus

9 hours ago

We absolutely need cryptographic model pinning and input/output signing. Now that there is per user acknowledged model degradation and by its nature non-deterministic behavior user to user or key to key beyond that of normal ML non-determinism. I hit on more of this here: open.substack.com/pub/obsecurus/…

0 0 0 150 0

View Details

Matt Lehman @obsecurus

17 hours ago

@EnoReyes This is just the first transparent confirmation of it. How do we verify as consumers of model outputs that they are indeed sourced from a specific pinned model or even proof machine learning at all? obsecurus.substack.com/p/the-api-is-a…

0 0 2 36 0

View Details

Matt Lehman @obsecurus

23 hours ago

@james406 you forgot tastemaxxing

0 0 0 37 0

View Details

Matt Lehman @obsecurus

23 hours ago

Fable 5 applying forced token maxing by converting to a Python strategy over RTK. I've personally found Python generation far less reliable and reliable than Golang, Rust, or TypeScript. I think this is primarily because of the strong typing, but perhaps even moreso the abundance of opinionated community content about the "right" way to implement something that likely went into training.

1 0 1 51 0

View Details

Matt Lehman @obsecurus

a day ago

x.com/i/article/2064…

1 1 5 33K 0

View Details

Matt Lehman @obsecurus

3 days ago

@theallinpod @altcap @andrewdfeldman @Will4Planet @Jason @chamath @cerebras @planet @Jason has mentioned the concerns about his team building 3 copies of the same thing as a weakness. I think this is the wrong take. It's can be a net benefit, you're missing a convergence event, how do you turn them into a superset? This is a first principle reshape of SWE.

0 0 0 48 0

View Details

Matt Lehman @obsecurus

3 days ago

Forget "token maxing", I've derived this new agentic concept of "outcome maxing". This novel concept is where you focus on producing real world results, embrace stakeholder empathy, and maximize value by scaling desired outcomes for your customers.

0 0 0 48 0

View Details

Matt Lehman @obsecurus

2 weeks ago

@dguido I do think however it’s useful as a baseline of expectations. Like if the answer isn’t 100% success on discovery that is a bit telling. Especially for very well bugs for which ample training documentation (blog posts, p0c, etc.) exist.

0 0 0 158 0

View Details

Matt Lehman @obsecurus

3 weeks ago

The details of AIxCC challenges are now live at archive.aicyberchallenge.com/challenges Huge thanks to all the challenge authors who contributed, and the teams who helped review content.

0 0 0 43 0

View Details

Matt Lehman @obsecurus

a month ago

If you haven’t re-evaluated delivery services lately for Instacart, Uber, Door Dash, it’s probably a good time to do so. Seeing extreme price increases. +36% just for the item, then nearly 2-3x for delivery from 1/4 mile away. 4x if you take the max suggested tip.

1 0 1 142 0

View Details

Matt Lehman @obsecurus

3 months ago

@elonmusk @StarbaseTX At first glance I thought it was elevated like this.

2 1 33 3K 0

View Details

Matt Lehman @obsecurus

3 months ago

generated company-profile.md

0 0 0 31 0

View Details

Matt Lehman @obsecurus

3 months ago

@huggingface might be an interesting endeavor for you also?

0 0 0 30 0

View Details

Matt Lehman @obsecurus

3 months ago

Does anyone know if there are any models specifically trained on data before GPT-3 was widely available? I know Sam has mentioned that GPT-4 could potentially be this but I think we need a site and a model like "before[.]ai" or similar that is specifically dedicated to continual training on ONLY datasets validated to be available before GPT-3 or some other clear line in the sand where we know models we're generating a significant volume of data on the internet. If we don't do this as some sort of public good, open-source, shared offering I think attempting to baseline understanding and separate raw human content from model generated content is going to be impossible. This is not about generating a large extremely capable model for free, but about human vs. AI/ML provenance. This seems like the sort of thing an academic institution or standards agency like NIST, etc. could help maintain with the support of large model vendors @sama @DarioAmodei @elonmusk

1 0 1 73 0

View Details

OpenSSF @openssf

4 months ago

New podcast 🎙️ How did the AI Cyber Challenge go from skepticism to success? Start with AIxCC Part 1 – From Skepticism to Success and hear how #AIxCC reshaped thinking around AI + cybersecurity. Part 1 kicks off a 4-episode series: hubs.la/Q042tjVx0