Daniel Galvez @memorypaladin
AI Developer Technology Engineer at NVIDIA. Speech recognition and computer architecture nerd. All opinions are my own, not my employer's. Joined January 2019-
Tweets138
-
Followers415
-
Following42
-
Likes3
@ezyang Yeah, that's my concern. I worry that it's already pretty obvious which kernels aren't batch invariant in LLM workloads.
I'm thinking of internship projects for this summer. If we build a dynamic binary translation tool to detect whether any cuda kernel in your workload is not batch invariant, would that be useful to anyone? You would run it just like compute-sanitizer. thinkingmachines.ai/blog/defeating…
I'm speaking at GTC on Thursday at 2 PM about debugging cuda graph errors in pytorch. I will be talking about several tools in pytorch and CUDA that can detect and eliminate classes of errors for you. Will be fun. nvidia.com/gtc/session-ca…
@cloud11665 @tenderizzation It's more like... it's surprising how competitive eager mode run via a cuda graph can be with torch.compile with max-autotune at times.
@tenderizzation @cloud11665 There's also subtle issues with NCCL user buffer registration as well. All of the edge cases and the need to work with arbitrary kernels is the reason why no one has pulled this off for generic code yet. And some have complained at me about the need for torch.compile as well.
@tenderizzation @cloud11665 But anyway it is at a point where I would be happy to merge it as an experimental feature after a lot of code cleanup. (And of course we need Elias Ellison to say yes!) There's lots of interest, so it's a renewed priority for me. Feel free to AMA.
It's really incredible how much I learn every day from my colleagues at NVIDIA on mapping ML compute to hardware. It's a special place.
@mike64_t Need to learn the problems first before you can make a solution for it. There are lots of complications I've learned only by working in detail with torch.compile and with colleagues on Megatron. e.g., torch.compile cannot obviously enforce PGAS for "symmetric memory" right now.
Is Amazon DeepFleet actually a generative transformer model trained to solve NP-Hard problems, or is it just a classic Vehicle Routing Problem (VRP) solver being branded as "AI"? Would be cool if the former, but no real substance in the announcement. aboutamazon.com/news/operation…
@marksaroufim Oooh, lots of random things I don't know. Thanks.
FYI, when you're building pytorch at HEAD from source, the versions of optional dependencies like triton for your given commit are listed at: github.com/pytorch/pytorc…
@CUDAHandbook To be clear, the idea is that you would use this to infer for an arbitrary program what data is being used by any particular kernel as a form of runtime analysis., not to try to do cuda managed memory all over.
@CUDAHandbook A fun thing about virtualization technology is that you can presumably "infer" at runtime the pages referenced by each CUDA kernel if you intercept every address lookup, thus telling you the "segments" used. Maybe it's hopeless for data-dependent workloads, though.
Really appreciate how adding a comment to c10/core/Allocator.h in pytorch causes over 1000 files to recompile. Someone really needs to make ccache comment-aware, if that's not impossible or otherwise impractical.
If you want to get rid of the "I0702 10:00:03.704527 2389475 torch/_subclasses/_fake_tensor.py:3179]" prefix in your torch logs, you need to set the environment variable TORCH_LOGS_FORMAT="%(message)s".
@_rozzai I have not. It seems that the typical design pattern in CUDA programs is that you have a single CPU thread driving all GPU work. Pytorch uses a separate thread for backward by default docs.pytorch.org/docs/stable/ge… but it never runs at the same time as the main thread.
Since ASLR makes the CPU allocations change with every process run, the location of the GPU virtual memory heap cannot be the same across every process run, as CPU memory might have been allocation already where your GPU memory previously was.
Dorsa @dorsa_rohani
10K Followers 2K Following machine learner @tenstorrent, prev @nvidia. and something new_
Robert Scoble @Scobleizer
587K Followers 50K Following San Francisco/Silicon Valley AI | Robots, holodecks, BCIs, analysis of new things | Ex-Microsoft, Rackspace, Fast Company | Wrote eight books about the future.
daki @daki_natteko
0 Followers 5K Following
We/z @wheezyair
406 Followers 1K Following I sell RTL dev trajectories in training datasets to semiconductor manufacturers and AI labs training Agents on chip design, I also climb skyscrapers and towers
tirth @__tirthap
4 Followers 265 Following
Oliver @0xAABBCC
0 Followers 229 Following
tejas krishna @tejaskrshna
12 Followers 2K Following
Svarupa @Svarupa2000
50 Followers 329 Following Hardware-Native Compute Architecture for Real-Time Adaptation in Robotics and Autonomous Systems | US Patents Filed
Tisha Wangui @WanguiTish51872
1 Followers 7 Following Interested in intelligence, Curious about Real-Time Speech AI | ML & Voice Tech
RUIHAN LIU @RuihanL68875
0 Followers 69 Following
Jacob Rintamaki @jacobrintamaki
10K Followers 3K Following now texas | semianalysis, elad gil, stanford, cwru | opinions are my own. not investment advice.
gf dz @gfdz6688
1 Followers 104 Following
Abraham A @Codingtreats
23 Followers 2K Following
Abdullah Sarkar @the_asarkar
1 Followers 185 Following
Alek Dimitriev @tensor_rotator
5K Followers 2K Following Inference @AnthropicAI, prev Gemini @Google, prev prev PhD @UTAustin
Stefan @0x573F
302 Followers 442 Following
Adam Baranowski @adamkbaranowski
343 Followers 490 Following Inference @poolsideai. Opinions are my own.
ishan @0xishand
2K Followers 2K Following inference (dynamo + sglang) @nvidia | prev. @brevdev (acq.) @agora_io (acq.), @columbia | my views ≠ employer views
Ameen Patel @Ameen_ml
2K Followers 2K Following Inference @PrimeIntellect, prev @togethercompute, @AmazonScience, @uwaterloo
Bob @polyidan
31 Followers 3K Following
Rudrashis Gorai @RudrashisGorai
30 Followers 320 Following ai & embedded sys @ucirvine | iiest shibpur
Teknium 🪽 @Teknium
99K Followers 6K Following Cofounder and Lead Engineer - Hermes Agent @NousResearch, prev @StabilityAI Github: https://t.co/LZwHTUFwPq HuggingFace: https://t.co/sN2FFU8PVE
!.! @xypyth
44 Followers 7K Following
Totally Indifferent @toatesmagoats
215 Followers 3K Following
En @En42750878
0 Followers 516 Following
Sky Lee @SkyLee010101010
3 Followers 166 Following
Wesley Maa @wesleymaa
223 Followers 98 Following bay area refugee — perpetually sleeping through terrible weather. building robots and satellites @columbia | prev. @columbia dropout
helen @mathemakitten
3K Followers 248 Following 💫🌷 my secret superpower is that i code while i cry better than anyone does
Expect Value @ExpectValue
121 Followers 2K Following Acquire and operate software/SaaS businesses. Student of valuation, probability, risk/reward. I like cashflow, reading, board games, lifelong learning.
JulianHuang @huangzhilin1221
6 Followers 242 Following
Jerome Ku @KuJerome
50 Followers 6K Following
Piotr Mazurek @tugot17
4K Followers 893 Following inteligence involution @liquidai; from first principles https://t.co/7biJRXvk53; ex @Aleph__Alpha (acc. by @cohere)
コムギ @kksspoi
100 Followers 603 Following 生成AI系、PC自作er、ビルド作業が好きです。よろしくお願いします。 #stablediffusion #comfyui #AI
Miguel @Kaweees1
1K Followers 2K Following Prev. Roboticist @dimensionalos, Humanoids & Chip design agents @nvidia GEAR.
SemiAnalysis @SemiAnalysis_
107K Followers 27 Following
tsuki @tensorcore
3K Followers 462 Following Hiroyuki Ootomo. High-precision GEMM emulation on Tensor Cores. Work at #76B900. Cooking @cp_async. Hai-to-Yoka: https://t.co/jAdudlZfnb
Mark Saroufim @marksaroufim
16K Followers 996 Following Amdahl comes for us all mts & co-founder @coreautoai @pytorch @gpu_mode enjoyer
Thien Tran @gaunernst
3K Followers 260 Following
Sam Rodriques @SGRodriques
21K Followers 232 Following Director and CEO at FutureHouse and Edison Scientific. Building an AI scientist. https://t.co/aNx8D1QmfN. https://t.co/rQYoPOwV8Q
Shane Gu @shaneguML
50K Followers 2K Following Gemini/Omni Thinking, Senior Staff AE @GoogleDeepMind. 🇯🇵-born 🇨🇳🇨🇦. ex: Gemini 1.5-3, GPT-4 @OpenAI, Google Robotics (JP: @shanegJP). Personal opinions.
Nicholas Wilt @CUDAHandbook
7K Followers 76 Following Nicholas Wilt was on the inception team for CUDA, wrote The CUDA Handbook, and writes at https://t.co/YkR71W07I7
Justin Zhang @Justin__Zhang
52 Followers 87 Following Systems @ NVIDIA. Ex-medical school. In Healthcare, Robotics, Semis
Joe Ryu @jesryu
408 Followers 148 Following 🚀Building physical superintelligence. 🤖Engineer & entrepreneur in Physical AI.
Daniel Han @danielhanchen
33K Followers 2K Following Building @UnslothAI • Making open-source LLMs faster, better & more accessible • YC S24 • ex-NVIDIA ML
Tessa Barton @tessybarton
4K Followers 1K Following Designer: GPU purse @gpupurse Member: technical staff @thinkymachines Prev: @mosaicML @NYTimes @BrownU
Rob Ghilduta @robertghilduta
2K Followers 2K Following
Shinji Watanabe @shinjiw_at_cmu
5K Followers 371 Following I'm working at CMU (2021-). I was working at NTT (2001-2011), MERL (2012-2017), and JHU (2017-2020). Speech and Audio Processing is my main research topic.
Vaibhav (VB) Srivasta... @reach_vb
50K Followers 282 Following founder mode @OpenAI | ex @huggingface | F1 fan | Here for @at_sofdog’s wisdom | *opinions my own
sjtugz @sjtugz1011
5 Followers 105 Following
Kaichao You @KaichaoYou
9K Followers 141 Following Ph.D. from Tsinghua University. Core maintainer of @vllm_project . Co-Founder & Chief Scientist @Inferact .
emozilla @theemozilla
12K Followers 1K Following catholic, ai researcher, co-founder/cto of @NousResearch alignment: whatever the opposite of yudkowsky + bryan johnson is. blessed be God in all his designs.
Piotr Żelasko @PiotrZelasko
2K Followers 734 Following AI + Speech @ Nvidia. PhD @ AGH-UST, ex-JHU. My interests: speech processing technology; ML/AI software engineering. Building OSS for Speech AI.
Ross Daly @Ross11235
12 Followers 295 Following Formal methods, synthesis, hardware-software codesign, compilers, quilting software. Universally quantified statements are almost always false.
Fabricated Knowledge ... @fabknowledge
32K Followers 850 Following Simplifying the world of semiconductor investing in the age of AI. Part of the @semianalysis_ gang. all views are personal.
Vijay @__tensorcore__
2K Followers 617 Following Systems and GPU Performance Mechanic - TBD Ex. CUTLASS 3.x / 4.x etc
James Hill-Khurana @jtvhk
5K Followers 7K Following Eclectic. Curious about machine learning, tech history, design, HCI and biomimicry. Prev, philosophy + cogsci, @uwaterloo.
Irrational Analysis @insane_analyst
40K Followers 100 Following Engineering-Driven Investment Analysis
Nando de Freitas @NandoDF
111K Followers 898 Following I seek to understand intelligence, agency and awareness, and build AI aligned with compassion, freedom, universal human empowerment, and progress science.
Longhorn @never_released
15K Followers 145 Following Supposedly "Kernel/Hypervisor Engineer" @ Amazon EC2 Core Compute
Asianometry @asianometry
18K Followers 313 Following My name is Jon and I run the Asianometry YouTube channel. You can email me at [email protected]
Dylan Patel @dylan522p
138K Followers 1K Following SemiAnalysis Boutique AI Infrastructure Research and Consulting DMs are open for consulting, quotes, or to talk shop, Opinions my own
Smerity @Smerity
32K Followers 3K Following ML x science. Founding Member of Technical Staff at @PrometheusInc. Prev @midjourney, @SFResearch, @CommonCrawl. @Harvard '14, @Sydney_Uni '11. 🇦🇺 in SF.
Scott Gray @scottgray76
9K Followers 880 Following GPU Geek at @OpenAI. I have a long standing interest in neuroscience and its application to machine learning. He/Him.
Hal Daumé III @haldaume3
26K Followers 332 Following https://t.co/jIZyWtJ8gS Human-centered AI, NLP & ML. Director @trails_ai. Prof @umdCS, member of @CLIPumd @HCIL_umd, former @MSFTresearch. Fun: 🧗🧑🍳🧘⛷️🏕️.
Noah A. Smith @nlpnoah
17K Followers 192 Following NLP&ML researcher. Prof @uwcse @uwnlp & helper @allen_ai @ai2_allennlp. Have not been accepting new followers here for some time, please try a better platform.
Chris Dyer @redpony
10K Followers 1K Following i live in Angel, London. I am a researcher at DeepMind. I play the cello. 𒌓𒈪𒊭𒄠𒋗 𒆪𒌦𒄭𒁺 𒌓 𒌨𒊑𒅇𒈬𒅆𒋢𒌨𒅇𒈨𒇷𒅍You might like
