In this second lecture, held on April 9, 2026, Brad Gerstner of Altimeter Capital and Sunny Madra of Nvidia (previously Groq) discuss the economics of GPUs and inference. Anything that is my personal opinion will be denoted with italics.
\nCourse Page
\nWatch the Lecture
The main question for the lecture is fairly simple: how does the cost of inference keep falling when both model size and demand keep going up?
\nGerstner starts with a long-term view of technology. His slides show GDP per capita roughly doubling every 25 years, while technology’s share of GDP grows from about 5% in 1998 to an estimated 15% in 2030.
\nHe also compares investment returns over the previous 15 years:
\nHis point is that compute sits at the root of these changes. Most of the gap between tech and non-tech appeared after 2020. I have a hard time seeing this as a compute story and not a market run up that has little to do with value….but whatevs
\nGerstner then introduces Madra, describing him as a founder who keeps getting acquired by larger companies. Madra was president of Groq before joining Nvidia.
\nGroq founder Jonathan Ross previously worked at Google, where he helped develop the company’s Tensor Processing Unit (TPU).
\nMadra describes Groq’s chip as a deterministic dataflow system. A compiler decides where the calculations will happen before the program runs. This is useful for large language models because generating tokens requires quite a bit of math.
\nThe amount of work needed to generate one token depends partly on the number of parameters in the model and the amount of context it must process.
\nMadra separates inference into two parts:
\nThese operations put different demands on the hardware. Madra says it could make sense to use different machines for each one.
\nGroq’s design has a large amount of SRAM, which is fast memory located directly on the chip. Nvidia has a wider collection of chips and systems designed to work together. Madra also discusses NVLink Fusion, which allows custom processors to connect to Nvidia’s systems.
\nIn the example he gives, combining the two approaches can generate 2.5 times as many tokens using the same amount of electricity.
\nGerstner asks what will push the unit cost of inference down. Madra gives three answers:
\nLithography, the process used to manufacture increasingly small features on chips, is beginning to reach physical limits. One response is to make the chips larger. Cerebras produces wafer-scale chips that Madra compares in size to pizza boxes.
\nHardware development is also happening alongside model development. Better hardware makes larger models possible, while larger models create demand for more hardware.
\nMadra says models with as many as 10 trillion parameters are coming. Even a 50-fold increase in capacity might not be enough over five years if models continue to get bigger.
\nThere are already tens of trillions of tokens being generated each year. This was in April. Appetite for tokens has increased like they predicted, if not more so.
\nGerstner says OpenAI initially had negative gross margins because inference was expensive. As the cost of inference came down, the models became more useful and people became more willing to pay for them.
\nThe models are also beginning to take actions rather than only answering questions. That increases the number of tokens used for each task.
\nMadra says there are two things that people outside the industry may not see yet. First, today’s public models were not trained on the newest hardware. Second, companies are still figuring out how to get more work from the models they already have.
\nA system that can take a problem and work on it continuously will use far more tokens than a chatbot waiting for someone to ask a question. Both alluded to the fact that there isn’t enough compute, and probably won’t be for some time. Cost goes down….great, but I want 1,000X the number of tokens.
\nGerstner points to Anthropic’s rapid revenue growth as evidence that willingness to pay is increasing along with model capability.
\nMadra argues that the current models and chips do not show where demand will settle. Newer models will be trained on newer hardware (Blackwell at the time of this lecture), and companies will get better at putting those models to work.
\nThis does not answer whether every company or infrastructure investment will earn a good return. It does explain why falling inference costs do not necessarily mean lower total spending.
\nGerstner says the Industrial Revolution disrupted many existing jobs, but people found new ways to create value.
\nHe gives a more immediate example from Altimeter: he would not hire someone who does not use Excel. His argument is that some forms of IQ may become easier to obtain, making EQ and problem-solving more valuable.
\nMadra uses mathematics as another example. If AI systems begin making new discoveries, the people and organizations that know how to use those discoveries could have an advantage.
\nThey both gave pretty crappy answers. But I think the answer is….we don’t know. And that’s ok to give as an answer, but they don’t do that.
\nGerstner says Apple’s AI strategy is risky because language models are not yet good enough to do everything locally on a device. Privacy is also a major consideration for the company.
\nThe case for Apple is its existing base of devices. If smaller models improve enough, products such as Siri could become much more useful without sending every request to a data center.
\nPower remains a problem. Madra says an 8-billion-parameter model running on a phone can drain the battery in about 30 minutes.
\nI personally feel that Apple is doing a great job holding off. Apple has always succeeded by thoughtfully doing something, not slapping new crap everywhere…ahem Samsung.
\nGerstner criticizes fear-based arguments intended to produce regulatory capture (cough…Dario….cough). At the same time, he says people should not put their heads in the sand.
\nHe takes Dario Amodei and Sam Altman seriously when they describe how quickly the models are improving. The rate of change, in his view, is becoming fairly parabolic.
\nA student presents two possibilities. Nvidia could protect its margins and accept slower revenue growth, or lower its margins and maintain a larger share of the market.
\nGerstner says Nvidia has a strong product schedule and much of its capacity is already booked for the next eight quarters. He also points out that people once said Nvidia could never become a $1 trillion company.
\nCompetitors such as Groq, Cerebras, and Google’s TPU can succeed without Nvidia failing. Gerstner describes this as the good part of capitalism.
\nThis student can’t see the forest for the trees. He sees the GPU market as a zero sum game. Nvidia has never worked like that. Silly premise.
\nThat is it for this week. I will keep posting my course notes and comments on this page.
\n", "date_published": "2026-08-05T22:32:02+08:00", "url": "https://robertritz.com/2026/08/05/lecture-the-gpu-economy.html", "tags": ["AI Supercycle"] }, { "id": "http://robertritz.micro.blog/2026/07/29/lecture-economics-of-the-ai.html", "title": "Lecture 1: Economics of the AI Supercycle", "content_html": "In this first lecture instructor, held on April 2, 2026, Apoorv Agrawal (currently Altimeter Capital, prev Palantir) goes over the format of the course and the purpose of the course with some primer concepts for the class. Anything that is my personal opinion will be denoted with italics.
\nCourse Page\nMaterials for this week:
\nThe course is primary guest speakers covering different areas of the current AI supercycle. Speakers are in areas such as GPUs, SaaS, infrastructure, etc.
\nWe are at the start of a massive supercycle. Agrawal makes the contention that this cycle will be larger than cloud, mobile, and the internet.
\nAgrawal asks some questions from the class then continues the rest of the time with a discussion.
\nA lot of money is going to CAPEX right now (capital expenditure), the so called 5 layer cake Jensen Huang has referred to (energy, chips, power, interconnect, memory). This CAPEX is amortized (at least for GPUs) over about 5-6 years. Is it worth it?
\nHow do you get to >$10 per user per year? Agrawal makes the point that knowledge work isn’t the answer, advertising is.
\nSoftware gets 80-90% gross margins. AI services at billions in revenue still isn’t profitable. Average margins for AI companies are much lower, around 30%.
\nAmazon took 8 years to build out AWS, and during that time people were asking if Amazon would go bankrupt. This is probably because Amazon had famously near 0 net profit for many years post IPO, instead focusing on reinvesting to increase revenue.
\nIn cloud most value accrues in apps, but in AI its at the semiconductor layer. Agrawal states that gross margins for semis is 75% (0-30% for AI apps layer). This is the inverse of the cloud period. Also, semis are built for 5-6 year depreciation cycles, but apps generate revenue now. Mobile supercycle had the same inflated CAPEX early in the cycle.
\n![\"\"](\"https://cdn.uploads.micro.blog/169387/2026/9208deb1b0.png\")
\nA student made a point that Google is one of the only complete AI stacks today, they have TPUs, apps, existing customer base, etc.
\nAgrawal makes the point to pay attention to earnings calls. CAPEX has an equilibrium (limited to free cashflow and cash reserves and also the expected revenue gains from additional CAPEX), so earnings calls will signal a change in this equilibrium.
\nThis lecture was in April 2026 and this is somewhat prescient but perhaps underestimated the frothiness of the AI CAPEX splurge. Google just had its very first period of negative free cashflow since 2004.
\nAgrawal says:
\nBecause of Google’s power in chips, apps, and existing customer base it might be likely that Google wins here.
\nInference is dependent on when humans are awake, so this causes the usage to be unpredictable. This was how AWS got started (by using excess capacity since Amazon didn’t have traffic at night), so I feel like this isn’t as big of a problem as he states it is.
\nASICs are application specific integrated circuits. Groq (acquired by Nvidia in Dec 2025) and Cerebras (Jan 2026 deal with OpenAI) are two that come to mind.
\nThere is 300 billion in revenue to go around. There wasn’t much discussion here around this topic, perhaps more in a later lecture.
\n2024: 90B vs 2026: 435B in revenue. But the shape of the revenue hasn’t changed much.
\n![\"\"](\"https://cdn.uploads.micro.blog/169387/2026/7519def291.png\")
\nYet profits remain in the semis later.
\nChatGPT has most users, then Gemini. Consumer apps are dwarfing ChatGPT, even with all of the hype.
\n![\"\"](\"https://cdn.uploads.micro.blog/169387/2026/be8d7835fe.png\")
\nChatGPT is closer to the niche app category (like Spotify and X) than it is to Netflix or YouTube.
\nIn ChatGPT you have to go and ask a question. This is very active and most people don’t prefer to engage in apps like this. To increase revenue Agrawal makes the point that these companies should use ads rather than subscriptions, and he thinks this is where it is heading.
\nIntent is much cleared in AI chat apps, so sure revenue per ad will be higher, but the inventory will be much lower as well.
\nThat’s it for this week. I’ll keep posting my course notes and comments on this page.
\n", "date_published": "2026-07-29T14:25:24+08:00", "url": "https://robertritz.com/2026/07/29/lecture-economics-of-the-ai.html", "tags": ["AI Supercycle"] } ] }