In this second lecture, held on April 9, 2026, Brad Gerstner of Altimeter Capital and Sunny Madra of Nvidia (previously Groq) discuss the economics of GPUs and inference. Anything that is my personal opinion will be denoted with italics.

Course Page
Watch the Lecture

The main question for the lecture is fairly simple: how does the cost of inference keep falling when both model size and demand keep going up?

The GPU economy

Gerstner starts with a long-term view of technology. His slides show GDP per capita roughly doubling every 25 years, while technology’s share of GDP grows from about 5% in 1998 to an estimated 15% in 2030.

He also compares investment returns over the previous 15 years:

• Technology companies: about 15% per year
• Non-technology companies: about 6% per year

His point is that compute sits at the root of these changes. Most of the gap between tech and non-tech appeared after 2020. I have a hard time seeing this as a compute story and not a market run up that has little to do with value….but whatevs

Gerstner then introduces Madra, describing him as a founder who keeps getting acquired by larger companies. Madra was president of Groq before joining Nvidia.

How Groq works

Groq founder Jonathan Ross previously worked at Google, where he helped develop the company’s Tensor Processing Unit (TPU).

Madra describes Groq’s chip as a deterministic dataflow system. A compiler decides where the calculations will happen before the program runs. This is useful for large language models because generating tokens requires quite a bit of math.

The amount of work needed to generate one token depends partly on the number of parameters in the model and the amount of context it must process.

Prefill and decode are different problems

Madra separates inference into two parts:

• Prefill processes the prompt and its context.
• Decode generates the response one token at a time.

These operations put different demands on the hardware. Madra says it could make sense to use different machines for each one.

Groq’s design has a large amount of SRAM, which is fast memory located directly on the chip. Nvidia has a wider collection of chips and systems designed to work together. Madra also discusses NVLink Fusion, which allows custom processors to connect to Nvidia’s systems.

In the example he gives, combining the two approaches can generate 2.5 times as many tokens using the same amount of electricity.

Will inference keep getting cheaper?

Gerstner asks what will push the unit cost of inference down. Madra gives three answers:

1. Improvements in the supply chain
2. Work by hardware and systems engineers
3. Power

Lithography, the process used to manufacture increasingly small features on chips, is beginning to reach physical limits. One response is to make the chips larger. Cerebras produces wafer-scale chips that Madra compares in size to pizza boxes.

Hardware development is also happening alongside model development. Better hardware makes larger models possible, while larger models create demand for more hardware.

Madra says models with as many as 10 trillion parameters are coming. Even a 50-fold increase in capacity might not be enough over five years if models continue to get bigger.

There are already tens of trillions of tokens being generated each year. This was in April. Appetite for tokens has increased like they predicted, if not more so.

Cost goes down, usage goes up

Gerstner says OpenAI initially had negative gross margins because inference was expensive. As the cost of inference came down, the models became more useful and people became more willing to pay for them.

The models are also beginning to take actions rather than only answering questions. That increases the number of tokens used for each task.

Madra says there are two things that people outside the industry may not see yet. First, today’s public models were not trained on the newest hardware. Second, companies are still figuring out how to get more work from the models they already have.

A system that can take a problem and work on it continuously will use far more tokens than a chatbot waiting for someone to ask a question. Both alluded to the fact that there isn’t enough compute, and probably won’t be for some time. Cost goes down….great, but I want 1,000X the number of tokens.

Is AI a bubble?

Gerstner points to Anthropic’s rapid revenue growth as evidence that willingness to pay is increasing along with model capability.

Madra argues that the current models and chips do not show where demand will settle. Newer models will be trained on newer hardware (Blackwell at the time of this lecture), and companies will get better at putting those models to work.

This does not answer whether every company or infrastructure investment will earn a good return. It does explain why falling inference costs do not necessarily mean lower total spending.

Q&A

How should people prepare for this?

Gerstner says the Industrial Revolution disrupted many existing jobs, but people found new ways to create value.

He gives a more immediate example from Altimeter: he would not hire someone who does not use Excel. His argument is that some forms of IQ may become easier to obtain, making EQ and problem-solving more valuable.

Madra uses mathematics as another example. If AI systems begin making new discoveries, the people and organizations that know how to use those discoveries could have an advantage.

They both gave pretty crappy answers. But I think the answer is….we don’t know. And that’s ok to give as an answer, but they don’t do that.

What is going on with Apple?

Gerstner says Apple’s AI strategy is risky because language models are not yet good enough to do everything locally on a device. Privacy is also a major consideration for the company.

The case for Apple is its existing base of devices. If smaller models improve enough, products such as Siri could become much more useful without sending every request to a data center.

Power remains a problem. Madra says an 8-billion-parameter model running on a phone can drain the battery in about 30 minutes.

I personally feel that Apple is doing a great job holding off. Apple has always succeeded by thoughtfully doing something, not slapping new crap everywhere…ahem Samsung.

How should AI CEOs talk about AGI?

Gerstner criticizes fear-based arguments intended to produce regulatory capture (cough…Dario….cough). At the same time, he says people should not put their heads in the sand.

He takes Dario Amodei and Sam Altman seriously when they describe how quickly the models are improving. The rate of change, in his view, is becoming fairly parabolic.

What is Nvidia’s long-term business model?

A student presents two possibilities. Nvidia could protect its margins and accept slower revenue growth, or lower its margins and maintain a larger share of the market.

Gerstner says Nvidia has a strong product schedule and much of its capacity is already booked for the next eight quarters. He also points out that people once said Nvidia could never become a $1 trillion company.

Competitors such as Groq, Cerebras, and Google’s TPU can succeed without Nvidia failing. Gerstner describes this as the good part of capitalism.

This student can’t see the forest for the trees. He sees the GPU market as a zero sum game. Nvidia has never worked like that. Silly premise.

Course notes

That is it for this week. I will keep posting my course notes and comments on this page.

My son used Claude Code for a while to help him fix mods and do some other small things, mostly gaming related, on his home computer. Since I switched to Codex a few weeks ago I convinced him to switch as well. It was easy since he uses perhaps once a day to a few times a week. He is 10 so he can type reasonably, but he prefers Wispr Flow to dictate (I set him up with my unused account since I use a local app on my Mac now).

Last night he was looking for a mod for Minecraft, which is focused on quickly switching between keybind profiles by hitting a hotkey. He found one, but it was buggy and didn’t work well. So I said….just make it yourself. He was super skeptical at first, but I said Codex with Terra could easily do it.

I sat him down with and helped him with the first prompt and showed him how to phrase it so it would actually make it and not just brainstorm endlessly. He set it to work and it took two turns to get it to launch and work properly, and another 30 minutes or so of work to polish it into something he really really likes. He is over the moon at what he did and he thinks it is amazing. He specifically directed the model to ensure it complies with server rules and doesn’t give him an unfair advantage (this impressed me).

As a 10 year old he is somewhat exasperated at the huge amount of AI generated content out there and he generally considers it a menace. But this got to him and showed him the real power. Now I’m convincing him to polish it a bit more then release it on Curse Forge, a popular modding platform, so others can benefit from it.

If he would have had to learn enough Java to make this work on his own, it would have taken him two weeks of work, and he would have given up after a few hours of learning. Part of me feels like learning how to code would have been better, because it would have taught him how to learn a new skill. Another part of me thinks that is stupid, and that I don’t even code anymore. What matters is the process, asking the right questions, and ensuring you make something that is quality.

So my son “coded” his first Minecraft mod.

The visa bond program for the United States started impacting Mongolia in April 2026. This program requires B1/2 visa applicants to post a bond of $10,000-$20,000 dollars and limits visa terms from three months to a year. This was a pilot program, and it was just made permanent.

It’s annoying for us, to say the least. At least for now, when my wife’s 10 year visa expires, she will need to go through this process. Since she was 18 or 19 she has gone to the US many, many times. In fact she lived there for 8 years, which is when I met her. We were even married there.

We go to the US to visit family, and to do business, every year. One of the largest jewelry trade shows in the US is in Las Vegas each year and she attends. In fact most of the purchases for her jewelry business in Mongolia come from US suppliers.

I understand the State Department’s reasoning for this decision. I’d argue that this program, as it is currently structured, will probably cost the State Department more money. For all those thousands of Mongolians who travel to the US regularly, and who don’t break the rules, they will now have to go to the US Embassy in Ulaanbaatar once a year (in the best case) and do a visa interview. It is going to overload the already stressed system they have.

It would make much more sense to be able to waive this program for those with a long and obviously strong visa history.

Back in 2015 when my wife and I were married, she had to get a visa to go to the US. We made no secret of the plan to get married in Texas, and we planned to go on our honeymoon and then come right back to Mongolia. The consular officer basically said that she wasn’t allowed to get married in the US, which isn’t correct for obvious reasons. But the reason they said this is because once you are married to a US citizen and in the US you can file for an adjustment of status and then stay. That wasn’t our intention of course, but for a consular officer it’s certainly a bit of a weird thing to hear from a visa applicant in a country where most people would love to immigrate to the US.

Anyways, I tell this story to point out that my wife has had ample opportunity over the past 20 years to immigrate legally. She didn’t. So this program makes little sense other than to create more hassle for us and the consular officers at the embassy.

Funny enough, all immigrant visa issuance from Mongolia are paused anyways. So even if we wanted to move our family to the US we couldn’t…

One of the very nice things in Mongolian summer is the wild strawberries! Last night we made jam with about 3 liters of strawberries.

In this first lecture instructor, held on April 2, 2026, Apoorv Agrawal (currently Altimeter Capital, prev Palantir) goes over the format of the course and the purpose of the course with some primer concepts for the class. Anything that is my personal opinion will be denoted with italics.

Course Page Materials for this week:

• The Economics of Generative AI (2024)
• The Economics of Generative AI: Two Years Later
• State of Consumer AI Part 1
• State of Consumer AI Part 2
• State of Consumer AI Part 3

Format of the course

The course is primary guest speakers covering different areas of the current AI supercycle. Speakers are in areas such as GPUs, SaaS, infrastructure, etc.

Why take this course?

We are at the start of a massive supercycle. Agrawal makes the contention that this cycle will be larger than cloud, mobile, and the internet.

Where is the $ in AI today?

Agrawal asks some questions from the class then continues the rest of the time with a discussion.

Q: Is this CAPEX generating revenue?

A lot of money is going to CAPEX right now (capital expenditure), the so called 5 layer cake Jensen Huang has referred to (energy, chips, power, interconnect, memory). This CAPEX is amortized (at least for GPUs) over about 5-6 years. Is it worth it?

• Google: $100/user/year
• Meta: $70
• OpenAI: $10

How do you get to >$10 per user per year? Agrawal makes the point that knowledge work isn’t the answer, advertising is.

How this revolution is different

Software gets 80-90% gross margins. AI services at billions in revenue still isn’t profitable. Average margins for AI companies are much lower, around 30%.

Amazon took 8 years to build out AWS, and during that time people were asking if Amazon would go bankrupt. This is probably because Amazon had famously near 0 net profit for many years post IPO, instead focusing on reinvesting to increase revenue.

In cloud most value accrues in apps, but in AI its at the semiconductor layer. Agrawal states that gross margins for semis is 75% (0-30% for AI apps layer). This is the inverse of the cloud period. Also, semis are built for 5-6 year depreciation cycles, but apps generate revenue now. Mobile supercycle had the same inflated CAPEX early in the cycle.

Don’t underestimate Google

A student made a point that Google is one of the only complete AI stacks today, they have TPUs, apps, existing customer base, etc.

Agrawal makes the point to pay attention to earnings calls. CAPEX has an equilibrium (limited to free cashflow and cash reserves and also the expected revenue gains from additional CAPEX), so earnings calls will signal a change in this equilibrium.

This lecture was in April 2026 and this is somewhat prescient but perhaps underestimated the frothiness of the AI CAPEX splurge. Google just had its very first period of negative free cashflow since 2004.

Agrawal says:

• Google won the internet supercycle
• Apple won mobile
• Meta won social
• Oligopoly in cloud

Because of Google’s power in chips, apps, and existing customer base it might be likely that Google wins here.

Training and inference are unpredictable

Inference is dependent on when humans are awake, so this causes the usage to be unpredictable. This was how AWS got started (by using excess capacity since Amazon didn’t have traffic at night), so I feel like this isn’t as big of a problem as he states it is.

Where are all the ASICs?

ASICs are application specific integrated circuits. Groq (acquired by Nvidia in Dec 2025) and Cerebras (Jan 2026 deal with OpenAI) are two that come to mind.

There is 300 billion in revenue to go around. There wasn’t much discussion here around this topic, perhaps more in a later lecture.

Where will value accrue in AI?

2024: 90B vs 2026: 435B in revenue. But the shape of the revenue hasn’t changed much.

Yet profits remain in the semis later.

AI Apps - Weekly active users

ChatGPT has most users, then Gemini. Consumer apps are dwarfing ChatGPT, even with all of the hype.

ChatGPT is closer to the niche app category (like Spotify and X) than it is to Netflix or YouTube.

Is knowledge work where most people are?

In ChatGPT you have to go and ask a question. This is very active and most people don’t prefer to engage in apps like this. To increase revenue Agrawal makes the point that these companies should use ads rather than subscriptions, and he thinks this is where it is heading.

Intent is much cleared in AI chat apps, so sure revenue per ad will be higher, but the inventory will be much lower as well.

Course notes

That’s it for this week. I’ll keep posting my course notes and comments on this page.

My son is doing GT Anywhere, which is an offshoot of Alpha School. The Timeback platform kept crashing on him for two of the lessons. I suspected a browser cache issue because of how it loads (via an embedded Electron browser in the app).

Deleted the cache in the container and it worked. Had some help from Codex to tell me where to look.

I truly think Timeback, Alpha, and GT School are the way forward in education (it’s working great for my son). I expected there would be some rough edges with something so new.

Even with all of these rough edges my son loves it. After class his current favorite thing to do is play table tennis.

Last Friday was broken machine day. Two of our six-sided hole machines, used for inserting holes and slots into furniture pieces, broke down.

For the first machine one of the spindles wouldn’t raise or lower. These are controlled electronically from the computer via an pneumatic system. We suspected either a computer issue, electrical, or a solenoid valve which controls the pneumatic air flow.

After a lot of troubleshooting I was convinced it was the solenoid valve. Our factory manager climbed up and checked on it.

Solenoid valves have these little buttons that allow you to activate them manually. This way you can see if the valve is working without an electrical trigger.

They have these little red LEDs telling you when it is being triggered. This is a “two headed” (no idea if that is what they are actually called) which allows airflow two ways. Often these are just open closed situations but this one allows airflow to move the spindle up, and another to move it down.

The electrical part was working, but the solenoid valve itself was borked. We heard it clicking but no movement.

We bought a new one at a local supply store but it was the wrong electrical type! These come in AC or DC varieties. The one we bought was for AC 220V but we needed it to work with DC 20V. Well it turns out we can just take the electrical ends from the broken valve and put it on the new valve. I’m going to research later to see why this is.

Anyways after a few hours we got that fixed. On to the next machine!

The next machine gave an error code, which is usually very helpful. The sensor on the lower spindle wasn’t responding, which usually mean the sensor is broken. These are little sensor that determine if a pneumatic cylinder is in the up or down position (or extended or collapsed depending on the orientation).

We replaced the sensor with a spare we have, but it still didn’t work! So time to trace wires.

We ended up finding out where one sensor wire was split into two, and one of the connections came off. This was nicely spliced too, but I think wear over repeated use made it come loose. We spliced it again and taped it up. I need to remember to bring my wire shrink insulation to the factory for the future.

Oh well it’s all good now! We’ve gotten quite good at troubleshooting these machines over the past year or so. We keep backup parts because a new part that isn’t available locally (like sensors) would take about 10 days to get here from China.

I’m quite surprised at how the old troubleshooting skills I learned from building PCs back in the day, or by working at an IT outsourcing company nearly 20 years ago helps with this stuff. It’s pretty much all the same.

Only 3% of American households are paying for an AI subscription. If LLMs make it so much faster and easier to get work done, why isn’t everyone using them?

This X post from Sam Altman sums it up nicely:

chatgpt work is remarkable, and “work” undersells it.

from my phone i sent:

“use all my chat history to figure out ideas for a long weekend trip with 8 friends, plan the best three options, make a full-stack site where the 9 of us can coordinate on what we would want to do in each place and decide where to go, and then after we get to group agreement make reservations. draft an email in my gmail i can send out to my friends when the site is ready.”

it…just worked.

I’m sure it did work. I’ve made so many personal utilities that are either single-use (i.e., throwaway) or that I use every week. I’m struck by the prompt.

Sam had to think through his plans, the range of options he wanted, the output he wanted, whether he needed coordination, and how he wanted to get the results. In other words, he thought like a builder. This is a mindset that, in my experience, is pretty uncommon.

When judging whether a product or service is techie, builder-oriented, or non-mainstream, I usually check with my wife. She is super smart, high-agency, and can generally look at something and immediately tell whether it will help her in life or work. I’ve been trying to teach her how to use ChatGPT Work to get things done for the past week or so. She has used ChatGPT for quick answers, brainstorming, and similar queries for some time now. But she hasn’t really used it to produce work yet.

I convinced her to try making a series of presentations for a course she is teaching. The process has been relatively bumpy, I must say. First, her prompt was too vague, and the LLM dutifully made everything in one go, with predictably middling results. Then I recommended that she start with one slide deck and, once she was happy with the result, ask ChatGPT to use it as a guide for the others. That worked better, but it took 30 minutes to make a single slide deck because of all the checks. It worked, but she was pretty underwhelmed. In her mind, she could have done a better job herself in maybe 45 minutes, so why wait 30? After making it herself, she would also understand the thought process behind the slides and be better able to present them. Fair point.

I think a better, chunkier, and more specific prompt at the beginning would have gotten her 90% of the way there, with perhaps 10 minutes spent writing out what she specifically wanted. But then I realized that she didn’t exactly know what she wanted until she started doing it. Well, I should say that she knew what she wanted, but she didn’t know how to articulate it as a series of instructions an LLM would understand.

Without a lot of coaching, I just can’t see more than a small percentage of people thinking this way. It’s weird. When they want to pay their bills, most people don’t think about the mechanics of finding the bills, paying them through a payment system, and recording that each bill was paid. Builders think about those things.

Until LLMs can do that builder-style thinking for ordinary users, I don’t see my wife, or people like her, choosing to turn to one most of the time.

Excellent post today from wakamoleguy about how LLms are bad at writing blog posts. Quick and simple argument about information theory that I believe is correct, and more importantly, this great snippet:

So let’s go back to the beginning: you have an idea, but writing is hard! Write anyways; write poorly. Use an LLM for research and feedback. Do not simply hand it your idea and expect it to generate something enjoyable or effective to read. And while you’re at it, lean into the imperfections, because even if they aren’t information-theoretical depictions of your ideas, they help make your writing interesting, unexpected, and yes, surprising. And that is what makes it fun to read.

I’m working on a longer form data driven blog post over on my nascent Substack publication, and I’m finding it hard to write like I used to. I’m writing then asking the LLM to give feedback. It often wants to make my data points overly specific, which I find annoying. Nevertheless it finds typos and areas I didn’t make all that much sense, so it’s helpful! This way I don’t have to bug a colleague or my wife to read it first!

Get to writing!

Neil Howe and Christian Ford write about ice cream consumption:

Over the last 50 years, US ice cream consumption has steadily declined. In 1975, the average American consumed 18.2 pounds of ice cream. In 2025, that figure fell to only 12.0 pounds. That’s a -34.1% decline.

This is quite sad to me. One of my fondest memories with my grandparents is eating dinner and having a scoop of vanilla ice cream afterwards (usually with chocolate syrup on top). My grandfather would have a scoop as well. Some doctors even believe it helps prevent Type II diabetes.