⏱️ Lectura: 12 min
On August 23, 2026, Paul Graham posted a one-sentence reply on X that racked up more than 642,800 views and 11,000 likes in under 24 hours: if he were 17, he’d set aside applications and learn to build LLMs from scratch, training the most powerful models the hardware he could access would allow.
📑 En este artículo
The reply sparked an exchange with Yann LeCun, a scientist renowned for his contributions to deep learning, who proposed a different path: understand why an LLM can write an essay but can’t clean a room, and use that question to search for architectures that go beyond transformers. The clash between these two positions matters for any developer today deciding between using an AI API or digging into how it actually works under the hood.
TL;DR
- Paul Graham posted a tweet on August 23, 2026 that surpassed 642,800 views and 11,000 likes in under 24 hours.
- Graham states that at 17 he’d learn to build LLMs from scratch and train them with the best hardware available.
- He clarified in the thread that he would NOT found a startup at 17: he’d first build a deep knowledge base about LLMs.
- Yann LeCun responded with an opposing stance: study why an LLM can’t clean a room, and look beyond transformers.
- The exchange reopens the debate between mastering current architecture versus searching for new architectures beyond LLMs.
- nanoGPT and llm.c, projects by Andrej Karpathy, let you train a GPT-style model from scratch in a few hundred lines.
- The paper ‘Attention Is All You Need’ (2017) remains the technical foundation of every modern LLM, including GPT and Claude.
- For developers in Latin America, the realistic starting point is accessible GPUs like Colab, Kaggle, or vast.ai, not owning clusters.
What happened
It all started with a simple question someone asked Paul Graham: what would he do if he were 17 today. His reply, posted on August 23, 2026 on X, was direct: he’d learn to build LLMs from scratch and train the most powerful models he could with whatever hardware he had access to. The tweet racked up 642,800 views, 450 replies, 614 reposts, 11,000 likes, and 5,000 bookmarks in under a day, an unusual level of reach even for someone with Graham’s audience.
Sixteen hours later he expanded on the idea in a second tweet: what he would NOT do is try to found a startup at 17. In his words, he’d first build the knowledge base needed to found something later, because the business ideas that come from deeply understanding LLMs are categorically better than the ones a teenager can have with the general knowledge he had at that age.
The most-discussed reply came from Yann LeCun, who framed the problem from a different angle: he’d try to understand why LLMs can write essays but can’t clean a room, and use that gap as a guide for what to study in college and grad school. His proposal points toward architectures and methods that can learn to act in the physical world as fast as an LLM learns text patterns, something current transformers don’t handle well.
Context and background
Graham isn’t an AI academic: he’s the co-founder of Y Combinator and one of the most widely read essayists in the startup world, with pieces like How to Start a Startup that shaped a generation of founders. That his advice for a 17-year-old in 2026 is to learn to build LLMs from scratch rather than code applications or go straight to founding a company marks a notable shift from his own historical message, which centered on shipping products fast and getting users.
Yann LeCun, for his part, is Meta’s chief AI scientist and one of the three researchers who won the 2018 Turing Award for laying the foundations of modern deep learning, alongside Geoffrey Hinton and Yoshua Bengio. He’s also one of the most consistent critics of current LLM limitations: he’s argued for years that autoregressive text learning isn’t enough to build systems with physical reasoning or real planning, and has publicly pushed alternative architectures centered on world models.
The contrast between the two isn’t just a difference of opinion between two influential figures: it reflects a real tension within the AI community in 2026 over whether the next decade of progress comes from scaling transformers or from pursuing something different.
Technical details and performance
When Graham says “build LLMs from scratch,” he doesn’t mean using an API, but implementing every piece of the process: a tokenizer, an embedding table, attention layers, and a training loop with gradient descent. The most cited reference for doing this in practice is nanoGPT, by Andrej Karpathy, a repository that reproduces the GPT-2 architecture in a few hundred lines of Python and PyTorch, and its even more minimalist version llm.c, which does the same in pure C with no framework dependencies.
The starting point for any LLM is turning text into numbers. A character tokenizer is the simplest possible version:
# minimal character tokenizer, the base of any LLM built from scratch
text = open("corpus.txt", encoding="utf-8").read()
chars = sorted(set(text))
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for ch, i in stoi.items()}
encode = lambda s: [stoi[c] for c in s]
decode = lambda ids: "".join(itos[i] for i in ids)
data = encode(text)
print(f"vocabulary: {len(chars)} symbols, corpus: {len(data)} tokens")This turns each character in the corpus into an integer. Production LLMs use subword tokenizers (BPE), but the principle, mapping symbols to indices, is the same. With those indices as input, the next block is the heart of the transformer architecture: causal self-attention.
import torch
import torch.nn as nn
class TransformerBlock(nn.Module):
def __init__(self, embed_dim=384, n_heads=6, context=256):
super().__init__()
self.attention = nn.MultiheadAttention(embed_dim, n_heads, batch_first=True)
self.norm1 = nn.LayerNorm(embed_dim)
self.mlp = nn.Sequential(
nn.Linear(embed_dim, 4 * embed_dim),
nn.GELU(),
nn.Linear(4 * embed_dim, embed_dim),
)
self.norm2 = nn.LayerNorm(embed_dim)
mask = torch.triu(torch.ones(context, context), diagonal=1).bool()
self.register_buffer("causal_mask", mask)
def forward(self, x):
t = x.shape[1]
atn, _ = self.attention(x, x, x, attn_mask=self.causal_mask[:t, :t])
x = self.norm1(x + atn)
x = self.norm2(x + self.mlp(x))
return xThis block is the building unit stacked 6, 12, or even 96 times depending on model size. The causal mask forces each token to only “see” previous tokens, the property that makes a transformer an autoregressive model like GPT, capable of predicting the next token one at a time. This same idea, formalized in the 2017 paper Attention Is All You Need, is the technical foundation shared by GPT, Claude, and virtually every modern LLM.
| Path | When to use it | Advantage | Limitation |
|---|---|---|---|
| Use an API (OpenAI, Anthropic) | Build a product fast without touching model weights | Zero infrastructure, immediate results | Doesn’t teach how the model works internally |
| Fine-tuning an open model | Adapt an existing LLM (Llama, Qwen) to your own domain | Requires far less compute than training from scratch | Still depends on someone else’s architecture decisions |
| Build from scratch (nanoGPT, llm.c) | Learn the transformer architecture in depth, as Graham suggests | Deep understanding of tokenization, attention, and training | Requires your own GPU, time, and tolerance for debugging gradients |
flowchart TD
A["Raw text"] --> B["BPE or character tokenizer"]
B --> C["Embeddings + position"]
C --> D["Transformer blocks attention and MLP"]
D --> E["Output layer logits"]
E --> F(("Next token prediction"))⚠️ Heads up: training an LLM competitive with GPT-5 or Claude isn’t feasible with a single home GPU: the difference lies in the scale of data and compute, not just the model code.
How to get started/try it
The shortest path to putting Graham’s advice into practice is cloning nanoGPT and training a small model on a toy corpus like tinyshakespeare. The steps are the same on Windows, macOS, and Linux once Python 3 and PyTorch are installed:
# Windows (PowerShell), macOS, and Linux: same commands, python3 on Unix
git clone https://github.com/karpathy/nanoGPT.git
cd nanoGPT
pip install torch numpy transformers datasets tiktoken wandb tqdm
# prepare the sample dataset (tinyshakespeare)
python data/shakespeare_char/prepare.py
# train a small model on local CPU or GPU
python train.py config/train_shakespeare_char.py --device=cpu --compile=FalseOn a laptop without a GPU, training is slow but works; on a free T4 GPU from Google Colab or Kaggle Notebooks, the same experiment drops from hours to minutes. For larger models, services like vast.ai or RunPod rent GPUs by the hour without committing to your own hardware, the most realistic route for a developer in Latin America who wants to experiment without buying a $2,000 card.
To confirm training is actually learning, the nanoGPT script prints the loss every so many iterations: if it drops steadily (for example from 4.5 to 1.8 in the first thousand iterations on tinyshakespeare), the model is adjusting its weights correctly. Generating sample text with python sample.py --out_dir=out-shakespeare-char is the most direct way to verify the model produces something coherent with the training corpus.
💡 Tip: you can train a small version of nanoGPT on tinyshakespeare for free on a Google Colab T4 GPU in under an hour, at no cost.
Impact and analysis
Graham’s tweet, beyond its viral reach, puts into words a concrete question many junior developers face: whether it’s worth becoming an expert at using AI APIs to build products fast, or investing time in understanding the transformer architecture in depth. His argument is that the best business ideas of the next decade will come from people who understand LLMs internally, not from people who only know how to call them via API.
LeCun’s reply adds an important nuance: building LLMs from scratch today means reproducing a 2017 architecture that’s already well understood, not necessarily standing at the frontier of research. His personal bet, tied to the world models he’s publicly championed for years, is that the next leap in capability won’t come from scaling transformers but from architectures that learn from physical interaction with the environment, not just from text.
Neither position is incompatible with the other in practice: understanding how a transformer works internally (tokenization, attention, training) is the necessary foundation for later questioning its limits or proposing something different, which is exactly what LeCun suggests doing in college. The real difference lies in the order and the goal: Graham points toward founding something with that knowledge, LeCun points toward researching beyond it.
What’s next
The exchange doesn’t change anything immediately in the industry, but it is a signal of where two influential figures are looking in 2026: one toward the practical application of LLMs to found businesses, the other toward research into architectures that overcome their current limits. Both conversations are likely to keep fueling the growth of open educational resources like nanoGPT and llm.c, which are already standard references in university courses and machine learning bootcamps.
For a developer just starting out, the practical takeaway is less dramatic than the debate on X: having the technical foundation of how an LLM works (tokenization, attention, training) is useful regardless of whether the end goal is founding a startup on that base or researching what comes after transformers.
📖 Summary on Telegram: View summary.
Try it yourself: clone nanoGPT and run python train.py config/train_shakespeare_char.py today to watch an LLM training from scratch on your own machine.
Frequently Asked Questions
What does “building an LLM from scratch” actually mean?
Implementing every component of the model (tokenizer, embeddings, attention blocks, training loop) instead of using an API or an already-trained model. Projects like nanoGPT and llm.c show the full process in readable code.
Do I need an expensive GPU to get started?
Not for a small practice model. Google Colab and Kaggle Notebooks offer free GPUs powerful enough to train a GPT-style model on a small corpus like tinyshakespeare in under an hour.
What is nanoGPT and who made it?
It’s an open source repository by Andrej Karpathy (former OpenAI and Tesla scientist) that reproduces the GPT-2 architecture in a few hundred lines of PyTorch, designed as reproducible educational material.
Why does Yann LeCun disagree with Paul Graham?
LeCun argues that current LLMs, despite writing fluent text, don’t handle physical reasoning or real-world planning tasks well, and proposes studying architectures beyond transformers rather than perfecting the current ones.
Does this advice apply if I’m no longer 17?
Yes: the technical foundation (tokenization, attention, training) is the same regardless of age. What changes is the time horizon available to invest in learning it before applying it to a concrete project.
Where can I learn the transformer architecture step by step?
The original paper Attention Is All You Need, the nanoGPT code, and the PyTorch documentation on multi-head attention are the most direct starting points.
References
- Paul Graham’s original tweet on X: the August 23, 2026 post about what he’d do at 17.
- nanoGPT on GitHub: educational implementation of a GPT-2-style model in PyTorch.
- Attention Is All You Need (arXiv, 2017): the paper that introduced the transformer architecture.
- PyTorch MultiheadAttention documentation: technical reference for the attention module used in the example.
📱 Enjoying this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day. @programacion
0 Comments