JEPA: What Is LeCun’s Architecture That Doesn’t Generate Pixels

⏱️ Reading time: 13 min

Yann LeCun has spent years insisting that generative models, the same ones that produce text or images token by token, are not the path toward artificial intelligence with common sense. On October 29, 2026, he published, together with Randall Balestriero, the mathematical proof of that intuition. Their paper on arXiv shows that the JEPA architecture they both designed converges to the same representation of the world regardless of the input modality, something that next-token prediction achieves with much less efficiency.

📑 En este artículo

1. TL;DR
2. What Is the JEPA Architecture?
3. Why LeCun’s Predictive Approach Matters
4. How JEPA Works Internally
5. Practical Examples: What a JEPA Prediction Looks Like in Code
6. How to Get Started with I-JEPA and V-JEPA
7. JEPA vs. Generative and Contrastive Models
8. Real Use Cases: From Image Classification to World Models
9. Common Mistakes and Best Practices
10. Going Deeper: The Platonic Hypothesis and the Representation Theorem
11. Frequently Asked Questions
    1. Does the JEPA Architecture Replace Generative Transformers?
    2. What’s the Difference Between I-JEPA and V-JEPA?
    3. Does JEPA Need Negative Pairs Like Contrastive Learning?
    4. Why Does Next-Token Prediction Align Worse with Vision, According to LeCun’s Paper?
    5. Can JEPA Be Used for Text, Not Just Images and Video?
    6. Can JEPA Generate New Images or Video?
12. References

That proof is not the subject of this article. What matters for a developer is understanding what makes this architecture different, one that Meta already uses in production in I-JEPA and V-JEPA. Its central bet is predicting representations instead of pixels.

TL;DR

• JEPA predicts the latent representation of a hidden part, without reconstructing pixels or tokens.
• Meta uses this design in I-JEPA for images and in V-JEPA for video, in production.
• A target encoder with EMA weights and stop-gradient prevents representational collapse.
• Positive views need 42 times fewer samples than next-token prediction to align representations.
• Unlike diffusion, JEPA predicts embeddings to understand the world, not to generate new content.

What Is the JEPA Architecture?

The JEPA architecture is a family of self-supervised learning models that predicts the latent representation of a missing part of the data from a visible part, without generating the original data pixel by pixel. LeCun proposed it in 2022 as an alternative to generative models.

LeCun presented the idea in his essay A Path Towards Autonomous Machine Intelligence, where he argued that an intelligent system needs a world model capable of predicting without spending compute on irrelevant details. Reconstructing every pixel of a video, for example, forces the model to learn camera noise or background textures that don’t help it understand the scene.

JEPA avoids that waste by predicting in the space of representations (embeddings) instead of the space of raw data. In practice, this means training two encoders that never share a direct gradient, and a predictor that connects both. The result is a model that learns what information matters in a scene without anyone telling it what’s important.

Why LeCun’s Predictive Approach Matters

The problem JEPA solves is concrete: training vision models without millions of human labels. Generative image models (diffusion, variational autoencoders) also learn without labels, but they spend their capacity reconstructing every texture and every pixel of noise, exactly what later needs to be ignored for tasks like classification or segmentation.

The Balestriero and LeCun paper puts numbers behind that intuition. Using a corpus of 145,000 image descriptions and 940 million web images, they found that positive views (two observations of the same scene) align representations using 42 times fewer samples than next-token prediction trained on the same text.

That explains why language models align with vision only at large scale. Next-token prediction is, according to the authors themselves, a JEPA with “bad views,” fragments of the same document instead of real observations of a shared scene. A text model with just 109 million parameters trained with good views aligned better with the DINOv2 and CLIP image encoders than any language model evaluated, including one with 72 billion parameters.

For anyone training models on a limited budget, the lesson is practical. Choosing an architecture that predicts in latent space saves compute and labeled data, two resources that are scarce outside large labs.

How JEPA Works Internally

The architecture has three pieces. The context encoder processes the visible part of the input, a block of patches from an image or a video clip. The target encoder processes the hidden part, but its weights are not trained by gradient, they’re updated as an exponential moving average (EMA) of the context encoder’s weights. The predictor takes the context encoder’s output and tries to guess the target encoder’s output.

flowchart TD
    A["Input (image or video)"] --> B["Context encoder"]
    A --> C["Target encoder (EMA)"]
    B --> D["Context embedding"]
    C --> E["Target embedding"]
    D --> F["Predictor"]
    F --> G["Predicted embedding"]
    G --> H["Loss in latent space"]
    E --> H

The training loss compares that prediction against the real target embedding, almost always with an L2 or smooth L1 distance, never against the original pixels. Since the target encoder never receives a direct gradient and the predictor sees less information than the target encoder, the system can’t memorize a constant output to cheat, and that detail is what prevents representational collapse, the most common failure in this type of training.

💭 Key point: Without the asymmetry between the context encoder and the target encoder (EMA plus stop-gradient), the model finds the shortest path: always predicting the same vector and minimizing the loss without learning anything.

I-JEPA divides an image into patches and hides several large, contiguous blocks, not random scattered patches. Hiding large blocks forces the predictor to reason about the structure of the scene, where an object ends, what’s behind it, instead of copying texture from the neighboring patch, which is what happens if the gap is tiny.

flowchart LR
    A["Image divided into patches"] --> B["Visible context block"]
    A --> C["Hidden target blocks"]
    B --> D["Context encoder"]
    C --> E["Target encoder"]
    D --> F["Predictor predicts C's embeddings"]
    E --> G["C's real embeddings"]
    F --> H["Compare with L2 loss"]
    G --> H

V-JEPA extends the same idea to time. Instead of hiding only a spatial region, it hides blocks that span several consecutive frames, so the predictor has to model both the structure of the scene and its evolution, the first step toward what Meta calls a world model.

Practical Examples: What a JEPA Prediction Looks Like in Code

JEPA’s core mechanism, predicting one embedding from another, can be illustrated with a minimal PyTorch snippet. This isn’t the actual code from Meta’s repositories, it’s a skeleton that shows the shape of the problem.

import torch
import torch.nn as nn

class Predictor(nn.Module):
    def __init__(self, dim=768):
        super().__init__()
        self.proj = nn.Linear(dim, dim)

    def forward(self, context_embedding, mask_tokens):
        return self.proj(context_embedding + mask_tokens)

context_embedding = torch.randn(1, 196, 768)   # embeddings of the visible block
mask_tokens = torch.randn(1, 196, 768)         # positional tokens of the hidden block
predictor = Predictor()

predicted_embedding = predictor(context_embedding, mask_tokens)
print(predicted_embedding.shape)

The output is torch.Size([1, 196, 768]): one batch, 196 patches (14×14 in a 224×224 image with 16-pixel patches), and 768 dimensions per embedding. The real target embedding must have that same shape so they can be compared with the loss function.

How to Get Started with I-JEPA and V-JEPA

To run the official code you need Python 3.9 or higher, Git, and, for serious training, a GPU with CUDA support. To just load a pretrained checkpoint, a CPU is enough, though much slower. The following steps are for Linux and macOS; on Windows they run the same way inside WSL.

git clone https://github.com/facebookresearch/ijepa.git
cd ijepa
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

For V-JEPA the repository is different: git clone https://github.com/facebookresearch/jepa.git, followed by the same three steps of setting up a virtual environment and installing dependencies.

There’s no single generic command to confirm the installation, because each repository exposes its own evaluation script. The sign that everything loaded correctly is that script finishing and printing the final metric (the linear probe accuracy, in the case of I-JEPA) instead of breaking with a traceback of missing dependencies.

JEPA vs. Generative and Contrastive Models

JEPA isn’t the only way to learn without labels. Compared with two other families, the difference becomes clearer.

flowchart TD
    subgraph Generativo["Generative model"]
    P1["Partial input"] --> P2["Decoder"]
    P2 --> P3["Reconstruction in pixel space"]
    end
    subgraph JEPA["JEPA Architecture"]
    J1["Partial input"] --> J2["Predictor"]
    J2 --> J3["Prediction in latent space"]
    end

The choice isn’t ideological. If the final goal is generating an image or text, a generative model is still the right tool. If the goal is for a system to understand a scene in order to later act on it, a robot, a self-driving car, predicting in latent space pays off more per training sample.

Real Use Cases: From Image Classification to World Models

Meta released I-JEPA in 2023 as its first vision model trained with this architecture, designed for classification and segmentation tasks with few available labels. The original paper showed it competing with contrastive models while using a fraction of their training compute.

V-JEPA, released in 2024, extended the idea to video for tasks like action recognition and understanding physical motion. The V-JEPA paper presents it as a step toward models that understand cause and effect in the physical world, what LeCun describes as “world models”: systems capable of predicting what happens after an action without having seen that exact sequence before.

The most concrete use case today is pretraining: training first with JEPA on unlabeled data and then fine-tuning a small head for the final task, with far fewer labeled images or videos than a model trained from scratch would need.

Common Mistakes and Best Practices

The most cited error in self-supervised learning literature is representational collapse: the model learns to produce the same embedding for any input, minimizing the loss without learning anything useful. JEPA avoids it with the asymmetry between encoders, but that asymmetry isn’t automatic. If the target encoder updates too fast (an EMA with low momentum) or too slow, the risk of collapse returns.

Another common mistake is confusing JEPA with contrastive learning. SimCLR and CLIP need negative pairs, examples that should NOT look alike, and carefully designed data augmentations. I-JEPA and V-JEPA don’t use explicit negatives; the architectural asymmetry does that job.

⚠️ Watch out: The real limitation, and the reason JEPA doesn’t replace generative models, is that its embeddings can’t be used to generate content directly. If the project needs an image generator or a chatbot, a generative model is still the right piece.

Going Deeper: The Platonic Hypothesis and the Representation Theorem

The strongest result in the Balestriero and LeCun paper is what they call the platonic representation theorem. The authors show that two JEPAs trained separately on different modalities, text and image, for example, converge to the same representation up to a rotation, as long as their positive pairs are “good views,” two real observations of the same scene. With bad views, arbitrary fragments of a document, the convergence has a bounded limit but doesn’t disappear.

The practical consequence, not just theoretical, is that this rotation can be calculated without retraining anything. The authors took a text encoder and an image encoder trained separately, calculated the rotation that aligns them, and built a caption generator that achieved a CIDEr score of 0.838 on COCO, without the two encoders ever having seen an image-text pair during training. On a pairless benchmark, that same text JEPA outperformed the zero-shot performance of every language model evaluated up to 8 billion parameters, with 64% against a maximum of 56% top-5 on ImageNet-100.

This doesn’t mean the JEPA architecture completely resolves LeCun’s platonic hypothesis. The paper itself clarifies that the misalignment bound depends on how “good” the view is, and building good views for each new modality, audio, robot sensors, remains an open design problem, not a universal recipe.

Your next step: clone the I-JEPA repository, load a pretrained checkpoint, and run the linear probing script on your own small dataset to see how many labels you actually need for acceptable accuracy.

Frequently Asked Questions

Does the JEPA Architecture Replace Generative Transformers?

No. JEPA solves the problem of learning representations without labels; generative transformers remain the tool when the final goal is producing new text, images, or audio. They’re complementary, not direct rivals.

What’s the Difference Between I-JEPA and V-JEPA?

I-JEPA works on static images and hides spatial blocks of patches. V-JEPA extends the idea to video, hiding blocks that extend over time so the model also learns the dynamics of a scene.

Does JEPA Need Negative Pairs Like Contrastive Learning?

No. SimCLR and CLIP compare positive pairs against explicit negatives. I-JEPA and V-JEPA avoid collapse through the asymmetry between the context encoder and the target encoder, without needing negatives.

Why Does Next-Token Prediction Align Worse with Vision, According to LeCun’s Paper?

Because its training pairs are fragments of the same text document, not two real observations of a shared scene. The paper describes it as a JEPA with “bad views,” and that’s why it needs much more scale to align with image models.

Can JEPA Be Used for Text, Not Just Images and Video?

Yes. The Balestriero and LeCun paper itself trains a 109-million-parameter text JEPA with positive views and compares it directly against language models with up to 72 billion parameters.

Can JEPA Generate New Images or Video?

Not directly. Its embeddings describe a scene, but they aren’t designed to reconstruct pixels. Generating new content from those embeddings requires coupling a decoder or a separate generative model.

References

• Balestriero and LeCun, “Proving and Improving the Platonic Representation Hypothesis with JEPAs” (arXiv, 2026): the paper that motivates this article.
• Assran et al., “Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture” (arXiv, 2023): the original I-JEPA paper.
• Bardes et al., “V-JEPA: Video Joint-Embedding Predictive Architecture for Self-Supervised Visual Learning” (arXiv, 2024): the original V-JEPA paper.
• Official I-JEPA repository on GitHub: code and pretrained checkpoints.
• Official V-JEPA repository on GitHub: code and pretrained checkpoints.

📱 Like this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.

Featured image: Foto de Logan Voss en Unsplash