⏱️ Lectura: 12 min
Meta and AMD just solved one of the most expensive problems in large-scale language model training: what to do when a node goes down in the middle of a job that has been running for days. The answer is called PyTorch Monarch, and as of July 6, 2026, it also runs on AMD Instinct GPUs with ROCm, not just on NVIDIA hardware with CUDA.
📑 En este artículo
Before this port, a memory error on a single GPU forced a full cluster restart from the last checkpoint. With Monarch, healthy nodes keep training while the failed node recovers and rejoins the job, without losing accumulated progress.
TL;DR
- AMD and Meta ported PyTorch Monarch to Instinct GPUs with ROCm, extending the runtime beyond CUDA, according to the July 6, 2026 announcement.
- The port uses hipify_torch to convert the CUDA C++ bridge to HIP and links against RCCL instead of NCCL.
- The Rust runtime runs on Tokio and coordinates actors, process meshes, and tensor sharding.
- ROCm has no static equivalent to libcudart_static.a: Monarch links amdhip64 dynamically instead of statically.
- The hierarchical failure model isolates crashes by actor: local recovery takes seconds, not minutes.
- RDMA integration stays on libibverbs; with GPU_PLATFORM=rocm only the GPU bindings change from CUDA to HIP.
- AMD’s prior work had already achieved 96.16% scaling efficiency on a 1024 MI325 GPU cluster training DeepSeek-V3-671B.
Introduction
PyTorch Monarch is an experimental runtime that lets a single Python program orchestrate hundreds or thousands of GPUs at once, as if they were a single machine. Instead of writing explicit distributed code for each node, the developer writes a sequential script and Monarch handles distributing it, running it in parallel, and recovering it if something fails.
The joint announcement from AMD and Meta on July 6, 2026 documents how Monarch’s GPU runtime and distributed communication stack were ported from CUDA to ROCm, AMD’s compute platform for its Instinct accelerators. This is the first time Monarch’s single-controller model has left the NVIDIA ecosystem.
What Happened
AMD’s engineering team (Chaojun Hou, Liz Li, Zachary Streeter, Xinyu Kang, Lei Zhang, Yuankai Chen, Yao Fu, Wen Chen, Zhenyu Gu, and Andy Luo), together with the Monarch team at Meta (Matthias Reso, Hamid Shojanazeri, and collaborators), published the full port on the official PyTorch blog on July 6, 2026.
The work touched three distinct layers of the stack:
- Collective communications. They used the
hipify_torchtool to automatically convert the CUDA C++ bridge to HIP, and linked the result against RCCL, AMD’s library that replicates the NCCL API. - GPU memory management. They extended the build system to detect the platform at compile time and redirect each call to the CUDA driver toward its HIP equivalent.
- RDMA integration. With the
GPU_PLATFORM=rocmvariable, the RDMA path based onlibibverbsremains intact; the only thing that changes are the GPU bindings, from CUDA to HIP, for GPU-direct transfers.
Two design decisions stand out in the port. First, ROCm has no static equivalent to libcudart_static.a: while CUDA links cudart_static directly, the ROCm build links amdhip64 dynamically, and both platforms use dlopen for GPU driver functions (hipMemCreate, cuMemCreate, and similar), which keeps the same runtime contract on both sides.
Second, instead of maintaining two separate versions of the Rust bindings, the team let hipify_torch rewrite the C/C++ headers and had bindgen automatically generate the types with HIP names (hipError_t, hipDeviceptr_t, hipStream_t). A compatibility shim in Rust unifies these types with their CUDA equivalents, so there is a single source of truth instead of two diverging forks.
💡 Tip: if you’re building Monarch from source, the GPU_PLATFORM=rocm variable is the only switch you need to touch: the rest of the platform detection is automatic in the build system.
Background and Context
The problem Monarch tackles is not new. Periodic checkpointing, saving the model’s complete state to storage at set intervals, is the standard fault-tolerance strategy in distributed training. It works, but it has a cost that grows with scale:
| Traditional Checkpointing Problem | Impact |
|---|---|
| Write overhead | Saving hundreds of gigabytes of state consumes time and I/O bandwidth |
| Lost compute | All progress since the last checkpoint is lost with every failure |
| Idle cluster | The entire cluster waits while the failed node is replaced and the job restarts |
| Scalability limit | The larger the cluster, the higher the chance a failure will occur within any given checkpoint interval |
This isn’t an abstract problem for AMD and Meta. In prior work, AMD had already demonstrated near-linear scaling for FP8 training, with 96.16% scaling efficiency on a 1024 MI325 GPU cluster training DeepSeek-V3-671B. Scaling compute stopped being the bottleneck: surviving failures while scaling is what remained to be solved.
Monarch’s architecture separates two things that used to be coupled: the parallelism strategy within each training replica, and the fault-tolerance mechanism across replicas. That separation is what allows a failure to be isolated instead of propagating to the entire cluster.
Technical Details and Performance
Monarch’s architecture operates on four levels:
- Python API. A single program interface: the developer writes sequential Python code and gets distributed GPU execution.
- Monarch Runtime. Manages actors and process meshes, supervision trees, and tensor sharding.
- Rust Runtime on Tokio. Guarantees performance and memory safety at the execution layer.
- Infrastructure. Integrates with RDMA, RCCL/NCCL, SLURM, Kubernetes, and SkyPilot.
The actor model is the key piece. Each actor has private state, so a crash doesn’t automatically propagate to other actors. Failures are handled at the lowest possible level of a supervision hierarchy, and recovery is fast: seconds for a local restart, minutes only if the failure escalates up the supervision tree.
flowchart TD
A["Python API"] --> B["Monarch Runtime: actors and process mesh"]
B --> C["Rust Runtime (Tokio)"]
C --> D["RDMA / RCCL / NCCL"]
D --> E[("AMD Instinct or NVIDIA GPU Cluster")]
subgraph Infrastructure
D
E
endThe following table summarizes where each component changed when moving from CUDA to ROCm:
| Component | CUDA Path (NVIDIA) | ROCm Path (AMD) |
|---|---|---|
| Collective communications | Direct link against NCCL | hipify_torch converts the C++ bridge and links against RCCL |
| GPU memory management | Native calls to the CUDA driver (cuMemCreate) | The build system redirects to the HIP equivalents (hipMemCreate) |
| RDMA integration | CUDA bindings over libibverbs | Same libibverbs path, GPU bindings changed to HIP |
| Runtime linking | Static against libcudart_static.a | No static equivalent; amdhip64 links dynamically |
⚠️ Heads up: if your build pipeline assumes a CUDA-style static runtime, you’ll need to adjust your packaging: ROCm depends on amdhip64 as a dynamic library, so the final binary needs that dependency available at runtime, not just at compile time.
Getting Started
Monarch is still an experimental project, so trying it out today means building it from source and pointing the build at the right backend.
Linux with ROCm (Supported Path)
# Clone the repository and enter the directory
git clone https://github.com/pytorch-labs/monarch.git
cd monarch
# Point the build at the AMD runtime instead of CUDA
export GPU_PLATFORM=rocm
export ROCM_PATH=/opt/rocm
pip install -e .macOS (Without a Real AMD GPU)
ROCm doesn’t run natively on macOS. To work on Monarch’s Python code without executing real kernels, install the package in CPU-only mode and connect to a remote Linux cluster for training runs:
pip install -e . --no-deps
# GPU calls are dispatched to the remote cluster via SSH/SLURMWindows
Windows also lacks native ROCm support. The practical path is to use WSL2 with GPU passthrough (still experimental for AMD) or connect via SSH to a Linux node on the training cluster.
A minimal example, a stateful actor running locally:
from monarch.actor import Actor, endpoint, this_proc
class Contador(Actor):
def __init__(self):
self.valor = 0
@endpoint
def incrementar(self, paso: int) -> int:
self.valor += paso
return self.valor
proc = this_proc()
contador = proc.spawn("contador", Contador)
print(contador.incrementar.call(5).get())This actor keeps its state (self.valor) in an isolated process. Calling it from the main script looks just like calling a regular function, even though internally it travels through the Rust runtime.
A more realistic example, a process mesh that survives the failure of a trainer:
from monarch.actor import Actor, endpoint
from monarch.mesh import ProcessMesh
class EntrenadorGPU(Actor):
@endpoint
def paso_entrenamiento(self, batch):
return self.modelo.step(batch)
mesh = ProcessMesh.from_slurm(nodos=64, gpus_por_nodo=8)
entrenadores = mesh.spawn("entrenador", EntrenadorGPU)
try:
resultados = entrenadores.paso_entrenamiento.call(lote_actual).get()
except ActorCrashError as fallo:
mesh.reemplazar_nodo(fallo.nodo_id)
resultados = entrenadores.paso_entrenamiento.call(lote_actual).get()When an EntrenadorGPU actor crashes, the exception is caught in the controller process, not in the other 511 processes in the mesh. The controller replaces the node and retries that step, without touching the rest of the training.
To confirm your build linked against ROCm and not CUDA, check the compiled binary’s dynamic dependencies:
ldd $(python3 -c "import monarch, os; print(os.path.dirname(monarch.__file__))")/_monarch_rust*.so | grep -i hipIf the build is correct, that line should list libamdhip64.so. A CUDA build would instead show libcudart.so.
Impact and Analysis
For AMD, this port is one more piece of a broader goal: making distributed training software independent of whether the cluster is NVIDIA. Every framework ported to ROCm lowers the cost of switching GPU vendors for a research team, which is especially relevant when AI compute demand outstrips available GPU supply.
For Meta, expanding Monarch beyond CUDA validates its bet on the single-controller model as a general paradigm, not a solution tied to a single hardware vendor. If the same Python script can orchestrate both NVIDIA and AMD GPUs, a research team can move between clusters without rewriting its training, evaluation, and reinforcement learning logic.
That said, the port has limits worth naming. RCCL replicates the NCCL API, but it’s not a perfect mirror: performance parity on each collective (all-reduce, all-gather, reduce-scatter) depends on how optimized each specific operation is in RCCL, and that can vary by cluster topology. Also, Monarch is still an experimental runtime: the actor model adds a layer of indirection that makes sense at large scale, but is unnecessary complexity for single-node training, where a traditional script with torch.distributed is still simpler to debug.
💭 Key point: Monarch’s real gain isn’t compute speed, it’s cluster utilization: fewer paid GPU-hours sitting idle waiting for a restart.
What’s Next
AMD and Meta describe this port as a step toward bringing the Monarch runtime to a broader hardware ecosystem, suggesting ROCm won’t be the last non-CUDA backend. The decoupled design between the Python API, the Rust runtime, and the network infrastructure makes it easier to add new communication backends without rewriting the layer the developer sees.
In the short term, work will most likely focus on closing the parity gap between RCCL and NCCL on the collective operations that matter most for large-scale training, and on better documenting the integration with SLURM and Kubernetes on ROCm, given that both orchestrators are already supported at Monarch’s infrastructure layer.
📖 Summary on Telegram: View summary
If you have access to a node with an AMD Instinct GPU and ROCm installed, clone the repository and run the actor example with GPU_PLATFORM=rocm: it takes less than ten minutes to confirm whether your local setup is ready to try Monarch.
Frequently Asked Questions
What is PyTorch Monarch?
It’s an experimental runtime from Meta that lets you control an entire GPU cluster from a single Python program, using an actor model and process meshes instead of explicit per-node distributed code.
Why did it need to be ported to ROCm?
Because the original runtime only ran on CUDA. Porting it to ROCm allows it to be used on AMD Instinct GPU clusters, the hardware behind large-scale training projects like AMD’s MI325 cluster.
What does single-controller mean in this context?
It means a single Python program, running in a controller process, orchestrates execution across all nodes in the cluster, instead of each node independently running its own copy of the training script.
Does Monarch replace NCCL or RCCL?
No. Monarch relies on them: it uses NCCL on NVIDIA GPUs and RCCL on AMD GPUs. Monarch is the orchestration and fault-tolerance layer on top of those collective communication libraries.
Can Monarch be used outside of Meta today?
The code is available as an open project and can be built from source, but AMD and Meta describe it as a runtime under active development, not yet a stable, general-purpose product.
What’s the difference between a traditional checkpoint and Monarch’s recovery?
A traditional checkpoint restarts the entire job from the last saved point when a node fails. Monarch isolates the failure in the affected actor and lets the rest of the cluster keep training while that node recovers.
References
- PyTorch Blog: official announcement of the Monarch port to ROCm, published on July 6, 2026 by the AMD and Meta teams.
- GitHub: pytorch-labs/monarch: the Monarch project repository.
- ROCm Documentation: official documentation for AMD’s ROCm platform.
- PyTorch: the framework’s official site and technical blog.
📱 Enjoy this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.
Imagen destacada: Foto de Caspar Camille Rubin en Unsplash
0 Comments