# PyTorch 2.15: the NCCL rewrite becomes the default, and what a CPU box can check

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/pytorch-2-15
> date: 2026-10-31
> tags: pytorch, distributed-training, nccl, fsdp, torch-compile, apple-silicon

PyTorch 2.15 came out on 28 October. The [release blog](https://pytorch.org/blog/pytorch-2-15-release-blog/) counts 3,117 commits from 583 contributors since 2.14, and the @PyTorch post announcing it led with the distributed work, NVGEMM, Apple Silicon and accelerator support. A release this size is mostly a list, and a list is hard to read for what matters to you. So I did two things. I grouped what the blog names and picked the changes I think carry weight. Then I took the claims that a CPU box can see and checked them against the real wheels.

<RepoCard repo="pytorch/pytorch" />

Labels used throughout: **measured** is something I ran or counted myself, **reported** is the blog's or the authors' statement that I did not re-run, **reasoned** is my inference from either. My runs are on a shared 4-vCPU cloud VM with no GPU, CPython 3.12, two threads. I installed `torch==2.14.1+cpu` and `torch==2.15.0+cpu` from the PyTorch CPU index into separate directories and ran the same scripts against each. I could not read the full GitHub release notes: the sandbox returns 403 for `github.com`, so everything below about what changed comes from the blog, which names five backwards-incompatible changes, and from the wheels themselves.

## What the blog actually says, by area

The blog has fifteen feature sections and five deprecation notes. I counted the distinct PR numbers cited under each feature section and grouped the sections by who they serve.

<ChangeMap />

**Measured.** The sections cite 108 PR numbers in total. Against 3,117 commits, the blog names about 3.5% of the release, and a PR cited in two sections is counted twice, so that is a ceiling. The counts say where the writing effort went, not where the engineering went. Apple Silicon draws more citations (25) than NVIDIA (21), which tells you who wrote a lot of small PRs more than it tells you the relative importance of Apple's GPU.

Two things are absent from the blog's feature list that you might expect from the previous three releases. There is no `torch.compile` or Inductor performance section on its own. The compiler shows up inside other items: Dynamo gets Python 3.15 support, Inductor autotunes FlyDSL templates on AMD, and NVGEMM is a GEMM backend that Inductor can choose. And there is no training-speed headline number for any model. The numbers the blog does give are all on Apple Silicon.

## The six that matter

I picked these by asking what changes the behaviour of a job I would already be running, then what changes what I can run at all, then what is new but narrow.

### 1. The rewritten NCCL backend is now the default

**Reported.** In 2.14 the rewritten backend, ported from torchcomms, was an opt-in preview called `nccl2`. In 2.15 every process group created with the backend name `nccl` uses it. Two rollback paths exist: set `TORCH_DIST_USE_NCCL2=0`, or ask for the `nccl-legacy` backend by name. Communicators are now created eagerly with the process group instead of lazily at the first collective. Reconfiguration, the step where a failed rank is replaced or the group shrinks and grows, can opt into a fast path that needs NCCL 2.29 or newer, so surviving ranks skip a full bootstrap. The same reconfiguration is enabled on ROCm with RCCL 2.30.7.

The second half of this change is debuggability. A backend-agnostic flight recorder writes a post-mortem to disk when a collective hangs, and watchdog health is published to the debug server, so a timeout on any rank makes every rank dump its recorder. Anyone who has chased a hang on 512 GPUs knows why: the rank that times out is rarely the rank that caused it. All of this is labelled **API Unstable** in the blog.

Here is the resolution table as I understand it from the blog. It is reported, not run.

<BackendSwitch />

**Why it matters, reasoned.** Making a rewrite the default is a bet that the preview period found the problems. Eager communicator creation changes startup behaviour: failures that used to show up at the first all-reduce now show up when the group is created, which is better, and also means a job with a misconfigured network fails earlier and in a different place than your runbook says. If you operate training clusters, the first thing I would do is run one representative job with `TORCH_DIST_USE_NCCL2=0` and one without, and diff the logs.

**Measured, and its limit.** I could not exercise any of this. The CPU wheel reports `dist.is_nccl_available() == False`, and `torch.distributed.Backend.backend_list` on 2.15 is `undefined, gloo, nccl, xccl, ucc, mpi, fake` on both wheels. What I can say from the Python sources: the string `nccl2` appears in one file under `torch/distributed` in 2.14.1 and in four in 2.15.0, `TORCH_DIST_USE_NCCL2` and `nccl-legacy` each appear once in both, and the 2.15.0 wheel has two new directories, `_transport` and `_nccl4py`. That fits the blog and proves nothing about speed or stability. The flight recorder was already a package in 2.14.1, so the change is in what feeds it and when it dumps, not that it exists.

### 2. One-sided transports and a NIXL backend

**Reported.** `torch.distributed` gains a Python transport API for one-sided data movement: one side reads or writes the other's memory without the other posting a matching receive. Peers connect through a store and need neither a process group nor collectives. `Transport.read_stream` and `write_stream` enqueue the transfer on a CUDA stream, so it can be captured by `torch.cuda.graph`. NIXL is the first backend. **Measured:** the `_transport` directory is in the 2.15.0 wheel and absent from 2.14.1. The leading underscore and the **API Unstable** label say what to expect.

**Reasoned.** This is the building block for disaggregated serving, where a prefill machine pushes a KV cache to a decode machine, and for parameter servers that never stall a trainer. Neither needs a collective, and a collective forces every participant to show up. That the transfer is graph-capturable is the interesting part: the usual cost of one-sided movement in a latency-sensitive loop is the launch overhead around it.

### 3. FSDP2 and pipelining: control over memory and what gets communicated

**Reported.** Four changes, all labelled **API Unstable**:

- `MixedPrecisionPolicy(param_dtype_override_fn=...)` lets precision-sensitive parameters, the blog names MoE router gates and norm weights, stay fp32 while the rest trains in bf16, still with one all-gather per `fully_shard`.
- A whole gradient-accumulation step can be captured in one CUDA graph.
- Gradients accumulate directly into `.grad` and follow each parameter's `grad_dtype`.
- Pipeline parallelism allocates receive buffers just before each receive instead of up front for every microbatch, releases forward outputs once their sends finish, and takes a `max_outstanding_sends` budget. FSDP gradient reduction can overlap with the next compute.

**Measured.** `MixedPrecisionPolicy.__dataclass_fields__` is `param_dtype, reduce_dtype, output_dtype, cast_forward_inputs` on 2.14.1 and gains `param_dtype_override_fn` on 2.15.0. The text `grad_dtype` appears in `_fsdp_api.py` in 2.15.0 and not in 2.14.1.

**Why it matters, reasoned.** Keeping a router gate in fp32 inside an otherwise bf16 model is a recurring fix for mixture-of-experts instability, and until now it meant either splitting the module into separate FSDP units, with extra all-gathers, or patching dtypes by hand. The dtype change is also the one change in this item that is *not* opt-in: the blog lists it under backwards-incompatible changes because the default behaviour with `reduce_dtype=None` moves.

### 4. NVGEMM for NVFP4 decode

**Reported.** NVGEMM, the CuTeDSL GEMM backend that arrived in 2.14, now folds the scalar dequantization factor of ModelOpt NVFP4 layers into the GEMM, which removes one pointwise launch per layer during decode. A shape-aware candidate policy covers Blackwell decode shapes that nvMatmulHeuristics misses. Kernels can use programmatic dependent launch, so a Triton consumer starts while the producer GEMM is finishing. The headline is "faster 4-bit LLM decoding on Blackwell". There is no number for it in the blog.

**Reasoned.** At decode, a batch of one token multiplies a tiny activation by a large weight. Each layer is then bound by launching and memory traffic more than by arithmetic, so deleting a launch per layer is worth something, in proportion to the number of layers. A 60-layer model with four 4-bit linears per layer saves on the order of 240 launches per token. I did not measure launch cost, and I cannot, so I will not turn that into milliseconds. **Measured:** the only visible trace on the CPU wheel is an Inductor configuration key, `_nvgemm_max_profiling_configs_default`, present in both versions. This is the item where the blog gives you the least to check, and it requires a Blackwell GPU.

### 5. Apple Silicon: FP8, streams and the only numbers in the blog

**Reported.** This is the one section with figures. More MPS operators move from MPSGraph to native Metal kernels. A single-row softmax over a 128,256-entry vocabulary drops by 8x, upsample backward is up to 21.7x faster, and reductions share one set of kernels, making `norm` full reductions more than 100x faster. For decode, `F.linear` now uses the existing gemv kernels (1.23x geometric mean, up to 5x on small projections), and the new MPS `grouped_mm` has a gemv path giving 1.32 to 1.34x MoE decode throughput on an M3 Ultra. QR of a 512 by 512 matrix falls from about 7 seconds to 1.5 ms.

New surface: `torch.float8_e4m3fn` and `_scaled_mm` (tensorwise or rowwise scaling) on MPS, and a public stream API modelled on CUDA's: `torch.mps.Stream`, `current_stream`, `set_stream`, the `torch.mps.stream` context manager, and `Tensor.record_stream`.

**Measured.** On 2.14.1, `dir(torch.mps)` contains no name with "stream". On 2.15.0 it contains `Stream, current_stream, default_stream, set_stream, stream, streams`. That is the API surface, on a CPU wheel, with no Apple GPU to run it on; `torch.backends.mps.is_available()` is `False` here. The fp8 cast and a 16 by 32 `_scaled_mm` already ran on CPU in 2.14.1, so FP8 itself is not new to PyTorch, only to MPS.

**Reasoned.** The speedups are real in kind: a single-row softmax or a QR going from seconds to milliseconds means the old path was doing something poorly suited to the shape, and replacing it with a direct kernel is the fix. Shapes of that kind (batch one, one vocabulary row, a small projection) are the decode loop of a local LLM. But the quoted ratios are the authors' own, on the hardware they chose. I would expect the geometric mean (1.23x) to describe a typical model better than the 100x outliers do.

### 6. Platform breadth: ROCm, Intel, CUDA 13, Python 3.15

**Reported.** FlyDSL joins CuTeDSL, Triton and Helion as a kernel DSL PyTorch can drive: on AMD gfx950 Inductor autotunes FlyDSL templates for GEMM, ragged grouped matmul and MXFP4/MXFP8 `scaled_mm` alongside Triton and ATen. On gfx1250 (MI450) Inductor uses the Tensor Data Mover. ROCm 10.0 wheels arrive next to 7.14, all built on AMD's TheRock packages, which install ROCm as a pip dependency rather than bundling it. Intel XPU gets SYCL free-function kernels, event sharing across processes, and `MemPool` policies. CUDA 13.2 becomes the default wheel with 13.4 beside it, and the 13.4 wheels add experimental support for NVIDIA Rubin. Dynamo supports Python 3.15.

**Measured.** In the 2.15.0 wheel, 10 files under `torch/_inductor` have `flydsl` in their name, against none in 2.14.1. `torch.xpu.Event` gains `ipc_handle` and `from_ipc_handle`, and `torch.xpu.MemPool.__init__` gains a `no_split` argument. A Dynamo version gate reads `sys.version_info >= (3, 15)` in 2.14.1 and `>= (3, 16)` in 2.15.0, which is where the upper bound for supported Pythons moved.

## The breaking-changes list, and which items I could test

The blog lists five.

1. **Python 3.10 is gone.** **Measured:** the PyPI wheel tags for 2.14.1 are `cp310` to `cp314`; for 2.15.0 they are `cp311` to `cp315`. The blog adds that remaining 3.10 code paths, including in Dynamo, will be removed in coming releases.
2. **`nccl` selects the rewritten backend.** Covered above; untested.
3. **C++ warnings from `torch.ops` and custom ops now propagate to Python.** I ran a mismatched `aten.add` and got the same `RuntimeError` on both wheels, which is an error and not a warning, so it does not test the item. Untested.
4. **macOS 14 is no longer supported for MPS.** A statement about CI, not code. Untested.
5. **FSDP2 separates sharded and unsharded gradient dtypes.** The new `grad_dtype` text is in the wheel; the changed default is not something I ran.

Two of the five I could confirm from the artifact itself. Two are about GPUs or a Mac. One I tried and could not provoke.

## Fourteen probes on two wheels

Here is the full set: ten claims where the two wheels answer differently, three where they answer the same, and one I could not probe at all.

<ProbeTable />

The three that match are as informative as the ten that differ. FP8 conversion and `scaled_mm`, and `F.grouped_mm`, are not new to CPU. The first `grouped_mm` probe I wrote used unaligned shapes and failed with `strides should be multiple of 16 bytes` on both wheels; with bf16 and 16-byte-aligned shapes it ran on both. That is a property of the op, not a change.

## A torch.compile test that did not settle anything

The blog has no compiler headline, but I wanted to know whether a plain `torch.compile` regressed or improved on CPU. The model is four pre-norm transformer blocks, width 256, 4 heads, batch 4, sequence 128, no gradients, two threads. I timed eager and compiled calls alternately in the same process, 60 each, so that slow stretches on a shared box hit both.

**Measured.**

| wheel | cold compile | eager median | compiled median | eager over compiled |
|---|---|---|---|---|
| 2.14.1, run 1 | 29.6 s | 126.8 ms | 112.6 ms | 1.13x |
| 2.14.1, run 2 | 2.2 s (warm cache) | 51.0 ms | 44.8 ms | 1.14x |
| 2.15.0, run 1 | 26.6 s | 35.4 ms | 27.5 ms | 1.29x |

A second 2.15.0 run did not finish inside my time limit. The medians moved by a factor of three between runs because other jobs were using the same four cores, and an earlier, non-interleaved pass gave eager times of 31.8 and 40.5 ms on 2.14.1 against 60.5 and 61.7 ms on 2.15.0, which would make 2.15 look twice as slow if I believed it. I do not. Compiled output differed from eager by at most 1.67e-06 on both wheels.

So: compiled beats eager by about 1.1 to 1.3x on this model on both versions, and I have one 2.15 data point for the ratio. The 1.13 versus 1.29 gap is inside what a loaded machine produces. I would not cite it as an improvement. A usable comparison needs an idle machine, a dozen runs per wheel and a fixed clock, none of which I had.

## What to do with this

If you run NCCL jobs, test with the variable set both ways before upgrading the fleet, and read your hang runbook against the new flight recorder. If you run FSDP2 on mixed-precision MoE models, read the `grad_dtype` note first, since it changes a default. If you serve on Apple Silicon, the MPS gains are the best-documented in the release, and the numbers are the authors'. If you are on Python 3.10, you are done. Nothing here asks you to change code before you upgrade, with the possible exception of the FSDP2 dtype change.

What I could not do: run anything on a GPU, so the NCCL rewrite, NVGEMM, FlyDSL and the transports are checked for existence only; read the full GitHub release notes, which would list far more than the five breaking changes the blog does; or measure the Apple Silicon figures. The workflow is cheap enough to repeat: pull both wheels, diff `dir()` and file lists, and put a name to each claim. For profiling the new CUDA-graph attribution once you have a GPU, my [torch.profiler walkthrough](/articles/torch-profiler) is the place to start.