PyTorch 2.15: the NCCL rewrite becomes the default, and what a CPU box can check

mdjsonmcpcopy↗ai

2026-10-31 · 13 min · pytorch · distributed-training · nccl · fsdp · torch-compile · apple-silicon

Why read this

Solid· top 85%

Probes fourteen release claims on two real wheels and says plainly which could not be tested; the rest is a careful reading of the blog.

• Facts measured from source
• Runs on a laptop CPU
• Widely used

·Developer tools & infra·BSD-3-Clause·Practitioner roundup

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

1 of 3: Partial mechanism

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

2 of 3: A widely used model, tool or lab release

Only here?

1 of 3: Some original analysis

Score 53 of 100, ranked 522 of 666 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

PyTorch 2.15 came out on 28 October. The release blog counts 3,117 commits from 583 contributors since 2.14, and the @PyTorch post announcing it led with the distributed work, NVGEMM, Apple Silicon and accelerator support. A release this size is mostly a list, and a list is hard to read for what matters to you. So I did two things. I grouped what the blog names and picked the changes I think carry weight. Then I took the claims that a CPU box can see and checked them against the real wheels.

Labels used throughout: measured is something I ran or counted myself, reported is the blog's or the authors' statement that I did not re-run, reasoned is my inference from either. My runs are on a shared 4-vCPU cloud VM with no GPU, CPython 3.12, two threads. I installed torch==2.14.1+cpu and torch==2.15.0+cpu from the PyTorch CPU index into separate directories and ran the same scripts against each. I could not read the full GitHub release notes: the sandbox returns 403 for github.com, so everything below about what changed comes from the blog, which names five backwards-incompatible changes, and from the wheels themselves.

What the blog actually says, by area

The blog has fifteen feature sections and five deprecation notes. I counted the distinct PR numbers cited under each feature section and grouped the sections by who they serve.

Measured. The sections cite 108 PR numbers in total. Against 3,117 commits, the blog names about 3.5% of the release, and a PR cited in two sections is counted twice, so that is a ceiling. The counts say where the writing effort went, not where the engineering went. Apple Silicon draws more citations (25) than NVIDIA (21), which tells you who wrote a lot of small PRs more than it tells you the relative importance of Apple's GPU.

Two things are absent from the blog's feature list that you might expect from the previous three releases. There is no torch.compile or Inductor performance section on its own. The compiler shows up inside other items: Dynamo gets Python 3.15 support, Inductor autotunes FlyDSL templates on AMD, and NVGEMM is a GEMM backend that Inductor can choose. And there is no training-speed headline number for any model. The numbers the blog does give are all on Apple Silicon.

The six that matter

I picked these by asking what changes the behaviour of a job I would already be running, then what changes what I can run at all, then what is new but narrow.

1. The rewritten NCCL backend is now the default

Reported. In 2.14 the rewritten backend, ported from torchcomms, was an opt-in preview called nccl2. In 2.15 every process group created with the backend name nccl uses it. Two rollback paths exist: set TORCH_DIST_USE_NCCL2=0, or ask for the nccl-legacy backend by name. Communicators are now created eagerly with the process group instead of lazily at the first collective. Reconfiguration, the step where a failed rank is replaced or the group shrinks and grows, can opt into a fast path that needs NCCL 2.29 or newer, so surviving ranks skip a full bootstrap. The same reconfiguration is enabled on ROCm with RCCL 2.30.7.

The second half of this change is debuggability. A backend-agnostic flight recorder writes a post-mortem to disk when a collective hangs, and watchdog health is published to the debug server, so a timeout on any rank makes every rank dump its recorder. Anyone who has chased a hang on 512 GPUs knows why: the rank that times out is rarely the rank that caused it. All of this is labelled API Unstable in the blog.

Here is the resolution table as I understand it from the blog. It is reported, not run.

Why it matters, reasoned. Making a rewrite the default is a bet that the preview period found the problems. Eager communicator creation changes startup behaviour: failures that used to show up at the first all-reduce now show up when the group is created, which is better, and also means a job with a misconfigured network fails earlier and in a different place than your runbook says. If you operate training clusters, the first thing I would do is run one representative job with TORCH_DIST_USE_NCCL2=0 and one without, and diff the logs.

Measured, and its limit. I could not exercise any of this. The CPU wheel reports dist.is_nccl_available() == False, and torch.distributed.Backend.backend_list on 2.15 is undefined, gloo, nccl, xccl, ucc, mpi, fake on both wheels. What I can say from the Python sources: the string nccl2 appears in one file under torch/distributed in 2.14.1 and in four in 2.15.0, TORCH_DIST_USE_NCCL2 and nccl-legacy each appear once in both, and the 2.15.0 wheel has two new directories, _transport and _nccl4py. That fits the blog and proves nothing about speed or stability. The flight recorder was already a package in 2.14.1, so the change is in what feeds it and when it dumps, not that it exists.

2. One-sided transports and a NIXL backend

Reported. torch.distributed gains a Python transport API for one-sided data movement: one side reads or writes the other's memory without the other posting a matching receive. Peers connect through a store and need neither a process group nor collectives. Transport.read_stream and write_stream enqueue the transfer on a CUDA stream, so it can be captured by torch.cuda.graph. NIXL is the first backend. Measured: the _transport directory is in the 2.15.0 wheel and absent from 2.14.1. The leading underscore and the API Unstable label say what to expect.

Reasoned. This is the building block for disaggregated serving, where a prefill machine pushes a KV cache to a decode machine, and for parameter servers that never stall a trainer. Neither needs a collective, and a collective forces every participant to show up. That the transfer is graph-capturable is the interesting part: the usual cost of one-sided movement in a latency-sensitive loop is the launch overhead around it.

3. FSDP2 and pipelining: control over memory and what gets communicated

Reported. Four changes, all labelled API Unstable:

• MixedPrecisionPolicy(param_dtype_override_fn=...) lets precision-sensitive parameters, the blog names MoE router gates and norm weights, stay fp32 while the rest trains in bf16, still with one all-gather per fully_shard.
• A whole gradient-accumulation step can be captured in one CUDA graph.
• Gradients accumulate directly into .grad and follow each parameter's grad_dtype.
• Pipeline parallelism allocates receive buffers just before each receive instead of up front for every microbatch, releases forward outputs once their sends finish, and takes a max_outstanding_sends budget. FSDP gradient reduction can overlap with the next compute.

Measured. MixedPrecisionPolicy.__dataclass_fields__ is param_dtype, reduce_dtype, output_dtype, cast_forward_inputs on 2.14.1 and gains param_dtype_override_fn on 2.15.0. The text grad_dtype appears in _fsdp_api.py in 2.15.0 and not in 2.14.1.

Why it matters, reasoned. Keeping a router gate in fp32 inside an otherwise bf16 model is a recurring fix for mixture-of-experts instability, and until now it meant either splitting the module into separate FSDP units, with extra all-gathers, or patching dtypes by hand. The dtype change is also the one change in this item that is not opt-in: the blog lists it under backwards-incompatible changes because the default behaviour with reduce_dtype=None moves.

4. NVGEMM for NVFP4 decode

Reported. NVGEMM, the CuTeDSL GEMM backend that arrived in 2.14, now folds the scalar dequantization factor of ModelOpt NVFP4 layers into the GEMM, which removes one pointwise launch per layer during decode. A shape-aware candidate policy covers Blackwell decode shapes that nvMatmulHeuristics misses. Kernels can use programmatic dependent launch, so a Triton consumer starts while the producer GEMM is finishing. The headline is "faster 4-bit LLM decoding on Blackwell". There is no number for it in the blog.

Reasoned. At decode, a batch of one token multiplies a tiny activation by a large weight. Each layer is then bound by launching and memory traffic more than by arithmetic, so deleting a launch per layer is worth something, in proportion to the number of layers. A 60-layer model with four 4-bit linears per layer saves on the order of 240 launches per token. I did not measure launch cost, and I cannot, so I will not turn that into milliseconds. Measured: the only visible trace on the CPU wheel is an Inductor configuration key, _nvgemm_max_profiling_configs_default, present in both versions. This is the item where the blog gives you the least to check, and it requires a Blackwell GPU.

5. Apple Silicon: FP8, streams and the only numbers in the blog

Reported. This is the one section with figures. More MPS operators move from MPSGraph to native Metal kernels. A single-row softmax over a 128,256-entry vocabulary drops by 8x, upsample backward is up to 21.7x faster, and reductions share one set of kernels, making norm full reductions more than 100x faster. For decode, F.linear now uses the existing gemv kernels (1.23x geometric mean, up to 5x on small projections), and the new MPS grouped_mm has a gemv path giving 1.32 to 1.34x MoE decode throughput on an M3 Ultra. QR of a 512 by 512 matrix falls from about 7 seconds to 1.5 ms.

New surface: torch.float8_e4m3fn and _scaled_mm (tensorwise or rowwise scaling) on MPS, and a public stream API modelled on CUDA's: torch.mps.Stream, current_stream, set_stream, the torch.mps.stream context manager, and Tensor.record_stream.

Measured. On 2.14.1, dir(torch.mps) contains no name with "stream". On 2.15.0 it contains Stream, current_stream, default_stream, set_stream, stream, streams. That is the API surface, on a CPU wheel, with no Apple GPU to run it on; torch.backends.mps.is_available() is False here. The fp8 cast and a 16 by 32 _scaled_mm already ran on CPU in 2.14.1, so FP8 itself is not new to PyTorch, only to MPS.

Reasoned. The speedups are real in kind: a single-row softmax or a QR going from seconds to milliseconds means the old path was doing something poorly suited to the shape, and replacing it with a direct kernel is the fix. Shapes of that kind (batch one, one vocabulary row, a small projection) are the decode loop of a local LLM. But the quoted ratios are the authors' own, on the hardware they chose. I would expect the geometric mean (1.23x) to describe a typical model better than the 100x outliers do.

6. Platform breadth: ROCm, Intel, CUDA 13, Python 3.15

Reported. FlyDSL joins CuTeDSL, Triton and Helion as a kernel DSL PyTorch can drive: on AMD gfx950 Inductor autotunes FlyDSL templates for GEMM, ragged grouped matmul and MXFP4/MXFP8 scaled_mm alongside Triton and ATen. On gfx1250 (MI450) Inductor uses the Tensor Data Mover. ROCm 10.0 wheels arrive next to 7.14, all built on AMD's TheRock packages, which install ROCm as a pip dependency rather than bundling it. Intel XPU gets SYCL free-function kernels, event sharing across processes, and MemPool policies. CUDA 13.2 becomes the default wheel with 13.4 beside it, and the 13.4 wheels add experimental support for NVIDIA Rubin. Dynamo supports Python 3.15.

Measured. In the 2.15.0 wheel, 10 files under torch/_inductor have flydsl in their name, against none in 2.14.1. torch.xpu.Event gains ipc_handle and from_ipc_handle, and torch.xpu.MemPool.__init__ gains a no_split argument. A Dynamo version gate reads sys.version_info >= (3, 15) in 2.14.1 and >= (3, 16) in 2.15.0, which is where the upper bound for supported Pythons moved.

The breaking-changes list, and which items I could test

The blog lists five.

1. Python 3.10 is gone. Measured: the PyPI wheel tags for 2.14.1 are cp310 to cp314; for 2.15.0 they are cp311 to cp315. The blog adds that remaining 3.10 code paths, including in Dynamo, will be removed in coming releases.
2. nccl selects the rewritten backend. Covered above; untested.
3. C++ warnings from torch.ops and custom ops now propagate to Python. I ran a mismatched aten.add and got the same RuntimeError on both wheels, which is an error and not a warning, so it does not test the item. Untested.
4. macOS 14 is no longer supported for MPS. A statement about CI, not code. Untested.
5. FSDP2 separates sharded and unsharded gradient dtypes. The new grad_dtype text is in the wheel; the changed default is not something I ran.

Two of the five I could confirm from the artifact itself. Two are about GPUs or a Mac. One I tried and could not provoke.

Fourteen probes on two wheels

Here is the full set: ten claims where the two wheels answer differently, three where they answer the same, and one I could not probe at all.

The three that match are as informative as the ten that differ. FP8 conversion and scaled_mm, and F.grouped_mm, are not new to CPU. The first grouped_mm probe I wrote used unaligned shapes and failed with strides should be multiple of 16 bytes on both wheels; with bf16 and 16-byte-aligned shapes it ran on both. That is a property of the op, not a change.

A torch.compile test that did not settle anything

The blog has no compiler headline, but I wanted to know whether a plain torch.compile regressed or improved on CPU. The model is four pre-norm transformer blocks, width 256, 4 heads, batch 4, sequence 128, no gradients, two threads. I timed eager and compiled calls alternately in the same process, 60 each, so that slow stretches on a shared box hit both.

Measured.

A second 2.15.0 run did not finish inside my time limit. The medians moved by a factor of three between runs because other jobs were using the same four cores, and an earlier, non-interleaved pass gave eager times of 31.8 and 40.5 ms on 2.14.1 against 60.5 and 61.7 ms on 2.15.0, which would make 2.15 look twice as slow if I believed it. I do not. Compiled output differed from eager by at most 1.67e-06 on both wheels.

So: compiled beats eager by about 1.1 to 1.3x on this model on both versions, and I have one 2.15 data point for the ratio. The 1.13 versus 1.29 gap is inside what a loaded machine produces. I would not cite it as an improvement. A usable comparison needs an idle machine, a dozen runs per wheel and a fixed clock, none of which I had.

What to do with this

If you run NCCL jobs, test with the variable set both ways before upgrading the fleet, and read your hang runbook against the new flight recorder. If you run FSDP2 on mixed-precision MoE models, read the grad_dtype note first, since it changes a default. If you serve on Apple Silicon, the MPS gains are the best-documented in the release, and the numbers are the authors'. If you are on Python 3.10, you are done. Nothing here asks you to change code before you upgrade, with the possible exception of the FSDP2 dtype change.

What I could not do: run anything on a GPU, so the NCCL rewrite, NVGEMM, FlyDSL and the transports are checked for existence only; read the full GitHub release notes, which would list far more than the five breaking changes the blog does; or measure the Apple Silicon figures. The workflow is cheap enough to repeat: pull both wheels, diff dir() and file lists, and put a name to each claim. For profiling the new CUDA-graph attribution once you have a GPU, my torch.profiler walkthrough is the place to start.