Multiple GFX Targets¶

Two different things, easy to confuse:

	What It Produces
`PYTORCH_ROCM_ARCH='gfx950;gfx1201'`	One fat build, both code objects in one binary, one set of tags
`ARCHES='gfx950 gfx1201'`	Two builds, two separately-tagged sets of artifacts

PYTORCH_ROCM_ARCH picks the architecture for one build and defaults to gfx950. ARCHES is the list the arches-* wrappers fan out over, and defaults to gfx950 gfx1201 gfx1151 gfx1250.

make arches-push-all                      # every target, every arch
make arches-push-wheels                   # just the wheel artifacts
make arches-build-pytorch ARCHES=gfx1201  # narrow it while iterating
make arches-list                          # what ARCHES resolves to

arches-% accepts any target name. Each architecture is a full pass over every target, so treat the list as a multiplier on build time; the passes run sequentially because they contend for the same builders.

`gfx1151` Is The Card snoc-strix Actually Has¶

RDNA3.5 (Strix Halo). Unlike gfx950 and gfx1250, it is in the fleet — see manifests/*.yml or make list-nodes for which node has what — so it can be GPU-tested rather than only cross-compiled, including with GPU_HOST=snoc-strix when the default gpu_node is busy.

ROCm 10.1 ships eleven -gfx1151 packages, so rocm-prune has something to keep. AITER v0.1.28 lists it as Experimental and still recommends installing it on RDNA parts, with the caveat that drives the PREBUILD_KERNELS choice below: most CK and ASM kernels are CDNA-only, so level 2 fails here as it does on gfx1201. Level 0 is the default.

It is not RDNA4 despite being gfx11-and-up — IS_RDNA4 tests for a gfx12 substring, so USE_ROCM_CK_SDPA_GFX12 stays off.

`gfx1250` Is In The List Without Being In The Fleet¶

The builders cross-compile — no node has a gfx950 card, yet the fleet builds it daily — so having no card only means make gpu-test cannot cover it. The ROCm 10.1 toolchain compiles it:

clang -x hip --offload-arch=gfx1250 -nogpulib -c   # exits 0

`gfx1250` Is Not RDNA, Despite The Number¶

This is the one place the architecture naming actively misleads, and the build branches on it:

ARCH_RDNA_TEST = replace(PYTORCH_ROCM_ARCH, "gfx125", "")
IS_RDNA4       = ARCH_RDNA_TEST != replace(ARCH_RDNA_TEST, "gfx12", "")

gfx1250 and gfx1251 are gfx12-numbered but MI450-class datacenter parts: upstream LLVM gates amdgpu-sched-strategy=coexec on gfx1250 and gfx950 together, and gives gfx1250 tensor instructions no RDNA part has. A bare gfx12 test applies RDNA-only settings to a datacenter card, which is why the strip comes first.

AITER's kernel prebuild is level 0 on every architecture: build time is the binding constraint here, and the prebuild buys first-inference latency rather than capability, since AITER JIT-compiles on first use either way.

A measurement forced that before cost chose it. On gfx1201 all 13 844 kernels compiled and the build failed in the JIT module phase — 6 of 68 modules carry CDNA-only code and setup.py treats a failure as fatal, so bdist_wheel exited 1 after 51 minutes with no wheel. Reproduced twice; upstream lists gfx1201 as Experimental.

PREBUILD_KERNELS pins a level for one build. Level 2 produces no wheel at all on gfx1201 and gfx125x, so a global pin is only safe on CDNA.

The Layer Cache Is Scoped By Architecture¶

CACHE_ARCH keeps each architecture's registry cache separate, because before it existed alternating architectures cost two cold builds every time. triton-wheel stays on the shared ref — it compiles no device code — and pytorch-base does not. See Caching.

Per-Architecture Pip Index¶

Every wheel here has an architecture-independent filename. torch is torch-…-cp314-cp314-linux_x86_64.whl whether it was built for gfx950 or gfx1201, and amd_aiter is 131 MB for one architecture and 481 MB for the other. In one directory they overwrite each other and --find-links serves whichever landed last. So the architecture lives in the path.

make wheel-index                  # fetch per arch, generate the indexes
make wheel-index-check            # install each one into a throwaway venv
python3 -m http.server 8081 --directory dist

pip install \
  --index-url http://<host>:8081/index/gfx950/ \
  --extra-index-url https://pypi.org/simple \
  "rocm-torch[device-gfx950]"

rocm-torch is a metapackage generated per index by .builders/wheelindex.py; its device-<arch> extra pins every wheel in that index at once, and each index declares only its own architecture's extra, so asking gfx950's index for [device-gfx1201] fails rather than installing mismatched binaries.

Wheels are fetched by architecture-specific tag, never :latest — written by whichever architecture was pushed last, with the wheels inside indistinguishable afterwards, which is the failure this index exists to prevent.

--extra-index-url is not optional

--index-url replaces PyPI outright, and the first ordinary dependency — numpy, via triton-kernels — then fails to resolve. download.pytorch.org has the same property and the same workaround.

Publishing The Index Publicly¶

make wheels-publish ARCH=gfx950 puts one architecture on the public internet. It is a separate decision from building, so it is in no group and no push-all; run make wheel-index first and this publishes what that fetched, rather than fetching again.

The binaries and the index go to different places, and the split is forced rather than chosen. A wheel here is 300–500 MB — amd_aiter for gfx950 is 495 MB — and GitHub refuses any file over 100 MB in git. So gh-pages carries the index, which is 28 KB of HTML, and the wheels go to a release, where an asset may be 2 GB and does not count against the repository.

wheelindex.py --wheel-url already existed for exactly this: it sets the href prefix, and every link already carries a #sha256= fragment, so an offsite asset is still verified by pip at install time.

One release per architecture. Assets within a release share a namespace, and these wheel filenames are architecture-independent — two architectures' torch wheels are the same string and differ by 50 MB. In one release the second upload would collide with the first. That is the same problem the per-architecture index path exists to solve, so it gets the same answer.

make wheel-index                       # fetch and build the indexes
make wheels-publish-dry ARCH=gfx950    # show what would happen
make wheels-publish ARCH=gfx950        # release + gh-pages

pip install \
  --index-url https://sbates130272.github.io/rocm-bake/data/wheels/gfx950/ \
  --extra-index-url https://pypi.org/simple \
  "rocm-torch[device-gfx950]"

A downloaded asset does not say which GPU it is for

The filenames say which ROCm, and not which architecture. torch-2.15.0a0+git31a77fc.rocm10.1.0-cp314-cp314-linux_x86_64.whl is the same string for gfx950 and gfx1201, and the two files differ by about 40MB of device code. Installing through the index is always correct, because the index path and the rocm-torch metapackage both carry the architecture. Downloading a release asset by hand is not.

Most wheels carry the ROCm version in their own version string — torchvision, atom, cupy and lmcache as +rocm10.1.0, and torch, triton and amd_aiter appended to the commit they were built from, as in 2.15.0a0+git31a77fc.rocm10.1.0. vllm carries upstream's own +rocm101.

Three do not, each for its own reason. nixl and triton_kernels are pure Python and contain no ROCm code to qualify. nixl_rocm does contain ROCm code, but its version comes from meson with no environment hook to extend, and its distribution name already says rocm — so it is the one wheel here identified by name rather than by version.

The architecture is in none of them. It could be — a local version segment takes as many dot-separated parts as it likes — but it is deliberately not, because then torch would resolve differently per machine from a single index, which is precisely what the per-arch index path and the metapackage exist to express.

Why `device-<arch>` And Not A Variant¶

This is the shape TheRock ships ROCm in, rather than the WheelNext variant mechanism. Variants (PEP 817/825) are the better long-term answer, but both PEPs are Draft, installer support is prototype-only forks of pip and uv, and AMD's published provider matches PyTorch 2.9 on ROCm 6.3/6.4 against a gfx list without gfx950.

The extra is spelled device-<arch> so that when variants do land the architecture is already a first-class coordinate.

CI builds two architectures' indexes on every pull request and asserts that each installs its own and only its own.

Multiple GFX Targets¶

gfx1151 Is The Card snoc-strix Actually Has¶

gfx1250 Is In The List Without Being In The Fleet¶

gfx1250 Is Not RDNA, Despite The Number¶