Serving Stack

vLLM, LMCache and NIXL, built from source against this repository's torch rather than installed from published wheels.

They have to be: their native extensions link against torch's C++ ABI, so a wheel built for AMD's ROCm 7.x torch and one built here cannot share a process.

What Gets Built

Target	What It Is
nixl-wheel	NIXL plus UCX, compiled from source with --with-rocm
cupy-wheel	CuPy for ROCm; LMCache's MP server imports it at module scope
torchvision-wheel	torchvision for ROCm; timm pulls it in and transformers imports it at module scope
vllm / vllm-wheel	The serving image and its wheel
lmcache / lmcache-wheel	The multi-process cache runtime and its wheel

make push-serving              # nixl + cupy + torchvision + vllm + lmcache
make build-vllm                # just vLLM, single builder, no push
make fetch-nixl-wheel          # pull a wheel artifact into ./dist
make push-serving VLLM_REF=v0.31.0 NIXL_REF=v1.5.0

Every ref here is a release tag, never a branch: a moving ref would be served from the layer cache indefinitely once built, so an "update" would silently be a no-op.

Ubuntu's libucx is compiled without ROCm memory-type support, so NIXL bundles its own: Dockerfile.nixl compiles UCX with --with-rocm and auditwheel bundles it. Without that, every device-memory transfer bounces through host staging.

Running It

serving/docker-compose.yml runs vLLM and the LMCache MP server as a pair over DOCKER_HOST=ssh://$GPU_HOST, so the file stays under version control here while the engine runs on the machine with the GPU.

make serve-stack               # vLLM + LMCache MP
make serve-vllm                # vLLM alone — the no-offload baseline
make serve-wait                # block until /health answers
make serve-test                # prove the cache is actually working
make serve-test-l2             # prove the disk tier serves blocks
make serve-logs                # both containers, merged, live
make serve-status              # what is running on GPU_HOST
make serve-down                # stop and remove

The compose file owns the runtime defaults the way rocm-bake.hcl owns the build ones, and the Makefile reads ${NAME:-default} out of it.

Variable	Default	What It Does
KV_TIER	cpu	cpu = L1 only, file = L1 plus an L2 on disk
TRANSFER_MODE	lmcache_driven	auto, engine_driven or lmcache_driven
HSA_IPC_LEGACY	1	Required for cross-container HIP IPC
VLLM_DEV_MODE	0	1 exposes /reset_prefix_cache, which serve-test needs
L1_SIZE_GB	8	LMCache's host-RAM tier
CHUNK_SIZE	256	Tokens per cached chunk
MAX_MODEL_LEN	8192
GPU_UTIL	0.90
SERVE_MODEL	Qwen/Qwen2.5-3B-Instruct
VLLM_BIND	0.0.0.0 from the Makefile, 127.0.0.1 in compose

A variable is passed only when actually set — an exported empty variable is still set, so VAR= would override the compose default with "".

VLLM_BIND differs deliberately: 127.0.0.1 is right locally, but driving it from another machine needs the port off loopback. Narrow it again, or tunnel, if the host is not on a trusted network.

HSA_ENABLE_IPC_MODE_LEGACY decides whether this works at all

It governs whether the two containers can share KV memory. When it is wrong the stack fails at startup, and no amount of TRANSFER_MODE tuning will fix it.

The Tags Come From The GPU Host

SERVE_ARCH asks GPU_HOST which GPU it has, via rocminfo over SSH. The build default is gfx950 while the serving host here is a gfx1201 RX 9070 XT, so deriving the serving tag from the build architecture asked Zot for an image nobody had built — the only symptom being a pull failure naming a plausible-looking tag. It falls back to the build architecture when the host is unreachable.

/health Is Not Evidence

LMCache on ROCm has a failure mode where the endpoint serves every request correctly and caches nothing — in LMCache#4419 a reporter ran eight days at a zero hit rate with no error anywhere.

make serve-test sends the same long prefix twice, requires the second prefill to be measurably cheaper, then reads LMCache's own log for the store and the retrieve — vLLM's own prefix caching can make the timing alone look fine. serve-test-l2 goes further, emptying LMCache's L1 as well as vLLM's VRAM between requests so a hit can only have come off disk; it needs KV_TIER=file.

make serve-logs merges both streams tagged by container rather than using compose logs. The useful lines alternate — vLLM says what it asked for, LMCache whether it had it — and reading them apart once hid LMCache dying on a missing CuPy while vLLM waited five minutes before reporting a generic engine error.

Running The Images Directly

# vLLM. AITER is installed but off by default.
docker run --rm --device /dev/kfd --device /dev/dri -p 8000:8000 \
    -e VLLM_ROCM_USE_AITER=1 \
    snoc-strix.fold-leaffish.ts.net:8080/rocm-pytorch/vllm:latest \
    serve Qwen/Qwen3-8B

# LMCache's MP deployment: a controller holding the cluster-wide index,
# and one or more cache servers holding the blocks. The image has no
# default role — name the CLI you want.
docker run --rm -p 9000:9000 \
    snoc-strix.fold-leaffish.ts.net:8080/rocm-pytorch/lmcache:latest \
    lmcache_controller --host 0.0.0.0 --port 9000

docker run --rm --device /dev/kfd --device /dev/dri -p 5555:5555 \
    snoc-strix.fold-leaffish.ts.net:8080/rocm-pytorch/lmcache:latest \
    lmcache server --host 0.0.0.0 --port 5555

The cache server lost its console script in v0.5.7

LMCache v0.5.7 restructured the package: the single-process engine moved to lmcache.inprocess.v1.*, the multiprocess stack took over lmcache.v1.*, and the standalone lmcache_server entry point was dropped in favour of the lmcache server subcommand. Both roles are argparse and take flags; --transport defaults to zmq, so pass --transport grpc for the gRPC path.

Two Upstream Constraints

Both projects' requirement files carry CUDA-only or x86-wheel-only pins — cufile-python, nvtx, tilelang, amd-quark and others. These are stripped before the build, which also keeps them out of the wheel's install_requires. The lists are the VLLM_DROP_REQS and LMCACHE_DROP_REQS build args, so a dependency can be let back in once it has a usable wheel.

Upstream publishes torchvision only as a CUDA wheel, so this pipeline builds it, the way it builds torch and triton. It used to take PyPI's: timm pulls torchvision in through vLLM's build requirements, its _C_stable.so links libtorch_cuda and libc10_cuda, and import torchvision failed at load time. Nothing in the text-model path touches it, so the build and the import checks stayed green while vision and video models would have failed at serve time — and serving papered over it by swapping in the CPU wheel at container start.

torchvision-wheel replaces all of that. Dockerfile.torchvision builds it FROM the PyTorch image with FORCE_CUDA=1 (a BuildKit RUN has no /dev/kfd, so torch.cuda.is_available() is false and without it the wheel ships no device code at all) and asserts the result carries code objects for the target architecture before publishing. /etc/pip-constraints.txt now pins torchvision alongside torch and triton, to a +rocm<version> local version PyPI cannot offer, so any later resolution that reaches for the CUDA build fails loudly. The proof it worked is in Dockerfile.vllm: the LMCache connector import check, which could not run while torchvision was broken, is back.