Domestic GPU Ecosystems: Ascend, Cambricon, and Hardware Selection

The Problem

You are planning a new training or inference cluster. The default answer is "NVIDIA H100 with CUDA and vLLM." Then one of three things happens.

Supply disappears. H100 lead times stretch, pricing spikes, or a quota policy limits imports. You are asked to evaluate a second supplier: Huawei Ascend, Cambricon, Hygon, Moore Threads, Baidu Kunlun, or Alibaba Hanguang.

The spec sheet looks good on paper. A domestic card advertises comparable FP16 TFLOPS and more HBM. But TFLOPS only count if your model's operators have optimized kernels on that stack. A missing flash_attention, unsupported bfloat16, or an unimplemented custom loss can turn a fast card into a science project.

Migration cost is hidden. Porting CUDA code to another stack is rarely a syntax change. It is a sequence of operator-level and numerical-precision surprises: torch.matmul may dispatch to a different kernel, default dtype behavior may differ, and distributed training collectives may have different performance cliffs. The cost is in coverage and alignment, not in renaming cuda() to npu().

The lesson builds a procurement scorer that forces these trade-offs into the open. The artifact is a small Python program that reads a JSON card table, scores each candidate against a workload profile, and reports both a ranked shortlist and the reason each rejected card failed.

The Concept

The accelerator is only half the product

A training or inference node is the combination of silicon plus software. The same hardware can be unusable, slow, or competitive depending on the stack:

All numbers in the table are approximate and should be treated as as-of-2025 snapshots. Prices, peak TFLOPS, and software maturity change quickly; the lesson ships a JSON card table precisely so you can update them.

Operator coverage decides "can it run," not peak FLOPs

Peak TFLOPS is the speed of a perfectly utilized tensor core. Real workloads are a mix of memory bandwidth, kernel launch overhead, communication, and operator availability. The question to ask first is:

Does the stack have a correct, optimized implementation of every operator in my model's forward and backward graphs?

If the answer is no, the card is not a slower H100; it is a porting project. That is why the scorer uses an ecosystem score (a 0–1 approximate maturity rating) as a first-class input, not a tie-breaker.

Compatibility layers reduce syntax cost, not semantics cost

Several domestic stacks advertise CUDA compatibility:

• torch_npu lets PyTorch code call torch.Tensor.to("npu") and largely reuse PyTorch APIs.
• Moore Threads' MUSA provides CUDA-like programming interfaces and migration tools.
• Hygon DCU follows the ROCm / HIP path, so many CUDA kernels compile with hipify.

These layers remove the mechanical rewrite, but they do not guarantee that:

• every operator has the same numeric precision,
• the same fused kernels exist,
• distributed collectives perform identically,
• bfloat16 or gradient scaling behave the same way.

The real migration cost is verifying the compute graph end-to-end and measuring accuracy drift.

Memory: the first hard gate

For inference, the per-GPU memory footprint is approximately:

where N is parameter count, b is bytes per parameter (2 for fp16/bf16), and the KV cache is:

with batch size B, sequence length S, hidden dimension H, number of layers L, and precision bytes b. The factor 2 comes from key and value tensors.

For training, reuse the lesson 35 memory budget. Mixed-precision Adam keeps about 16 bytes per parameter for model state:

Activations add another term proportional to batch size, sequence length, hidden dimension, and layers. The scorer uses a simplified activation estimate to keep the code self-contained.

If the model and KV cache do not fit, no amount of TFLOPS helps. The scorer rejects cards with a memory reason when the required card count exceeds a practical ceiling.

Compute: throughput after ecosystem and memory

Once memory fits, check whether the available compute meets the latency or wall-clock target. For inference under a compute-bound assumption:

where P is per-GPU peak TFLOP/s, \eta is the achieved Model FLOPs Utilization, and G is the number of GPUs. The target tokens per second is B \cdot 1000 / \text{ms-per-token}.

For training, total FLOPs are approximately:

and wall-clock time is:

The scorer uses these rules of thumb, with conservative MFU defaults, to estimate whether a candidate can meet the latency or training-time target.

Cost and power are procurement constraints

The scorer treats per-card hourly cost and total datacenter power as hard gates. A card that is fast and fits memory is still rejected if it breaks the budget or blows the rack power limit.

Build It

The lesson ships a command-line hardware-selection scorer. It reads a JSON card table (code/cards.json by default, or any path you provide) and scores each accelerator against a WorkloadProfile.

Run it:

consoleCopypython3 main.py

The demo evaluates three procurement scenarios:

1. 7B inference with vLLM, H100 shortage. H100 is removed from the candidate list to model unavailability. Small-VRAM cards are rejected for memory, non-NVIDIA cards are rejected for ecosystem, and the cheapest feasible NVIDIA card rises to the top of the cost ranking.
2. 13B fine-tuning. Training memory dominates; small-VRAM consumer cards are rejected because the required card count exceeds the practical ceiling.
3. 7B training in an Ascend datacenter. torch_npu is required, so NVIDIA cards are rejected for ecosystem. Among Ascend candidates, the scorer weighs memory headroom, total cost, and rack power.

The JSON card table uses this schema:

jsonCopy{
  "name": "Huawei Ascend 910B 64GB",
  "peak_fp16_tflops": 320.0,
  "vram_gb": 64.0,
  "memory_bw_gb_s": 1600.0,
  "interconnect_bw_gb_s": 392.0,
  "ecosystem_score": 0.75,
  "supports_vllm": false,
  "supports_torch_npu": true,
  "hourly_usd": 1.50,
  "tdp_kw": 0.40
}

Edit the file, add your own cards, or pass a custom table path to the scorer functions.

Verify It

code/tests/test_main.py locks down the scorer's decision logic:

• A card with too little VRAM is eliminated with reason memory.
• A card that does not support a required framework (vLLM or torch_npu) is eliminated with reason ecosystem.
• A card that exceeds the per-card budget is eliminated with reason budget.
• A configuration that exceeds the datacenter power limit is eliminated with reason power.
• Loading and saving the JSON card table is lossless.
• The demo script runs end-to-end and exits 0.

Why This Matters for Engineers

• Second-source hardware is a real requirement. Supply uncertainty, procurement rules, or datacenter constraints can make NVIDIA unavailable. The decision is not "is it as good as H100?" but "can it run my workload at acceptable cost and effort?"
• Software ecosystem is the dominant variable. A card with more TFLOPS but missing kernels is not a faster card; it is a porting and validation task. Ask for an operator coverage matrix before you ask for a benchmark.
• Migration cost is in coverage and precision alignment. CUDA-compatible syntax removes the easy 20% of work. The remaining 80% is verifying that every operator, dtype, and distributed primitive behaves the same way.
• Memory is the first gate, compute is the second, cost and power are the third. The scorer forces this order. A card that fails the first gate should not clutter the performance discussion.
• Keep the card table under version control. Prices, availability, and driver maturity change. A JSON table that the team updates is more useful than a one-time spreadsheet.

Hints

• Start with a minimal card table containing only the accelerators you can actually obtain, then add others for sensitivity analysis.
• If a card is rejected for memory, try reducing batch size or sequence length before discarding it; memory is often the easiest knob.
• If a card is rejected for ecosystem, ask the vendor for a specific operator checklist for your model architecture, not a general compatibility claim.
• The default MFU assumptions are conservative. Replace them with measured MFU from a representative micro-benchmark once you have access to the hardware.
• When procurement is uncertain, run the scorer with optimistic and pessimistic ecosystem scores to see which candidates are robust to small changes in maturity.

Key Terms