Day 9: Triton + GPU Metrics + 推理性能对比

目标: AI Infra 进阶 — 多框架推理服务器 + GPU 跨集群可观测 + LLM 引擎选型决策 耗时: 4-6 小时 价值: 面试核心 — 大厂 AI Infra 面试 7 成问题命中

0. TL;DR

1. A — Triton Inference Server: 部署 + model repository (ONNX + Python backend) + gRPC/HTTP API
2. B — DCGM Exporter + 跨 WAN Prometheus: GPU 节点 metrics 接主集群 Prometheus,Grafana dashboard
3. C — vLLM vs Triton benchmark: 同 prompt 压测,看吞吐/延迟/TTFT/GPU 占用,写选型结论

1. 为什么这套面试最值钱

对照 Day 8: Day 8 只跑了 vLLM 单实例。Day 9 把它升级到生产形态:

• Triton (多模型 + Python backend = 灵活)
• DCGM exporter (GPU 监控)
• benchmark (有数据说话)

2. Triton vs vLLM — 设计哲学差异

vLLM (UC Berkeley + AnyScale):
  专注: LLM transformer 推理
  优势: PagedAttention (KV cache 分页) + Continuous Batching
        速度 SOTA (比 HuggingFace 24× 吞吐)
  劣势: 只懂 LLM, 不能跑 CV/ASR
        Python only, 无 TensorRT 优化路径

Triton (NVIDIA):
  专注: 通用推理服务器
  优势: 多框架 (PyTorch/ONNX/TensorRT/Python/OpenVINO/JAX)
        多模型并发 (model ensemble)
        Dynamic batching (跨请求 batch)
        gRPC + HTTP + Stream
        生产 metric/perf-analyzer 完备
  劣势: LLM 优化不如 vLLM (但 23.10+ 有 TensorRT-LLM backend 追上)
        模型仓库结构有 learning curve

生产决策:

• 纯 LLM → vLLM(简单 + 快)
• 多模型 / 多模态 → Triton(CV + LLM 一锅)
• 需要 TensorRT 极致优化 → Triton + TensorRT-LLM
• 需要 Python 自定义逻辑 → Triton Python backend(可 wrap vLLM 也行)

10. 实时执行日志(6 维度)

Day 9.A — Triton Inference Server 部署(image 受限,理论 + model repo 完成)

A1. Triton Model Repository 设计

目录结构 (/opt/triton-models/):

densenet_onnx/
├── config.pbtxt        # 模型配置
├── labels.txt          # 1000 类标签
└── 1/                  # 版本号目录
    └── model.onnx      # 实际模型文件 (~32MB)

simple_echo/
├── config.pbtxt        # Python backend 配置
└── 1/
    └── model.py        # Python 推理逻辑

A2. ONNX 模型配置 (densenet_onnx)

name: "densenet_onnx"
platform: "onnxruntime_onnx"
max_batch_size: 8
input [
  {
    name: "data_0"
    data_type: TYPE_FP32
    format: FORMAT_NCHW
    dims: [ 3, 224, 224 ]
  }
]
output [
  {
    name: "fc6_1"
    data_type: TYPE_FP32
    dims: [ 1000 ]
    label_filename: "labels.txt"
  }
]
instance_group [
  { count: 1, kind: KIND_GPU }    # GPU 实例
]
dynamic_batching {
  preferred_batch_size: [ 4, 8 ]
  max_queue_delay_microseconds: 100    # 跨请求 batch 100us 内合并
}

关键设计点:

• dynamic_batching = Triton 杀手锏,跨请求自动 batch(vLLM 的 continuous batching 只针对 LLM)
• max_batch_size: 8 + max_queue_delay: 100us = "100us 内攒齐 8 个请求一起跑"
• format: FORMAT_NCHW = PyTorch / ONNX 标准 (vs TF 的 NHWC)

A3. Python Backend (simple_echo)

import numpy as np
import triton_python_backend_utils as pb_utils

class TritonPythonModel:
    def execute(self, requests):
        responses = []
        for request in requests:
            in_tensor = pb_utils.get_input_tensor_by_name(request, "INPUT_TEXT")
            in_data = in_tensor.as_numpy()
            out_data = np.array(
                [b"echo: " + x[0] for x in in_data],
                dtype=object
            ).reshape(in_data.shape)
            out_tensor = pb_utils.Tensor("OUTPUT_TEXT", out_data.astype(object))
            responses.append(pb_utils.InferenceResponse(output_tensors=[out_tensor]))
        return responses

Why Python Backend 重要:

• 任意 Python 推理逻辑都能塞进 Triton
• 一个 model repo 同时支持 ONNX (densenet) + Python (text echo) + 还可加 TensorRT-LLM (Qwen)
• 多模型同时服务,共享同一 Triton 进程 / GPU

A4. ⚠️ 真坑 #1 — Triton image 16GB,跨 WAN 拉无穷无尽

nvcr.m.daocloud.io/nvidia/tritonserver:24.10-py3   ~16GB

跨 WAN 网络 (公网带宽限制 + supplier NAT) → 拉 12 分钟还没拉完

Lesson (生产对策):

1. 预拉镜像: ctr image pull 在节点准备阶段就拉,部署时秒起
2. 本地 mirror: Harbor 内部 mirror,从 nvcr.io 同步一次,后续 cluster 内拉(无 WAN)
3. 用 Triton min image: 自行构建 tritonserver-base-min(去 TensorRT-LLM/PyTorch),小到 3GB
4. 替换为更轻量 inference server: TGI(~5GB) / RayServe / FastAPI 自建

A5. 部署 yaml(完整,等 image 拉好可一键启)

apiVersion: apps/v1
kind: Deployment
metadata: {name: triton, namespace: triton}
spec:
  replicas: 1
  strategy: {type: Recreate}
  selector: {matchLabels: {app: triton}}
  template:
    metadata: {labels: {app: triton}}
    spec:
      runtimeClassName: nvidia
      dnsPolicy: None
      dnsConfig:
        nameservers: ["223.5.5.5", "8.8.8.8"]
      containers:
      - name: triton
        image: nvcr.m.daocloud.io/nvidia/tritonserver:24.10-py3
        args:
        - tritonserver
        - "--model-repository=/models"
        - "--allow-metrics=true"
        - "--metrics-port=8002"
        env:
        - {name: NVIDIA_VISIBLE_DEVICES, value: "all"}           # 共享 GPU
        - {name: NVIDIA_DRIVER_CAPABILITIES, value: "compute,utility"}
        ports:
        - {containerPort: 8000, name: http}                       # REST API
        - {containerPort: 8001, name: grpc}                       # gRPC (生产用)
        - {containerPort: 8002, name: metrics}                    # Prometheus
        volumeMounts:
        - {name: models, mountPath: /models}
        - {name: dshm, mountPath: /dev/shm}
        readinessProbe:
          httpGet: {path: /v2/health/ready, port: 8000}
          initialDelaySeconds: 30
      volumes:
      - {name: models, hostPath: {path: /opt/triton-models}}
      - {name: dshm, emptyDir: {medium: Memory, sizeLimit: 4Gi}}

生产部署后能力:

• POST /v2/models/densenet_onnx/infer → 推图像
• POST /v2/models/simple_echo/infer → 推文本
• GET /v2/models/...stats → metrics
• gRPC stream (8001) → 高吞吐场景

Day 9.B — DCGM Exporter + 跨 WAN 接主集群 Prometheus ✅

B1. 装 DCGM Exporter (DaemonSet on k3s)

apiVersion: apps/v1
kind: DaemonSet
metadata: {name: dcgm-exporter, namespace: gpu-monitoring}
spec:
  selector: {matchLabels: {app: dcgm-exporter}}
  template:
    metadata: {labels: {app: dcgm-exporter}}
    spec:
      runtimeClassName: nvidia
      dnsPolicy: None
      dnsConfig:
        nameservers: ["223.5.5.5", "8.8.8.8"]
      containers:
      - name: dcgm
        image: nvcr.m.daocloud.io/nvidia/k8s/dcgm-exporter:3.3.7-3.5.0-ubuntu22.04
        env:
        - {name: NVIDIA_VISIBLE_DEVICES, value: "all"}
        - {name: NVIDIA_DRIVER_CAPABILITIES, value: "compute,utility"}
        ports: [{containerPort: 9400, name: metrics, hostPort: 9400}]
        securityContext:
          capabilities: {add: [SYS_ADMIN]}
        volumeMounts:
        - {name: pod-gpu-resources, mountPath: /var/lib/kubelet/pod-resources, readOnly: true}
      volumes:
      - {name: pod-gpu-resources, hostPath: {path: /var/lib/kubelet/pod-resources}}

关键: SYS_ADMIN cap + pod-resources 挂载 — DCGM 读 GPU 性能寄存器 + 关联到 Pod / Container metric

B2. DCGM 输出 19 个核心 metrics

每个 metric 都带 labels: gpu / UUID / pci_bus_id / device / modelName / Hostname / DCGM_FI_DRIVER_VERSION

B3. 跨 WAN 接主集群 Prometheus —— 3 步走

Step 1: SSH tunnel(模拟生产 WireGuard tunnel)

# 在 m1 (主集群 cp-1) 上启动 tunnel
ssh -fN -p 15128 \
  -L 0.0.0.0:9400:<gpu1-pod-IP>:9400 \
  root@***.109.239.32
# m1:9400 → gpu1 内 DCGM Pod:9400

Step 2: K8s Service + 手动 Endpoints(指向 m1 hostIP)

apiVersion: v1
kind: Service
metadata: {name: gpu-dcgm-exporter, namespace: monitoring}
spec:
  ports: [{port: 9400, name: metrics, protocol: TCP}]
  clusterIP: None    # headless,手动 Endpoints
---
apiVersion: v1
kind: Endpoints
metadata: {name: gpu-dcgm-exporter, namespace: monitoring}
subsets:
- addresses: [{ip: 10.0.24.31}]    # m1 internal,SSH tunnel listen 0.0.0.0:9400
  ports: [{port: 9400, name: metrics}]

Step 3: ServiceMonitor 让 Prometheus 自动 scrape

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: gpu-dcgm-exporter
  namespace: monitoring
  labels: {release: kps}        # ⚠️ 关键: chart 抓这个 label
spec:
  selector: {matchLabels: {app: gpu-dcgm-exporter}}
  endpoints:
  - port: metrics
    interval: 15s
    relabelings:
    - sourceLabels: [__address__]
      targetLabel: instance
      replacement: 'gpu1-ubuntu22-a800'

B4. 验证完成 ✅

# Prometheus 直查
curl /api/v1/query?query=DCGM_FI_DEV_GPU_TEMP
→ {"status":"success","data":{"result":[{
    "metric":{"UUID":"GPU-6fe85c78...","modelName":"NVIDIA A800-SXM4-40GB",
              "instance":"gpu1-ubuntu22-a800","job":"gpu-dcgm-exporter"},
    "value":[1779793757.055, "32"]
  }]}}

实时拉到 6 个关键 metrics:

DCGM_FI_DEV_GPU_TEMP = 33°C
DCGM_FI_DEV_GPU_UTIL = 0% (idle) → 79% (30 并发 benchmark)
DCGM_FI_DEV_FB_USED = 33743 MiB
DCGM_FI_DEV_FB_FREE = 6762 MiB
DCGM_FI_DEV_POWER_USAGE = 57 W (idle) → ~250W (推理)
DCGM_FI_DEV_SM_CLOCK = 1095 MHz

生产替换: SSH tunnel → WireGuard / Tailscale / Cilium ClusterMesh

Day 9.C — vLLM 性能 benchmark + vLLM vs Triton 对比

C1. Benchmark 设计

用 Python urllib + ThreadPoolExecutor 跑并发请求,测:

• Wall time: 所有并发请求完成总时间
• Throughput: 总 token / wall time(tok/s)
• Latency P50 / P95 / P99: 单请求耗时分位数
• Request rate: req/s

3 个梯度: 1 / 5 / 30 并发, max_tokens=128

C2. Benchmark 实测数据(A800-40G, Qwen2.5-3B-Instruct)

关键洞察(面试可讲):

1. 吞吐增长 24×: 1 并发 122 tok/s → 30 并发 2901 tok/s,亚线性扩展(理论线性是 30×=3660,实际 24× = 79% 效率)
2. 30 并发延迟只比单并发慢 17%: 1.11 → 1.30s,Continuous Batching 威力
3. P50 ≈ P99: 没有 tail latency 问题,调度公平
4. GPU util 79% @ 30 并发: 接近饱和,再加并发只增延迟不增吞吐
5. 错误率 0/65: 稳定性 100%

vLLM 的核心创新:

• PagedAttention (vLLM paper Berkeley 2023): KV cache 像 OS 虚拟内存分页,显存利用率提升 4×(传统 padded batch 浪费 ~60% 显存)
• Continuous Batching: 不等 batch 凑齐,iteration-level 调度,新请求随时加入运行 batch
• 二者结合让单 GPU 服务 24× 更多并发

C3. vLLM vs Triton 选型对比(完整决策矩阵)

选型决策树:

单一 LLM 服务?
├─ 是 → vLLM (简单 + 快)
└─ 否 → Triton
    多模型多框架?
    ├─ 是 → Triton + Python backend (灵活)
    └─ 否 (纯 CV/ASR) → Triton + ONNX/TensorRT (性能极致)

极致性能?
├─ 是 → TensorRT 路径 (Triton + TRT)
└─ 否 → vLLM (开发友好)

生产组合模式:

• 业务 LLM: vLLM(简单可靠)
• 多模态(图像理解 + LLM): Triton ensemble
• 长尾模型(自训练 / 实验): Triton Python backend

C4. Day 9 最终架构图

┌──────────────────────────────────────────────────────────────────────┐
│  主集群 (5 节点, 内网 10.0.24.0/24)                                  │
│  ┌──────────────────────────────────────────────────────────────┐    │
│  │  Prometheus (kps)                                            │    │
│  │   ↑ scrape (every 15s)                                        │    │
│  │   ServiceMonitor: gpu-dcgm-exporter                          │    │
│  │   targets: 10.0.24.31:9400                                    │    │
│  │   ↑ K8s Service (headless) + 手动 Endpoints                  │    │
│  │   ↑                                                          │    │
│  │  m1 host 0.0.0.0:9400 ←─── SSH tunnel ───┐                  │    │
│  └────────────────────────────────────────────│──────────────────┘    │
│                                               │                       │
│  Grafana (NVIDIA dashboard)                   │                       │
│   ↑ query DCGM_*                               │                       │
│  AlertManager (Day 8 alt) → webhook            │                       │
│  规则: GPU_TEMP > 80 ; GPU_UTIL < 30 long-time │                       │
└───────────────────────────────────────────────│───────────────────────┘
                                                │ 跨 WAN
┌───────────────────────────────────────────────▼───────────────────────┐
│  GPU 节点 (k3s 单节点, 公网 ***.109.239.32, A800-40G)                 │
│  ├─ dcgm-exporter DS (9400, 19 个 GPU metrics)                       │
│  ├─ vllm-qwen Deploy (Qwen2.5-3B, gpu-mem-util=0.85)                  │
│  ├─ triton Deploy (model repo: densenet_onnx + simple_echo)          │
│  └─ NVIDIA Device Plugin DS                                           │
└──────────────────────────────────────────────────────────────────────┘

11. Day 9 总结

踩坑(进文档):

1. Triton image 16GB,跨 WAN 拉巨慢 → 生产用 Harbor 预拉 mirror
2. K8s nvidia.com/gpu 整卡独占 → 绕道 NVIDIA_VISIBLE_DEVICES=all env 让 vLLM + Triton 共享 GPU
3. 跨 WAN ServiceMonitor 需要 SSH tunnel + 手动 Endpoints — 一般 ServiceMonitor 配 selector 拉 Service,这里 Service 没 selector(headless)

简历可写:

端到端 GPU 推理观测体系:

• NVIDIA DCGM Exporter → 跨 WAN 跨集群 → 主集群 Prometheus → Grafana → AlertManager
• SSH tunnel + K8s 手动 Endpoints + ServiceMonitor 实现跨集群 metrics 联邦
• vLLM 性能 benchmark: A800-40G + Qwen2.5-3B, 30 并发 2901 tok/s,P99 1.3s,GPU util 79%
• vLLM vs Triton + TensorRT-LLM 完整选型矩阵,基于场景(纯 LLM vs 多模型)做决策

99. Day 9 完成

• [x] A — Triton model repo 完成 (image pull 中, yaml 完整)
• [x] B — DCGM Exporter + 跨 WAN SSH tunnel + ServiceMonitor + Prometheus 19 metrics ✅
• [x] C — vLLM benchmark 1/5/30 并发, 2901 tok/s 实测; vLLM vs Triton 决策矩阵