acoustic-carpenter-78188
04/28/2026, 4:41 PMVLLMAppEnvironment and connect Claude Code to them:
• examples/genai/vllm/vllm_gpt_oss_claude.py — openai/gpt-oss-20b on L40s:1, 131k context
• examples/genai/vllm/vllm_qwen_claude.py — Qwen/Qwen2.5-7B-Instruct on L4:1, 32k context (lighter-weight option)
What the examples do
• Prefetch the model from HuggingFace into remote storage via flyte.prefetch.hf_model.
• Serve the model with VLLMAppEnvironment, with the flags Claude Code needs:
• --enable-auto-tool-choice
• --tool-call-parser openai (gpt-oss) / hermes (Qwen)
• --served-model-name my-model
• --max-model-len sized for Claude Code's system prompt
• --gpu-memory-utilization 0.92
• Scale to zero when idle (replicas=(0, 1), scaledown_after=300).
Connecting Claude Code
vLLM 0.11 only exposes the OpenAI-compatible API (/v1/chat/completions); Claude Code speaks the Anthropic API (/v1/messages). The docstrings document using claude-code-router (ccr) as a local translator:
npm install -g @musistudio/claude-code-router
# configure ~/.claude-code-router/config.json with the deployed app URL
ccr code
ccr runs a local proxy on 127.0.0.1:3456, translates Anthropic ↔️ OpenAI, and launches Claude Code pointed at it.
Notes
• gpt-oss-20b ships in MXFP4 quantization which requires GPU compute capability ≥ 8.0 (Ampere+). T4 (sm_7.5) is not supported; L4, L40S, A10G, A100, H100 all work.
• For the Qwen example's smaller 32k window, run /compact inside Claude Code on long sessions to keep the prompt under the limit.
• Reference: https://docs.vllm.ai/en/stable/serving/integrations/claude_code/
Test plan
• flyte deploy examples/genai/vllm/vllm_gpt_oss_claude.py vllm_app succeeds and the Knative revision becomes Ready
• flyte.prefetch.hf_model(repo="openai/gpt-oss-20b") completes and the served model loads from remote storage
• vLLM /v1/models returns my-model with max_model_len=131072
• ccr code launches Claude Code; messages are routed through the proxy and answered by my-model (verified via vllm:request_success_total metrics)
• Tool-using prompt round-trips successfully through Claude Code → ccr → vLLM
• Repeat the above for vllm_qwen_claude.py
flyteorg/flyte-sdkacoustic-carpenter-78188
05/01/2026, 1:09 AM