<#1017 Add vLLM + Claude Code integration example>...
# flyte-github
a
#1017 Add vLLM + Claude Code integration example Pull request opened by pingsutw Summary Screenshot 2026-04-28 at 11 15 05 PM Screenshot 2026-04-28 at 11 14 42 PM Adds two end-to-end examples that serve OSS models via
VLLMAppEnvironment
and connect Claude Code to them: •
examples/genai/vllm/vllm_gpt_oss_claude.py
—
openai/gpt-oss-20b
on
L40s:1
, 131k context •
examples/genai/vllm/vllm_qwen_claude.py
—
Qwen/Qwen2.5-7B-Instruct
on
L4:1
, 32k context (lighter-weight option) What the examples do • Prefetch the model from HuggingFace into remote storage via
flyte.prefetch.hf_model
. • Serve the model with
VLLMAppEnvironment
, with the flags Claude Code needs: •
--enable-auto-tool-choice
•
--tool-call-parser openai
(gpt-oss) /
hermes
(Qwen) •
--served-model-name my-model
•
--max-model-len
sized for Claude Code's system prompt •
--gpu-memory-utilization 0.92
• Scale to zero when idle (
replicas=(0, 1)
,
scaledown_after=300
). Connecting Claude Code vLLM 0.11 only exposes the OpenAI-compatible API (
/v1/chat/completions
); Claude Code speaks the Anthropic API (
/v1/messages
). The docstrings document using
claude-code-router
(
ccr
) as a local translator: npm install -g @musistudio/claude-code-router # configure ~/.claude-code-router/config.json with the deployed app URL ccr code
ccr
runs a local proxy on
127.0.0.1:3456
, translates Anthropic ↔️ OpenAI, and launches Claude Code pointed at it. Notes •
gpt-oss-20b
ships in MXFP4 quantization which requires GPU compute capability ≥ 8.0 (Ampere+). T4 (sm_7.5) is not supported; L4, L40S, A10G, A100, H100 all work. • For the Qwen example's smaller 32k window, run
/compact
inside Claude Code on long sessions to keep the prompt under the limit. • Reference: https://docs.vllm.ai/en/stable/serving/integrations/claude_code/ Test plan •
flyte deploy examples/genai/vllm/vllm_gpt_oss_claude.py vllm_app
succeeds and the Knative revision becomes Ready •
flyte.prefetch.hf_model(repo="openai/gpt-oss-20b")
completes and the served model loads from remote storage • vLLM
/v1/models
returns
my-model
with
max_model_len=131072
•
ccr code
launches Claude Code; messages are routed through the proxy and answered by
my-model
(verified via
vllm:request_success_total
metrics) • Tool-using prompt round-trips successfully through Claude Code → ccr → vLLM • Repeat the above for
vllm_qwen_claude.py
flyteorg/flyte-sdk