<#1076 fix(genai): gate Flyte blob streaming on mo...
# flyte-github
a
#1076 fix(genai): gate Flyte blob streaming on model_path for vLLM and SGLang apps Pull request opened by cosmicBboy This pull request refines the model loading logic and documentation for both SGLang and vLLM plugin environments, clarifying how streaming from object storage is handled when using remote model paths versus Hugging Face paths. It ensures that Flyte blob streaming is only enabled when a remote model path is provided, and updates tests to verify this behavior. Additionally, it improves logging and the registration logic for the vLLM custom model loader. Model loading and streaming logic improvements: • Updated the
stream_model
parameter documentation and logic in both
SGLangAppEnvironment
and
VLLMAppEnvironment
to clarify that Flyte blob streaming is only enabled when a remote
model_path
is provided, and is ignored for Hugging Face-only paths. • Changed the internal logic to set streaming-related environment variables and arguments only when both
stream_model
is True and
model_path
is set, preventing accidental streaming activation for Hugging Face-only models. Testing improvements: • Enhanced tests for both plugins to assert that streaming is disabled (
FLYTE_MODEL_LOADER_STREAM_SAFETENSORS == "false"
) when only a Hugging Face path is used, and that no streaming-specific arguments are present in such cases. vLLM custom loader registration and logging: • Changed the registration of the
FlyteModelLoader
to occur only if both
REMOTE_MODEL_PATH
and
STREAM_SAFETENSORS
are set, preventing unnecessary loader registration. • Improved logging in the vLLM loader shim to provide visibility into environment variable values and prefetching behavior. flyteorg/flyte-sdk