#839 custom model served Pull request opened by dansola Adds a model serving example that demonstrates how to serve a PyTorch model in .pt2 format using a custom loader. The key pattern shown is loading an

ExportedProgram

in

@env.on_startup

, applying

move_to_device_pass

to retarget from CPU to GPU, and defining preprocessing transforms manually as opposed to relying on

AutoImageProcessor

or

from_pretrained

. Also includes a warmup forward pass to pre-compile CUDA kernels before the first request, and always-on replica scaling to keep the instance warm. model generation task: https://demo.hosted.unionai.cloud/v2/domain/development/project/daniel/runs/r4j2x9mqvnd2v4fctc8z testing endpoint:

curl -s https:
//ancient-tooth-e8bed.apps.demo.hosted.unionai.cloud/health
{"status":"ready","model_loaded":true,"device":"cuda"

curl -o /tmp/dog.jpg <https://upload.wikimedia.org/wikipedia/commons/thumb/2/26/YellowLabradorLooking_new.jpg/1200px-YellowLabradorLooking_new.jpg>                   
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-100  246k  100  246k    0     0   690k      0 --:--:-- --:--:-- --:--:--  691k

curl -X POST h
<ttps://ancient-tooth-e8bed.apps.demo.hosted.unionai.cloud/predict> -F "file=@/tmp/dog.jpg"                  
{"predictions":[{"class_index":208,"confidence":0.1902},{"class_index":227,"confidence":0.0201},{"class_index":209,"confidence":0.0149},{"class_index":248,"confidence":0.0098},{"class_index":207,"confidence":0.0083}]}

flyteorg/flyte-sdk