#961 add: example of FastAPI and TokenBatcher in Flyte App environment Pull request opened by cosmicBboy Summary • Add a new example

examples/ml/batch_inference_fastapi.py

that demonstrates how to use

FastAPI

within a

<http://flyte.App|flyte.App>

environment for batintence. • The example shows how to integrate

vLLM

and

TokenBatcher

to maximize GPU utilization by aggregating concurrent HTTP requests into optimal batches. Details This pattern is useful for developers looking to serve models as persistent services using Flyte, leveraging the high throughput of the

TokenBatcher

. flyteorg/flyte-sdk