Apache beam is really good for model inference if you are using python ml packages. The sdk is in python so like no annoying spark Java errors or needing to use special python UDFs or building spark transformers.
I want to use it to batch score models in parallel using this
https://beam.apache.org/documentation/sdks/python-machine-learning/.
Also using GCPs dataflow is cool because auto scaling features are incredible. Vertical and horizontal scaling. Plus you can use GPUs.