Hey, does anybody know how can i pass multiple val...
# ask-the-community
h
Hey, does anybody know how can i pass multiple values to task_config param in @task decorator i.e. i want to pass Pod configuration as well as Spark configuration?
d
Hi @HIMANSHU JOSHI, unfortunately this is not supported. The
task_config
argument indirectly defines what
task_type
the task is. Passing
Pod
as well as
Spark
would be conflicting here. Can you elaborate on your use-case here? The Spark k8s operator supports
Pod
level configuration through a number of k8s configuration values in the spark config - you can very likely configure whatever you're looking for using these.
h
oh sorry for misunderstanding i thought flyte will spawn a pod which will then spawn driver / executor so thought of attaching labels to both. got to know we don't need to define pod labels just labels on spark job will work. thanks
btw do you know DockerFile structure for spark task in flyte we're building custom image so needed that for same?
d
No problem! Flyte actually creates a SparkJob CRD directly, which is then evaluated using the spark on k8s operator and Flyte periodically monitors. So it doesn't require an unnecessary Pod in the interim. And for an example, the spark flytesnacks directory would be a great place to start.
100 Views