Hey, does anybody know how can i pass multiple values to task_config param in @task decorator i.e. i want to pass Pod configuration as well as Spark configuration?

Hi @HIMANSHU JOSHI, unfortunately this is not supported. The

task_config

argument indirectly defines what

task_type

the task is. Passing

Pod

as well as

Spark

would be conflicting here. Can you elaborate on your use-case here? The Spark k8s operator supports

Pod

level configuration through a number of k8s configuration values in the spark config - you can very likely configure whatever you're looking for using these.

oh sorry for misunderstanding i thought flyte will spawn a pod which will then spawn driver / executor so thought of attaching labels to both. got to know we don't need to define pod labels just labels on spark job will work. thanks

No problem! Flyte actually creates a SparkJob CRD directly, which is then evaluated using the spark on k8s operator and Flyte periodically monitors. So it doesn't require an unnecessary Pod in the interim. And for an example, the spark flytesnacks directory would be a great place to start.