Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hey, does anybody know how can i pass multiple values to task_config param in @task decorator i.e. i want to pass Pod configuration as well as Spark configuration?

Hi <@U04NTATNBR7>, unfortunately this is not supported. The `task_config` argument indirectly defines what `task_type` the task is. Passing `Pod` as well as `Spark` would be conflicting here. Can you elaborate on your use-case here? The Spark k8s operator supports `Pod` level configuration through a number of k8s configuration values in the spark config - you can very likely configure whatever you're looking for using these.

oh sorry for misunderstanding i thought flyte will spawn a pod which will then spawn driver / executor so thought of attaching labels to both. got to know we don't need to define pod labels just labels on spark job will work. thanks

btw do you know DockerFile structure for spark task in flyte we're building custom image so needed that for same?

No problem! Flyte actually creates a SparkJob CRD directly, which is then evaluated using the spark on k8s operator and Flyte periodically monitors. So it doesn't require an unnecessary Pod in the interim. And for an example, the <https://github.com/flyteorg/flytesnacks/tree/master/cookbook/integrations/kubernetes/k8s_spark|spark flytesnacks directory> would be a great place to start.