Another issue this causes is when I run the workflow that has spark tasks, I have to supply --service-account spark e.g.

pyflyte run --remote --service-account spark --image spark_image ...

But this will cause other flyte tasks to have permission issues. How can we provide the --service-account spark param in the code for the spark task to avoid this? Will the following work?

@task(
    container_image=".../flyte-pyspark:latest",
    task_config=Spark(
        spark_conf={...
        },
        service_account: "spark",
    ),
)

IIRC, we don’t support passing the SA to the specific task. mind creating an issue for it?

@glamorous-carpet-83516, I would like to do that. Since this is the first time for me to create an issue, would you mind tell me where and how?

here. https://github.com/flyteorg/flyte/issues. thanks!

⭐ Create a new Flyte Core Feature issue: https://github.com/flyteorg/flyte/issues/new?assignees=&labels=enhancement%2Cuntriaged&template=feature_request.yaml&title=%5BCore+feature%5D+

@glamorous-carpet-83516 created https://github.com/flyteorg/flyte/issues/3241

The spark service account will have the required perms to create executor pods.

this will also be doable once the work to add a pod template to the task decorator is one.

@thankful-minister-83577 @tall-lock-23197, thanks. What would be the reasons why the tasks got denied access to AWS resources when spark SA is specified as opposed to default SA, and the possible solutions to this?

k8s service accounts in eks are linked to iam roles.

Thanks

https://github.com/flyteorg/flyte/issues/3123 is the issue btw.