Hi all, I am trying to execute a single task in a workflow (not the entire workflow) with elevated rights in my k8s cluster. For this I created an additional k8s role, service account and binding similar to the spark setup and am now trying to work out how to make Flyte launch the pod with that service account. I tried using the pod plugin and with the following task decorator:

@task(
    requests=Resources(mem="512Mi", cpu="1"),
    limits=Resources(mem="2Gi", cpu="1"),
    task_config=Pod(
        pod_spec=V1PodSpec(
            containers=[V1Container(name="primary")],
            service_account="<sa-name>",
            service_account_name="<sa-name>"),
        primary_container_name="primary",
    ),
)

However, when executing the workflow it seems that Flyte can no longer fetch the serialised Task inputs from the Flyte GCS bucket and fails with:

Error from command '['gsutil', 'cp', '<gs://flyte-store/metadata/propeller/default-development-fddb5e602ce594338828/n1/data/inputs.pb>', '/tmp/flyte-tz9k8etn/sandbox/local_flytekit/inputs.pb']':
...
raise exceptions.CommunicationError(\napitools.base.py.exceptions.CommunicationError: Could not reach metadata service: Forbidden\n

Interestingly when I put

default

as

sa-name

everything works fine and the two k8s service accounts are linked to the same gcp-service-account. I would very much appreciate any pointers towards debugging this or other ways of setting up the Task to be executed with the new service account. Cheers, Seb

Can you check if step6 has been done for sa-name from this doc https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to Assuming for default sa you have already done this step .

@Sebastian Schulze, let us know if giving a custom service account in the launch form works.

@Samhita Alla the hint by Prafulla already solved my issue. I did indeed miss the additional binding between the gke and k8s accounts. With that in place the decorator I wanted to use did its job.

@Yee @Kevin Su @Samhita Alla from the above task_config of the @task decorator code sample, it seems it’s possible to supply the SA for spark task decorator. Is that correct? If so, what is the syntax like?

pod template can only be used in pod task for now. we are going to support new feature that allow you to override all other kinds of task. then you can override the SA for your spark task