Hi all, I am trying to execute a single task in a ...
# ask-the-community
s
Hi all, I am trying to execute a single task in a workflow (not the entire workflow) with elevated rights in my k8s cluster. For this I created an additional k8s role, service account and binding similar to the spark setup and am now trying to work out how to make Flyte launch the pod with that service account. I tried using the pod plugin and with the following task decorator:
Copy code
@task(
    requests=Resources(mem="512Mi", cpu="1"),
    limits=Resources(mem="2Gi", cpu="1"),
    task_config=Pod(
        pod_spec=V1PodSpec(
            containers=[V1Container(name="primary")],
            service_account="<sa-name>",
            service_account_name="<sa-name>"),
        primary_container_name="primary",
    ),
)
However, when executing the workflow it seems that Flyte can no longer fetch the serialised Task inputs from the Flyte GCS bucket and fails with:
Copy code
Error from command '['gsutil', 'cp', '<gs://flyte-store/metadata/propeller/default-development-fddb5e602ce594338828/n1/data/inputs.pb>', '/tmp/flyte-tz9k8etn/sandbox/local_flytekit/inputs.pb']':
...
raise exceptions.CommunicationError(\napitools.base.py.exceptions.CommunicationError: Could not reach metadata service: Forbidden\n
Interestingly when I put
default
as
sa-name
everything works fine and the two k8s service accounts are linked to the same gcp-service-account. I would very much appreciate any pointers towards debugging this or other ways of setting up the Task to be executed with the new service account. Cheers, Seb
p
Can you check if step6 has been done for sa-name from this doc https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to Assuming for default sa you have already done this step .
You are allowed to override the service account from the launch form in flyteconsole From flytectl create execution also you can do the same https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_create_execution.html Adding @Kevin Su @Samhita Alla for the flytekit part
s
@Sebastian Schulze, let us know if giving a custom service account in the launch form works.
s
@Samhita Alla the hint by Prafulla already solved my issue. I did indeed miss the additional binding between the gke and k8s accounts. With that in place the decorator I wanted to use did its job.
👍 3
f
@Yee @Kevin Su @Samhita Alla from the above task_config of the @task decorator code sample, it seems it’s possible to supply the SA for spark task decorator. Is that correct? If so, what is the syntax like?
RE:
Copy code
task_config=Pod(
        pod_spec=V1PodSpec(
            containers=[V1Container(name="primary")],
            service_account="<sa-name>",
            service_account_name="<sa-name>"),
        primary_container_name="primary",
    ),
k
pod template can only be used in pod task for now. we are going to support new feature that allow you to override all other kinds of task. then you can override the SA for your spark task
365 Views