Hi all, I am back with yet another question about ...
# flyte-deployment
e
Hi all, I am back with yet another question about spark. I am trying to upgrade our
flyte-core
deployments from
v0.18.2
to
v1.0.1
. So far I didn't have any issues with regular tasks. But when I'm trying to execute spark tasks, I see that the default service account is assigned to the spark tasks. Is there something I need to change when upgrading to 1.+ versions? For context, I am using
flytekit
0.26.1. Thanks in advance!
p
Hi @Emirhan Karagül, The version of flytekit is old and you should upgrade to latest for fixes around this feature. Also you can set the workflow execution config for your spark workflows or you can assign it to project-domain which will allow you to use the spark service account for all workflows running under that project-domain https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_workflow-execution-config.html Also there is proposal to have such settings at project level too
k
@Emirhan Karagül preference is that you move to Flyte 1 throughout- though @Prafulla Mahindrakar this should work right
e
Thanks for the quick responses. I tried upgrading flytekit to 1.+ versions as well, but I still get the default service account during spark task execution. I suspect this is happening because we are using remote API to register/execute entities.
To give some more context@Ketan (kumare3): I also tested this with pyflyte serialize + register. During workflow execution, spark task is executed with the default service account. Didn't debug it further yet.
k
Are you seeing the service account in security context
p
Hi @Emirhan Karagül did you upgrade all the components to 1.0.1
yes @Ketan (kumare3) the current way that Emirhan is doing also should work. Just wanted to confirm if all the component bits are latest too expecially flyteadmin and flytekit
e
Yes, my flyte related requirements are:
flyteidl==1.0.1
flytekit==1.0.3
flytekitplugins-kfpytorch==1.0.3
flytekitplugins-spark==1.0.3
For the security context, when the task is extracted the
_security_ctx
field is
None
p
what about your flyte pods images and there versions . Are they also using 1.0.1 components
e
Yep
For sanity check, I downgraded flyte-core to v0.18.2 and flyte components to v.0.26. Now the spark task is assigned the spark
serviceAccount
.
If this is any useful to you guys as well, we can make a short call and I can demonstrate how I am registering the tasks, and how the service account is not being assigned. Also would be open to creating a PR if there is any bug on flyte side.
p
Sure whenever you are available we can sync up here https://meet.google.com/acz-ypcj-ynu
@Ketan (kumare3) so in Emirhan’s case the workflow has default service account but one of the task which is a spark task is also using the same service account. Emirhan pointed to this code used by spark https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/spark/spark.go#L95-L99 And i believe that it had been working fine for him until this change went in for admin to set the default service account in app config https://github.com/flyteorg/flyteadmin/blob/83d5ea767e7ca204c65be475f18ea73cfc159d5f/pkg/runtime/application_config_provider.go#L34 and hence uses the default service account even for the spark subtask. Do we ask users to set this in the task template ?
e
We were also thinking about creating a more flexible UI for the spark tasks here. Maybe we can adapt it such that the service account can be passed under the podSpec along with tolerations.
k
Hmm what do you mean by Ui
e
Sorry, I meant the task signature.
110 Views