https://flyte.org
Join Slack
Hi all, I am back with yet another question about spark. I am trying to upgrade our `flyte-core` dep...
# flyte-deployment
w
Hi all, I am back with yet another question about spark. I am trying to upgrade our 
flyte-core
deployments from 
v0.18.2
to 
v1.0.1
. So far I didn't have any issues with regular tasks. But when I'm trying to execute spark tasks, I see that the default service account is assigned to the spark tasks. Is there something I need to change when upgrading to 1.+ versions? For context, I am using 
flytekit
0.26.1. Thanks in advance!
i
Hi @worried-farmer-76794, The version of flytekit is old and you should upgrade to latest for fixes around this feature. Also you can set the workflow execution config for your spark workflows or you can assign it to project-domain which will allow you to use the spark service account for all workflows running under that project-domain https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_workflow-execution-config.html Also there is proposal to have such settings at project level too
f
@worried-farmer-76794 preference is that you move to Flyte 1 throughout- though @icy-agent-73298 this should work right
w
Thanks for the quick responses. I tried upgrading flytekit to 1.+ versions as well, but I still get the default service account during spark task execution. I suspect this is happening because we are using remote API to register/execute entities.
To give some more context@freezing-airport-6809: I also tested this with pyflyte serialize + register. During workflow execution, spark task is executed with the default service account. Didn't debug it further yet.
f
Are you seeing the service account in security context
i
Hi @worried-farmer-76794 did you upgrade all the components to 1.0.1
yes @freezing-airport-6809 the current way that Emirhan is doing also should work. Just wanted to confirm if all the component bits are latest too expecially flyteadmin and flytekit
w
Yes, my flyte related requirements are: 
flyteidl==1.0.1
flytekit==1.0.3
flytekitplugins-kfpytorch==1.0.3
flytekitplugins-spark==1.0.3
For the security context, when the task is extracted the 
_security_ctx
field is 
None
i
what about your flyte pods images and there versions . Are they also using 1.0.1 components
w
Yep
For sanity check, I downgraded flyte-core to v0.18.2 and flyte components to v.0.26. Now the spark task is assigned the spark 
serviceAccount
.
If this is any useful to you guys as well, we can make a short call and I can demonstrate how I am registering the tasks, and how the service account is not being assigned. Also would be open to creating a PR if there is any bug on flyte side.
i
Sure whenever you are available we can sync up here https://meet.google.com/acz-ypcj-ynu
@freezing-airport-6809 so in Emirhan’s case the workflow has default service account but one of the task which is a spark task is also using the same service account. Emirhan pointed to this code used by spark https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/spark/spark.go#L95-L99 And i believe that it had been working fine for him until this change went in for admin to set the default service account in app config https://github.com/flyteorg/flyteadmin/blob/83d5ea767e7ca204c65be475f18ea73cfc159d5f/pkg/runtime/application_config_provider.go#L34 and hence uses the default service account even for the spark subtask. Do we ask users to set this in the task template ?
w
We were also thinking about creating a more flexible UI for the spark tasks here. Maybe we can adapt it such that the service account can be passed under the podSpec along with tolerations.
f
Hmm what do you mean by Ui
w
Sorry, I meant the task signature.
162 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by