Hi all,
I am back with yet another question about ...
# flyte-deploymente
Emirhan Karagül
05/30/2022, 4:21 PMHi all,
I am back with yet another question about spark. I am trying to upgrade our
flyte-corev0.18.2v1.0.1flytekitp
Prafulla Mahindrakar
05/30/2022, 4:35 PMHi @Emirhan Karagül,
The version of flytekit is old and you should upgrade to latest for fixes around this feature.
Also you can set the workflow execution config for your spark workflows or you can assign it to project-domain which will allow you to use the spark service account for all workflows running under that project-domain https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_workflow-execution-config.html
Also there is proposal to have such settings at project level too
k
Ketan (kumare3)
05/30/2022, 4:49 PM
@Emirhan Karagül preference is that you move to Flyte 1 throughout- though @Prafulla Mahindrakar this should work right
e
Emirhan Karagül
05/30/2022, 7:09 PMThanks for the quick responses. I tried upgrading flytekit to 1.+ versions as well, but I still get the default service account during spark task execution. I suspect this is happening because we are using remote API to register/execute entities.
Emirhan Karagül
05/30/2022, 7:49 PMTo give some more context@Ketan (kumare3): I also tested this with pyflyte serialize + register. During workflow execution, spark task is executed with the default service account. Didn't debug it further yet.
k
Ketan (kumare3)
05/31/2022, 12:50 AM
Are you seeing the service account in security context
p
Prafulla Mahindrakar
05/31/2022, 3:24 AMHi @Emirhan Karagül did you upgrade all the components to 1.0.1
Prafulla Mahindrakar
05/31/2022, 5:26 AMyes @Ketan (kumare3) the current way that Emirhan is doing also should work. Just wanted to confirm if all the component bits are latest too expecially flyteadmin and flytekit
e
Emirhan Karagül
05/31/2022, 8:25 AMYes, my flyte related requirements are:
flyteidl==1.0.1flytekit==1.0.3flytekitplugins-kfpytorch==1.0.3flytekitplugins-spark==1.0.3Emirhan Karagül
05/31/2022, 8:26 AMFor the security context, when the task is extracted the
_security_ctxNonep
Prafulla Mahindrakar
05/31/2022, 8:26 AMwhat about your flyte pods images and there versions . Are they also using 1.0.1 components
e
Emirhan Karagül
05/31/2022, 8:27 AMYep
Emirhan Karagül
05/31/2022, 8:44 AMFor sanity check, I downgraded flyte-core to v0.18.2 and flyte components to v.0.26. Now the spark task is assigned the spark
serviceAccountEmirhan Karagül
05/31/2022, 8:46 AMIf this is any useful to you guys as well, we can make a short call and I can demonstrate how I am registering the tasks, and how the service account is not being assigned. Also would be open to creating a PR if there is any bug on flyte side.
p
Prafulla Mahindrakar
05/31/2022, 9:10 AMSure whenever you are available we can sync up here https://meet.google.com/acz-ypcj-ynu
Prafulla Mahindrakar
05/31/2022, 2:43 PM@Ketan (kumare3) so in Emirhan’s case the workflow has default service account but one of the task which is a spark task is also using the same service account. Emirhan pointed to this code used by spark https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/spark/spark.go#L95-L99
And i believe that it had been working fine for him until this change went in for admin to set the default service account in app config https://github.com/flyteorg/flyteadmin/blob/83d5ea767e7ca204c65be475f18ea73cfc159d5f/pkg/runtime/application_config_provider.go#L34
and hence uses the default service account even for the spark subtask.
Do we ask users to set this in the task template ?
e
Emirhan Karagül
06/01/2022, 10:06 AMWe were also thinking about creating a more flexible UI for the spark tasks here. Maybe we can adapt it such that the service account can be passed under the podSpec along with tolerations.
k
Ketan (kumare3)
06/01/2022, 1:40 PM
Hmm what do you mean by Ui
e
Emirhan Karagül
06/01/2022, 1:40 PMSorry, I meant the task signature.
162 Views