Hi all I am back with yet another question about spark I am

Question

Hi all I am back with yet another question about spark I am trying to upgrade our `flyte core` deployments from `v0 18 2` to `v1 0 1` So far I didn t have any issues with regular tasks But when I m trying to execute spark tasks I see that the default service account is assigned to the spark tasks Is there something I need to change when upgrading to 1 + versions For context I am using `flytekit` 0 26 1 Thanks in advance

Prafulla Mahindrakar · Answer

Hi < Emirhan Karagül> The version of flytekit is old and you should upgrade to latest for fixes around this feature Also you can set the workflow execution config for your spark workflows or you can assign it to project domain which will allow you to use the spark service account for all workflows running under that project domain Also there is proposal to have such settings at project level too

Ketan (kumare3) · Answer

< Emirhan Karagül> preference is that you move to Flyte 1 throughout though < Prafulla Mahindrakar> this should work right

Emirhan Karagül · Answer

Thanks for the quick responses I tried upgrading flytekit to 1 + versions as well but I still get the default service account during spark task execution I suspect this is happening because we are using remote API to register execute entities

Emirhan Karagül · Answer

To give some more context< Ketan kumare3 > I also tested this with pyflyte serialize + register During workflow execution spark task is executed with the default service account Didn t debug it further yet

Ketan (kumare3) · Answer

Are you seeing the service account in security context

Prafulla Mahindrakar · Answer

Hi < Emirhan Karagül> did you upgrade all the components to 1 0 1

Prafulla Mahindrakar · Answer

yes < Ketan kumare3 > the current way that Emirhan is doing also should work Just wanted to confirm if all the component bits are latest too expecially flyteadmin and flytekit

Emirhan Karagül · Answer

Yes my flyte related requirements are `flyteidl==1 0 1` `flytekit==1 0 3` `flytekitplugins kfpytorch==1 0 3` `flytekitplugins spark==1 0 3`

Emirhan Karagül · Answer

For the security context when the task is extracted the ` security ctx` field is `None`

Prafulla Mahindrakar · Answer

what about your flyte pods images and there versions Are they also using 1 0 1 components

Emirhan Karagül · Answer

Yep

Emirhan Karagül · Answer

For sanity check I downgraded flyte core to v0 18 2 and flyte components to v 0 26 Now the spark task is assigned the spark `serviceAccount`

Emirhan Karagül · Answer

If this is any useful to you guys as well we can make a short call and I can demonstrate how I am registering the tasks and how the service account is not being assigned Also would be open to creating a PR if there is any bug on flyte side

Prafulla Mahindrakar · Answer

Sure whenever you are available we can sync up here <https meet google com acz ypcj ynu>

Prafulla Mahindrakar · Answer

< Ketan kumare3 > so in Emirhan s case the workflow has default service account but one of the task which is a spark task is also using the same service account Emirhan pointed to this code used by spark And i believe that it had been working fine for him until this change went in for admin to set the default service account in app config and hence uses the default service account even for the spark subtask Do we ask users to set this in the task template

Emirhan Karagül · Answer

We were also thinking about creating a more flexible UI for the spark tasks <https flyte org slack com archives CREL4QVAQ p1644362333173139 thread ts=1644260738 361739 amp cid=CREL4QVAQ|here> Maybe we can adapt it such that the service account can be passed under the podSpec along with tolerations

Ketan (kumare3) · Answer

Hmm what do you mean by Ui

Emirhan Karagül · Answer

Sorry I meant the task signature