• Emirhan Karagül

    Emirhan Karagül

    2 months ago
    Hi all, I am back with yet another question about spark. I am trying to upgrade our
    flyte-core
    deployments from
    v0.18.2
    to
    v1.0.1
    . So far I didn't have any issues with regular tasks. But when I'm trying to execute spark tasks, I see that the default service account is assigned to the spark tasks. Is there something I need to change when upgrading to 1.+ versions? For context, I am using
    flytekit
    0.26.1. Thanks in advance!
  • p

    Prafulla Mahindrakar

    2 months ago
    Hi @Emirhan Karagül, The version of flytekit is old and you should upgrade to latest for fixes around this feature. Also you can set the workflow execution config for your spark workflows or you can assign it to project-domain which will allow you to use the spark service account for all workflows running under that project-domain https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_workflow-execution-config.html Also there is proposal to have such settings at project level too
  • Ketan (kumare3)

    Ketan (kumare3)

    2 months ago
    @Emirhan Karagül preference is that you move to Flyte 1 throughout- though @Prafulla Mahindrakar this should work right
  • Emirhan Karagül

    Emirhan Karagül

    2 months ago
    Thanks for the quick responses. I tried upgrading flytekit to 1.+ versions as well, but I still get the default service account during spark task execution. I suspect this is happening because we are using remote API to register/execute entities.
  • To give some more context@Ketan (kumare3): I also tested this with pyflyte serialize + register. During workflow execution, spark task is executed with the default service account. Didn't debug it further yet.
  • Ketan (kumare3)

    Ketan (kumare3)

    2 months ago
    Are you seeing the service account in security context
  • p

    Prafulla Mahindrakar

    2 months ago
    Hi @Emirhan Karagül did you upgrade all the components to 1.0.1
  • yes @Ketan (kumare3) the current way that Emirhan is doing also should work. Just wanted to confirm if all the component bits are latest too expecially flyteadmin and flytekit
  • Emirhan Karagül

    Emirhan Karagül

    2 months ago
    Yes, my flyte related requirements are:
    flyteidl==1.0.1
    flytekit==1.0.3
    flytekitplugins-kfpytorch==1.0.3
    flytekitplugins-spark==1.0.3
  • For the security context, when the task is extracted the
    _security_ctx
    field is
    None
  • p

    Prafulla Mahindrakar

    2 months ago
    what about your flyte pods images and there versions . Are they also using 1.0.1 components
  • Emirhan Karagül

    Emirhan Karagül

    2 months ago
    Yep
  • For sanity check, I downgraded flyte-core to v0.18.2 and flyte components to v.0.26. Now the spark task is assigned the spark
    serviceAccount
    .
  • If this is any useful to you guys as well, we can make a short call and I can demonstrate how I am registering the tasks, and how the service account is not being assigned. Also would be open to creating a PR if there is any bug on flyte side.
  • p

    Prafulla Mahindrakar

    2 months ago
    Sure whenever you are available we can sync up here https://meet.google.com/acz-ypcj-ynu
  • @Ketan (kumare3) so in Emirhan’s case the workflow has default service account but one of the task which is a spark task is also using the same service account. Emirhan pointed to this code used by spark https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/spark/spark.go#L95-L99 And i believe that it had been working fine for him until this change went in for admin to set the default service account in app config https://github.com/flyteorg/flyteadmin/blob/83d5ea767e7ca204c65be475f18ea73cfc159d5f/pkg/runtime/application_config_provider.go#L34 and hence uses the default service account even for the spark subtask. Do we ask users to set this in the task template ?
  • Emirhan Karagül

    Emirhan Karagül

    2 months ago
    We were also thinking about creating a more flexible UI for the spark tasks here. Maybe we can adapt it such that the service account can be passed under the podSpec along with tolerations.
  • Ketan (kumare3)

    Ketan (kumare3)

    2 months ago
    Hmm what do you mean by Ui
  • Emirhan Karagül

    Emirhan Karagül

    2 months ago
    Sorry, I meant the task signature.