https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
r

Rob Ulbrich

08/31/2023, 12:01 PM
Hi Community! we want to use the MPI Plugin in order to launch distributed learning tasks on GCP. That plugin is using the MPI operator from Kubeflow. The MPI operator spawns launcher and worker pods. The launcher pod then needs to access the Flyte GCS bucket. Since we work with K8s service accounts to provide access to the GCS Flyte bucket the launcher pod cannot access the Flyte bucket, because we did not find a way to tell the MPI operator that it should use a certain K8s service account to launch the Launcher and worker pods. Do you have a solution for this? It is quite a blocker for us at the moment. Thanks for your advice!
k

Ketan (kumare3)

08/31/2023, 1:55 PM
Is this tensorflow or pytorch
Also let me take a look, cc @Kevin Su ?
f

Fabio Grätz

08/31/2023, 2:16 PM
You could bind the respective GCP service account via workload identities to the default service account used by the launcher and worker as a workaround maybe? 🤔
k

Ketan (kumare3)

08/31/2023, 2:25 PM
@Rob Ulbrich what service account are you passing in pyflyte run
r

Rob Ulbrich

08/31/2023, 2:30 PM
The example workflow was tensorflow
Currently, we do not pass the service account directly, but have a service account per workspace that is set in the configuration
k

Ketan (kumare3)

08/31/2023, 2:35 PM
Hmm is that the default service account?
If you bind it to default it should work? Cc @Fabio Grätz
f

Fabio Grätz

08/31/2023, 2:36 PM
We don’t use MPI but we do use the default service account in every domain namespace and bind a GCP service account to it.
r

Rob Ulbrich

08/31/2023, 2:37 PM
pyflyte run --remote -p workflow -d dev --service-account default  mnist.py horovod_training_wf
f

Fabio Grätz

08/31/2023, 2:38 PM
🤔
r

Rob Ulbrich

08/31/2023, 2:38 PM
So I just ran the MPI Workflow with the 
--service-account
flag. The service account to use is 
default
.
But when you look at the service account, that the launcher is using, it is a totally different
f

Fabio Grätz

08/31/2023, 2:38 PM
Same happens when you don’t specify the service account in the pyflyte command?
r

Rob Ulbrich

08/31/2023, 2:39 PM
Yes
f

Fabio Grätz

08/31/2023, 2:39 PM
It appears as if there is a service account created temporarily for each node/execution.
What is the service account of the worker?
k

Ketan (kumare3)

08/31/2023, 2:41 PM
We will have to look at the training operator 😤
f

Fabio Grätz

08/31/2023, 2:41 PM
The launcher might need a dedicated service account because it creates other pods? (Suggested by the name)
I wonder whether the workers have the default service account then at least.
Could you paste the manifest for the 
SparkApplication
?
r

Rob Ulbrich

08/31/2023, 2:42 PM
Okay, the worker use the service account, that I specify in 
pyflyte
image.png
f

Fabio Grätz

08/31/2023, 2:43 PM
kubectl -n <namespace name> get serviceaccount default -o yaml
What is the output of this?
r

Rob Ulbrich

08/31/2023, 2:45 PM
Copy code
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    <http://iam.gke.io/gcp-service-account|iam.gke.io/gcp-service-account>: flyte-workflow-dev-sa@<redacted>
  creationTimestamp: "2023-08-03T07:15:09Z"
  name: default
  namespace: dev
  resourceVersion: "676089"
  uid: dd7f67ce-2530-4c59-883d-2e08bebeebbb
f

Fabio Grätz

08/31/2023, 2:45 PM
Ok workflow identities are there
r

Rob Ulbrich

08/31/2023, 2:45 PM
The 
default
service acccont is the one that we want to use
f

Fabio Grätz

08/31/2023, 2:45 PM
Assuming the required IAM permissions are configured too.
r

Rob Ulbrich

08/31/2023, 2:45 PM
It has the permissions to access the GCS buckets
Sure
For Pod Tasks everything works fine
f

Fabio Grätz

08/31/2023, 2:45 PM
Ok 👍
I don’t know enough about MPI tbh 😕
But does the launcher really need to access anything in GCP?
Or is its job only to create other pods?
r

Rob Ulbrich

08/31/2023, 2:46 PM
Yes, I see error logs in the launcher pod, that it needs to access flyte data from the Flyte GCS bucket
f

Fabio Grätz

08/31/2023, 2:46 PM
Probably the code for fast registration 🤔
k

Ketan (kumare3)

08/31/2023, 2:47 PM
@Fabio Grätz MPI does not need a launcher - it’s a peer to peer protocol
f

Fabio Grätz

08/31/2023, 2:47 PM
Yes but what does the launcher pod do? 🤔
k

Ketan (kumare3)

08/31/2023, 2:48 PM
Yes @Rob Ulbrich does a regular pod task work with pyflyte run, if so this should work
Beats me for launcher
r

Rob Ulbrich

08/31/2023, 2:50 PM
I guess we need the launcher pod, because I am running Flyte's own example code, where it uses the launcher 😉
Copy code
@task(
    container_image="europe-west4-docker.pkg.dev/<redacted>/flyte/mpi-mnist:latest",
    task_config=MPIJob(
        launcher=Launcher(
            replicas=1,
        ),
        worker=Worker(
            replicas=2,
        ),
    ),
    retries=3,
    requests=Resources(cpu="1", mem="1000Mi"),
    limits=Resources(cpu="2", mem="4000Mi")
)
I suppose the launcher communicates to the workers and sends data to them and receives the learned parameters and brings everything toegether
k

Ketan (kumare3)

08/31/2023, 2:56 PM
Haha, I think the launcher maybe a misnomer. Let us try today
f

Fabio Grätz

08/31/2023, 3:02 PM
Sorry, I’m mixing spark and MPI now 🤦‍♂️
@Dimss consider trying the v2 controller, which doesn’t depend on ServiceAccounts 🙂
Which version of mpi operator or training operator (?) are you running?
r

Rob Ulbrich

08/31/2023, 3:08 PM
Seems to be image version v1-855e096
f

Fabio Grätz

08/31/2023, 3:09 PM
But mpi operator or training operator?
r

Rob Ulbrich

08/31/2023, 3:10 PM
Copy code
<https://github.com/kubeflow/training-operator>
using branch v1.7-branch
Standalone
The training operator actually spawns the launcher and the workers
f

Fabio Grätz

08/31/2023, 3:13 PM
Mh ok 😕 The issue above suggested that there are versions that need a service account for the launcher whereas others don’t. But the training operator is definitely newer than the mpi operator. We don’t use mpi, I honestly don’t know the answer, sorry 🤷‍♂️
@Byron Hsu @Yubo Wang you guys use mpi, or am I mistaken?
k

Ketan (kumare3)

08/31/2023, 4:10 PM
ohh right they do
also maybe @Rahul Mehta?
@Samhita Alla also has run i think
r

Rahul Mehta

08/31/2023, 4:11 PM
Ah we actually are opting to use dask instead of the Kubeflow training operator
y

Yubo Wang

08/31/2023, 4:25 PM
we do, but unfortunately we don’t use k8s service account. Our MPI job talks to hadoop only, no cloud storage😅
f

Fabio Grätz

08/31/2023, 5:28 PM
Do you build the code into the image/aka no fast registration?
y

Yubo Wang

08/31/2023, 5:38 PM
we have our internal blob storage, it uses certs for authentication. And we mount the certs through some internal service
r

Rob Ulbrich

09/01/2023, 2:09 PM
I found where the problem is located in the training operator and created an issue in the Github repository.
But I actually found another issue: If there is a sidecar container, then MPI jobs can't either be spawned, because the launcher connects to the workers using 
kubectl exec
. It then connects to the sidecar container instead to the main container where the worker is located
k

Ketan (kumare3)

09/01/2023, 3:23 PM
@Rob Ulbrich so what do you want to train torch?
If so check out flytekit plugin for elastic
r

Rob Ulbrich

09/04/2023, 7:56 AM
We have a proprietary ML app that needs MPI, can't use flytekit plugin for elastic
I will create an issue for Flyte
k

Ketan (kumare3)

09/04/2023, 3:01 PM
Please file one with details - we will try to reproduce it
r

Rob Ulbrich

09/05/2023, 7:38 AM
I have decided to create another issue to the kubeflow team for the training operator, because it is easier to fix for them
Powered by