our use case is that we cant just define our service in the task because we have dependencies that a...

late-traffic-56555

12/19/2023, 9:19 AM

our use case is that we cant just define our service in the task because we have dependencies that aren't compatible with flytekit so we have to build our training service separately

tall-lock-23197

12/19/2023, 9:56 AM

have you looked at this doc? https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/kftensorflow_plugin/tf_mnist.html#run-distributed-tensorflow-training why is there no

TFJob

late-traffic-56555

12/19/2023, 9:58 AM

yes I have seen this, they use:

Copy code

TfJob(worker=Worker(replicas=1), ps=PS(replicas=1), chief=Chief(replicas=1)),

This means flyte will run the code in the task in the image. However our code has conflicts with flytekit so we need to build a separate image for it

tall-lock-23197

12/19/2023, 10:09 AM

i think you need to add flytekit to the images you supply to chief and worker as well.

late-traffic-56555

12/19/2023, 10:10 AM

I did that but the TFJob spec that gets built has a command that tries to run pyflyte-execute and points to the serialized python function for the task

late-traffic-56555

12/19/2023, 10:10 AM

which I dont know how to inject into the image

late-traffic-56555

12/19/2023, 10:11 AM

alternatively is there an example anywhere that uses the

image

attribute of `Chief`/`Ps` ?

tall-lock-23197

12/19/2023, 10:15 AM

i don't think so. @damp-lion-88352 / @glamorous-carpet-83516 / @limited-raincoat-94253 any idea how to implement this?

🙏 1

damp-lion-88352

12/19/2023, 10:29 AM

I will take a look, maybe in 1 week

late-traffic-56555

12/19/2023, 1:04 PM

it seems that when you provide an image to TfJob Chief/worker it doesn't actually package the task module into the image

limited-raincoat-94253

12/19/2023, 6:50 PM

Hi @late-traffic-56555 did you examine the destination directory in your running pod’s container? Are you sure the task modules are not in there?

late-traffic-56555

01/02/2024, 8:32 AM

yeah I couldnt find them there

late-traffic-56555

01/02/2024, 8:33 AM

@damp-lion-88352 did you have a chance to look at this?

damp-lion-88352

01/02/2024, 8:34 AM

still working on other issues, probably will take a look at friday. You can take a look at this file. https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-kf-tensorflow/flytekitplugins/kftensorflow/task.py

👍 1

damp-lion-88352

01/02/2024, 8:34 AM

I don't think we currently support specifying image to TFJob Chief/Worker

damp-lion-88352

01/02/2024, 8:35 AM

If it is doable, you can create PR to support this feature.

damp-lion-88352

01/02/2024, 8:35 AM

https://github.com/flyteorg/flyte/blob/master/flyteplugins/go/tasks/plugins/k8s/kfoperators/tensorflow/tensorflow.go

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.