our use case is that we cant just define our service in the task because we have dependencies that aren't compatible with flytekit so we have to build our training service separately

have you looked at this doc? https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/kftensorflow_plugin/tf_mnist.html#run-distributed-tensorflow-training why is there no

TFJob

?

yes I have seen this, they use:

TfJob(worker=Worker(replicas=1), ps=PS(replicas=1), chief=Chief(replicas=1)),

This means flyte will run the code in the task in the image. However our code has conflicts with flytekit so we need to build a separate image for it

i think you need to add flytekit to the images you supply to chief and worker as well.

I did that but the TFJob spec that gets built has a command that tries to run pyflyte-execute and points to the serialized python function for the task

i don't think so. @L godlike / @Kevin Su / @Yubo Wang any idea how to implement this?

I will take a look, maybe in 1 week

it seems that when you provide an image to TfJob Chief/worker it doesn't actually package the task module into the image

Hi @Alykhan Tejani did you examine the destination directory in your running pod’s container? Are you sure the task modules are not in there?

yeah I couldnt find them there

still working on other issues, probably will take a look at friday. You can take a look at this file. https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-kf-tensorflow/flytekitplugins/kftensorflow/task.py