I am giving the `kftensorflow` plugin a try and struggling t

Question

I am giving the `kftensorflow` plugin a try and struggling to setup correctly I don t see issues on github so I figured I would try here to see if anyone can catch anything I am on flytepropeller 1 6 1 and flytekit 1 7 0 Details in thread

Len Strnad · Answer

I am using tensorflows MultiWorkerMirrored strategy which only requires workers as far as I understand where one is randomly chosen to be the chief This is what I am seeing ``` task task config=TfJob worker=Worker replicas=5 requests=Resources cpu= 15 mem= 30Gi gpu= 2 limits=Resources cpu= 15 mem= 30Gi gpu= 2 restart policy=RestartPolicy FAILURE ps=PS replicas=0 chief=Chief replicas=0 run policy=RunPolicy clean pod policy=CleanPodPolicy RUNNING ``` fails to schedule In the training operator I am seeing ```INFO <http TFJob kubeflow org|TFJob kubeflow org> azg7k57nsljsq6fwkljk n5 0 n5 n0 0 not found tfjob redwood development azg7k57nsljsq6fwkljk n5 0 n5 n0 0 unable to fetch TFJob ``` and in the console I am seeing ``` RuntimeExecutionError max number of system retry attempts 51 50 exhausted ``` Then I try to specify a chief in case my assumption about needing workers is incorrect ``` task task config=TfJob worker=Worker replicas=5 requests=Resources cpu= 15 mem= 30Gi gpu= 2 limits=Resources cpu= 15 mem= 30Gi gpu= 2 restart policy=RestartPolicy FAILURE ps=PS replicas=0 worker=Chief replicas=1 requests=Resources cpu= 15 mem= 30Gi gpu= 2 limits=Resources cpu= 15 mem= 30Gi gpu= 2 restart policy=RestartPolicy FAILURE run policy=RunPolicy clean pod policy=CleanPodPolicy RUNNING ``` but I see the same issue that it can t schedule and the training operator is sad Then I try to specify a parameter server as follows ``` task task config=TfJob worker=Worker replicas=5 requests=Resources cpu= 15 mem= 30Gi gpu= 2 limits=Resources cpu= 15 mem= 30Gi gpu= 2 restart policy=RestartPolicy FAILURE ps=PS replicas=1 requests=Resources cpu= 7 mem= 15Gi gpu= 1 limits=Resources cpu= 7 mem= 15Gi gpu= 1 restart policy=RestartPolicy NEVER chief=Chief replicas=1 requests=Resources cpu= 7 mem= 15Gi gpu= 1 limits=Resources cpu= 7 mem= 15Gi gpu= 1 restart policy=RestartPolicy ALWAYS run policy=RunPolicy clean pod policy=CleanPodPolicy RUNNING ``` Then things schedule but hang indefinitely because my guess is that the multiworkermirrored strategy is confused why there is a PS in the TF CONFIG

Len Strnad · Answer

cc < Yubo Wang> I think you added this functionality pray

Ketan (kumare3) · Answer

yes also cc < Fabio Grätz> who use this a lot

Yubo Wang · Answer

Let me reproduce your issue < Len Strnad>

Len Strnad · Answer

Our training operator is v1 5 0 with the most basic deployment since we are testing this out for now

Yubo Wang · Answer

interesting < Len Strnad> can you do `kubectl describe tfjob` with ``` task task config=TfJob worker=Worker replicas=5 requests=Resources cpu= 15 mem= 30Gi gpu= 2 limits=Resources cpu= 15 mem= 30Gi gpu= 2 restart policy=RestartPolicy FAILURE ps=PS replicas=0 chief=Chief replicas=0 run policy=RunPolicy clean pod policy=CleanPodPolicy RUNNING ```

Yubo Wang · Answer

ah I think the versions are messed up can you try to use the propeller from the latest release

Len Strnad · Answer

Will `describe` in one sec have to fast register and wait for nodes

Len Strnad · Answer

Is there a point in time you want me to run `describe`

Yubo Wang · Answer

just after the job launched

Yubo Wang · Answer

after your tfjob is created

Len Strnad · Answer

Everything is dated for yesterday

Len Strnad · Answer

The workflow error is ```Workflow redwood development flyte redwood tensorflow train eval workflow failed RuntimeExecutionError max number of system retry attempts 51 50 exhausted Last known status message failed at Node n5 n0 RuntimeExecutionError failed during plugin execution caused by failed to execute handle for plugin tensorflow panic when executing a plugin tensorflow Stack goroutine 1063 running runtime debug Stack usr local go src runtime debug stack go 24 +0x65 go src github com flyteorg flytepropeller pkg controller nodes task handler go 389 +0xfe panic 0x2291e00 0x3fb42a0 usr local go src runtime panic go 838 +0x207 0x0 0xc0097e7f68 0x7fd9e339c108 0xc0049c2c90 0x2bd3228 0xc008799d40 go pkg mod github com flyteorg flyteplugins v1 0 67 go tasks plugins k8s kfoperators tensorflow tensorflow go 203 +0xc8 0x2bbeb18 0xc008797f80 0x2bcaf00 0xc008ea80c0 0xc011b2c248 go src github com flyteorg flytepropeller pkg controller nodes task k8s plugin manager go 283 +0xc83 0xa 0x2bbfbb8 0x401b590 0x2ba3e00 0xc0006d3860 0x7fd9bc1f5780 0xc000d6a4e0 0x2bcf670 0xc000dd5c90 go src github com flyteorg flytepropeller pkg controller nodes task k8s plugin manager go 338 +0x685 0x2bbeb18 0xc008797d10 0x2bc07d8 0xc001e54140 0x3f658e8 go src github com flyteorg flytepropeller pkg controller nodes task handler go 396 +0x184 0xc0013fab10 0x2baac78 0xc00152d180 0xc001549f80 0xc001549fb0 0xc00163a000 0x2bc07d8 0xc0021a6140 0xc000621080 go src github com flyteorg flytepropeller pkg controller nodes task handler go 398 +0x9a 0xc0013fab10 0x2baac78 0xc00152d180 0xc001549f80 0xc001549fb0 0xc00163a000 0x2bc07d8 0xc0021a6140 0xc000621080 go src github com flyteorg flytepropeller pkg controller nodes task handler go 666 +0x1ba5 0xc00138bd40 0xc001638310 0x0 0xc0005aa5c0 0x4 0x4 0xc001638330 go src github com flyteorg flytepropeller pkg controller nodes dynamic handler go 70 +0xd8 0xc00138bd40 0xc001638310 0x0 0xc0005aa5c0 0x4 0x4 0xc001638330 go src github com flyteorg flytepropeller pkg controller nodes dynamic handler go 224 +0x9d0 0x2bbeb18 0xc008797890 0x2bc05d8 0xc0013cf180 0xc008ea8000 0x2bd4d70 0xc008f03e10 go src github com flyteorg flytepropeller pkg controller nodes executor go 460 +0x157 0x2bbeb18 0xc008797890 0xc008ea8000 0x2bc05d8 0xc0013cf180 go src github com flyteorg flytepropeller pkg controller nodes executor go 593 +0x227 0x2bbeb18 0xc008797890 0x2baade0 0xc008797860 0xc008ea8000 0x2bc05d8 0xc0013cf180 go src github com flyteorg flytepropeller pkg controller nodes executor go 820 +0x3c5 0x2bbeb18 0xc008797350 0x2bcf238 0xc005e3d4f0 0x2baade0 0xc008797860 0x2bbee28 0xc0027a5400 0x2bccb50 go src github com flyteorg flytepropeller pkg controller nodes executor go 1018 +0x705 0x2bbeb18 0xc008797350 0x2bccc90 0xc008dfda40 0x2bd4d70 0xc008f03ba0 0x2bccb50 0xc00c17c900 go src github com flyteorg flytepropeller pkg controller nodes branch handler go 148 +0x409 0x2bbeb18 0xc008797350 0x2bbedf0 0xc008f51e00 0x2bccc90 0xc008dfda40 0x2bbee28 0xc0027a5400 go src github com flyteorg flytepropeller pkg controller nodes branch handler go 103 +0x949 0x2bbeb18 0xc008797350 0x2bccc90 0xc008dfda40 go src github com flyteorg flytepropeller pkg controller nodes branch handler go 115 +0x137 0x2bbeb18 0xc008797350 0x2bc0598 0xc00163a840 0xc008dfda40 0x2bd4d70 0xc008f03ba0 go src github com flyteorg flytepropeller pkg controller nodes executor go 460 +0x157 0x2bbeb18 0xc008797350 0xc008dfda40 0x2bc0598 0xc00163a840 go src github com flyteorg flytepropeller pkg controller nodes executor go 593 +0x227 0x2bbeb18 0xc008797350 0x2ba3040 0xc0027a5400 0xc008dfda40 0x2bc0598 0xc00163a840 go src github com flyteorg flytepropeller pkg controller nodes executor go 820 +0x3c5 0x2bbeb18 0xc00bd6a750 0x2bcf238 0xc008545630 0x2ba3040 0xc0027a5400 0x2bbee28 0xc0027a5400 0x2bccb50 go src github com flyteorg flytepropeller pkg controller nodes executor go 1018 +0x705 0x2bbeb18 0xc00bd6a750 0x2bcf238 0xc008545630 0x2ba3040 0xc0027a5400 0x2bbee28 0xc0027a5400 0x2bccb50 go src github com flyteorg flytepropeller pkg controller nodes executor go 858 +0x3c5 0x2bbeb18 0xc00bd6a750 0x2bcf238 0xc008545630 0x2ba3040 0xc0027a5400 0x2bbee28 0xc0027a5400 0x2bccb50 go src github com flyteorg flytepropeller pkg controller nodes executor go 1025 +0x935 0x2bbeb18 0xc00bd6a750 0xc0027a5400 go src github com flyteorg flytepropeller pkg controller workflow executor go 147 +0x1b3 0x2bbeb18 0xc00bd6a750 0xc0027a5400 go src github com flyteorg flytepropeller pkg controller workflow executor go 393 +0x40f 0x2bbeb18 0xc00bd6a750 0xc00e4277d0 0x214b2a0 go src github com flyteorg flytepropeller pkg controller handler go 142 +0x18e 0x2bbeb18 0xc00bd6a2a0 0xc0027a4a00 go src github com flyteorg flytepropeller pkg controller handler go 143 +0x495 0x2bbeb18 0xc00bd6a2a0 0xc003165710 0x13 0xc003165724 0x14 go src github com flyteorg flytepropeller pkg controller handler go 259 +0xe4a 0xc00e427f28 0x214b2a0 0xc0091943a0 go src github com flyteorg flytepropeller pkg controller workers go 88 +0x510 0x2bbeb18 0xc00bd6a2a0 go src github com flyteorg flytepropeller pkg controller workers go 99 +0xf1 0x2bbeb18 0xc002b32120 go src github com flyteorg flytepropeller pkg controller workers go 115 +0xbd go src github com flyteorg flytepropeller pkg controller workers go 150 +0x59 created by go src github com flyteorg flytepropeller pkg controller workers go 147 +0x285 ```

Yubo Wang · Answer

interesting so the tfjob is never created

Len Strnad · Answer

I think things are failing when trying to create the tfjob crd for this job and therefore doesn t register at all Just a guess

Len Strnad · Answer

yeah

Yubo Wang · Answer

can you try the latest release of propeller

Len Strnad · Answer

Yeah that is what I will try next but I won t be able to until next week and might need help from < Bernhard Stadlbauer> my colleague

Len Strnad · Answer

I ll try that and post back here in a few days or sooner if possible

Yubo Wang · Answer

I will improve the doc on matching versions of flytekit and flytepropeller later today

Len Strnad · Answer

Sweet Thanks a ton for the help Im very excited to get this up and running

Len Strnad · Answer

Not to derail but are there any plans to support an update to tf cluster replicas It would be nice to let the task args define the number of workers I suppose one can always modify the deployment with python kubernetes in the task body

Ketan (kumare3) · Answer

Aaha just for this < Fabio Grätz> < Bernhard Stadlbauer> and < Byron Hsu> are working on config overrides

Len Strnad · Answer

We are on flyte propeller version `v1 1 96` Do we think a possible fix has been introduced since I ll try to upgrade to `v1 1 98` in any case

Yubo Wang · Answer

that propeller version should be good do you have more logs you can provide

Len Strnad · Answer

`1 1 96` should be good

Len Strnad · Answer

Possibly I ll see what comes up when I try `1 1 98` today

Yubo Wang · Answer

my versions are flytekit 1 7 0 and propeller 1 1 96 and it is working for me I can investigate more with more logs

Len Strnad · Answer

Are you using `interruptible=True` for your task

Yubo Wang · Answer

no

Yubo Wang · Answer

I can test with that too

Len Strnad · Answer

We just added a non interruptible node pool I ll try with interruptible=False just in case at some point today