Hey yall I m running into the same stack

Question

Hey yall I m running into the same stack <https github com flyteorg flyte issues 2786|trace outlined on this issue> and seemed to be addressed I m on a recent version of flyte propeller data catalog admin 1 1 78 but trying to use map task for 30k inputs with concurrency=10 Am I stressing the k8s array beyond it s limitation

Yee · Answer

can you post the specific stack trace you re seeing even if it looks almost exactly the same as the one in the issue unless the versions are exactly the same things like line numbers will have changed etc it s helpful to see exactly what s going on

Anooj Patel · Answer

Sure ```Workflow apfelstrudel development apfelstrudel flyte workflows search workflow fragmentlevel hpo deeplearning cpu failed RuntimeExecutionError max number of system retry attempts 51 50 exhausted Last known status message failed at Node n1 RuntimeExecutionError failed during plugin execution caused by failed to execute handle for plugin k8s array panic when executing a plugin k8s array Stack goroutine 232 running runtime debug Stack usr local go src runtime debug stack go 24 +0x65 go src github com flyteorg flytepropeller pkg controller nodes task handler go 390 +0xfe panic 0x2276320 0x3f6e0d0 usr local go src runtime panic go 838 +0x207 go pkg mod github com flyteorg flytestdlib v1 0 15 bitarray bitset go 33 0xc01dd23a70 0x2ba3080 0xc00b0aa2c0 0xc01652ab40 0x279b758 go pkg mod github com flyteorg flyteplugins v1 0 45 go tasks plugins array core metadata go 33 +0x1e1 0xc0007ae680 0x2b89030 0xc004753760 0x2b89030 0xc004753810 0x2b96d18 0xc01dd23a70 0x2ba3080 0xc00b0aa2c0 go pkg mod github com flyteorg flyteplugins v1 0 45 go tasks plugins array k8s executor go 96 +0x268 0x2b96d18 0xc01dd236e0 0x2b99458 0xc0048ee810 0x0 go src github com flyteorg flytepropeller pkg controller nodes task handler go 397 +0x178 0xc0014c90f8 0x2b83398 0xc0014baa60 0xc001683920 0xc001683950 0xc001683980 0x2b98998 0xc020838640 0xc0003a4000 go src github com flyteorg flytepropeller pkg controller nodes task handler go 399 +0x9a 0xc0014c90f8 0x2b83398 0xc0014baa60 0xc001683920 0xc001683950 0xc001683980 0x2b98998 0xc020838640 0xc0003a4000 go src github com flyteorg flytepropeller pkg controller nodes task handler go 672 +0x1de5 0xc0004000d0 0xc0005b29c0 0x0 0xc000fa4000 0x4 0x4 0xc0005b29e0 go src github com flyteorg flytepropeller pkg controller nodes dynamic handler go 70 +0xd8 0xc0004000d0 0xc0005b29c0 0x0 0xc000fa4000 0x4 0x4 0xc0005b29e0 go src github com flyteorg flytepropeller pkg controller nodes dynamic handler go 224 +0x9d0 0x2b96d18 0xc01dd23050 0x2b98798 0xc000a38280 0xc0318a0000 0x2baced0 0xc004faa4e0 go src github com flyteorg flytepropeller pkg controller nodes executor go 460 +0x157 0x2b96d18 0xc01dd23050 0xc0318a0000 0x2b98798 0xc000a38280 go src github com flyteorg flytepropeller pkg controller nodes executor go 593 +0x227 0x2b96d18 0xc01dd23050 0x2b7b760 0xc00a42cf00 0xc0318a0000 0x2b98798 0xc000a38280 go src github com flyteorg flytepropeller pkg controller nodes executor go 820 +0x3c5 0x2b96d18 0xc01dd22a20 0x2ba7398 0xc043119770 0x2b7b760 0xc00a42cf00 0x2b96ff0 0xc00a42cf00 0x2ba4d50 go src github com flyteorg flytepropeller pkg controller nodes executor go 1018 +0x705 0x2b96d18 0xc01dd22a20 0x2ba7398 0xc043119770 0x2b7b760 0xc00a42cf00 0x2b96ff0 0xc00a42cf00 0x2ba4d50 go src github com flyteorg flytepropeller pkg controller nodes executor go 858 +0x3c5 0x2b96d18 0xc01dd22a20 0x2ba7398 0xc043119770 0x2b7b760 0xc00a42cf00 0x2b96ff0 0xc00a42cf00 0x2ba4d50 go src github com flyteorg flytepropeller pkg controller nodes executor go 1025 +0x935 0x2b96d18 0xc01dd22a20 0x2ba7398 0xc043119770 0x2b7b760 0xc00a42cf00 0x2b96ff0 0xc00a42cf00 0x2ba4d50 go src github com flyteorg flytepropeller pkg controller nodes executor go 858 +0x3c5 0x2b96d18 0xc01dd22a20 0x2ba7398 0xc043119770 0x2b7b760 0xc00a42cf00 0x2b96ff0 0xc00a42cf00 0x2ba4d50 go src github com flyteorg flytepropeller pkg controller nodes executor go 1025 +0x935 0x2b96d18 0xc01dd22a20 0xc00a42cf00 go src github com flyteorg flytepropeller pkg controller workflow executor go 147 +0x1b3 0x2b96d18 0xc01dd22a20 0xc00a42cf00 go src github com flyteorg flytepropeller pkg controller workflow executor go 393 +0x40f 0x2b96d18 0xc01dd22a20 0xc00567b7d0 0x2130000 go src github com flyteorg flytepropeller pkg controller handler go 142 +0x18e 0x2b96d18 0xc01dd225d0 0xc013aa6500 go src github com flyteorg flytepropeller pkg controller handler go 143 +0x495 0x2b96d18 0xc01dd225d0 0xc01764c9c0 0x18 0xc01764c9d9 0x14 go src github com flyteorg flytepropeller pkg controller handler go 259 +0xe4a 0xc00567bf28 0x2130000 0xc045a12cc0 go src github com flyteorg flytepropeller pkg controller workers go 88 +0x510 0x2b96d18 0xc01dd225d0 go src github com flyteorg flytepropeller pkg controller workers go 99 +0xf1 0x2b96d18 0xc019212030 go src github com flyteorg flytepropeller pkg controller workers go 115 +0xbd go src github com flyteorg flytepropeller pkg controller workers go 150 +0x59 created by go src github com flyteorg flytepropeller pkg controller workers go 147 +0x285 ``` is the error at the top of the workflow panel and ```max number of system retry attempts 51 50 exhausted system failure ``` for the map task side panel

Anooj Patel · Answer

For versions ``` gt kubectl describe po | grep Image Image Image Image Image Image Image Image Image Image Image Image ```

Dan Rammer (hamersaw) · Answer

Hey < Anooj Patel> this certainly seems like a bug Very similar to the one you linked but should be an easier fix In the logs do you see anything like `array size gt max allowed requested v allowed v ` My guess is that this lookup phase is returning an error which is not caught here will submit a bug fix soon but want to see if we can workaround

Yee · Answer

i see

Yee · Answer

give us some time to investigate

Yee · Answer

give dan rather some time to investigate slightly smiling face

Anooj Patel · Answer

< Dan Rammer hamersaw > < Yee> thanks for the heads up I ll relaunch and rummage through the logs for it

Dan Rammer (hamersaw) · Answer

This maximum array size is also configured with the `maxArrayJobSize` value at something like `plugins k8s array maxArrayJobSize` and then other specific configuration if caching is enabled is it The default value is like 5000 but I m sure you ve increased it

Dan Rammer (hamersaw) · Answer

just to confirm i was able to reproduce this locally with a job that exceeds the `maxArrayJobSize` parameter

Anooj Patel · Answer

I m a newb with parsing logs with k9s and gcp lemme try again I ll reach out to our infra team and try to up this value I ll report back here thanks < Dan Rammer hamersaw >

Anooj Patel · Answer

in terms of flyte propeller lt gt datacatalog robustness would we expect strange behavior if we up the limit to 30k

Dan Rammer (hamersaw) · Answer

There should be nothing to worry about with the scale In maptasks basically what happens is that it begins a sequential cache lookup process in the background then periodically checks if it has completed None of the subtasks will be started until the cache lookup for all of them is complete So it may take a little longer to start truly executing the subtasks but there shouldn t be any correctness or robustness issues

Anooj Patel · Answer

amazing Thanks Dan

Dan Rammer (hamersaw) · Answer

fix for the panic is submitted <https github com flyteorg flyteplugins pull 352>