Enabling task cache for Map tasks throws the error in thread

Question

Enabling task cache for Map tasks throws the error in thread Is this an issue with my config or a real bug Thanks pray

varsha Parthasarathy · Answer

My map task config ``` flytekit task limits=flytekit Resources cpu= 1 gpu= 1 mem= 70Gi requests=flytekit Resources cpu= 1 gpu= 1 mem= 70Gi retries=1 task config=Pod pod spec=V1PodSpec containers= V1Container name= primary node selector= <http l5 lyft com pool|l5 lyft com pool> eks pdx pool gpu tolerations= V1Toleration effect= NoSchedule key= reserved operator= Equal value= gpu primary container name= primary ``` the above works as expected

varsha Parthasarathy · Answer

If i include cache then it crashes with below error ``` flytekit task cache=True cache version= 0 0 0 limits=flytekit Resources cpu= 1 gpu= 1 mem= 70Gi requests=flytekit Resources cpu= 1 gpu= 1 mem= 70Gi retries=1 task config=Pod pod spec=V1PodSpec containers= V1Container name= primary node selector= <http l5 lyft com pool|l5 lyft com pool> eks pdx pool gpu tolerations= V1Toleration effect= NoSchedule key= reserved operator= Equal value= gpu primary container name= primary ```

varsha Parthasarathy · Answer

```Workflow avperceptionworkflows dev src perception scene workflows ground truth ground truth offline workflows GroundTruthOfflinePCPWorkFlow failed RuntimeExecutionError max number of system retry attempts 51 50 exhausted Last known status message failed at Node n1 RuntimeExecutionError failed during plugin execution caused by failed to execute handle for plugin k8s array panic when executing a plugin k8s array Stack goroutine 982 running runtime debug Stack usr local go src runtime debug stack go 24 +0x65 go src github com flyteorg flytepropeller pkg controller nodes task handler go 375 +0xfe panic 0x1f45600 0x395a540 usr local go src runtime panic go 838 +0x207 go pkg mod github com flyteorg flytestdlib v1 0 4 bitarray bitset go 33 0xc07cb88f60 0x27a88c0 0xc0887b6420 0xc0d58fafc0 0x23d8110 go pkg mod github com flyteorg flyteplugins v1 0 8 go tasks plugins array core metadata go 33 +0x1e1 0xc000b34a80 0x278ff70 0xc00054a8f0 0x278ff70 0xc00054aa50 0x279cfd0 0xc07cb88f60 0x27a88c0 0xc0887b6420 go pkg mod github com flyteorg flyteplugins v1 0 8 go tasks plugins array k8s executor go 94 +0x225 0x279cfd0 0xc07cb88d20 0x279f4b8 0xc000d17230 0x0 go src github com flyteorg flytepropeller pkg controller nodes task handler go 382 +0x178 0xc00180e648 0x278ab08 0xc000baea40 0xc000e76fc0 0xc000e77050 0xc000e77080 0x279ea38 0xc044cc7cc0 0xc000bf2580 go src github com flyteorg flytepropeller pkg controller nodes task handler go 384 +0x9a 0xc00180e648 0x278ab08 0xc000baea40 0xc000e76fc0 0xc000e77050 0xc000e77080 0x279ea38 0xc044cc7cc0 0xc000bf2580 go src github com flyteorg flytepropeller pkg controller nodes task handler go 616 +0x182b 0xc001d6c1a0 0xc000d1fbe0 0x0 0xc0012460c0 0x4 0x4 0xc000d1fc00 go src github com flyteorg flytepropeller pkg controller nodes dynamic handler go 70 +0xd8 0xc001d6c1a0 0xc000d1fbe0 0x0 0xc0012460c0 0x4 0x4 0xc000d1fc00 go src github com flyteorg flytepropeller pkg controller nodes dynamic handler go 220 +0x9d0 0x279cfd0 0xc07cb88690 0x279e8b8 0xc000c2c000 0xc063448d80 0x27b12b8 0xc0641ae0d0 go src github com flyteorg flytepropeller pkg controller nodes executor go 382 +0x157 0x279cfd0 0xc07cb88690 0xc063448d80 0x279e8b8 0xc000c2c000 go src github com flyteorg flytepropeller pkg controller nodes executor go 512 +0x227 0x279cfd0 0xc07cb88690 0x2783470 0xc0999a5400 0xc063448d80 0x279e8b8 0xc000c2c000 go src github com flyteorg flytepropeller pkg controller nodes executor go 736 +0x3c5 0x279cfd0 0xc07cb88240 0x27ac5a8 0xc0a5113e50 0x2783470 0xc0999a5400 0x2783498 0xc0999a5400 0x27a99d0 go src github com flyteorg flytepropeller pkg controller nodes executor go 934 +0x705 0x279cfd0 0xc07cb88240 0x27ac5a8 0xc0a5113e50 0x2783470 0xc0999a5400 0x2783498 0xc0999a5400 0x27a99d0 go src github com flyteorg flytepropeller pkg controller nodes executor go 774 +0x3c5 0x279cfd0 0xc07cb88240 0x27ac5a8 0xc0a5113e50 0x2783470 0xc0999a5400 0x2783498 0xc0999a5400 0x27a99d0 go src github com flyteorg flytepropeller pkg controller nodes executor go 941 +0x935 0x279cfd0 0xc07cb88240 0x27ac5a8 0xc0a5113e50 0x2783470 0xc0999a5400 0x2783498 0xc0999a5400 0x27a99d0 go src github com flyteorg flytepropeller pkg controller nodes executor go 774 +0x3c5 0x279cfd0 0xc07cb88240 0x27ac5a8 0xc0a5113e50 0x2783470 0xc0999a5400 0x2783498 0xc0999a5400 0x27a99d0 go src github com flyteorg flytepropeller pkg controller nodes executor go 941 +0x935 0x279cfd0 0xc07cb88240 0xc0999a5400 go src github com flyteorg flytepropeller pkg controller workflow executor go 147 +0x1b3 0x279cfd0 0xc07cb88240 0xc0999a5400 go src github com flyteorg flytepropeller pkg controller workflow executor go 393 +0x40f 0x279cfd0 0xc07cb88240 0xc0a4151848 0x1e5a080 go src github com flyteorg flytepropeller pkg controller handler go 130 +0x18e 0x279cfd0 0xc0954dbc50 0xc0999a4a00 go src github com flyteorg flytepropeller pkg controller handler go 131 +0x459 0x279cfd0 0xc0954dbc50 0xc0b25c0318 0x3 0xc0b25c031c 0x14 go src github com flyteorg flytepropeller pkg controller handler go 205 +0x86d 0xc0a4151f28 0x1e5a080 0xc08a8d8bd0 go src github com flyteorg flytepropeller pkg controller workers go 88 +0x510 0x279cfd0 0xc0954dbc50 go src github com flyteorg flytepropeller pkg controller workers go 99 +0xf1 0x279cfd0 0xc049036c00 go src github com flyteorg flytepropeller pkg controller workers go 115 +0xbd go src github com flyteorg flytepropeller pkg controller workers go 150 +0x59 created by go src github co```

varsha Parthasarathy · Answer

< Ketan kumare3 > = Appreciate any insights pray Having cache for map tasks would be extremely helpful not to reompute everything

Dan Rammer (hamersaw) · Answer

< varsha Parthasarathy> this is definitely a bug There should be no issue with caching in map tasks it is fully supported

varsha Parthasarathy · Answer

Since I specify tolerations as above could that interfere with caching feature

Dan Rammer (hamersaw) · Answer

Can you provide a little more context please did this failure occur on the first execution or when using cached data how large is the maptask fanout ie how many input items

Dan Rammer (hamersaw) · Answer

tolerations shouldn t be an issue

Ketan (kumare3) · Answer

< Dan Rammer hamersaw > I actually see subtasks caching is not working

Ketan (kumare3) · Answer

Let s talk

varsha Parthasarathy · Answer

gt did this failure occur on the first execution or when using cached data when using cache data gt how large is the maptask fanout ie how many input items 6 subtasks for map task

varsha Parthasarathy · Answer

Size of the sub task doesn t matter even if I give 1 sub task the above panic mode happens

Dan Rammer (hamersaw) · Answer

^^ ah ok good to know

varsha Parthasarathy · Answer

Ahh any workarounds in the meantime < Ketan kumare3 >

Ketan (kumare3) · Answer

I do not know

Ketan (kumare3) · Answer

But we will get through this quickly

varsha Parthasarathy · Answer

sounds good thanks pray

Ketan (kumare3) · Answer

< varsha Parthasarathy> < Dan Rammer hamersaw > sorry this is different I see another problem when caching was not working for subtasks

Ketan (kumare3) · Answer

This is a panic that causes failure

varsha Parthasarathy · Answer

could it be with the way I specify tolerations

varsha Parthasarathy · Answer

but without caching the tolerations work fine hence i am confused

Dan Rammer (hamersaw) · Answer

internally the maptask maintains a bitset of subtask status and indexes thereof that need to be computed because of a cache miss according the stacktrace here there is an error where we attempt to access that bitset in an index that doesn t exist unfortunately rather than sending an error the bitset lib panics which is another issue

Dan Rammer (hamersaw) · Answer

so the panic is being caused because there is a bug in how we initially lookup into the cache for each subtask

Dan Rammer (hamersaw) · Answer

i m looking into it right now

varsha Parthasarathy · Answer

Sounds good let me know if you want to try something I will be available all morning PST to help pray

Dan Rammer (hamersaw) · Answer

< varsha Parthasarathy> do you know what version of FlytePropeller you re running I m doing some testing locally and unable to repro I know we here relatively recently

Dan Rammer (hamersaw) · Answer

it looks like this would have gotten pushed through to FlytePropeller <https github com flyteorg flytepropeller releases tag v1 1 33|v1 1 33> on Sept 9th

varsha Parthasarathy · Answer

Hi < Dan Rammer hamersaw > ``` datacatalog version = v1 0 1 flyteadmin version = v1 1 21 flyteconsole version = v1 1 0 flytecopilot version = v0 0 26 flytepropeller version = v1 1 21 ```

varsha Parthasarathy · Answer

Is it enough if we update FlytePropeller or should any other version needs an update

Dan Rammer (hamersaw) · Answer

You should be able to just update FlytePropeller It doesn t look like there were any breaking changes introduced between 1 21 and 1 33

Dan Rammer (hamersaw) · Answer

I should note that there does seem to be a bug in displaying the correct cache status in the UI right now for subtasks I ll get on a PR to fix that

Alex Pozimenko · Answer

< Dan Rammer hamersaw > after upgrading propeller to v1 1 33 we see long queued times 30 min is not unusual

Alex Pozimenko · Answer

console shows status running but there s no pod and the Logs tab shows queued

Alex Pozimenko · Answer

cc < Ketan kumare3 >

Ketan (kumare3) · Answer

Sorry catching up

Ketan (kumare3) · Answer

Hmm I guess need to looks at logs

Ketan (kumare3) · Answer

Can jump on a call

Dan Rammer (hamersaw) · Answer

< Alex Pozimenko> are the long queued times for map tasks only Or for all tasks

Alex Pozimenko · Answer

i have an example of spark task

Alex Pozimenko · Answer

map task from another workflow looks fine

Alex Pozimenko · Answer

call

Alex Pozimenko · Answer

Dan Rammer (hamersaw) · Answer

I can hop on for a few minutes here but only have 10 minutes or so

Ketan (kumare3) · Answer

I can hop in 10

Ketan (kumare3) · Answer

< Alex Pozimenko> I am trying to join ^

Alex Pozimenko · Answer

< Dan Rammer hamersaw > < Ketan kumare3 > we re having similar problem spark tasks and nested worklfows won t start Spark operator looks healthy no exceptions in logs no pending pods but queue latency is high and growing and propeller free workers count is 0

Alex Pozimenko · Answer

is there a limit on how many driver pods we can run at the same time

Alex Pozimenko · Answer

i see 100+

Alex Pozimenko · Answer

for one workflow i see spark application in SUBMITTED state but nothing in the spark operator logs

Dan Rammer (hamersaw) · Answer

< Alex Pozimenko> it sounds like it could be a starvation problem Do you know how many workflows are currently running Also what is the And finally how many workers do you have configured for your FlytePropeller instance

Dan Rammer (hamersaw) · Answer

If it is indeed starvation we can easily <https docs flyte org en latest deployment cluster config performance html automatic scale out|scale up the number of flytepropeller instances> Just want to make sure it s not something else and we re just throw more resources at it fixing

Alex Pozimenko · Answer

flyte propeller free workers count was at 0

Alex Pozimenko · Answer

we ran 100+ spark workflows and perhaps 100 more of non spark

Alex Pozimenko · Answer

total completed about 1k wf s hourly

Dan Rammer (hamersaw) · Answer

gt flyte propeller free workers count was at 0 do you know how many workers you have propeller configured with and then the round latency is pretty important here i think were you able to find that prometheus metric

Alex Pozimenko · Answer

gt do you know how many workers you have propeller configured with 64 what is the round latency

Dan Rammer (hamersaw) · Answer

Interesting We can certainly increase the number of workers on propeller Some users are running up to 500 and it is scaling find in their setup All you need to do is change the configuration and roll the propeller deployment to pick up the new config Maybe try 128 256 etc This should reduce the queueing times and probably have some free workers

Alex Pozimenko · Answer

do I need to bump cpu mem as well right now we have 8CPU 16G memory limilt 1 1G request

Dan Rammer (hamersaw) · Answer

The round latency is an end to end timestamp of propeller evaluating a workflow So it works by periodically reading the CRD and checking nodes status scheduling new nodes etc Each time it does this is one round Each worker only works on a single workflow at a time so if the round latencies are large it reduces the throughput of workflow evaluations This would cause longer queued times as you are seeing Per the link I sent previously this is exposed in a prometheus metric with something like `sum rate flyte propeller all round raw ms 5m by wf `

Dan Rammer (hamersaw) · Answer

gt do I need to bump cpu mem as well right now we have 8CPU 16G memory limilt 1 1G request that s difficult to say right now depending on configuration propeller can quite a bit of memory for things like caching workflow definitions blobstore caching etc however we aren t seeing issues with cpu utilization being high thankfully increasing the number of workers has no effect or very little on memory utilization so if 16G has been fine up until now i would say your fine i think 8 cpu would be plenty for 256 workers but it might be something to keep an eye on

Alex Pozimenko · Answer

i have propeller workflow acceptance latency and transition latency These two should correlate with the round latency right

Dan Rammer (hamersaw) · Answer

gt i have propeller workflow acceptance latency and transition latency These two should correlate with the round latency right also a difficult question sweat smile theoretically if the round latency is high then workers will take longer to process each workflow so the acceptance latency and transition latencies will be high as well however this is only the case if the number of free workers hits zero if you have workers available then the round latency can be higher and the acceptance and transition latencies should not be significantly affected

Alex Pozimenko · Answer

what would be a scenario when round latency is high but acceptance and transition are low

Ketan (kumare3) · Answer

Usually 1 Workflow has a large fan out and max parallelism is not user 2 Kube client is blocking and is hanging to create pods or kube api is slow 3 S3 read write is slow

Ketan (kumare3) · Answer

Admin write is slow

Alex Pozimenko · Answer

thanks We seem to be running into scaling limitations of spark operator Can you recommend some resources about scaling up spark operator or maybe give some suggestions slightly smiling face Typical scenario it runs fine for a while but then CPU jumps to 100 I bumped CPU limit from 16 to 24 to 32 but the pattern repeats just seem to take a bit longer to get to that state

Ketan (kumare3) · Answer

Hmm I think it s the webhook

Ketan (kumare3) · Answer

Nothing else can take that much cpu

Alex Pozimenko · Answer

Alex Pozimenko · Answer

no mem pressure

Alex Pozimenko · Answer

i wonder if changing `resync interval` and or `controller threads` may help

Alex Pozimenko · Answer

currently at 90 and 240

Ketan (kumare3) · Answer

It will but high cpu indicated something else