Hi Everyone! A bit of a vague question but I figured I'd ask: Is there a way to investigate why an execution takes so much longer than the same code running in a method on my machine?
I have a preprocessing method that takes ~1 minute to run on my laptop, but has taken ~40 minutes to finish running as part of my execution
05/25/2022, 2:18 AM
hey @Ruksana Kabealo have you explored the timeline view in the console? That should better show how time is broken down between tasks
as for why it's slower on flyte, you could be resource constrained if you're requesting a significant fraction of the default namespace quota - this can affect scheduling. but you can easily bump the quota if that's the case
05/25/2022, 2:47 AM
That seems odd @Ruksana Kabealo . Has to be something else
06/01/2022, 8:15 PM
@katrina Yes - the timeline view shows me that the preprocessing task takes around 30 minutes to execute (with increased resources - previously it was 40 minutes). The "preprocessing task" is comprised of 3 smaller tasks (shown in image attached) - each of which is a relatively simple operation (for example, one is zero meaning time series data). I have increased the available resources and only seen around a 7 minute speedup compared to the last time I ran it.
06/01/2022, 8:27 PM
this is really weird, would you have some time for a chat
we have never seen this, must be something with the pod
can you check the pod in the back?
kubectl get pods -n delaieliane-development | grep ahrsdvrz
are you also saying that for the entire pipeline it is 1 minute locally?
06/01/2022, 8:33 PM
@Ketan (kumare3) Edit: My bad, I actually do not have time right now. I would be available tomorrow before 3PM EST for a zoom chat. Attached is the result of that command.
And the preprocessing step takes 1 minute locally - there is a feature extraction step in the pipeline that takes a few minutes (don't have an exact number rn but I'd say 2-4 minutes if memory serves) locally - it takes 19 minutes on the cluster I have