Channels
#ask-the-community
Title
r
Ruksana Kabealo
05/25/2022, 2:09 AMHi Everyone! A bit of a vague question but I figured I'd ask: Is there a way to investigate why an execution takes so much longer than the same code running in a method on my machine?
I have a preprocessing method that takes ~1 minute to run on my laptop, but has taken ~40 minutes to finish running as part of my execution
k
katrina
05/25/2022, 2:18 AM
hey @Ruksana Kabealo have you explored the timeline view in the console? That should better show how time is broken down between tasks
as for why it's slower on flyte, you could be resource constrained if you're requesting a significant fraction of the default namespace quota - this can affect scheduling. but you can easily bump the quota if that's the case
👍 1
k
Ketan (kumare3)
05/25/2022, 2:47 AM
That seems odd @Ruksana Kabealo . Has to be something else
r
Ruksana Kabealo
06/01/2022, 8:15 PM@katrina Yes - the timeline view shows me that the preprocessing task takes around 30 minutes to execute (with increased resources - previously it was 40 minutes). The "preprocessing task" is comprised of 3 smaller tasks (shown in image attached) - each of which is a relatively simple operation (for example, one is zero meaning time series data). I have increased the available resources and only seen around a 7 minute speedup compared to the last time I ran it.
k
Ketan (kumare3)
06/01/2022, 8:27 PM
this is really weird, would you have some time for a chat
we have never seen this, must be something with the pod
can you check the pod in the back?
kubectl get pods -n delaieliane-development | grep ahrsdvrzare you also saying that for the entire pipeline it is 1 minute locally?
r
Ruksana Kabealo
06/01/2022, 8:33 PM@Ketan (kumare3) Edit: My bad, I actually do not have time right now. I would be available tomorrow before 3PM EST for a zoom chat. Attached is the result of that command.
And the preprocessing step takes 1 minute locally - there is a feature extraction step in the pipeline that takes a few minutes (don't have an exact number rn but I'd say 2-4 minutes if memory serves) locally - it takes 19 minutes on the cluster I have
k
Ketan (kumare3)
06/01/2022, 10:28 PM
ok something is definitely wrong
let me DM you
7 Views