Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hi Everyone! A bit of a vague question but I figured I'd ask: Is there a way to investigate why an execution takes so much longer than the same code running in a method on my machine?

I have a preprocessing method that takes ~1 minute  to run on my laptop, but has taken ~40 minutes to finish running as part of my execution

hey <@U037D5QF1V5> have you explored the timeline view in the console? That should better show how time is broken down between tasks

as for why it's slower on flyte, you could be resource constrained if you're requesting a significant fraction of the default namespace quota - this can affect scheduling. but you can easily bump the quota if that's the case

That seems odd <@U037D5QF1V5> . Has to be something else

<@UPBBNMXD1> Yes - the timeline view shows me that the preprocessing task takes around 30 minutes to execute (with increased resources - previously it was 40 minutes). The "preprocessing task" is comprised of 3 smaller tasks (shown in image attached) - each of which is a relatively simple operation (for example, one is zero meaning time series data). I have increased the available resources and only seen around a 7 minute speedup compared to the last time I ran it.

this is really weird, would you have some time for a chat

we have never seen this, must be something with the pod

`kubectl get pods -n delaieliane-development | grep ahrsdvrz`

are you also saying that for the entire pipeline it is 1 minute locally?

<@UNZB4NW3S> Edit: My bad, I actually do not have time right now. I would be available tomorrow before 3PM EST for a zoom chat. Attached is the result of that command.

And the preprocessing step takes 1 minute locally - there is a feature extraction step in the pipeline that takes a few minutes (don't have an exact number rn but I'd say 2-4 minutes if memory serves) locally - it takes 19 minutes on the cluster I have