Thread
#ask-the-community
    Ruksana Kabealo

    Ruksana Kabealo

    4 months ago
    Hi Everyone! A bit of a vague question but I figured I'd ask: Is there a way to investigate why an execution takes so much longer than the same code running in a method on my machine?
    I have a preprocessing method that takes ~1 minute to run on my laptop, but has taken ~40 minutes to finish running as part of my execution
    k

    katrina

    4 months ago
    hey @Ruksana Kabealo have you explored the timeline view in the console? That should better show how time is broken down between tasks
    as for why it's slower on flyte, you could be resource constrained if you're requesting a significant fraction of the default namespace quota - this can affect scheduling. but you can easily bump the quota if that's the case
    Ketan (kumare3)

    Ketan (kumare3)

    4 months ago
    That seems odd @Ruksana Kabealo . Has to be something else
    Ruksana Kabealo

    Ruksana Kabealo

    3 months ago
    @katrina Yes - the timeline view shows me that the preprocessing task takes around 30 minutes to execute (with increased resources - previously it was 40 minutes). The "preprocessing task" is comprised of 3 smaller tasks (shown in image attached) - each of which is a relatively simple operation (for example, one is zero meaning time series data). I have increased the available resources and only seen around a 7 minute speedup compared to the last time I ran it.
    Ketan (kumare3)

    Ketan (kumare3)

    3 months ago
    this is really weird, would you have some time for a chat
    we have never seen this, must be something with the pod
    can you check the pod in the back?
    kubectl get pods -n delaieliane-development | grep ahrsdvrz
    are you also saying that for the entire pipeline it is 1 minute locally?
    Ruksana Kabealo

    Ruksana Kabealo

    3 months ago
    @Ketan (kumare3) Edit: My bad, I actually do not have time right now. I would be available tomorrow before 3PM EST for a zoom chat. Attached is the result of that command.
    And the preprocessing step takes 1 minute locally - there is a feature extraction step in the pipeline that takes a few minutes (don't have an exact number rn but I'd say 2-4 minutes if memory serves) locally - it takes 19 minutes on the cluster I have
    Ketan (kumare3)

    Ketan (kumare3)

    3 months ago
    ok something is definitely wrong
    let me DM you