https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
announcements
  • n

    Nicholas LoFaso

    12/10/2021, 2:33 AM
    Hi, I’m trying to call a
    @dynamic
    task from a
    map_task
    and am seeing a strange error. Is there a limitation I should be aware of?
    [1/1] currentAttempt done. Last Error: UNKNOWN::failed to read data from dataDir [<gs://mybucket/metadata/propeller/nickflyte-dev-fc4174c9bef1a4c2caa5/n0/data/0/0/outputs.pb>]. Error: path:<gs://mybucket/metadata/propeller/nickflyte-dev-fc4174c9bef1a4c2caa5/n0/data/0/0/outputs.pb>: not found
    The attached code fails on our GCS cluster. Other map_tasks that don’t call a dynamic task work fine
    experiment.py.zip
    k
    • 2
    • 29
  • s

    Sharon Gong

    12/13/2021, 2:30 PM
    hi there! we created a subclass of TypedSchema that writes to other storages besides GCS/local. i'm getting this error when running a workflow that creates this subclass as an output. this worked up until recently, so wondering if anything changed with TypedSchema recently?
    [3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):
    
          File "/home/jovyan/.local/lib/python3.7/site-packages/flytekit/common/exceptions/scopes.py", line 165, in system_entry_point
            return wrapped(*args, **kwargs)
          File "/home/jovyan/.local/lib/python3.7/site-packages/flytekit/core/base_task.py", line 514, in dispatch_execute
            ) from e
    
    Message:
    
        Failed to convert return value for var o0 for function onemodel.models.subscribers._flyte.subscribers_task with error <class 'AttributeError'>: '_TypedJasperSchema' object has no attribute '_remote_path'
    k
    • 2
    • 6
  • b

    Ben Konz

    12/14/2021, 10:06 PM
    Hi! Are there plans on supporting
    BranchNode
    and
    WorkflowNode
    in the local executor for tests? (https://github.com/flyteorg/flytekit-java/blob/3cf3ee4ac27a3e1bad231be2fa9a30b8afa[…]/src/main/java/org/flyte/localengine/ExecutionNodeCompiler.java) It's difficult to write tests for our sub-workflows without support for these in the local executor
    y
    k
    n
    • 4
    • 3
  • z

    Zach Palchick

    12/16/2021, 12:25 AM
    quick question about the sandbox. If I'd like to run a workflow where the input to that workflow is a flytefile, where can I place that file in the sandbox environment so it can be found? This is what the UI looks like
    y
    • 2
    • 6
  • h

    Haytham Abuelfutuh

    12/16/2021, 8:02 PM
    Welcome, officially, to Flyte, MethaneSAT… Beyond being naturally excited to see the list of Flyte Users/Contributors grow, I get extra excited to see such big ambitious projects with a goal of making life better build on Flyte. To space and beyond, @Nicholas LoFaso & Team! 🛰️ :flyte:
    🚀 11
    🎉 4
    🛰️ 1
    💯 5
    😛artyparrot: 7
    :excellent: 4
    🔥 8
    ❤️ 7
    🙌 11
    n
    • 2
    • 1
  • p

    Paul Dittamo

    12/17/2021, 9:22 PM
    Hello 👋 is there a way to avoid starting a new scheduled workflow if the previous one is still running?
    g
    d
    • 3
    • 3
  • s

    Sören Brunk

    12/20/2021, 5:19 PM
    We're seeing flytepropeller auth failures after installation/upgrade (via helm) with a message like this:
    E1220 16:10:36.391892       1 workers.go:102] error syncing 'mandant1-development/f3359d6b5cd941830000': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken]
    It leaves the system in a weird state because there's no hard failure. flytepropeller ist running after all so our monitoring does not trigger an alarm, but no workflow execution is happening. After a flytepropeller restart, it suddenly starts working again. So after some digging I found out why this happens in our setup:
    flyte-secret-auth
    is populated with
    .Values.secrets.adminOauthClientCredentials.clientSecret
    during installation which is set to the placeholder
    foobar
    in
    values.yaml
    . We set the secret value dynamically though with a helm hook during installation because we need to fetch the real client-secret from Keycloak. That happens only after flytepropeller is deployed and it seems that flytepropeller does not reload the secret on changes. Since
    flyte-secret-auth
    is managed by helm, this happens again on every
    helm upgrade
    . I see mainly two (non exclusive) ways to improve this behavior: • Remove the default
    clientSecret
    and only create
    flyte-secret-auth
    via helm if the value is actually set. Only mount
    flyte-secret-auth
    if external auth is enabled. That would cause flytepropeller to fail to start until
    flyte-secret-auth
    is created by other means. • Trigger a flytepropeller reload when
    flyte-secret-auth
    changes. Any thoughts on this? Happy to contribute here but I'd like to discuss the best way forward with you first.
    h
    • 2
    • 3
  • j

    Jake Neyer

    12/20/2021, 6:48 PM
    Hey I'm having trouble registering a workflow. It looks like the tasks are registering alright:
    flytectl register files --project chariot-sdk-test --domain development --archive out.tar.gz --version v2
    
     ---------------------------------------------------------------------- --------- ------------------------------------------------------------
    | NAME (4)                                                             | STATUS  | ADDITIONAL INFO                                            |
     ---------------------------------------------------------------------- --------- ------------------------------------------------------------
    | /tmp/register024275035/0_pm.nb.new.ipynb_1.pb                        | Success | Successfully registered file                               |
     ---------------------------------------------------------------------- --------- ------------------------------------------------------------
    | /tmp/register024275035/1_new.ipynb_1.pb                              | Success | Successfully registered file                               |
     ---------------------------------------------------------------------- --------- ------------------------------------------------------------
    | /tmp/register024275035/2_workflows.new_workflow.nb_to_python_wf_2.pb | Failed  | Error registering file due to rpc error: code =            |
    |                                                                      |         | Internal desc = failed to compile workflow for             |
    |                                                                      |         | [resource_type:WORKFLOW project:"chariot-sdk-test"         |
    |                                                                      |         | domain:"development"                                       |
    |                                                                      |         | name:"workflows.new_workflow.nb_to_python_wf" version:"v2" |
    |                                                                      |         | ] with err entry not found                                 |
     ---------------------------------------------------------------------- --------- ------------------------------------------------------------
    | /tmp/register024275035/3_workflows.new_workflow.nb_to_python_wf_3.pb | Failed  | Error registering file due to rpc error: code =            |
    |                                                                      |         | NotFound desc = entry not found                            |
     ---------------------------------------------------------------------- --------- ------------------------------------------------------------
    Any ideas why that might happen?
    y
    • 2
    • 15
  • j

    Jake Neyer

    12/21/2021, 3:01 AM
    I have a workflow that is successfully writing outputs but the CRD remains in "Workflow Started", the UI shows "Running" and the pod falls into a "NotReady" state. What triggers the CRD to fall into a error or completed status?
    👍 1
    h
    • 2
    • 5
  • e

    Endre Karlson

    12/21/2021, 10:40 AM
    Hi all! 🙂 Trying to start a flyte sandbox locally with the command from the Guide but getting
    +-------------------------------------+---------+-----------+
    |               SERVICE               | STATUS  | NAMESPACE |
    +-------------------------------------+---------+-----------+
    | flyte-contour-contour-certgen-xggqx | Pending | flyte     |
    +-------------------------------------+---------+-----------+
    ^ it's just stuck there
    y
    • 2
    • 7
  • z

    Zach Palchick

    12/21/2021, 11:12 PM
    I've gotten around the map_task "single input" limitation before by using a data class to consolidate all the inputs together, but it doesn't look like I can use those as inputs to the shell task since you can't (or I think you can't) deference multiple times (i.e.
    {{ .<http://inputs.my|inputs.my>__input__data_class.input_1 }}
    )
    k
    • 2
    • 6
  • n

    Nicholas LoFaso

    12/22/2021, 3:57 PM
    Hi all, how can I tell if a map_task is successfully cached? As far as I can tell this bugfix should now allow for caching, but the UI doesn’t give an indication that it was saved.
    g
    h
    d
    • 4
    • 3
  • a

    Avshalom Manevich

    12/23/2021, 8:34 AM
    Anyone here having issues with running the sandbox environmnet + flytesnacks on a M1 Mac?
    y
    k
    • 3
    • 12
  • a

    Avshalom Manevich

    12/23/2021, 4:17 PM
    Follow up question (sorry for the spam): I got the sandbox to work, but the hello world task from flytesnacks fails.
    h
    y
    • 3
    • 7
  • k

    Ketan (kumare3)

    12/25/2021, 10:58 PM
    Hi Community, What a year it has been. I am extremely proud of this community and especially how supportive everyone is & how much have we accomplished together. We have grown to many production deployments and seen such amazing use-cases fulfilled by Flyte, that I could not have ever imagined. I think it is still very early for this community. Even tough we started in 2020, 2021 is really when we created the community. Looking ahead I am extremely excited about the launchpad we have created and whats to come. Earlier this year I started a company called union.ai, with a mission to help orchestrate ML and data processing and make it accessible to everyone across the world using Flyte. The team at union is truly dedicated to ensure that every user of Flyte’s voice is heard. We have done over 20+ integration, 100s of features and 1000s of community members of joined us on this journey. In 2022, we promise to continue working even harder to make Flyte the best platform you will use for you orchestration needs. One way to do this to ensure we focus on the user experience and understand what our users are trying to get from the platform. They use Flyte, when they want resource-aware, reliable, serverless, multi-tenant and extensible platform for orchestrating really complex data and ml jobs. Along the way, as the community scaled, we at union had to choose the things that we could focus on and we had to let some potential new users go (as we could not fulfil their requests). But, with 2022, we will be working on more of these use cases and many more integrations and improvements (stay tuned). Wishing all of you a Merry Christmas and a happy new year! Yours Gratefully, Ketan
    🎉 7
    💯 6
    🍾 1
    👍 9
    ❤️ 15
    • 1
    • 1
  • a

    Alessandro Liparoti

    12/31/2021, 2:13 PM
    hey I am working on a backend webapi go plugin which runs a BigQuery query. I am trying to write an output for the task so whenever the query is successful I use
    outputWriter.Put(ctx, ioutils.NewInMemoryOutputReader(outputs, nil))
    to store a
    outputs
    structure to be visible in console. I do this in
    Status
    method (in this interface) whenever the job is completed. However it seems the protobuf is not written (thus not visible). I followed the same approach for a core plugin and it worked. Am I doing something wrong here in the way I am using the webapi plugin?
    g
    k
    h
    • 4
    • 4
  • k

    Ketan (kumare3)

    01/04/2022, 4:44 PM
    Folks, I am working on a new design that would allow users to run low latency workflows on Flyte and run existing simple python tasks and reduce / ammortize the overhead of running a pod. Would folks here be interested in helping me refine the design and the UX model? Goal: Users should be able to run sub second tasks and yet get the same level of visibility, fault tolerance and simplicity for their code. We might lose some isolation properties that running an individual pod offers, but we will continue to offer isolated environments, versioning and dynamic infrastructure.
    🙌 1
    • 1
    • 1
  • s

    Sonja Ericsson

    01/07/2022, 4:55 PM
    Hi! Is there an endpoint to list all executions across projects/domains? Like this, but without the project/domain parameters.
    k
    y
    • 3
    • 5
  • m

    Michael Cheng Jan Kao

    01/08/2022, 5:21 AM
    Hi everyone, I've got some questions hoping someone can give me a hand. Is there a way to control what gets registered? There are two scenarios in that this is helpful. 1. I have a workflow that has 2 sub-workflows, the sub workflows should not be used separately so I don't want them to be registered and visible in Flyte 2. How can I register a dynamic workflow? Thanks in advance.
    k
    • 2
    • 2
  • n

    Nicholas LoFaso

    01/10/2022, 3:43 PM
    Hi All, is there an example for how to configure the python logging package for use with
    @tasks
    ? I’m configuring the root logger, but it seems like that code is only running at registration time not execution time? See thread for example
    y
    g
    • 3
    • 47
  • s

    Sandra Youssef

    01/10/2022, 9:53 PM
    Hello everyone, Welcome back to our OSS Sync Ups! Join us tomorrow 1/11 at the adjusted time of 7:00pm PT (10:00am WIB, 8:30am IST on 1/12) to learn how Gojek uses Flyte for their ML platform, with @Pradithya Aria Pura. Zoom link: https://us06web.zoom.us/j/85635963908?pwd=N0k0ZUw1MkFadlBvT0VydlkvcXFyZz09 See you then! Flyte Team
    🙏 1
    k
    s
    • 3
    • 9
  • h

    Haytham Abuelfutuh

    01/12/2022, 3:00 PM
    Hey everyone! I’m on zoom for the next 30mins hosting Flyte Office Hours. If anyone has questions about deploying/trying/using Flyte, I’ll be happy to discuss!
    w
    k
    • 3
    • 3
  • k

    Ketan (kumare3)

    01/13/2022, 12:30 AM
    @Tim Bauer / @Haytham Abuelfutuh can we merge this one- https://github.com/flyteorg/flytesnacks/pull/463
    t
    h
    • 3
    • 3
  • e

    Eugene Cha

    01/13/2022, 6:48 AM
    what kubernetes versions (min, max) does flyte support?
    s
    p
    h
    • 4
    • 3
  • j

    Julien Bisconti

    01/17/2022, 2:12 PM
    Hi team, we are trying to compute the overhead time of the platform. Meaning:
    1 - (All workflows transition time / all workflows duration)
    We use this PromQL query:
    1 - 
      ( flyte:propeller:all:workflow:completion_latency_unlabeled_ms_sum
        + flyte:propeller:all:node:transition_latency_unlabeled_ms_sum
        + flyte:propeller:all:node:queueing_latency_unlabeled_ms_sum
        + flyte:propeller:all:workflow:acceptance_latency_unlabeled_ms_sum)
      / (flyte:propeller:all:workflow:failure_duration_unlabeled_ms_sum + flyte:propeller:all:workflow:success_duration_unlabeled_ms_sum)
    The numbers we get seem a bit off ( around 50%). Is the PromQL query correct? What do you recommend we use to measure that overhead time ?
    k
    • 2
    • 9
  • q

    Quinn Romanek

    01/19/2022, 5:31 PM
    How do I configure default resource limits for tasks?
    k
    • 2
    • 8
  • m

    Muhammad Daniyal

    01/23/2022, 11:10 AM
    Hello All. Hope you are doing well. I am a software developer (precisely python developer) and new to the domain of ML. During the journey, i encountered an error while applying flyte for workflow. i am working with spacy and at first it was just one file without any self defined functions and just calling 3rd party APIs. so i somewhat changed one sequential code to function to create tasks to get better understanding of flyte. well, in one function (task) i am returning a list of tuple. like traindata = [...('some text here',{'key': (1,3,'abc')})...] and wrote return type as typing.List[tuple]. but it is giving me an exception that generic types cannot be there. what should i do in this condition
    👋 1
    k
    o
    • 3
    • 16
  • m

    Muhammad Daniyal

    01/23/2022, 11:12 AM
    and another question, if i somehow manage to make work perfectly fine, what is the benefit i am getting from that?
    k
    • 2
    • 4
  • c

    Chen Yuanyuan

    01/24/2022, 2:57 AM
    Hi team, I am a DS from gojek, currently working on using Flyte for model pipeline. One question I came across is that the resource requested (cpu, memory) is hard coded in the decorator for each task for the current implementation, so it’s the same for different environments. What will be a better way to do overrides for each task in the workflow for different environments? Will it be possible to define it in a config file instead? cc @Pradithya Aria Pura @Michael Cheng Jan Kao
    k
    y
    p
    • 4
    • 31
  • h

    harsh patel

    01/24/2022, 3:34 AM
    How does flyte compare to prefect. I read up on prefect I find it alot easier to follow the docs on how to create external integrations like JIRA, ServiceNow etc etc
    k
    • 2
    • 4
Powered by Linen
Title
h

harsh patel

01/24/2022, 3:34 AM
How does flyte compare to prefect. I read up on prefect I find it alot easier to follow the docs on how to create external integrations like JIRA, ServiceNow etc etc
k

Ketan (kumare3)

01/24/2022, 3:51 AM
I answered about integrations separately
👍 1
But, I want to answer about direct comparison with Flyte. They similar yet different Flyte is implemented in golang, designed for container native execution and connecting external services. For external services it offers the idea of backend plugins- which allows separating plugins from user code (python library), but these backend plugins are harder to implement if you do not know golang On the other hand, the power is reproducibility , includes container orchestration and can run spark etc in k8s. We do not use something like dask to execute
Also Flyte supports cross language workflows
Cc @Smriti Satyan
View count: 14