https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
announcements
  • e

    Evan Sadler

    07/11/2022, 4:15 PM
    Hello! I am trying to use the Flyte Spark plugin with the SynapseML package with the local runner. It works until I try to add the
    spark.jars.packages
    to the config (see example below). There is some kind of connection refused error:
    io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused:
    . I imagine this has to do with some configuration that I don’t quite understand in the backend. Any help is much appreciated. I tested it out with another package and it had the same error.
    import datetime
    import random
    from operator import add
    
    import flytekit
    from flytekit import Resources, task, workflow
    
    
    from flytekitplugins.spark import Spark
    
    @task(
        task_config=Spark(
            # this configuration is applied to the spark cluster
            spark_conf={
                "spark.driver.memory": "8g",
                "spark.jars.repositories": "<https://mmlspark.azureedge.net/maven>", 
                "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:0.9.5"  # adding this causes problems
            }
        ),
        limits=Resources(mem="2000M"),
        cache_version="1",
    )
    def hello_spark(partitions: int) -> float:
        print("Starting Spark with Partitions: {}".format(partitions))
    
        n = 100000 * partitions
        sess = flytekit.current_context().spark_session
        count = (
            sess.sparkContext.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
        )
        pi_val = 4.0 * count / n
        print("Pi val is :{}".format(pi_val))
        return pi_val
    s
    • 2
    • 5
  • s

    Sandra Youssef

    07/11/2022, 7:39 PM
    Hi Flyers, Join us tomorrow for our very first Fireside Chat with Flyte Contributors, plus community updates and roadmap items in our biweekly community sync. Enjoy guest appearances from: • @Sugato Ray, PhD Candidate & UnionML contributor • @Matheus Moreno, ML Engineer at Hurb • @Robin Kahlow, ML Engineer at Ntropy • @Mike Zhong, SR Software Engineer at Embark Veterinary • @krishna Yerramsetty, Data Scientist at Infinome Biosciences Tuesday 7/12 at 9am PT Calendar Invite and Zoom Link See you there! Flyte Team
    🚀 7
  • b

    Bernhard Stadlbauer

    07/12/2022, 6:21 AM
    Hi! We are currently trying to add automatic tracing through datadog to the pods started through Flyte. To do so, we would need to set a few environment variables and mount a volume. While we can set the environment variables in the config, there is no way to add volumes here. Is there any other way that we're missing? The only other thing I could think of is switching all of our tasks to pod tasks, where we would have control over the whole spec, but that is something we would like to avoid if possible. Also happy to contribute here if this is something you'd see a need for.
    d
    a
    • 3
    • 17
  • r

    Rahul Mehta

    07/12/2022, 10:42 PM
    We've been putting flyte through its paces on high fan-out workflows (ie ~50 parallel subworkflows using
    @dynamic
    ) and we've noticed some flakiness w/ the graph view when expanding subworkflows. In lieu of that, it seems like the timeline view is better suited to finding failing nodes. Would there be interest in a contribution to improve filtering in the timeline view (ie. using the same categories in the graph view)?
    k
    j
    • 3
    • 9
  • s

    seunggs

    07/13/2022, 3:03 AM
    Is the name of the workflow (i.e. the function decorated with
    @workflow
    ) the unique identifier for that workflow (combined with project and domain)?
  • s

    seunggs

    07/13/2022, 3:04 AM
    i.e. you can’t have two workflows with the same name?
    s
    • 2
    • 4
  • p

    Prada Souvanlasy

    07/13/2022, 11:32 AM
    hello! assuming that we're in a
    @dynamic
    workflow, is there a way to define dependencies between tasks without relying on their respective outputs? i.e, we would like to run
    task2()
    after
    task1()
    even though the latter does not output anything. In a classic
    @workflow
    , we could rely on
    create_node()
    +
    >>
    but can't figure out the equivalent for
    @dynamic
    k
    r
    e
    • 4
    • 18
  • r

    Rémy Dubois

    07/13/2022, 3:59 PM
    Run a task on others error
    k
    • 2
    • 10
  • s

    Sandra Youssef

    07/13/2022, 5:45 PM
    Hi Flyers, Flyte will be at SciPy 2022 this week! @Niels Bantilan will talk about the basics of Flyte and its programming model that gives it productionizing strengths, in "*Reliable, Reproducible, Recoverable, and Auditable Machine Learning for Production Workloads with Flyte*" Thursday 7/14, 3:50pm CDT
    :flyte: 1
    😛artyparrot: 2
    💯 2
  • n

    Niels Bantilan

    07/14/2022, 3:48 PM
    Developing some memes for Scipy… what do you think?
    👍 4
    😛artyparrot: 4
    :flyte: 4
  • r

    Rahul Mehta

    07/15/2022, 6:03 PM
    Would there be any way to set up gang scheduling within a project/domain currently? I've seen something similar for Kubeflow's training operators using https://github.com/volcano-sh/volcano and was wondering if it'd be possible to achieve for Flyte workflows. For context, we'd like to create user-specific projects and enforce resource quotas on them in K8s; however, if we do this, we'd like to try to avoid multiple queued workflows contending for resources and let pods associated w/ a specific workflow get scheduled together. Another example of this would be https://github.com/palantir/k8s-spark-scheduler
    k
    • 2
    • 4
  • s

    Sandra Youssef

    07/15/2022, 8:26 PM
    Hi Flyers, Thank you to everyone who attended this week's Flyte OS Community Sync. The following recordings have been published: •

    Community Update▾

    - Martin Stein •

    Flyte Signaling and Gate Nodes▾

    - Dan Rammer • Fireside Chat: Recording editing and processing in progress. • Meeting notes and slides Keep an eye out for our upcoming Biotech-themed Community Sync on July 26th. See you there! Flyte Team
  • a

    austin

    07/15/2022, 8:50 PM
    Is there any established process/scripts [ and associated testing ] for syncing boilerplate ( there is https://github.com/flyteorg/flyte/issues/2561 ). At the moment, I’m specifically interested in https://github.com/flyteorg/boilerplate/tree/master/boilerplate/flyte/end2end being different from what is in https://github.com/flyteorg/flyte/tree/master/boilerplate/flyte/end2end …. Imagining that these are meant to be the same?
    s
    y
    • 3
    • 6
  • v

    Vijay Saravana

    07/16/2022, 1:26 AM
    Hello Flyte team, While running map tasks, few tasks successfully completed but few were in running state in flyte console for a long time. I checked GCP logs and it showed that the map task completed. kubectl also showed all pods had completed executing except one which was
    terminating
    . What could be the reason for this? Did the map task not complete because of the one pod ?
    👋 1
    k
    s
    +3
    • 6
    • 36
  • v

    Vijay Saravana

    07/16/2022, 1:28 AM
  • v

    Vijay Saravana

    07/16/2022, 1:30 AM
    Kubectl output
    vijay.jaishankervijay@MacBook-Pro ~ % kubectl get pods -n dev | grep ah62ztnhshgx4gqqfgdv
    ah62ztnhshgx4gqqfgdv-n0-0                 0/1     Completed                  0          50m
    ah62ztnhshgx4gqqfgdv-n1-0                 0/1     Completed                  0          46m
    ah62ztnhshgx4gqqfgdv-n2-0-0               0/1     Completed                  0          42m
    ah62ztnhshgx4gqqfgdv-n2-0-1               0/1     Completed                  0          42m
    ah62ztnhshgx4gqqfgdv-n2-0-10              0/1     Completed                  0          34m
    ah62ztnhshgx4gqqfgdv-n2-0-11              1/1     Terminating                0          34m
    ah62ztnhshgx4gqqfgdv-n2-0-12              0/1     Completed                  0          30m
    ah62ztnhshgx4gqqfgdv-n2-0-13              0/1     Completed                  0          30m
    ah62ztnhshgx4gqqfgdv-n2-0-14              0/1     Completed                  0          30m
    ah62ztnhshgx4gqqfgdv-n2-0-15              0/1     Completed                  0          30m
    ah62ztnhshgx4gqqfgdv-n2-0-16              0/1     Completed                  0          30m
    ah62ztnhshgx4gqqfgdv-n2-0-17              0/1     Completed                  0          30m
    ah62ztnhshgx4gqqfgdv-n2-0-18              0/1     Completed                  0          30m
    ah62ztnhshgx4gqqfgdv-n2-0-19              0/1     Completed                  0          30m
    ah62ztnhshgx4gqqfgdv-n2-0-2               0/1     Completed                  0          42m
    ah62ztnhshgx4gqqfgdv-n2-0-20              0/1     Completed                  0          28m
    ah62ztnhshgx4gqqfgdv-n2-0-21              0/1     Completed                  0          23m
    ah62ztnhshgx4gqqfgdv-n2-0-22              0/1     Completed                  0          23m
    ah62ztnhshgx4gqqfgdv-n2-0-23              0/1     Completed                  0          23m
    ah62ztnhshgx4gqqfgdv-n2-0-24              0/1     Completed                  0          23m
    ah62ztnhshgx4gqqfgdv-n2-0-25              0/1     Completed                  0          23m
    ah62ztnhshgx4gqqfgdv-n2-0-26              0/1     Completed                  0          23m
    ah62ztnhshgx4gqqfgdv-n2-0-27              0/1     Completed                  0          23m
    ah62ztnhshgx4gqqfgdv-n2-0-28              0/1     Completed                  0          23m
    ah62ztnhshgx4gqqfgdv-n2-0-29              0/1     Completed                  0          23m
    ah62ztnhshgx4gqqfgdv-n2-0-3               0/1     Completed                  0          42m
    ah62ztnhshgx4gqqfgdv-n2-0-30              0/1     Completed                  0          19m
    ah62ztnhshgx4gqqfgdv-n2-0-31              0/1     Completed                  0          19m
    ah62ztnhshgx4gqqfgdv-n2-0-32              0/1     Completed                  0          18m
    ah62ztnhshgx4gqqfgdv-n2-0-33              0/1     Completed                  0          18m
    ah62ztnhshgx4gqqfgdv-n2-0-34              0/1     Completed                  0          18m
    ah62ztnhshgx4gqqfgdv-n2-0-35              0/1     Completed                  0          18m
    ah62ztnhshgx4gqqfgdv-n2-0-36              0/1     Completed                  0          18m
    ah62ztnhshgx4gqqfgdv-n2-0-37              0/1     Completed                  0          18m
    ah62ztnhshgx4gqqfgdv-n2-0-38              0/1     Completed                  0          18m
    ah62ztnhshgx4gqqfgdv-n2-0-39              0/1     Completed                  0          13m
    ah62ztnhshgx4gqqfgdv-n2-0-4               0/1     Completed                  0          42m
    ah62ztnhshgx4gqqfgdv-n2-0-40              0/1     Completed                  0          13m
    ah62ztnhshgx4gqqfgdv-n2-0-41              0/1     Completed                  0          13m
    ah62ztnhshgx4gqqfgdv-n2-0-42              0/1     Completed                  0          13m
    ah62ztnhshgx4gqqfgdv-n2-0-43              0/1     Completed                  0          13m
    ah62ztnhshgx4gqqfgdv-n2-0-44              0/1     Completed                  0          13m
    ah62ztnhshgx4gqqfgdv-n2-0-45              0/1     Completed                  0          13m
    ah62ztnhshgx4gqqfgdv-n2-0-46              0/1     Completed                  0          13m
    ah62ztnhshgx4gqqfgdv-n2-0-47              0/1     Completed                  0          13m
    ah62ztnhshgx4gqqfgdv-n2-0-48              0/1     Completed                  0          6m55s
    ah62ztnhshgx4gqqfgdv-n2-0-49              0/1     Completed                  0          6m56s
    ah62ztnhshgx4gqqfgdv-n2-0-5               0/1     Completed                  0          42m
    ah62ztnhshgx4gqqfgdv-n2-0-6               0/1     Completed                  0          42m
    ah62ztnhshgx4gqqfgdv-n2-0-7               0/1     Completed                  0          42m
    ah62ztnhshgx4gqqfgdv-n2-0-8               0/1     Completed                  0          42m
    ah62ztnhshgx4gqqfgdv-n2-0-9               0/1     Completed                  0          42m
  • p

    Prada Souvanlasy

    07/18/2022, 8:59 AM
    hello! Is there a mean to add custom tags on the Flyte provided metrics mentioned there? The only available tag is currently the workflow name (
    wf
    ), but we would like to add dimensions to the relevant metrics depending of the workflow inputs or the custom labels provided while submitting the workflow. Those labels are only added on the relevant pods as far as I saw but that would be great if those labels could be somehow added to the metrics as well
    d
    k
    • 3
    • 13
  • e

    Eric Hsiao

    07/18/2022, 5:31 PM
    Hi all! I'm running into an issue where I'm trying to pass in a boolean value to a workflow. I'm on flytekit 1.1.0 and trying to run this command:
    pyflyte run my_task.py download --remote # this works
    pyflyte run my_task.py download # this does not work (remote=False)
    Here's an example of my workflow definition
    @workflow
    def download(remote: bool = False):
        ...
    And here's the error I'm getting
    line 191, in to_python_value
        raise TypeTransformerFailedError(f"Cannot convert literal {lv} to {self._type}")
    flytekit.core.type_engine.TypeTransformerFailedError: Cannot convert literal scalar {
      primitive {
      }
    }
     to <class 'bool'>
    Is there anything obvious that I'm doing wrong?
    y
    e
    g
    • 4
    • 26
  • s

    Sandra Youssef

    07/19/2022, 7:19 PM
    Hey everyone, Join us tomorrow for another Flyte Ecosystem Project UnionML OSS Planning Event. Bring your ideas, questions, and wish list! Wednesday 7/20 at 9am PT Calendar Invite and Zoom Link Also, check out UnionML on GitHub and read the docs. See you then!
  • s

    Sandra Youssef

    07/20/2022, 7:33 PM
    Hi Flyers, If you haven't already, check out the Flyte 1.1.0 milestone release featuring: - The all-new FlyteDecks - FlyteKit Optional type support, PyFlyte Run and PyFlyte Register - Backend improvements & bug fixes - UI improvements
  • s

    seunggs

    07/21/2022, 3:58 AM
    Is there a way to skip the y/n confirmation with flytectl?
    This action will overwrite an existing config file at [/.flyte/config.yaml]. Do you want to continue? [y/n]:
    y
    • 2
    • 2
  • s

    seunggs

    07/21/2022, 4:00 AM
    While running
    flytectl config init --host=<http://example.com|example.com>
  • b

    Bryan Yeung

    07/21/2022, 5:40 AM
    After setting up a cluster, with metallb + contour ingress (default install with helm flyte sandbox instructions), deploying a sample with command line receives a grpc error:
    pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2
    {"asctime": "2022-07-20 21:46:45,700", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1658378805.700485000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3261,\"referenced_errors\":[{\"created\":\"@1658378805.700485000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":167,\"grpc_status\":14}]}\"\n>, sleeping 200ms and retrying"}
    {"asctime": "2022-07-20 21:46:45,906", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses\"\n\tdebug_error_string = \"{\"created\":\"@1658378805.905927000\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3261,\"referenced_errors\":[{\"created\":\"@1658378805.905926000\",\"description\":\"failed to connect to all addresses\",\"file\":\"src/core/lib/transport/error_utils.cc\",\"file_line\":167,\"grpc_status\":14}]}\"\n>, sleeping 400ms and retrying"}
    Traceback (most recent call last):
      File "/Users/btyeung/.virtualenvs/flyte/bin/pyflyte", line 8, in <module>
        sys.exit(main())
      File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
        return self.main(*args, **kwargs)
      File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/run.py", line 516, in _run
        remote_entity = remote.register_script(
      File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/remote/remote.py", line 596, in register_script
        upload_location, md5_bytes = fast_register_single_script(
      File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/tools/script_mode.py", line 116, in fast_register_single_script
        upload_location = create_upload_location_fn(content_md5=md5)
      File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
        return super(SynchronousFlyteClient, self).create_upload_location(
      File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/clients/raw.py", line 42, in handler
        return fn(*args, **kwargs)
      File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/flytekit/clients/raw.py", line 859, in create_upload_location
        return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
      File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
        return _end_unary_response_blocking(state, call, False, None)
      File "/Users/btyeung/.virtualenvs/flyte/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
        raise _InactiveRpcError(state)
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
            status = StatusCode.UNAVAILABLE
            details = "failed to connect to all addresses"
            debug_error_string = "{"created":"@1658378806.310949000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3261,"referenced_errors":[{"created":"@1658378806.310944000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}]}"
    >
  • b

    Bryan Yeung

    07/21/2022, 5:41 AM
    The loadbalancer (metallb) successfully provided an IP address, and console is available through that address. Here is flyte config.yaml
    admin:
      # For GRPC endpoints you might want to use dns:///flyte.myexample.com
      endpoint: dns:///192.168.111.220
      insecure: true
    logger:
      show-source: true
      level: 0
    Here is a screenshot of the console:
    y
    p
    k
    • 4
    • 18
  • b

    Bryan Yeung

    07/21/2022, 5:47 AM
  • b

    Bernhard Stadlbauer

    07/21/2022, 2:24 PM
    Hi! I've been looking into the docs for Notifications (and also checked the corresponding admin code). From what I could gather, GCP support for creating events (in Pub/Sub) already exists, but there is currently no way of sending email notifications with GCP (i.e. no
    emailer
    for GCP). Just wanted to confirm whether that is correct or if I am missing something?
    k
    j
    +3
    • 6
    • 12
  • s

    Sandra Youssef

    07/21/2022, 7:27 PM
    Hi Flyers, ⚡Flyte will be featured tomorrow 7/22 on Kelsey Hightower's Twitter Space: Machine Learning in Production⚡ Join @Ketan (kumare3) and @Eduardo Apolinario (eapolinario), plus guest speakers @Fabio Grätz, @Evan Sadler, @Arno Hollosi, Jordan Ganoff and Gautam Kedia for insights into unified machine learning and data orchestration. 7/22 at 10am PT Event link (also QR code in the flyer)
    😛artyparrot: 1
    🔥 4
    ❤️ 9
  • j

    Jordan Ganoff

    07/21/2022, 9:39 PM
    👋 Hi everyone!
    👋 5
  • j

    Josh Kilts

    07/21/2022, 10:02 PM
    👋 Hello everyone. I'm exploring flyte as a potential tool for my team and have a question I can't seem find an answer to: Is there any mechanism for configuring the python runtime environment for a task? Other tools I've explored have allowed me to pass in a list of packages (or a requirements.txt) and I'm looking for something similar.
    y
    k
    • 3
    • 13
  • g

    George Snelling

    07/22/2022, 12:06 AM
    Hi everybody, I have a simple question that is very deployment- and workflow-specific for those of you who use Flyte in production: Do you know the mean and the median time for task executions throughout your organization? And if so, can you share? Either publicly if that makes sense for your organization (in this thread please), or privately via DM. If you don't know, that would be useful to understand too. Thanks!
Powered by Linen
Title
g

George Snelling

07/22/2022, 12:06 AM
Hi everybody, I have a simple question that is very deployment- and workflow-specific for those of you who use Flyte in production: Do you know the mean and the median time for task executions throughout your organization? And if so, can you share? Either publicly if that makes sense for your organization (in this thread please), or privately via DM. If you don't know, that would be useful to understand too. Thanks!
View count: 4