https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
announcements
  • h

    harsh patel

    01/24/2022, 3:35 AM
    Can we also easily create integrations with external apps as easily as prefect on Flyte too?
    k
    s
    • 3
    • 3
  • s

    Sandra Youssef

    01/24/2022, 3:48 PM
    Hey everyone, Join our OSS Sync Up guests @Stephen and @Tim Bauer tomorrow 1/25 at **8:30**am PT, who will share their experience of how Wolt deploys Flyte, in "*End-To-End ML With Flyte*". Also featured is our 2nd FlyteConsole UX/UI Townhall with @Jason Porter. Zoom Link and Calendar Invite See you there!
    z
    • 2
    • 4
  • q

    Quinn Romanek

    01/24/2022, 11:45 PM
    Do I need to run
    flytectl register files
    first?
    k
    k
    p
    • 4
    • 32
  • n

    Nicholas LoFaso

    01/25/2022, 7:53 PM
    Hi I would like to set an environment variable for every Pod that gets scheduled to k8s. What is the preferred way to do this? Specifically I’m trying to set the following for datadog metrics
    env:
        - name: DD_AGENT_HOST
          valueFrom:
              fieldRef:
                  fieldPath: status.hostIP
    y
    • 2
    • 17
  • k

    Ketan (kumare3)

    01/26/2022, 4:09 PM
    https://events.linuxfoundation.org/open-source-summit-north-america/program/cfp/
    👍 1
    • 1
    • 1
  • z

    Zach Palchick

    01/26/2022, 10:20 PM
    is there documentation on how to use container tasks beyond https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/raw_container.html . I'm trying with the sandbox env but not having luck. I've heard that it depends on the SYS_PTRACE capability, which may depend on one's K8s setup
    e
    • 2
    • 16
  • j

    JP Kosymna

    01/26/2022, 11:19 PM
    Hey all. I'm having trouble using a ShellTask with secrets. In the following task I would expect an environment variable possibly named
    AWS-ACCESS-KEY_AWS_ACCESS_KEY_ID
    to exist but it doesn't. What am I doing wrong here?
    task_with_secrets = ShellTask(
        debug=True,
        name="task_with_secrets",
        script='''
        set -ex
        env > out_env
        ''',
        inputs=kwtypes(source=str),
        output_locs=[
            OutputLocation(var="x", var_type=FlyteFile, location="out_env")
        ],
        secret_requests=[
            Secret(
                group='aws-access-key',
                key='AWS_ACCESS_KEY_ID',
                mount_requirement=Secret.MountType.ENV_VAR
            ),
            Secret(
                group='aws-access-key',
                key='AWS_SECRET_ACCESS_KEY',
                mount_requirement=Secret.MountType.ENV_VAR
            ),
        ]
    )
    k
    • 2
    • 27
  • s

    Sandra Youssef

    01/27/2022, 9:57 PM
    Deploying Flyte just got a whole lot easier - thanks to @Nitin Aggarwal's team at Run[X] developing Opta's IaC framework. https://blog.flyte.org/how-opta-makes-deploying-flyte-much-easier
    🙌 5
    ❤️ 6
    a
    k
    +2
    • 5
    • 5
  • s

    Sonja Ericsson

    01/31/2022, 3:07 PM
    Hi! We are comparing how often a workflow succeeds and gets accepted, i.e.,
    flyte:propeller:all:workflow:success_duration_ms_count
      vs 
    flyte:propeller:all:workflow:accepted
    and get some unexpected numbers. Successes is equal to 4345 for the last 30 days which is bigger than the number of accepted which is 4322. There are 0 failures also. Should they not be the same? Is there a better metric to use to be able to compare these numbers? I don’t find
    flyte:propeller:all:workflow:success
    for example
    d
    k
    • 3
    • 30
  • n

    Nicholas LoFaso

    01/31/2022, 6:00 PM
    Hi on my GCP Flyte deployment Flyte deletes dynamic tasks from k8s as soon as they complete whether they pass or fail. Regular python tasks are deleted after the GC Interval which is 30h on our cluster. This makes it very hard to debug dynamic tasks if something goes wrong because the pod is no longer in k8s. Is this something that I can stop Flyte from doing via a config option?
    y
    d
    • 3
    • 10
  • j

    jeev

    02/02/2022, 5:56 PM
    Is there a default task timeout? Does it happen to be 48 hours by some chance? Is it configurable?
    k
    k
    • 3
    • 14
  • a

    Alex Pozimenko

    02/04/2022, 12:49 AM
    hi there, I'm registering workflows with
    flyte-cli register-files ... --kubernetes-service-account=<SERVICE_ACCOUNT>
    but when I look at the running pods they all use default serviceAccount. I have this in .config also:
    [auth]
    kubernetes_service_account=<SERVICE_ACCOUNT>
    am I missing something?
    y
    • 2
    • 42
  • n

    Nicholas LoFaso

    02/07/2022, 7:39 PM
    @Haytham Abuelfutuh I saw your post a few weeks back saying that Flyte v0.19.2 helm chart would work with K8s 1.22+. I see this PR was successfully merged that makes the necessary helm updates. However, when I try and run using
    helm
    it does not seem to be pulling in all the changes. Specifically
    kind: Ingress
    and
    kind: CustomResourceDefinition
    are still using
    v1beta1
    . I have run
    helm repo update
    helm template --version v0.19.2 ...
    r
    h
    • 3
    • 11
  • a

    Alex Bain

    02/08/2022, 11:00 PM
    Hey Flyte team, we (at Woven Planet) are hoping to start making good use of Flyte map tasks. I went through the Flyte OSS roadmap issues and saw that a lot of recent work has been done to get them up to speed (such as https://github.com/flyteorg/flyte/issues/1838 to propagate the status of subtasks). However, I can see on the latest
    flyteconsole:v0.41.0
    that in the console, there are no log links to subtasks or any UI to show the status of subtasks. Can I ask how soon you plan to land these features in flyteconsole and if there are GitHub issues or PRs that I can follow / contribute to?
    k
    d
    j
    • 4
    • 25
  • y

    Yongchan Hong

    02/09/2022, 7:31 AM
    Did anyone tried using FlyteRemote? Specifically, did anyone tried registering? Can we register flyte workflow using FlyteRemote?
    👋 1
    s
    k
    • 3
    • 21
  • s

    Sandra Youssef

    02/09/2022, 6:11 PM
    Hi Flyers, Watch recordings of yesterday's OSS Sync Up on our Flyte YouTube channel: •

    Large Scale Processing with Flyte: Blackshark.ai's Semantic 3D Planet▾

    [Maarten de Jong & Team - Blackshark.ai] • ⚡*

    Structured Datasets▾

    *⚡[Yee Tong - Union.ai] •

    Community Updates▾

    [Ketan Umare - Union.ai] Many thanks to our guest speaker @Maarten de Jong and the team from Blackshark.ai!
    🙌 9
    k
    • 2
    • 1
  • k

    Ketan (kumare3)

    02/10/2022, 4:13 PM
    You can upgrade to higher version. Cc @Pradithya Aria Pura you guys already use dbt right?
    p
    • 2
    • 1
  • e

    Eduardo Apolinario (eapolinario)

    02/10/2022, 4:15 PM
    @Brandon Segal, can you update flytekit to 0.30.1? That should take care of this on the flytekit side. Can you also confirm which dbt version you're using?
    b
    k
    • 3
    • 3
  • s

    Stephen

    02/11/2022, 3:52 PM
    Hey quick question, how would you add a
    nodeSelector
    for a specific workflow/task? I’m not sure if taints are enough to trigger our cluster-autoscaler. I wanted to edit the pods on the fly but I can’t 😕
    k
    h
    j
    • 4
    • 17
  • s

    Sandra Youssef

    02/14/2022, 6:31 PM
    If you or someone you know would like to join Flyte's first virtual hackathon with the MLOps community, Engineering Labs, sign up by clicking "Join Hackathon". Build MLOps and Data products, have some fun, and win exciting prizes! Feb 22-March 22 More details in @Niels Bantilan's video below:

    https://youtu.be/bljxhoTuM60▾

    Feel free to spread the word!
    💯 4
    🎯 2
    a
    • 2
    • 2
  • k

    Kam Leung

    02/16/2022, 1:20 PM
    Hey all! In our quest to increase the number of plugins, we'd like to ask which plugins you want Flyte to be integrated with next. Here's what we are thinking, but feel free to let us know your ideas.
    k
    s
    • 3
    • 4
  • j

    JP Kosymna

    02/17/2022, 7:13 PM
    Hey all, I'm having trouble passing an empty list to a map_task. I think it might be a bug, hopefully there is a simple work around I could use for now. In the following example. Workflow
    test_fixed_array_map_task
    completes successfully in a few seconds, while workflow
    test_empty_array_map_task
    hangs indefinitely with the status QUEUED.
    from flytekit import map_task, task, workflow
    
    @task
    def say_hello(name: str) -> str:
        message = f"hello {name}"
        print(message)
        return message
    
    @workflow
    def test_empty_array_map_task() -> list[str]:
        names: list[str] = []
        messages = map_task(say_hello)(name=names)
        return messages
    
    @workflow
    def test_fixed_array_map_task() -> list[str]:
        names: list[str] = ['jp', 'ketan']
        messages = map_task(say_hello)(name=names)
        return messages
    Let me know if there is any additional information I can provide. Thanks
    y
    d
    k
    • 4
    • 13
  • k

    Ketan (kumare3)

    02/18/2022, 5:38 AM
    Folks, I have a question for you. We are working on revamping the flytekit configuration system. But, always try very hard not to break anyone. Would it be ok, if we changed flytekit.remote constructor and deprecate some configs. This may break some scripts that use flytekit.remote, but the fix will be very simple. cc @Maarten de Jong / @jeev / @Dylan Wilder
    m
    f
    j
    • 4
    • 8
  • s

    Sandra Youssef

    02/18/2022, 7:08 PM
    Engineering Labs is starting next week! If you haven't signed up, registration is still open! https://flyte.org/hackathon/ The hackathon with MLOps is starting Tuesday Feb 22nd, so feel free to ping us or @mlopscommunity any questions! mlops-community.slack.com (#engineeringlabs)
    👍 2
    👀 1
    s
    n
    • 3
    • 3
  • s

    satyanarayana m

    02/19/2022, 8:14 AM
    Hi #general i have started flyte sandbox but when I go to the localhost with portnumber 30081 in browser. I see an error that localhost didn't send any data And the flyte ui doesnt load. Being very new to docker I couldn't understand how to solve this problem. Can anyone help?
    y
    • 2
    • 1
  • a

    Ali Ahmed

    02/24/2022, 11:32 AM
    I don't where where to find the starting announcement 😞
    s
    n
    • 3
    • 6
  • g

    Greg Gydush

    02/24/2022, 7:55 PM
    In Flyte UI, is it possible to populate the description field of a workflow? Google style docstrings work for documenting parameters, but the description on the UI is always “This workflow has no description”
    y
    k
    +2
    • 5
    • 7
  • b

    Brandon Segal

    02/25/2022, 2:54 AM
    Hey I was working with a set of flyte tasks in a workflow and was couldn't figure out an error related to why tasks downstream couldn't find a FlyteFile? My workflow passes around a dataclass
    @dataclass_json
    @dataclass
    class Output:
        manifest: FlyteFile
        catalog: typing.Optional[FlyteFile]
        run_results: typing.Optional[FlyteFile]
        sources: typing.Optional[FlyteFile]
    My first task makes this object and passes it to the next task but the next task errors with an error that there is no file where the manifest file is. Would love to know if maybe I'm doing something I shouldn't be doing or if there is something I need to do to get this to work.
    👍 1
    k
    k
    • 3
    • 15
  • f

    Farman Pirzada

    02/27/2022, 6:07 PM
    is there a help channel?
    h
    • 2
    • 4
  • f

    Farman Pirzada

    02/27/2022, 6:09 PM
    if not, I'm trying to run this guide: https://docs.flyte.org/en/latest/getting_started/deploy.html I am on:
    flytectl sandbox start --source .
    And got to this screen below but it just hangs. I was on my VPN when I first tried it but went for another go after turning it off, in case it was an issue:
    h
    k
    y
    • 4
    • 6
Powered by Linen
Title
f

Farman Pirzada

02/27/2022, 6:09 PM
if not, I'm trying to run this guide: https://docs.flyte.org/en/latest/getting_started/deploy.html I am on:
flytectl sandbox start --source .
And got to this screen below but it just hangs. I was on my VPN when I first tried it but went for another go after turning it off, in case it was an issue:
I do see this section though so I can try this to see what I did wrong in case I missed something: https://docs.flyte.org/en/latest/community/troubleshoot.html#troubleshoot
My understanding of Flyte is limited, but I assumed that the source would be
flyte.config
which i have in my project so why would this message come up regarding that it could not find a config file?
h

Haytham Abuelfutuh

02/27/2022, 6:15 PM
It's looking for a different config file. It's not an error though.. expected to be empty the first time.. What happens if you answer yes to the delete existing cluster question?
f

Farman Pirzada

02/27/2022, 6:19 PM
So this most recent time of trying again I: 1. Said
n
to deleting brand new flyte cluster 2. Next message came up saying I needed to export two variables: a. Add KUBECONFIG and FLYTECTL_CONFIG to your environment variable 3. I re-ran the
flytectl sandbox start --source .
command, said
y
to deleting brand new flyte cluster and was able to get my docker container running
k

Ketan (kumare3)

02/27/2022, 9:08 PM
I think the y/n is misleading. Cc @Yuvraj does it every work when you say no
y

Yuvraj

02/28/2022, 4:20 AM
Yes, No is only useful if container already running. It will not perform restart.
View count: 30