Can we also easily create integrations with external apps as easily as prefect on Flyte too?

Hey everyone, Join our OSS Sync Up guests @Stephen and @Tim Bauer tomorrow 1/25 at **8:30**am PT, who will share their experience of how Wolt deploys Flyte, in "*End-To-End ML With Flyte*". Also featured is our 2nd FlyteConsole UX/UI Townhall with @Jason Porter. Zoom Link and Calendar Invite See you there!

Do I need to run

flytectl register files

first?

Hi I would like to set an environment variable for every Pod that gets scheduled to k8s. What is the preferred way to do this? Specifically I’m trying to set the following for datadog metrics

env:
    - name: DD_AGENT_HOST
      valueFrom:
          fieldRef:
              fieldPath: status.hostIP

is there documentation on how to use container tasks beyond https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/raw_container.html . I'm trying with the sandbox env but not having luck. I've heard that it depends on the SYS_PTRACE capability, which may depend on one's K8s setup

Hey all. I'm having trouble using a ShellTask with secrets. In the following task I would expect an environment variable possibly named

AWS-ACCESS-KEY_AWS_ACCESS_KEY_ID

to exist but it doesn't. What am I doing wrong here?

task_with_secrets = ShellTask(
    debug=True,
    name="task_with_secrets",
    script='''
    set -ex
    env > out_env
    ''',
    inputs=kwtypes(source=str),
    output_locs=[
        OutputLocation(var="x", var_type=FlyteFile, location="out_env")
    ],
    secret_requests=[
        Secret(
            group='aws-access-key',
            key='AWS_ACCESS_KEY_ID',
            mount_requirement=Secret.MountType.ENV_VAR
        ),
        Secret(
            group='aws-access-key',
            key='AWS_SECRET_ACCESS_KEY',
            mount_requirement=Secret.MountType.ENV_VAR
        ),
    ]
)

Deploying Flyte just got a whole lot easier - thanks to @Nitin Aggarwal's team at Run[X] developing Opta's IaC framework. https://blog.flyte.org/how-opta-makes-deploying-flyte-much-easier

Hi! We are comparing how often a workflow succeeds and gets accepted, i.e.,

flyte:propeller:all:workflow:success_duration_ms_count

  vs 

flyte:propeller:all:workflow:accepted

and get some unexpected numbers. Successes is equal to 4345 for the last 30 days which is bigger than the number of accepted which is 4322. There are 0 failures also. Should they not be the same? Is there a better metric to use to be able to compare these numbers? I don’t find

flyte:propeller:all:workflow:success

for example

Hi on my GCP Flyte deployment Flyte deletes dynamic tasks from k8s as soon as they complete whether they pass or fail. Regular python tasks are deleted after the GC Interval which is 30h on our cluster. This makes it very hard to debug dynamic tasks if something goes wrong because the pod is no longer in k8s. Is this something that I can stop Flyte from doing via a config option?

Is there a default task timeout? Does it happen to be 48 hours by some chance? Is it configurable?

hi there, I'm registering workflows with

flyte-cli register-files ... --kubernetes-service-account=<SERVICE_ACCOUNT>

but when I look at the running pods they all use default serviceAccount. I have this in .config also:

[auth]
kubernetes_service_account=<SERVICE_ACCOUNT>

am I missing something?

@Haytham Abuelfutuh I saw your post a few weeks back saying that Flyte v0.19.2 helm chart would work with K8s 1.22+. I see this PR was successfully merged that makes the necessary helm updates. However, when I try and run using

helm

it does not seem to be pulling in all the changes. Specifically

kind: Ingress

and

kind: CustomResourceDefinition

are still using

v1beta1

. I have run

helm repo update
helm template --version v0.19.2 ...

Hey Flyte team, we (at Woven Planet) are hoping to start making good use of Flyte map tasks. I went through the Flyte OSS roadmap issues and saw that a lot of recent work has been done to get them up to speed (such as https://github.com/flyteorg/flyte/issues/1838 to propagate the status of subtasks). However, I can see on the latest

flyteconsole:v0.41.0

that in the console, there are no log links to subtasks or any UI to show the status of subtasks. Can I ask how soon you plan to land these features in flyteconsole and if there are GitHub issues or PRs that I can follow / contribute to?

Did anyone tried using FlyteRemote? Specifically, did anyone tried registering? Can we register flyte workflow using FlyteRemote?

Hi Flyers, Watch recordings of yesterday's OSS Sync Up on our Flyte YouTube channel: •

Large Scale Processing with Flyte: Blackshark.ai's Semantic 3D Planet▾

[Maarten de Jong & Team - Blackshark.ai] • ⚡**⚡[Yee Tong - Union.ai] •

Community Updates▾

[Ketan Umare - Union.ai] Many thanks to our guest speaker @Maarten de Jong and the team from Blackshark.ai!

You can upgrade to higher version. Cc @Pradithya Aria Pura you guys already use dbt right?

@Brandon Segal, can you update flytekit to 0.30.1? That should take care of this on the flytekit side. Can you also confirm which dbt version you're using?

Hey quick question, how would you add a

nodeSelector

for a specific workflow/task? I’m not sure if taints are enough to trigger our cluster-autoscaler. I wanted to edit the pods on the fly but I can’t 😕

If you or someone you know would like to join Flyte's first virtual hackathon with the MLOps community, Engineering Labs, sign up by clicking "Join Hackathon". Build MLOps and Data products, have some fun, and win exciting prizes! Feb 22-March 22 More details in @Niels Bantilan's video below: Feel free to spread the word!

Hey all! In our quest to increase the number of plugins, we'd like to ask which plugins you want Flyte to be integrated with next. Here's what we are thinking, but feel free to let us know your ideas.

Hey all, I'm having trouble passing an empty list to a map_task. I think it might be a bug, hopefully there is a simple work around I could use for now. In the following example. Workflow

test_fixed_array_map_task

completes successfully in a few seconds, while workflow

test_empty_array_map_task

hangs indefinitely with the status QUEUED.

from flytekit import map_task, task, workflow

@task
def say_hello(name: str) -> str:
    message = f"hello {name}"
    print(message)
    return message

@workflow
def test_empty_array_map_task() -> list[str]:
    names: list[str] = []
    messages = map_task(say_hello)(name=names)
    return messages

@workflow
def test_fixed_array_map_task() -> list[str]:
    names: list[str] = ['jp', 'ketan']
    messages = map_task(say_hello)(name=names)
    return messages

Let me know if there is any additional information I can provide. Thanks

Folks, I have a question for you. We are working on revamping the flytekit configuration system. But, always try very hard not to break anyone. Would it be ok, if we changed flytekit.remote constructor and deprecate some configs. This may break some scripts that use flytekit.remote, but the fix will be very simple. cc @Maarten de Jong / @jeev / @Dylan Wilder

Engineering Labs is starting next week! If you haven't signed up, registration is still open! https://flyte.org/hackathon/ The hackathon with MLOps is starting Tuesday Feb 22nd, so feel free to ping us or @mlopscommunity any questions! mlops-community.slack.com (#engineeringlabs)

Hi #general i have started flyte sandbox but when I go to the localhost with portnumber 30081 in browser. I see an error that localhost didn't send any data And the flyte ui doesnt load. Being very new to docker I couldn't understand how to solve this problem. Can anyone help?

I don't where where to find the starting announcement 😞

In Flyte UI, is it possible to populate the description field of a workflow? Google style docstrings work for documenting parameters, but the description on the UI is always “This workflow has no description”

Hey I was working with a set of flyte tasks in a workflow and was couldn't figure out an error related to why tasks downstream couldn't find a FlyteFile? My workflow passes around a dataclass

@dataclass_json
@dataclass
class Output:
    manifest: FlyteFile
    catalog: typing.Optional[FlyteFile]
    run_results: typing.Optional[FlyteFile]
    sources: typing.Optional[FlyteFile]

My first task makes this object and passes it to the next task but the next task errors with an error that there is no file where the manifest file is. Would love to know if maybe I'm doing something I shouldn't be doing or if there is something I need to do to get this to work.

is there a help channel?