flyte-org #ask-the-community

Xin Shi

03/17/2023, 7:41 PM

Hi, I have a simple question: From the Flyte documentation and tutorial, it seems classes are not supported by Flyte. Is this correct? If not, is there any material that you can share about using classes in Flyte. Thanks!

Anindya Saha

03/17/2023, 10:24 PM

Hi Team, I am following the first example to launch a test workflow, I see the following error in the docker desktop dashboard console:

Copy code

"Error syncing pod, skipping" err="failed to \"StartContainer\" for \"f154ee96e6c1a4fed852-n0-0\" with ImagePullBackOff: \"Back-off pulling image \\\"<http://cr.flyte.org/flyteorg/flytekit:py3.10-1.2.11\\\|cr.flyte.org/flyteorg/flytekit:py3.10-1.2.11\\\>"\"" pod="flytesnacks-development/f154ee96e6c1a4fed852-n0-0" podUID=5952591c-fde8-47bd-a9c4-2885a01915a7
E0317 22:24:01.121900      71 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"f154ee96e6c1a4fed852-n0-0\" with ImagePullBackOff: \"Back-off pulling image \\\"<http://cr.flyte.org/flyteorg/flytekit:py3.10-1.2.11\\\|cr.flyte.org/flyteorg/flytekit:py3.10-1.2.11\\\>"\"" pod="flytesnacks-development/f154ee96e6c1a4fed852-n0-0" podUID=5952591c-fde8-47bd-a9c4-2885a01915a7

How do I resolve this ?

Ryo M

03/18/2023, 1:19 PM

Hi, Is each task running on different pod? I want to know how to share files saved in one task among tasks.

Harmen van Rossum

03/19/2023, 6:23 PM

Hi! For some reason, I’m unable to figure out how to run a map_task with a custom container_image. I’ve tried

.with_overrides(container_image='image:version')

and using a decorator for the task, but both approaches don’t seem to do anything (just the default image is used). What’s the appropriate way to do this? Thanks!

Ryo M

03/20/2023, 5:30 AM

Hi, I got the

OOMKilled

error, and I found many similar questions in this channel. According to the discussion, the reason is resource shortage, so I set

@task(limits=Resources(storage="20Gi", ephemeral_storage="20Gi", mem="40Gi", cpu="6"))

which is enough for the error task. But I still got the same error. I do not use GPU. Any other reasons for this?

Copy code

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f52215af773e248b0af3-n2-0] terminated with exit code (1). Reason [OOMKilled]. Message: 
asctime": "2023-03-20 05:09:46,909", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
tar: Removing leading `/' from member names
{"asctime": "2023-03-20 05:09:48,309", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
...

Nan Qin

03/20/2023, 3:23 PM

Hi, a question about using FlyteFile: I have two tasks (t1 & t2) in a workflow:

Copy code

@task
def t1() -> FlyteFile:
    # do something and save checkpoint to path/to/file
    return FlyteFile("path/to/file")

def t2(input: FlyteFile) -> float:
    checkpoint = torch.load(input)
    # do something
    return 1.0

@workflow
def wf() -> float:
    t1_output = t1()
    return t2(t1_output)

when executed, t1 output is stored in S3. In t2 I get the following error

Copy code

[Errno 2] No such file or directory: '/tmp/flytemlfm4mk1/local_flytekit/0569821759669848f705b8863309a4bb/checkpoint-epoch0001.pth'

it seems flyte dowloaded the file from S3 to a local temp folder, not sure why the file does not exist. any ideas?

George Horrell

03/20/2023, 11:01 PM

pyflyte

flytectl

question; is it possible to package and register from a

flyte-package.tgz

as two separate steps, all within

pyflyte

? Or is it necessary to pass the

flyte-package.tgz

file to flytectl after packaging in

pyflyte

seunggs

03/21/2023, 2:10 AM

Hi there seems to be conflicting values.yaml setup for flyte helm charts in the docs - is this the correct one to use? https://docs.flyte.org/en/latest/deployment/deployment/cloud_simple.html

Ben Rosand

03/21/2023, 11:45 AM

Hey everyone! Has anyone used flyte for model serving? I'm new to flyte but trying to set up some workflows with ray serve. Has anyone experimented with this before?

Eduardo Matus

03/21/2023, 2:06 PM

Hi everyone…. trying to setup the notifications following the guide but the helm template is not rendering correctly. I tried different ways to escape the ‘{{’ but none of them are working.. any thoughts?

Copy code

workflow_notifications:
  enabled: true
  config:
    notifications:
      type: aws
      region: "{{ .Values.userSettings.accountRegion }}"
      publisher:
        topicName: "arn:aws:sns:{{ .Values.userSettings.accountRegion }}:{{ .Values.userSettings.accountNumber }}:flyte_notification"
      processor:
        queueName: flyte_notification
        accountId: "{{ .Values.userSettings.accountNumber }}"
      emailer:
        subject: "Notice: Execution \"{{ name }}\" has {{ phase }} in \"{{ domain }}\"."
        #/{{ `{{` }} domain {{ `}}` }}/{{ `{{` }} launch_plan.name {{ `}}` }} has '{{ `{{` }} phase {{ `}}` }}'"
        sender: "<mailto:flyte-notification@mydomain.com|flyte-notification@mydomain.com>"

Aditya Sharma

03/21/2023, 2:40 PM

Hello everyone, we are trying to evaluate Flyte and wondering if it supports RBAC, not able to find anything related to it in the doc.

Martin Tschechne

03/21/2023, 3:41 PM

Hi everyone, flyte-novice here. Is it possible to check for

None

values in conditionals? I know about

is_true()

is_false()

and

is_()

, but checking for

None

seems not to be supported yet? Any help appreciated 🙏 Thanks!

Ketan (kumare3)

03/21/2023, 4:13 PM

The community sync is on Right now Learn how to serve with banana and also learn from Shift about their journey - https://zoom.us/j/93875115830?pwd=YWZWOHl1ODRRVjhjVkxSV0pmZkJaZz09

Visak

03/21/2023, 11:01 PM

Hi all, a flyte-newbie here , I want to write a workflow that runs the same task sequentially in a workflow. Any thoughts/pointers to examples as to how I can do this in Flyte? For example, here is a dummy case where I want to run the task ‘add_1’ sequentially, currently flute would run these simultaneously As there is no dependency between 2 calls

Jay Ganbat

03/21/2023, 11:10 PM

Anyone know if there is any preference between using

create_node

to create dependencies or use

chaining Flyte entity

seunggs

03/22/2023, 3:47 AM

Are there some docs on ingress for flyte if we need to use another ingress service like emissary-ingress? Looks like flyte-binary service has 8088 for http and 8089 for grpc, but using a custom domain like example.com, not sure how to map the routes to both. I tried just mapping 8088 and console works, but I got this error when I tried to run `flytectl register`:

Copy code

Error: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/plain; charset=utf-8"

Maybe this error is unrelated? Not sure

seunggs

03/22/2023, 3:48 AM

If there’s anything I can look at to figure this out, I’d really appreciate it!

mykyta luzan

03/22/2023, 10:07 AM

Hi, guys! Our team has Vault for Flyte as

secrets/common/flyte

but it’s not clear to me what is the SECRET_GROUP here?

common/flyte

Ferdinand von den Eichen

03/22/2023, 2:16 PM

Does Flyte have concept for handling errors at workflow execution time? For example it would be nice to be able to react on a failing pod • by trying something else (schedule pod with twice as much memory to train on) • by creating a datadog / new relic incident • in general just being able to run a task as a reaction would open up a world of use cases

Broder Peters

03/22/2023, 2:59 PM

How are you usually handling notifications around your e.g. failing workflows? I understand that the mail flow is the intended way right now or to listen to the events behind that flow, but am wondering as well if there are any long run plans? Is e.g. https://github.com/flyteorg/flyte/issues/2317 on some roadmap to open up the notification process?

Visak

03/22/2023, 5:11 PM

When using map_task with a large number of inputs I get the following error

array size > max allowed. requested [13947]. allowed [5000].

How can I address this? Is there a way I can increase this size or is there a workaround to the size limit? Appreciate any help!

Tim Sheiner

03/22/2023, 8:49 PM

Hi! I'm trying to run the example in the docs here: https://docs.flyte.org/projects/cookbook/en/latest/getting_started/analytics.html# When I register and run on the demo sandbox I get this error:

Copy code

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[ass5zvhclkmtr9hd55ks-n0-0] terminated with exit code (1). Reason [Error]. Message: 
trap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/root/workflows/ex_analytics.py", line 4, in <module>
    import pycountry
ModuleNotFoundError: No module named 'pycountry'
Traceback (most recent call last):
  File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 513, in fast_execute_task_cmd
    subprocess.run(cmd, check=True)
  File "/usr/local/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pyflyte-execute', '--inputs', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ass5zvhclkmtr9hd55ks/n0/data/inputs.pb>', '--output-prefix', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ass5zvhclkmtr9hd55ks/n0/data/0>', '--raw-output-data-prefix', '<s3://my-s3-bucket/data/7x/ass5zvhclkmtr9hd55ks-n0-0>', '--checkpoint-path', '<s3://my-s3-bucket/data/7x/ass5zvhclkmtr9hd55ks-n0-0/_flytecheckpoints>', '--prev-checkpoint', '""', '--dynamic-addl-distro', '<s3://my-s3-bucket/flytesnacks/development/TPFR2TGNNPMEAIZGLYXBVJ46S4======/scriptmode.tar.gz>', '--dynamic-dest-dir', '/root', '--resolver', 'flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'workflows.ex_analytics', 'task-name', 'clean_data']' returned non-zero exit status 1.
.

Anybody understand this?

Eduardo Matus

03/23/2023, 12:13 AM

Hi everyone… I had running flyte on EKS, but for some reason after trying to setup the notifications, can’t make the UI work. Tried using the configuration that had before (working) but flyte scheduler image and flyte admin can’t run. The error in scheduler that I get is this one:

Copy code

Init Containers:
  flytescheduler-check:
    Container ID:  <docker://a8fd892170ade6a0395de645ec513dd810c8d6af23aee1b6cefda29fd3c09aa>c
    Image:         <http://cr.flyte.org/flyteorg/flytescheduler-release:v1.2.0-b1|cr.flyte.org/flyteorg/flytescheduler-release:v1.2.0-b1>
    Image ID:      <docker-pullable://cr.flyte.org/flyteorg/flytescheduler-release@sha256:324a09a2a7ccd3dd50f334ac83ef183f83af2840c87807c562e506055a48e0ba>
    Port:          <none>
    Host Port:     <none>
    Command:
      flytescheduler
      precheck
      --config
      /etc/flyte/config/*.yaml
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Wed, 22 Mar 2023 21:04:50 -0300
      Finished:     Wed, 22 Mar 2023 21:04:54 -0300
    Ready:          False
    Restart Count:  7
    Environment:    <none>
    Mounts:
      /etc/db from db-pass (rw)
      /etc/flyte/config from config-volume (rw)
      /etc/secrets/ from auth (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wxvbg (ro)

Anyone can give me some directions?

seunggs

03/23/2023, 2:07 AM

A quick question - I have a failed execution pod stuck at

pending

state. Is this an expected behavior?

seunggs

03/23/2023, 2:08 AM

Shouldn’t the pod be destroyed if it fails?

Leiqing

03/23/2023, 8:36 AM

Hi team, I’m using

flytekit

to list projects, but it only gives me active projects, whereas with

flytectl

, I’m getting all projects by default and can filter for active projects with

--filter.fieldSelector "state=1"

Copy code

from flytekit.clients.friendly import SynchronousFlyteClient
from flytekit.configuration import Config
from flytekit.models.filters import Filter

client = SynchronousFlyteClient(Config.auto().platform)
print(client.list_projects_paginated())

How can I get all projects with

flytekit

Visak

03/23/2023, 3:34 PM

I have a dynamic task, that has a few chained tasks (using >> operator), when running I get the following error`RuntimeExecutionError: max number of system retry attempts [51/50] exhausted. Last known status message: [system] unable to read futures file, maybe corrupted, caused by: [system] Failed to read futures protobuf file., caused by: path/to/futures.pb: [LIMIT_EXCEEDED] limit exceeded. 22mb > 10mb.` Any thoughts on what might be going on here? Thanks!

Sabrina Lui

03/23/2023, 6:22 PM

Hello, I'd like to use the results of a

map_task

task with a

min_success_ratio

< 1.0 as input to another task. However, as the failed outputs are not filtered out of the results, flytekit is unable to convert them to inputs properly, resulting in errors like this and this. Since this happens in type engine code between tasks, even an intermediate task to filter out none values would fail. This blocks us from using

map_task

effectively in our workflows. Here is a min repro example: https://gist.github.com/sabrinalui/e5478b9557dcf370c84d5e57758b4c87 It seems

min_success_ratio

doesn't work locally but I attached the output when running with inputs

[true,true,false]

from our endpoint below. What would the lift be to (optionally) filter out failed tasks from the
map_task
output?

Frank Shen

03/23/2023, 10:32 PM

Hi, airflow-provider-flyte allows airflow dag to include flyte workflow or task, and sensor when flyte task is completed. What I need is the opposite: I need a flyte task to sensor when an airflow dag run is completed and then trigger the flyte downstream workflow or task. Is there any such airflow-sensor in flyte? If not, could anyone suggest any approach to achieve that? Thanks!

Rahul Mehta

03/23/2023, 10:50 PM

Is there a straightforward way to either override or set a template for the pod name for dynamic steps? It'd be very helpful to minimally be able to name the pods associated w/ a dynamic step with a common prefix for the workflow (rather than a random set of characters)