flyte-org #ask-the-community

ask-the-community

Eduardo Matus

03/23/2023, 12:13 AM

Hi everyone… I had running flyte on EKS, but for some reason after trying to setup the notifications, can’t make the UI work. Tried using the configuration that had before (working) but flyte scheduler image and flyte admin can’t run. The error in scheduler that I get is this one:

Init Containers:
  flytescheduler-check:
    Container ID:  <docker://a8fd892170ade6a0395de645ec513dd810c8d6af23aee1b6cefda29fd3c09aa>c
    Image:         <http://cr.flyte.org/flyteorg/flytescheduler-release:v1.2.0-b1|cr.flyte.org/flyteorg/flytescheduler-release:v1.2.0-b1>
    Image ID:      <docker-pullable://cr.flyte.org/flyteorg/flytescheduler-release@sha256:324a09a2a7ccd3dd50f334ac83ef183f83af2840c87807c562e506055a48e0ba>
    Port:          <none>
    Host Port:     <none>
    Command:
      flytescheduler
      precheck
      --config
      /etc/flyte/config/*.yaml
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Wed, 22 Mar 2023 21:04:50 -0300
      Finished:     Wed, 22 Mar 2023 21:04:54 -0300
    Ready:          False
    Restart Count:  7
    Environment:    <none>
    Mounts:
      /etc/db from db-pass (rw)
      /etc/flyte/config from config-volume (rw)
      /etc/secrets/ from auth (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wxvbg (ro)

Anyone can give me some directions?

seunggs

03/23/2023, 2:07 AM

A quick question - I have a failed execution pod stuck at

pending

state. Is this an expected behavior?

seunggs

03/23/2023, 2:08 AM

Shouldn’t the pod be destroyed if it fails?

Leiqing

03/23/2023, 8:36 AM

Hi team, I’m using

flytekit

to list projects, but it only gives me active projects, whereas with

flytectl

, I’m getting all projects by default and can filter for active projects with

--filter.fieldSelector "state=1"

from flytekit.clients.friendly import SynchronousFlyteClient
from flytekit.configuration import Config
from flytekit.models.filters import Filter

client = SynchronousFlyteClient(Config.auto().platform)
print(client.list_projects_paginated())

How can I get all projects with

flytekit

Visak

03/23/2023, 3:34 PM

I have a dynamic task, that has a few chained tasks (using >> operator), when running I get the following error`RuntimeExecutionError: max number of system retry attempts [51/50] exhausted. Last known status message: [system] unable to read futures file, maybe corrupted, caused by: [system] Failed to read futures protobuf file., caused by: path/to/futures.pb: [LIMIT_EXCEEDED] limit exceeded. 22mb > 10mb.` Any thoughts on what might be going on here? Thanks!

Sabrina Lui

03/23/2023, 6:22 PM

Hello, I'd like to use the results of a

map_task

task with a

min_success_ratio

< 1.0 as input to another task. However, as the failed outputs are not filtered out of the results, flytekit is unable to convert them to inputs properly, resulting in errors like this and this. Since this happens in type engine code between tasks, even an intermediate task to filter out none values would fail. This blocks us from using

map_task

effectively in our workflows. Here is a min repro example: https://gist.github.com/sabrinalui/e5478b9557dcf370c84d5e57758b4c87 It seems

min_success_ratio

doesn't work locally but I attached the output when running with inputs

[true,true,false]

from our endpoint below. What would the lift be to (optionally) filter out failed tasks from the
map_task
output?

Frank Shen

03/23/2023, 10:32 PM

Hi, airflow-provider-flyte allows airflow dag to include flyte workflow or task, and sensor when flyte task is completed. What I need is the opposite: I need a flyte task to sensor when an airflow dag run is completed and then trigger the flyte downstream workflow or task. Is there any such airflow-sensor in flyte? If not, could anyone suggest any approach to achieve that? Thanks!

Rahul Mehta

03/23/2023, 10:50 PM

Is there a straightforward way to either override or set a template for the pod name for dynamic steps? It'd be very helpful to minimally be able to name the pods associated w/ a dynamic step with a common prefix for the workflow (rather than a random set of characters)

Younes El Hjouji

03/24/2023, 11:57 AM

Hi Flyte friends, I am using pyflyte and flytectl to package and push workflows to my cluster. I can see the pushed workflows through flyte console When I attempt to launch any workflow, the first task fails after about 40 seconds with the error in the picture.

Bosco Raju

03/24/2023, 4:14 PM

Hi folks, I am running into an error when running a simple hello_world workflow. I am using a custom docker image. Any idea why?

FROM python:3.8-buster

WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root

ARG tag
ARG wandb_api_key
ARG wandb_username

ENV FLYTE_INTERNAL_IMAGE $tag
ENV WANDB_API_KEY $wandb_api_key
ENV WANDB_USERNAME $wandb_username

# Install the AWS cli separately to prevent issues with boto being written over
RUN pip3 install awscli

RUN apt-get update && apt-get install -y curl

ENV VENV /opt/venv
# Virtual environment
RUN python3 -m venv ${VENV}
ENV PATH="${VENV}/bin:$PATH"

# Install Python dependencies
COPY requirements.txt /root/.
RUN pip install -r /root/requirements.txt

# Copy the actual code
COPY src/ /root/src/

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[ab8dm4nttv49wgvn59kg-n0-0] terminated with exit code (1). Reason [Error]. Message: 
rtlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'site-packages.flytekit'
Traceback (most recent call last):
  File "/opt/venv/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 513, in fast_execute_task_cmd
    subprocess.run(cmd, check=True)
  File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pyflyte-execute', '--inputs', '<s3://senn-ai-mlops-flyte/metadata/propeller/flytesnacks-development-ab8dm4nttv49wgvn59kg/n0/data/inputs.pb>', '--output-prefix', '<s3://senn-ai-mlops-flyte/metadata/propeller/flytesnacks-development-ab8dm4nttv49wgvn59kg/n0/data/0>', '--raw-output-data-prefix', '<s3://senn-ai-mlops-flyte/data/sk/ab8dm4nttv49wgvn59kg-n0-0>', '--checkpoint-path', '<s3://senn-ai-mlops-flyte/data/sk/ab8dm4nttv49wgvn59kg-n0-0/_flytecheckpoints>', '--prev-checkpoint', '""', '--dynamic-addl-distro', '<s3://senn-ai-mlops-flyte/mo/flytesnacks/development/NRI2T5OSZMCFXKR4CUNLWO7MSM======/fastab502ef8d4ae75b6b5497a94633e8642.tar.gz>', '--dynamic-dest-dir', '/root', '--resolver', 'site-packages.flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'src.workflows.hello_world', 'task-name', 'say_hello']' returned non-zero exit status 1.

MUhwjR9VF3K2QJhL2VvNGoFY

03/24/2023, 8:58 PM

Is this a known issue? E: Fixed

$ docker pull <http://cr.flyte.org/flyteorg/flytescheduler-release:v1.4.3|cr.flyte.org/flyteorg/flytescheduler-release:v1.4.3>
5e5f900059c8: Download complete
0f96f76f982d: Download complete
88dae1745799: Downloading [>                                                  ]       0B/2.343kB
failed to copy: httpReadSeeker: failed open: failed to do request: Get "<https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:88dae1745799f4078f859b8ada55bc1f1757d10d9c97cdeb62f88cfc827c3a1f?se=2023-03-24T21%3A05%3A00Z&sig=rMhv3oYchaYycF260vI0pdx6pENdYsKpph9eXak%2BMPw%3D&sp=r&spr=https&sr=b&sv=2019-12-12>": x509: certificate is valid for *.<http://githubassets.com|githubassets.com>, <http://githubassets.com|githubassets.com>, not <http://pkg-containers.githubusercontent.com|pkg-containers.githubusercontent.com>

seunggs

03/24/2023, 11:36 PM

Hi, we can specify the number of gpus in flyte tasks, but how do we specify which type of gpu to use?

Taeef Najib

03/25/2023, 3:37 AM

Hi, I was trying to use a local

csv

file. But I got this error:

Message:

    [Errno 2] No such file or directory: 'data.csv'

User error.

What would be the workaround?

mykyta luzan

03/27/2023, 8:17 AM

Hi, guys! Could you help with s3 credentials. I can run hello_world example in sandbox env, but I am failing to do so on our local k8s cluster.

raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to get data from <s3://blabs-flyte-mgmt-metadata/dispute-resolution/development/BP7IZUPWYESUYFGRO52LE7HBDU======/scriptmode.tar.gz> to /root/ (recursive=False).

Original exception: Access Denied

We don’t have any auth yet for flyte. I assume I just haven’t specified s3 bucket endpoint, access_key_id, and secret_access_key. Where should I do this? In my

.flyte/config.yaml

? Or on

k8s/values.yaml

side via devops support? If so what is the template?

storage:
  access-key: <SECRET>
  auth-type: iam
  disable-ssl: false
  endpoint: <s3://blabs-flyte-mgmt-metadata>
  region: eu-central-1
  secret-key: <SECRET>

I don’t understand how to use this page https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.configuration.S3Config.html

Srinivas Venkattaramanujam

03/27/2023, 4:16 PM

Hi! I'm evaluating Flyte for our workflow orchestration needs. I'm curious what will happen when WorkflowExecutor tries to update the task status to FlyteAdmin but FlyteAdmin is unavailable for a prolonged duration (say 1 hour) for some reason. Will the WorkflowExecutor eventually update the correct task status to the FlyteAdmin when it comes back up? Or will the FlyteAdmin have an inconsistent state (say Running when the task has actually failed/succeeded when the FlyteAdmin was down)

justin hallquist

03/27/2023, 4:19 PM

what constitutes as a valid execution name? I am having a hard time finding the validator both in the docs and code

Error: rpc error: code = InvalidArgument desc = invalid name format: 370eb303160612

David Espejo (he/him)

03/27/2023, 5:35 PM

x-posting from #flyte-deployment: https://flyte-org.slack.com/archives/C01P3B761A6/p1679938476764839

varsha Parthasarathy

03/27/2023, 8:09 PM

Hi team, What is the suggestions/do’s & don’t for sharing flyte tasks across multiple workflows? WorkflowA has a taskA, registered with SHA1 WorkflowB uses taskA and has the same SHA1, when we register both workflows, get the error that task_id already exists with SHA1. How can we get around this issue?

Frank Shen

03/27/2023, 8:32 PM

Hello, Do you know if there is a flytekit plugins for airflow? I need to detect if an Airflow dag run is completed from Flyte. Thanks.

Ena Škopelja

03/28/2023, 12:05 PM

Hi all, I have a task returning a

StructuredDataset

that's failing if I turn on

cache_serialize

with this error:

[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):

      File "/opt/venv/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/opt/venv/lib/python3.9/site-packages/flytekit/core/base_task.py", line 572, in dispatch_execute
        raise TypeError(
    Failed to convert return value for var o0 for function {my task} with error <class 'pyarrow.lib.ArrowInvalid'>: ("Could not convert ('... ... (580 characters truncated) ... ...',) with type Row: did not recognize Python value type when inferring an Arrow data type", 'Conversion failed for column sequence with type object')

The sequence column of the dataframe I return is a string (no non-standard characters). Any idea on what could be causing this? Regular (no

cache_serialize

) cache works just fine.

Prashant Jain

03/28/2023, 12:18 PM

@Prashant Jain has left the channel

Andrew Korzhuev

03/28/2023, 2:16 PM

flyteidl 1.3.12 depends on protobuf<5.0.0 and >=4.21.1
    sagemaker 2.133.0 depends on protobuf<4.0 and >=3.1

Since 1.3 FlyteKit has version conflict with Sagemaker API and Tensorflow as well by deprecating protobuf 3 making it impossible to use those libraries together

HIMANSHU JOSHI

03/28/2023, 5:03 PM

Hey, does anybody know how can i pass multiple values to task_config param in @task decorator i.e. i want to pass Pod configuration as well as Spark configuration?

Samuel Bentley

03/28/2023, 7:24 PM

Hi everyone! I'm trying to run the workflow remotely as described in the Flyte Fundamentals docs, but I get an OpenSSL version error:

E0328 20:15:47.445709000 140704578418304 <http://ssl_transport_security.cc:1495]|ssl_transport_security.cc:1495]> Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.

I'm using Python 3.9, grpcio & grpc-status 1.48.2, pyOpenSSL 23.0.0 and against a local flyte-sandbox (latest version) Can anyone help? I'm trying to get this working as a proof-of-concept for wider use in my compnay and need to demonstrate that this works

Craig Amundsen

03/28/2023, 10:38 PM

Hello - I've been looking through the docs and searching here, and haven't figured out how, from flytekit, to query a flyte set up to get the execution status of a workflow that was started via FlyteRemote.emote.execute. Essentially, I want to be able to display in the system that kicked off the workflow if it's Pending, Running, Succeeded, etc. It seems like it should be possible. If someone could point me in the right direction, I'd be most appreciative.

Greg Gydush

03/29/2023, 12:01 AM

Hey all! With GCP, the logs before user code is executed are all coming up as severity=ERROR. I'd love to try to fix this at some point and was wondering if anyone has thoughts! Ideally this could be configurable too - I was talking to @Yee about this earlier and he suggested posting here for visibility! I also made a GitHub issue

HIMANSHU JOSHI

03/29/2023, 12:00 PM

Hey I've installed flyte on GKE it was running fine till now but i wanted to increase ResourceQuota for development/ staging and production namespace so i changed the config for the same but suddenly flyte-admin pod sync-cluster-resource container is failing with this error.

panic: Failed to sync cluster resources : no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>"

goroutine 1 [running]:
main.main()
/go/src/github.com/flyteorg/flyteadmin/cmd/main.go:13 +0x91

Victor Gustavo da Silva Oliveira

03/29/2023, 12:15 PM

Hello all I'm having an issue when trying to use reference tasks... When building image, I'm getting this error:

'ReferenceTask' object has no attribute '_task_function'

... Can someone help me? I would be very appreciated

Agath Emmanuel

03/29/2023, 1:18 PM

Hello all, I could find here that Authorization can be set up for Flyte. But in the doc in production grade Flyte deployment, its mentioned that "Authorization is not supported out-of-the-box in Flyte". Is this just for UI or am I missing something here. So does Flyte have Role based access control and Authorization support or can it be achieved, thinking in terms of production requirements?

Srinivas Venkattaramanujam

03/29/2023, 4:28 PM

hey! I'm back with another question 😅 Can multiple instances of FlyteAdmin be run simultaneously without issues (such as duplicated scheduled runs)? I found the following about the data plane but nothing about FlyteAdmin scalability_._ While FlyteAdmin serves the Workflow Execution API, it does not itself execute workflows. To launch workflow executions, FlyteAdmin sends the workflow DAG to the DataPlane. For added scalability and fault-tolerance, FlyteAdmin can be configured to load-balance workflows across multiple isolated data-plane clusters.

Title

Srinivas Venkattaramanujam

03/29/2023, 4:28 PM

Yee

03/29/2023, 8:16 PM

yes we’ve run asgs before for flyte admin. scheduler semantics are idempotent and stored as an execution in the database so no need to worry about that.

View count: 1