https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
ask-the-community
  • e

    Eduardo Matus

    03/23/2023, 12:13 AM
    Hi everyone… I had running flyte on EKS, but for some reason after trying to setup the notifications, can’t make the UI work. Tried using the configuration that had before (working) but flyte scheduler image and flyte admin can’t run. The error in scheduler that I get is this one:
    Init Containers:
      flytescheduler-check:
        Container ID:  <docker://a8fd892170ade6a0395de645ec513dd810c8d6af23aee1b6cefda29fd3c09aa>c
        Image:         <http://cr.flyte.org/flyteorg/flytescheduler-release:v1.2.0-b1|cr.flyte.org/flyteorg/flytescheduler-release:v1.2.0-b1>
        Image ID:      <docker-pullable://cr.flyte.org/flyteorg/flytescheduler-release@sha256:324a09a2a7ccd3dd50f334ac83ef183f83af2840c87807c562e506055a48e0ba>
        Port:          <none>
        Host Port:     <none>
        Command:
          flytescheduler
          precheck
          --config
          /etc/flyte/config/*.yaml
        State:          Waiting
          Reason:       CrashLoopBackOff
        Last State:     Terminated
          Reason:       Error
          Exit Code:    2
          Started:      Wed, 22 Mar 2023 21:04:50 -0300
          Finished:     Wed, 22 Mar 2023 21:04:54 -0300
        Ready:          False
        Restart Count:  7
        Environment:    <none>
        Mounts:
          /etc/db from db-pass (rw)
          /etc/flyte/config from config-volume (rw)
          /etc/secrets/ from auth (rw)
          /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wxvbg (ro)
    Anyone can give me some directions?
    k
    • 2
    • 4
  • s

    seunggs

    03/23/2023, 2:07 AM
    A quick question - I have a failed execution pod stuck at
    pending
    state. Is this an expected behavior?
    k
    • 2
    • 7
  • s

    seunggs

    03/23/2023, 2:08 AM
    Shouldn’t the pod be destroyed if it fails?
  • l

    Leiqing

    03/23/2023, 8:36 AM
    Hi team, I’m using
    flytekit
    to list projects, but it only gives me active projects, whereas with
    flytectl
    , I’m getting all projects by default and can filter for active projects with
    --filter.fieldSelector "state=1"
    from flytekit.clients.friendly import SynchronousFlyteClient
    from flytekit.configuration import Config
    from flytekit.models.filters import Filter
    
    client = SynchronousFlyteClient(Config.auto().platform)
    print(client.list_projects_paginated())
    How can I get all projects with
    flytekit
    ?
    s
    • 2
    • 2
  • v

    Visak

    03/23/2023, 3:34 PM
    I have a dynamic task, that has a few chained tasks (using >> operator), when running I get the following error`RuntimeExecutionError: max number of system retry attempts [51/50] exhausted. Last known status message: [system] unable to read futures file, maybe corrupted, caused by: [system] Failed to read futures protobuf file., caused by: path/to/futures.pb: [LIMIT_EXCEEDED] limit exceeded. 22mb > 10mb.` Any thoughts on what might be going on here? Thanks!
    d
    s
    • 3
    • 12
  • s

    Sabrina Lui

    03/23/2023, 6:22 PM
    Hello, I'd like to use the results of a
    map_task
    task with a
    min_success_ratio
    < 1.0 as input to another task. However, as the failed outputs are not filtered out of the results, flytekit is unable to convert them to inputs properly, resulting in errors like this and this. Since this happens in type engine code between tasks, even an intermediate task to filter out none values would fail. This blocks us from using
    map_task
    effectively in our workflows. Here is a min repro example: https://gist.github.com/sabrinalui/e5478b9557dcf370c84d5e57758b4c87 It seems
    min_success_ratio
    doesn't work locally but I attached the output when running with inputs
    [true,true,false]
    from our endpoint below. What would the lift be to (optionally) filter out failed tasks from the
    map_task
    output?
    k
    d
    • 3
    • 5
  • f

    Frank Shen

    03/23/2023, 10:32 PM
    Hi, airflow-provider-flyte allows airflow dag to include flyte workflow or task, and sensor when flyte task is completed. What I need is the opposite: I need a flyte task to sensor when an airflow dag run is completed and then trigger the flyte downstream workflow or task. Is there any such airflow-sensor in flyte? If not, could anyone suggest any approach to achieve that? Thanks!
    k
    • 2
    • 2
  • r

    Rahul Mehta

    03/23/2023, 10:50 PM
    Is there a straightforward way to either override or set a template for the pod name for dynamic steps? It'd be very helpful to minimally be able to name the pods associated w/ a dynamic step with a common prefix for the workflow (rather than a random set of characters)
    j
    r
    • 3
    • 7
  • y

    Younes El Hjouji

    03/24/2023, 11:57 AM
    Hi Flyte friends, I am using pyflyte and flytectl to package and push workflows to my cluster. I can see the pushed workflows through flyte console When I attempt to launch any workflow, the first task fails after about 40 seconds with the error in the picture.
    b
    k
    • 3
    • 9
  • b

    Bosco Raju

    03/24/2023, 4:14 PM
    Hi folks, I am running into an error when running a simple hello_world workflow. I am using a custom docker image. Any idea why?
    FROM python:3.8-buster
    
    WORKDIR /root
    ENV VENV /opt/venv
    ENV LANG C.UTF-8
    ENV LC_ALL C.UTF-8
    ENV PYTHONPATH /root
    
    ARG tag
    ARG wandb_api_key
    ARG wandb_username
    
    ENV FLYTE_INTERNAL_IMAGE $tag
    ENV WANDB_API_KEY $wandb_api_key
    ENV WANDB_USERNAME $wandb_username
    
    # Install the AWS cli separately to prevent issues with boto being written over
    RUN pip3 install awscli
    
    RUN apt-get update && apt-get install -y curl
    
    ENV VENV /opt/venv
    # Virtual environment
    RUN python3 -m venv ${VENV}
    ENV PATH="${VENV}/bin:$PATH"
    
    # Install Python dependencies
    COPY requirements.txt /root/.
    RUN pip install -r /root/requirements.txt
    
    # Copy the actual code
    COPY src/ /root/src/
    
    [1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
    [ab8dm4nttv49wgvn59kg-n0-0] terminated with exit code (1). Reason [Error]. Message: 
    rtlib._bootstrap>", line 1014, in _gcd_import
      File "<frozen importlib._bootstrap>", line 991, in _find_and_load
      File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
    ModuleNotFoundError: No module named 'site-packages.flytekit'
    Traceback (most recent call last):
      File "/opt/venv/bin/pyflyte-fast-execute", line 8, in <module>
        sys.exit(fast_execute_task_cmd())
      File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
        return self.main(*args, **kwargs)
      File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 513, in fast_execute_task_cmd
        subprocess.run(cmd, check=True)
      File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
        raise CalledProcessError(retcode, process.args,
    subprocess.CalledProcessError: Command '['pyflyte-execute', '--inputs', '<s3://senn-ai-mlops-flyte/metadata/propeller/flytesnacks-development-ab8dm4nttv49wgvn59kg/n0/data/inputs.pb>', '--output-prefix', '<s3://senn-ai-mlops-flyte/metadata/propeller/flytesnacks-development-ab8dm4nttv49wgvn59kg/n0/data/0>', '--raw-output-data-prefix', '<s3://senn-ai-mlops-flyte/data/sk/ab8dm4nttv49wgvn59kg-n0-0>', '--checkpoint-path', '<s3://senn-ai-mlops-flyte/data/sk/ab8dm4nttv49wgvn59kg-n0-0/_flytecheckpoints>', '--prev-checkpoint', '""', '--dynamic-addl-distro', '<s3://senn-ai-mlops-flyte/mo/flytesnacks/development/NRI2T5OSZMCFXKR4CUNLWO7MSM======/fastab502ef8d4ae75b6b5497a94633e8642.tar.gz>', '--dynamic-dest-dir', '/root', '--resolver', 'site-packages.flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'src.workflows.hello_world', 'task-name', 'say_hello']' returned non-zero exit status 1.
    d
    n
    s
    • 4
    • 21
  • m

    MUhwjR9VF3K2QJhL2VvNGoFY

    03/24/2023, 8:58 PM
    Is this a known issue? E: Fixed
    $ docker pull <http://cr.flyte.org/flyteorg/flytescheduler-release:v1.4.3|cr.flyte.org/flyteorg/flytescheduler-release:v1.4.3>
    5e5f900059c8: Download complete
    0f96f76f982d: Download complete
    88dae1745799: Downloading [>                                                  ]       0B/2.343kB
    failed to copy: httpReadSeeker: failed open: failed to do request: Get "<https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:88dae1745799f4078f859b8ada55bc1f1757d10d9c97cdeb62f88cfc827c3a1f?se=2023-03-24T21%3A05%3A00Z&sig=rMhv3oYchaYycF260vI0pdx6pENdYsKpph9eXak%2BMPw%3D&sp=r&spr=https&sr=b&sv=2019-12-12>": x509: certificate is valid for *.<http://githubassets.com|githubassets.com>, <http://githubassets.com|githubassets.com>, not <http://pkg-containers.githubusercontent.com|pkg-containers.githubusercontent.com>
    r
    • 2
    • 6
  • s

    seunggs

    03/24/2023, 11:36 PM
    Hi, we can specify the number of gpus in flyte tasks, but how do we specify which type of gpu to use?
    k
    k
    • 3
    • 5
  • t

    Taeef Najib

    03/25/2023, 3:37 AM
    Hi, I was trying to use a local
    csv
    file. But I got this error:
    Message:
    
        [Errno 2] No such file or directory: 'data.csv'
    
    User error.
    What would be the workaround?
    k
    k
    • 3
    • 7
  • m

    mykyta luzan

    03/27/2023, 8:17 AM
    Hi, guys! Could you help with s3 credentials. I can run hello_world example in sandbox env, but I am failing to do so on our local k8s cluster.
    raise FlyteAssertion(
    flytekit.exceptions.user.FlyteAssertion: Failed to get data from <s3://blabs-flyte-mgmt-metadata/dispute-resolution/development/BP7IZUPWYESUYFGRO52LE7HBDU======/scriptmode.tar.gz> to /root/ (recursive=False).
    
    Original exception: Access Denied
    We don’t have any auth yet for flyte. I assume I just haven’t specified s3 bucket endpoint, access_key_id, and secret_access_key. Where should I do this? In my
    .flyte/config.yaml
    ? Or on
    k8s/values.yaml
    side via devops support? If so what is the template?
    storage:
      access-key: <SECRET>
      auth-type: iam
      disable-ssl: false
      endpoint: <s3://blabs-flyte-mgmt-metadata>
      region: eu-central-1
      secret-key: <SECRET>
    I don’t understand how to use this page https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.configuration.S3Config.html
    y
    d
    • 3
    • 11
  • s

    Srinivas Venkattaramanujam

    03/27/2023, 4:16 PM
    Hi! I'm evaluating Flyte for our workflow orchestration needs. I'm curious what will happen when WorkflowExecutor tries to update the task status to FlyteAdmin but FlyteAdmin is unavailable for a prolonged duration (say 1 hour) for some reason. Will the WorkflowExecutor eventually update the correct task status to the FlyteAdmin when it comes back up? Or will the FlyteAdmin have an inconsistent state (say Running when the task has actually failed/succeeded when the FlyteAdmin was down)
    k
    • 2
    • 4
  • j

    justin hallquist

    03/27/2023, 4:19 PM
    what constitutes as a valid execution name? I am having a hard time finding the validator both in the docs and code
    Error: rpc error: code = InvalidArgument desc = invalid name format: 370eb303160612
    k
    k
    • 3
    • 6
  • d

    David Espejo (he/him)

    03/27/2023, 5:35 PM
    x-posting from #flyte-deployment: https://flyte-org.slack.com/archives/C01P3B761A6/p1679938476764839
  • v

    varsha Parthasarathy

    03/27/2023, 8:09 PM
    Hi team, What is the suggestions/do’s & don’t for sharing flyte tasks across multiple workflows? WorkflowA has a taskA, registered with SHA1 WorkflowB uses taskA and has the same SHA1, when we register both workflows, get the error that task_id already exists with SHA1. How can we get around this issue?
    s
    • 2
    • 1
  • f

    Frank Shen

    03/27/2023, 8:32 PM
    Hello, Do you know if there is a flytekit plugins for airflow? I need to detect if an Airflow dag run is completed from Flyte. Thanks.
    n
    e
    k
    • 4
    • 9
  • e

    Ena Škopelja

    03/28/2023, 12:05 PM
    Hi all, I have a task returning a
    StructuredDataset
    that's failing if I turn on
    cache_serialize
    with this error:
    [3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):
    
          File "/opt/venv/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
            return wrapped(*args, **kwargs)
          File "/opt/venv/lib/python3.9/site-packages/flytekit/core/base_task.py", line 572, in dispatch_execute
            raise TypeError(
        Failed to convert return value for var o0 for function {my task} with error <class 'pyarrow.lib.ArrowInvalid'>: ("Could not convert ('... ... (580 characters truncated) ... ...',) with type Row: did not recognize Python value type when inferring an Arrow data type", 'Conversion failed for column sequence with type object')
    The sequence column of the dataframe I return is a string (no non-standard characters). Any idea on what could be causing this? Regular (no
    cache_serialize
    ) cache works just fine.
    k
    • 2
    • 7
  • p

    Prashant Jain

    03/28/2023, 12:18 PM
    @Prashant Jain has left the channel
  • a

    Andrew Korzhuev

    03/28/2023, 2:16 PM
    flyteidl 1.3.12 depends on protobuf<5.0.0 and >=4.21.1
        sagemaker 2.133.0 depends on protobuf<4.0 and >=3.1
    Since 1.3 FlyteKit has version conflict with Sagemaker API and Tensorflow as well by deprecating protobuf 3 making it impossible to use those libraries together
    f
    k
    +2
    • 5
    • 19
  • h

    HIMANSHU JOSHI

    03/28/2023, 5:03 PM
    Hey, does anybody know how can i pass multiple values to task_config param in @task decorator i.e. i want to pass Pod configuration as well as Spark configuration?
    d
    • 2
    • 4
  • s

    Samuel Bentley

    03/28/2023, 7:24 PM
    Hi everyone! I'm trying to run the workflow remotely as described in the Flyte Fundamentals docs, but I get an OpenSSL version error:
    E0328 20:15:47.445709000 140704578418304 <http://ssl_transport_security.cc:1495]|ssl_transport_security.cc:1495]> Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
    I'm using Python 3.9, grpcio & grpc-status 1.48.2, pyOpenSSL 23.0.0 and against a local flyte-sandbox (latest version) Can anyone help? I'm trying to get this working as a proof-of-concept for wider use in my compnay and need to demonstrate that this works
    n
    d
    • 3
    • 16
  • c

    Craig Amundsen

    03/28/2023, 10:38 PM
    Hello - I've been looking through the docs and searching here, and haven't figured out how, from flytekit, to query a flyte set up to get the execution status of a workflow that was started via FlyteRemote.emote.execute. Essentially, I want to be able to display in the system that kicked off the workflow if it's Pending, Running, Succeeded, etc. It seems like it should be possible. If someone could point me in the right direction, I'd be most appreciative.
    g
    • 2
    • 3
  • g

    Greg Gydush

    03/29/2023, 12:01 AM
    Hey all! With GCP, the logs before user code is executed are all coming up as severity=ERROR. I'd love to try to fix this at some point and was wondering if anyone has thoughts! Ideally this could be configurable too - I was talking to @Yee about this earlier and he suggested posting here for visibility! I also made a GitHub issue
    y
    • 2
    • 22
  • h

    HIMANSHU JOSHI

    03/29/2023, 12:00 PM
    Hey I've installed flyte on GKE it was running fine till now but i wanted to increase ResourceQuota for development/ staging and production namespace so i changed the config for the same but suddenly flyte-admin pod sync-cluster-resource container is failing with this error.
    panic: Failed to sync cluster resources : no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "Role" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>", no matches for kind "RoleBinding" in version "<http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>"
    
    goroutine 1 [running]:
    main.main()
    /go/src/github.com/flyteorg/flyteadmin/cmd/main.go:13 +0x91
    k
    d
    k
    • 4
    • 14
  • v

    Victor Gustavo da Silva Oliveira

    03/29/2023, 12:15 PM
    Hello all I'm having an issue when trying to use reference tasks... When building image, I'm getting this error:
    'ReferenceTask' object has no attribute '_task_function'
    ... Can someone help me? I would be very appreciated
    k
    s
    • 3
    • 5
  • a

    Agath Emmanuel

    03/29/2023, 1:18 PM
    Hello all, I could find here that Authorization can be set up for Flyte. But in the doc in production grade Flyte deployment, its mentioned that "Authorization is not supported out-of-the-box in Flyte". Is this just for UI or am I missing something here. So does Flyte have Role based access control and Authorization support or can it be achieved, thinking in terms of production requirements?
    k
    • 2
    • 5
  • s

    Srinivas Venkattaramanujam

    03/29/2023, 4:28 PM
    hey! I'm back with another question 😅 Can multiple instances of FlyteAdmin be run simultaneously without issues (such as duplicated scheduled runs)? I found the following about the data plane but nothing about FlyteAdmin scalability_._ While FlyteAdmin serves the Workflow Execution API, it does not itself execute workflows. To launch workflow executions, FlyteAdmin sends the workflow DAG to the DataPlane. For added scalability and fault-tolerance, FlyteAdmin can be configured to load-balance workflows across multiple isolated data-plane clusters.
    y
    • 2
    • 1
Powered by Linen
Title
s

Srinivas Venkattaramanujam

03/29/2023, 4:28 PM
hey! I'm back with another question 😅 Can multiple instances of FlyteAdmin be run simultaneously without issues (such as duplicated scheduled runs)? I found the following about the data plane but nothing about FlyteAdmin scalability_._ While FlyteAdmin serves the Workflow Execution API, it does not itself execute workflows. To launch workflow executions, FlyteAdmin sends the workflow DAG to the DataPlane. For added scalability and fault-tolerance, FlyteAdmin can be configured to load-balance workflows across multiple isolated data-plane clusters.
y

Yee

03/29/2023, 8:16 PM
yes we’ve run asgs before for flyte admin. scheduler semantics are idempotent and stored as an execution in the database so no need to worry about that.
View count: 1