https://flyte.org logo
Join the conversationJoin Slack
Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
sig-large-models
workflow-building-ui-proj
writing-w-sfloris
Powered by Linen
ask-the-community
  • h

    HaoboGu

    05/21/2022, 9:47 AM
    Hello, I'm going to deploy my workflow on the local container. But the workflow fails with this message:
    [1/1] currentAttempt done. Last Error: USER::containers with unready status: [fba452891990446a7a49-n0-0]|Back-off pulling image "cosine:2ce90d3ad559b4e5b1981af8726f4d19eeedc835"
    Is there any way I can use to debug what's happened?
    đź‘€ 1
    k
    • 2
    • 18
  • a

    Attila Nagy

    05/23/2022, 10:00 PM
    Hi, When deploying with https://docs.flyte.org/en/latest/deployment/aws/opta.html#deployment-aws-opta, what's the canonical way for setting up an allowlist (list of permitted IPs/networks) for the internet-facing service, so the service isn't open for anyone on the internet?
    k
    j
    • 3
    • 6
  • s

    Stefan Avesand

    05/24/2022, 1:17 PM
    Good morning! We’re trying out Dynamic tasks for distributed computing/forecasting and notice it takes over 8 minutes for Flyte to kick off the first task. Is this normal? (@Kelsey Messer FYI)
    k
    d
    e
    • 4
    • 22
  • j

    Jonathan Lamiel

    05/24/2022, 9:27 PM
    Hey there, I’m a beginner using Flyte and I’m struggling to deploy it and to connect to my cluster with flytectl. Right now, I’ll be good not using https. I’m deploying on an EKS cluster using nginx. My config is basic with and I tried different options. see Below:
    ingress:
        separateGrpcIngress: true
        annotations:
          <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    It create 2 ingresses in my cluster and I can connect to the console in HTTP. I can also connect to admin with a port-forward… But I can’t connect to admin using flytectl using the same url as for the console. I really use:
    flytectl config init --host='xxxxx' --insecure
    But I got:
    Error: rpc error: code = Unavailable desc = connection closed
    I’m sure it’s something stupid but I can’t figure out what
    y
    k
    s
    • 4
    • 6
  • r

    Ruksana Kabealo

    05/25/2022, 2:09 AM
    Hi Everyone! A bit of a vague question but I figured I'd ask: Is there a way to investigate why an execution takes so much longer than the same code running in a method on my machine?
    k
    k
    • 3
    • 14
  • m

    MĂĽcahit

    05/25/2022, 2:28 PM
    Hi! I want to make creating Project with a certain default IAM role with less friction. My understanding is like this(excluding Domain for simplicity): 1-
    cluster_resource_manager.templates
    helm value contains all the default k8s resources for a namespace(including namespace itself and ServiceAccount which we use to specify an IAM role) 2- We can specify a default IAM role for the ServiceAccounts. 3- User creates a project using
    flytectl create project
    4- flyteadmin creates namespace, serviceaccount and all other resoruces under
    cluster_resource_manaager.templates
    helm value. Our projects need different default IAM roles and we want our users to not have to go to the created namespace and create a ServiceAccount then specify it for each execution. So, is there any way to create projects with different default IAM roles/ServiceAccounts?
    k
    p
    • 3
    • 8
  • s

    Stephen Fromm

    05/25/2022, 3:11 PM
    I can't seem to get this example in the User's Guide working: Flyte Basics: Tasks
    k
    p
    • 3
    • 43
  • s

    Shihgian Lee

    05/26/2022, 6:57 PM
    Hi everyone! I am new to Flyte. I am curious what tools or libraries everyone use in Flyte to evaluate their models. We have custom code that generates plots such as ROC to evaluate classification models. Is it easy to display plots generated from custom code in a task and display them on Flyte console? cc @Habiba Mohamed
    👍 2
    🙏 2
    k
    k
    • 3
    • 5
  • r

    Rupsha Chaudhuri

    05/27/2022, 8:32 PM
    Hey folks... I am new-ish to Flyte... in the sense I've built workflows before, but this is the first time I'm looking to do a Flyte based PoC at a new workplace. So I have some fairly basic questions: 1. Should I set up a new repo? Or can it be in a folder in an existing repo? What boilerplate code does it need? 2. For the PoC I'll set up a K8s cluster on-prem. Once I install MinIO and postgres on it, how should I be configuring everything?
    đź‘‹ 2
    ❤️ 4
    k
    v
    • 3
    • 13
  • j

    Jake Yoon

    05/30/2022, 7:07 AM
    Hi folks, I am trying to launch the flytesnack task, which called
    _bash.blastx
    , but it displays below logs. Can somebody let me know what's going on here? It's status stays
    unknown
    forever.
    c":"handlers.go:237"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-05-30T16:01:59+09:00"}
    {"json":{"src":"handlers.go:193"},"level":"debug","msg":"gRPC server info in logging interceptor [1a329842-ff6f-4268-8349-edd25113438c]method [/flyteidl.service.AdminService/CreateExecution]\n","ts":"2022-05-30T16:01:59+09:00"}
    {"json":{"src":"execution_manager.go:808"},"level":"debug","msg":"Launching single task execution with [resource_type:TASK project:\"flytesnacks\" domain:\"development\" name:\"_bash.blastx\" version:\"v0.3.82\" ]","ts":"2022-05-30T16:01:59+09:00"}
    {"json":{"exec_id":"a4jj4dgl6lk24nbn75vt","src":"execution_manager.go:381"},"level":"warning","msg":"Failed to fetch override values when assigning task resource default values for [resource_type:WORKFLOW project:\"flytesnacks\" domain:\"development\" name:\".flytegen._bash.blastx\" version:\"v0.3.82\" ]: Resource [{Project:flytesnacks Domain:development Workflow:.flytegen._bash.blastx LaunchPlan: ResourceType:TASK_RESOURCE}] not found","ts":"2022-05-30T16:01:59+09:00"}
    {"json":{"exec_id":"a4jj4dgl6lk24nbn75vt","src":"execution_manager.go:385"},"level":"debug","msg":"Assigning task requested resources for [resource_type:WORKFLOW project:\"flytesnacks\" domain:\"development\" name:\".flytegen._bash.blastx\" version:\"v0.3.82\" ]","ts":"2022-05-30T16:01:59+09:00"}
    {"json":{"src":"queues.go:43"},"level":"debug","msg":"refreshing execution queues","ts":"2022-05-30T16:02:33+09:00"}
    {"json":{"exec_id":"a4jj4dgl6lk24nbn75vt","src":"queues.go:73"},"level":"warning","msg":"Failed to fetch override values when assigning execution queue for [{ResourceType:WORKFLOW Project:flytesnacks Domain:development Name:.flytegen._bash.blastx Version:v0.3.82 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] with err: Resource [{Project:flytesnacks Domain:development Workflow:.flytegen._bash.blastx LaunchPlan: ResourceType:EXECUTION_QUEUE}] not found","ts":"2022-05-30T16:02:33+09:00"}
    {"json":{"exec_id":"a4jj4dgl6lk24nbn75vt","src":"queues.go:109"},"level":"info","msg":"found no matching queue for [{ResourceType:WORKFLOW Project:flytesnacks Domain:development Name:.flytegen._bash.blastx Version:v0.3.82 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]","ts":"2022-05-30T16:02:33+09:00"}
    {"json":{"exec_id":"a4jj4dgl6lk24nbn75vt","src":"execution_manager.go:529"},"level":"info","msg":"getting the workflow execution config from application configuration","ts":"2022-05-30T16:02:33+09:00"}
    Thanks in advance. P.S. I managed to launch the flyte on
    on premise k8s cluster
    using keycloak as auth backend, if it is relevant to anything.
    p
    • 2
    • 7
  • r

    Rupsha Chaudhuri

    06/01/2022, 5:01 PM
    Hey folks.. I'm working on a PoC and using the demo setup. Is there an easy way to add a python package into the image?
    k
    s
    • 3
    • 9
  • r

    Roberto Ruiz

    06/01/2022, 7:58 PM
    Hi. I'm getting OOMKilled when running the diabetes demo on Kubernetes. Describing the pod I see
    Limits:
          cpu:     1
          memory:  200Mi
        Requests:
          cpu:     1
          memory:  200Mi
    How can I increase this? My helm yaml already has
    task_resource_defaults:
        task_resources:
          defaults:
            cpu: 1000m
            memory: 1000Mi
            storage: 1000Mi
          limits:
            storage: 2000Mi
    k
    • 2
    • 7
  • r

    Rupsha Chaudhuri

    06/02/2022, 6:03 PM
    Hi all, if I have to chain 3 flyte tasks can I represent them as a >> b >> c or does it need to be a >> b, b >> c ?
    k
    • 2
    • 1
  • k

    Katrina P

    06/02/2022, 9:23 PM
    I have some colleagues that are running the flyte demo step by step in https://docs.flyte.org/en/latest/getting_started/index.html, but they're running into an issue
    Back-off pulling image "<http://ghcr.io/flyteorg/flytekit:py3.9-1.0.3|ghcr.io/flyteorg/flytekit:py3.9-1.0.3>
    as it seems that image tag doesn't exist. Is anyone else having this issue with running that getting started page?
    k
    e
    +2
    • 5
    • 18
  • r

    Rupsha Chaudhuri

    06/03/2022, 5:01 PM
    Hi folks.. I was trying to run a simple workflow on a remote cluster by following the instructions here but I'm running into an error
    flytectl config init --host={SOME_HOSTNAME}:30080 --storage
    Error: unknown flag: --storage
    ERRO[0000] unknown flag: --storage
    k
    • 2
    • 9
  • a

    Aleksei Potov

    06/03/2022, 10:35 PM
    I've spun up a Sandbox cluster (using
    flytectl sandbox start
    ) on one machine, and I'm trying to run an example wf from different machine (I've updated
    endpoint
    in
    ~/.flyte/config.yaml
    ).
    pyflyte run --remote example.py:wf --n 500 --mean 42 --sigma 21
    succeds and I get a link to the console. However when inspecting the execution I see this error:
    [1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
    [f31016626d79e40b2a4c-n0-0] terminated with exit code (1). Reason [Error]. Message: 
    _execute(cmd=cmd, s3_cfg=self.s3_cfg)
      File "/usr/local/lib/python3.8/site-packages/flytekit/extras/persistence/s3_awscli.py", line 51, in _update_cmd_config_and_execute
        return subprocess.check_call(anonymous_cmd, env=env)
      File "/usr/local/lib/python3.8/site-packages/flytekit/tools/subprocess.py", line 26, in check_call
        raise Exception(
    Exception: Called process exited with error code: 1.  Stderr dump:
    
    b'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\n'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
        sys.exit(fast_execute_task_cmd())
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 495, in fast_execute_task_cmd
        _download_distribution(additional_distribution, dest_dir)
      File "/usr/local/lib/python3.8/site-packages/flytekit/tools/fast_registration.py", line 102, in download_distribution
        file_access.get_data(additional_distribution, destination)
      File "/usr/local/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 427, in get_data
        raise FlyteAssertion(
    flytekit.exceptions.user.FlyteAssertion: Failed to get data from <s3://my-s3-bucket/36/flytesnacks/development/VGJSBXLCAE3JIKQYKY2O3N5IMQ======/scriptmode.tar.gz> to . (recursive=False).
    
    Original exception: Called process exited with error code: 1.  Stderr dump:
    
    b'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\n'
    .
    k
    • 2
    • 50
  • j

    Jonathan Lamiel

    06/05/2022, 2:42 PM
    Hello there we have an issue and we were wondering if there was a way to go around it we got the following: • We have a namespace with a 3Go memory limit and default task memory request to 1Go • We run 3 workflows running a first task were we had
    [1/1] currentAttempt done. Last Error: USER::task execution timeout [5m0s] expired
    error. The reason is that the pod was trying to mount a secret volume that didn’t exist (there was a typo in the secret name) • The problem is that the deployment on K8s of the those tasks was still available after those 5 minutes, and that for hours, keeping the 3Go for themselves. Making the other tasks waiting. • Ultimately, after that deployments was “removed” the other tasks were picked up I would expect Flyte to terminate the deployments right after the first error no? Freeing the ressource usage for other task? Any clue?
    k
    s
    d
    • 4
    • 15
  • a

    Abhinav Ayalur

    06/06/2022, 7:07 PM
    Hey Everyone! I'm trying to create workflows around video processing, splitting up the data into chunks and processing frames with ML then coalescing the information back together. There are different stages of extracting information from the video/images in parallel. Previously, our system relied on using google cloud functions to split up processing then had a worker that would keep track of all processes, which was a bit verbose. Trying to simplify it with Flyte, but unsure where to get started. Any tips would be appreciated. Thanks!
    k
    k
    • 3
    • 19
  • a

    Arshak Ulubabyan

    06/09/2022, 10:17 AM
    Hi Everyone! I’m trying to follow the “Getting Started” document, and on the “Executing Workflows on a Flyte Cluster” step, when I’m running the
    pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2"
    command, I’m getting an error:
    Traceback (most recent call last):
      File "/opt/anaconda3/envs/flytekit_test/bin/pyflyte", line 8, in <module>
        sys.exit(main())
      File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
        return self.main(*args, **kwargs)
      File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/click/core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/click/core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/flytekit/clis/sdk_in_container/run.py", line 473, in _run
        wf = remote.register_script(
      File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/flytekit/remote/remote.py", line 536, in register_script
        upload_location, md5_bytes = fast_register_single_script(
      File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/flytekit/tools/script_mode.py", line 117, in fast_register_single_script
        upload_location = create_upload_location_fn(content_md5=md5)
      File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
        return super(SynchronousFlyteClient, self).create_upload_location(
      File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/flytekit/clients/raw.py", line 40, in handler
        return fn(*args, **kwargs)
      File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/flytekit/clients/raw.py", line 834, in create_upload_location
        return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
      File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
        return _end_unary_response_blocking(state, call, False, None)
      File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
        raise _InactiveRpcError(state)
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
            status = StatusCode.UNAVAILABLE
            details = "failed to connect to all addresses"
            debug_error_string = "{"created":"@1654765336.255436000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3218,"referenced_errors":[{"created":"@1654765336.255435000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":165,"grpc_status":14}]}"
    k
    • 2
    • 4
  • r

    Rupsha Chaudhuri

    06/09/2022, 5:38 PM
    Hi team.. while running the workflow in the local sandbox, is there a way to view the k8s logs? The k8s dashboard asks for a token or a config
    k
    k
    e
    • 4
    • 36
  • r

    Ruksana Kabealo

    06/09/2022, 7:32 PM
    Hi all, I'm getting an OOM error and concerned the limits I'm attempting to provide aren't being taken. When I kubectl describe the pod that failed due to OOMKilled - I see that it says the pod has a memory limit of 2G. However, in the flyte-deployment-config.yaml that I deployed using helm, I have allocated for a memory limit of 8G. See attached images for screenshots of the limit and the description. None of the @tasks have any resource limits imposed. Any help would be much appreciated!
    n
    • 2
    • 2
  • r

    Rupsha Chaudhuri

    06/09/2022, 8:20 PM
    Hi folks... I have a
    @dynamic
    task that calls a
    @task
    task that's expected to return a dict. Instead I'm getting a
    flytekit.core.promise.Promise
    object back. I also don't see any of the logs from that task.. nor do I see the called task executing on the console. What's weird is that the same called task works as expected when invoked from other tasks in the same workflow. Can someone help me understand why a task would not execute and return a Promise instead?
    k
    • 2
    • 35
  • n

    Neal Feierabend

    06/09/2022, 10:28 PM
    Hi everyone! I'm wondering if others have had issues getting Keycloak to work as the OAuth2 Authorization Server. I have the console working correctly and can login through our Keycloak instance to the web ui, but when I try to enable the configMap.adminServer.auth.appAuth and thirdPartyConfig in the Helm chart, the Flyteadmin pod enters a crashloop on startup trying to create the oauth2ResourceServer (hitting this error point I think) because it is getting a "404" response. I've been able to see in access logs that it is looking for /auth/realms/myrealm/.well-known/oauth-authorization-server on the Keycloak instance and that endpoint doesn't exist, thus the 404. As far as I know, Keycloak only supports the .well-known/openid-configuration and not oauth-authorization-service endpoint, but I could be wrong since others seem to have it working? This is deploying with v1.0.2 of the helm chart, which uses v1.1.16 of flyteadmin, and v16 Keycloak instance.
    k
    p
    b
    • 4
    • 7
  • e

    Evan Sadler

    06/13/2022, 3:59 PM
    Doing a Flyte POC at HBO Max! Going to have some questions coming through in the next week or two when our instance is setup. I was curious if there was a way to dynamically set task resources within workflows. I have more information here https://github.com/flyteorg/flyte/discussions/2610. Thanks for the help!
    ❤️ 5
    k
    • 2
    • 17
  • r

    Rupsha Chaudhuri

    06/13/2022, 6:03 PM
    I have a workflow that has a conditional task A that executes if some parameter is true. Then there is a task B that needs to execute after A. Is there a way to represent this dependency without defining a sub-workflow for A? Looks like I can't represent this as condition >> node (syntax error)
    k
    k
    • 3
    • 16
  • s

    Sérgio de Melo Barreto Junior

    06/13/2022, 6:03 PM
    hello guys, I need to build a task that saves the data output from the processing in a table in BigQuery on Google cloud. can you help me? Do you have an operator ready or something?
    k
    a
    +3
    • 6
    • 15
  • t

    Tao He

    06/14/2022, 9:29 AM
    Hi folks, I'm wondering if there's any configuration that allows me to launch a flyte cluster without kubernetes/docker ?
    s
    • 2
    • 2
  • t

    Tao He

    06/14/2022, 9:31 AM
    I'm trying to modify some code snippets of flytekit and test a workflow against the customized version, how could I test the remote execution without building a new docker image?
    s
    • 2
    • 3
  • t

    Tao He

    06/14/2022, 9:31 AM
    I have noticed that there seems not much documentation for developers/contributors to get started...
    s
    • 2
    • 2
  • y

    Yaroslav

    06/14/2022, 10:26 AM
    Hi guys, was trying to follow instructions from https://docs.flyte.org/en/latest/getting_started/index.html to run
    pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2
    but it was failing on my local environment https://flyte-org.slack.com/files/U037VET9014/F03KJK10Y10/untitled.txt Flytectl was up & running itself. @Yuvraj suggested running
    flytectl config init
    and it helped. I wonder, is this smth. that needs to be added on a getting_started page?
    Untitled.txt
    y
    k
    e
    • 4
    • 7
Powered by Linen
Title
y

Yaroslav

06/14/2022, 10:26 AM
Hi guys, was trying to follow instructions from https://docs.flyte.org/en/latest/getting_started/index.html to run
pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2
but it was failing on my local environment https://flyte-org.slack.com/files/U037VET9014/F03KJK10Y10/untitled.txt Flytectl was up & running itself. @Yuvraj suggested running
flytectl config init
and it helped. I wonder, is this smth. that needs to be added on a getting_started page?
Untitled.txt
y

Yuvraj

06/14/2022, 10:27 AM
cc: @Eduardo Apolinario (eapolinario)
k

Ketan (kumare3)

06/14/2022, 1:51 PM
Welcome to the community @Yaroslav
đź’Ş 3
Thank you for posting the issue
y

Yaroslav

06/14/2022, 4:51 PM
also I keep getting wrong port in the URL after running
pyflyte run --remote ...
I get
Go to <http://localhost:30081/console/projects/flytesnacks/domains/development/executions/f0af4be9910484159b8d> to see execution in the console.
but my Flyte runs on
:30080
and URL with
:30081
is broken for me. Anyone knows what might be causing this?
e

Eduardo Apolinario (eapolinario)

06/14/2022, 10:54 PM
@Yaroslav,
flytectl demo start
should add a config file under
~/.flyte
. If that didn't happen then we'd have to investigate. For the time being I'm going to add a note to ensure we check that a config file is present before continuing the guide.
👍 1
I'm looking into the wrong port error now.
@Yaroslav, https://github.com/flyteorg/flytekit/pull/1064 should fix the issue. I'll ping here once we have a release that you can test, ok?
👍 1
View count: 5