Hello, I'm going to deploy my workflow on the local container. But the workflow fails with this message:

[1/1] currentAttempt done. Last Error: USER::containers with unready status: [fba452891990446a7a49-n0-0]|Back-off pulling image "cosine:2ce90d3ad559b4e5b1981af8726f4d19eeedc835"

Is there any way I can use to debug what's happened?

Hi, When deploying with https://docs.flyte.org/en/latest/deployment/aws/opta.html#deployment-aws-opta, what's the canonical way for setting up an allowlist (list of permitted IPs/networks) for the internet-facing service, so the service isn't open for anyone on the internet?

Good morning! We’re trying out Dynamic tasks for distributed computing/forecasting and notice it takes over 8 minutes for Flyte to kick off the first task. Is this normal? (@Kelsey Messer FYI)

Hey there, I’m a beginner using Flyte and I’m struggling to deploy it and to connect to my cluster with flytectl. Right now, I’ll be good not using https. I’m deploying on an EKS cluster using nginx. My config is basic with and I tried different options. see Below:

ingress:
    separateGrpcIngress: true
    annotations:
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx

It create 2 ingresses in my cluster and I can connect to the console in HTTP. I can also connect to admin with a port-forward… But I can’t connect to admin using flytectl using the same url as for the console. I really use:

flytectl config init --host='xxxxx' --insecure

But I got:

Error: rpc error: code = Unavailable desc = connection closed

I’m sure it’s something stupid but I can’t figure out what

Hi Everyone! A bit of a vague question but I figured I'd ask: Is there a way to investigate why an execution takes so much longer than the same code running in a method on my machine?

Hi! I want to make creating Project with a certain default IAM role with less friction. My understanding is like this(excluding Domain for simplicity): 1-

cluster_resource_manager.templates

helm value contains all the default k8s resources for a namespace(including namespace itself and ServiceAccount which we use to specify an IAM role) 2- We can specify a default IAM role for the ServiceAccounts. 3- User creates a project using

flytectl create project

4- flyteadmin creates namespace, serviceaccount and all other resoruces under

cluster_resource_manaager.templates

helm value. Our projects need different default IAM roles and we want our users to not have to go to the created namespace and create a ServiceAccount then specify it for each execution. So, is there any way to create projects with different default IAM roles/ServiceAccounts?

I can't seem to get this example in the User's Guide working: Flyte Basics: Tasks

Hi everyone! I am new to Flyte. I am curious what tools or libraries everyone use in Flyte to evaluate their models. We have custom code that generates plots such as ROC to evaluate classification models. Is it easy to display plots generated from custom code in a task and display them on Flyte console? cc @Habiba Mohamed

Hey folks... I am new-ish to Flyte... in the sense I've built workflows before, but this is the first time I'm looking to do a Flyte based PoC at a new workplace. So I have some fairly basic questions: 1. Should I set up a new repo? Or can it be in a folder in an existing repo? What boilerplate code does it need? 2. For the PoC I'll set up a K8s cluster on-prem. Once I install MinIO and postgres on it, how should I be configuring everything?

Hi folks, I am trying to launch the flytesnack task, which called

_bash.blastx

, but it displays below logs. Can somebody let me know what's going on here? It's status stays

unknown

forever.

c":"handlers.go:237"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2022-05-30T16:01:59+09:00"}
{"json":{"src":"handlers.go:193"},"level":"debug","msg":"gRPC server info in logging interceptor [1a329842-ff6f-4268-8349-edd25113438c]method [/flyteidl.service.AdminService/CreateExecution]\n","ts":"2022-05-30T16:01:59+09:00"}
{"json":{"src":"execution_manager.go:808"},"level":"debug","msg":"Launching single task execution with [resource_type:TASK project:\"flytesnacks\" domain:\"development\" name:\"_bash.blastx\" version:\"v0.3.82\" ]","ts":"2022-05-30T16:01:59+09:00"}
{"json":{"exec_id":"a4jj4dgl6lk24nbn75vt","src":"execution_manager.go:381"},"level":"warning","msg":"Failed to fetch override values when assigning task resource default values for [resource_type:WORKFLOW project:\"flytesnacks\" domain:\"development\" name:\".flytegen._bash.blastx\" version:\"v0.3.82\" ]: Resource [{Project:flytesnacks Domain:development Workflow:.flytegen._bash.blastx LaunchPlan: ResourceType:TASK_RESOURCE}] not found","ts":"2022-05-30T16:01:59+09:00"}
{"json":{"exec_id":"a4jj4dgl6lk24nbn75vt","src":"execution_manager.go:385"},"level":"debug","msg":"Assigning task requested resources for [resource_type:WORKFLOW project:\"flytesnacks\" domain:\"development\" name:\".flytegen._bash.blastx\" version:\"v0.3.82\" ]","ts":"2022-05-30T16:01:59+09:00"}
{"json":{"src":"queues.go:43"},"level":"debug","msg":"refreshing execution queues","ts":"2022-05-30T16:02:33+09:00"}
{"json":{"exec_id":"a4jj4dgl6lk24nbn75vt","src":"queues.go:73"},"level":"warning","msg":"Failed to fetch override values when assigning execution queue for [{ResourceType:WORKFLOW Project:flytesnacks Domain:development Name:.flytegen._bash.blastx Version:v0.3.82 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] with err: Resource [{Project:flytesnacks Domain:development Workflow:.flytegen._bash.blastx LaunchPlan: ResourceType:EXECUTION_QUEUE}] not found","ts":"2022-05-30T16:02:33+09:00"}
{"json":{"exec_id":"a4jj4dgl6lk24nbn75vt","src":"queues.go:109"},"level":"info","msg":"found no matching queue for [{ResourceType:WORKFLOW Project:flytesnacks Domain:development Name:.flytegen._bash.blastx Version:v0.3.82 XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}]","ts":"2022-05-30T16:02:33+09:00"}
{"json":{"exec_id":"a4jj4dgl6lk24nbn75vt","src":"execution_manager.go:529"},"level":"info","msg":"getting the workflow execution config from application configuration","ts":"2022-05-30T16:02:33+09:00"}

Thanks in advance. P.S. I managed to launch the flyte on

on premise k8s cluster

using keycloak as auth backend, if it is relevant to anything.

Hey folks.. I'm working on a PoC and using the demo setup. Is there an easy way to add a python package into the image?

Hi. I'm getting OOMKilled when running the diabetes demo on Kubernetes. Describing the pod I see

Limits:
      cpu:     1
      memory:  200Mi
    Requests:
      cpu:     1
      memory:  200Mi

How can I increase this? My helm yaml already has

task_resource_defaults:
    task_resources:
      defaults:
        cpu: 1000m
        memory: 1000Mi
        storage: 1000Mi
      limits:
        storage: 2000Mi

Hi all, if I have to chain 3 flyte tasks can I represent them as a >> b >> c or does it need to be a >> b, b >> c ?

I have some colleagues that are running the flyte demo step by step in https://docs.flyte.org/en/latest/getting_started/index.html, but they're running into an issue

Back-off pulling image "<http://ghcr.io/flyteorg/flytekit:py3.9-1.0.3|ghcr.io/flyteorg/flytekit:py3.9-1.0.3>

as it seems that image tag doesn't exist. Is anyone else having this issue with running that getting started page?

Hi folks.. I was trying to run a simple workflow on a remote cluster by following the instructions here but I'm running into an error

flytectl config init --host={SOME_HOSTNAME}:30080 --storage
Error: unknown flag: --storage
ERRO[0000] unknown flag: --storage

I've spun up a Sandbox cluster (using

flytectl sandbox start

) on one machine, and I'm trying to run an example wf from different machine (I've updated

endpoint

in

~/.flyte/config.yaml

).

pyflyte run --remote example.py:wf --n 500 --mean 42 --sigma 21

succeds and I get a link to the console. However when inspecting the execution I see this error:

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f31016626d79e40b2a4c-n0-0] terminated with exit code (1). Reason [Error]. Message: 
_execute(cmd=cmd, s3_cfg=self.s3_cfg)
  File "/usr/local/lib/python3.8/site-packages/flytekit/extras/persistence/s3_awscli.py", line 51, in _update_cmd_config_and_execute
    return subprocess.check_call(anonymous_cmd, env=env)
  File "/usr/local/lib/python3.8/site-packages/flytekit/tools/subprocess.py", line 26, in check_call
    raise Exception(
Exception: Called process exited with error code: 1.  Stderr dump:

b'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 495, in fast_execute_task_cmd
    _download_distribution(additional_distribution, dest_dir)
  File "/usr/local/lib/python3.8/site-packages/flytekit/tools/fast_registration.py", line 102, in download_distribution
    file_access.get_data(additional_distribution, destination)
  File "/usr/local/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 427, in get_data
    raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to get data from <s3://my-s3-bucket/36/flytesnacks/development/VGJSBXLCAE3JIKQYKY2O3N5IMQ======/scriptmode.tar.gz> to . (recursive=False).

Original exception: Called process exited with error code: 1.  Stderr dump:

b'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\n'
.

Hello there we have an issue and we were wondering if there was a way to go around it we got the following: • We have a namespace with a 3Go memory limit and default task memory request to 1Go • We run 3 workflows running a first task were we had

[1/1] currentAttempt done. Last Error: USER::task execution timeout [5m0s] expired

error. The reason is that the pod was trying to mount a secret volume that didn’t exist (there was a typo in the secret name) • The problem is that the deployment on K8s of the those tasks was still available after those 5 minutes, and that for hours, keeping the 3Go for themselves. Making the other tasks waiting. • Ultimately, after that deployments was “removed” the other tasks were picked up I would expect Flyte to terminate the deployments right after the first error no? Freeing the ressource usage for other task? Any clue?

Hey Everyone! I'm trying to create workflows around video processing, splitting up the data into chunks and processing frames with ML then coalescing the information back together. There are different stages of extracting information from the video/images in parallel. Previously, our system relied on using google cloud functions to split up processing then had a worker that would keep track of all processes, which was a bit verbose. Trying to simplify it with Flyte, but unsure where to get started. Any tips would be appreciated. Thanks!

Hi Everyone! I’m trying to follow the “Getting Started” document, and on the “Executing Workflows on a Flyte Cluster” step, when I’m running the

pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2"

command, I’m getting an error:

Traceback (most recent call last):
  File "/opt/anaconda3/envs/flytekit_test/bin/pyflyte", line 8, in <module>
    sys.exit(main())
  File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/flytekit/clis/sdk_in_container/run.py", line 473, in _run
    wf = remote.register_script(
  File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/flytekit/remote/remote.py", line 536, in register_script
    upload_location, md5_bytes = fast_register_single_script(
  File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/flytekit/tools/script_mode.py", line 117, in fast_register_single_script
    upload_location = create_upload_location_fn(content_md5=md5)
  File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/flytekit/clients/friendly.py", line 998, in get_upload_signed_url
    return super(SynchronousFlyteClient, self).create_upload_location(
  File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/flytekit/clients/raw.py", line 40, in handler
    return fn(*args, **kwargs)
  File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/flytekit/clients/raw.py", line 834, in create_upload_location
    return self._dataproxy_stub.CreateUploadLocation(create_upload_location_request, metadata=self._metadata)
  File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/opt/anaconda3/envs/flytekit_test/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1654765336.255436000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3218,"referenced_errors":[{"created":"@1654765336.255435000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":165,"grpc_status":14}]}"

Hi team.. while running the workflow in the local sandbox, is there a way to view the k8s logs? The k8s dashboard asks for a token or a config

Hi all, I'm getting an OOM error and concerned the limits I'm attempting to provide aren't being taken. When I kubectl describe the pod that failed due to OOMKilled - I see that it says the pod has a memory limit of 2G. However, in the flyte-deployment-config.yaml that I deployed using helm, I have allocated for a memory limit of 8G. See attached images for screenshots of the limit and the description. None of the @tasks have any resource limits imposed. Any help would be much appreciated!

Hi folks... I have a

@dynamic

task that calls a

@task

task that's expected to return a dict. Instead I'm getting a

flytekit.core.promise.Promise

object back. I also don't see any of the logs from that task.. nor do I see the called task executing on the console. What's weird is that the same called task works as expected when invoked from other tasks in the same workflow. Can someone help me understand why a task would not execute and return a Promise instead?

Hi everyone! I'm wondering if others have had issues getting Keycloak to work as the OAuth2 Authorization Server. I have the console working correctly and can login through our Keycloak instance to the web ui, but when I try to enable the configMap.adminServer.auth.appAuth and thirdPartyConfig in the Helm chart, the Flyteadmin pod enters a crashloop on startup trying to create the oauth2ResourceServer (hitting this error point I think) because it is getting a "404" response. I've been able to see in access logs that it is looking for /auth/realms/myrealm/.well-known/oauth-authorization-server on the Keycloak instance and that endpoint doesn't exist, thus the 404. As far as I know, Keycloak only supports the .well-known/openid-configuration and not oauth-authorization-service endpoint, but I could be wrong since others seem to have it working? This is deploying with v1.0.2 of the helm chart, which uses v1.1.16 of flyteadmin, and v16 Keycloak instance.

Doing a Flyte POC at HBO Max! Going to have some questions coming through in the next week or two when our instance is setup. I was curious if there was a way to dynamically set task resources within workflows. I have more information here https://github.com/flyteorg/flyte/discussions/2610. Thanks for the help!

I have a workflow that has a conditional task A that executes if some parameter is true. Then there is a task B that needs to execute after A. Is there a way to represent this dependency without defining a sub-workflow for A? Looks like I can't represent this as condition >> node (syntax error)

hello guys, I need to build a task that saves the data output from the processing in a table in BigQuery on Google cloud. can you help me? Do you have an operator ready or something?

Hi folks, I'm wondering if there's any configuration that allows me to launch a flyte cluster without kubernetes/docker ?

I'm trying to modify some code snippets of flytekit and test a workflow against the customized version, how could I test the remote execution without building a new docker image?

I have noticed that there seems not much documentation for developers/contributors to get started...

Hi guys, was trying to follow instructions from https://docs.flyte.org/en/latest/getting_started/index.html to run

pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2

but it was failing on my local environment https://flyte-org.slack.com/files/U037VET9014/F03KJK10Y10/untitled.txt Flytectl was up & running itself. @Yuvraj suggested running

flytectl config init

and it helped. I wonder, is this smth. that needs to be added on a getting_started page?