Hey all. I'm back with a quick question. I have a situation where a workflow fans out to bunch of nodes, 100+, at some point we're hitting the image pull qps limit and since it takes too long for the pod to start, flyte marks the node as failed and nukes the workflow. So my question is, is there a config I can set to have flyte wait longer for the pod to spin up before giving up?

been reading up on some of the new features and found out about this approve https://github.com/flyteorg/flytesnacks/blob/master/cookbook/core/control_flow/waiting_for_external_inputs.py#L150-L158. When I tried doing fast registration after copying pasting, it errors out with the following.

RPC Failed, with Status: StatusCode.INTERNAL
	details: failed to compile workflow for [resource_type:WORKFLOW project:"flytetester" domain:"development" name:"flyte.example.reporting_with_approval_wf" version:"unique-version" ] with err failed to compile workflow with err Collected Errors: 2
	Error 0: Code: ParameterNotBound, Node Id: end-node, Description: Parameter not bound [o0].
	Error 1: Code: VariableNameNotFound, Node Id: n3, Description: Variable [o0] not found on node [n3].

Why is GitHub not supported? And are there plans to support it in the future?

I am investigating a good way to manage two dependent workflows, developed by two separate teams. Imagine that the output of one task in workflow A is needed in another task in workflow B. - Let's say these workflows run on different scheduled launch plans, but B's task references A's task execution explicitly, such that B's task becomes dependent (downsteam) from A's task. Optimally this could even inform upstream workflow A about attached downstream tasks in B, allowing A developers to re-run the full effective pipeline, in cases of bugs or bad data in A's upstream. I understand that team 1 could build task 1 and team 2 could build task 2 - dependent on 1, in a common workflow, but the topology of our data flows would more or less require a mono-workflow structure, which seems to work against the features of launch plans and the workflow image/packaging/registration pattern.

Hello, Can Polars dataframe be used as flyte task input & output params out of the box? Since which version of flyte it’s supported. I am using flyte 1.2.4.

Another question about passing secrets to Flyte workflow executions. I have a service that kicks off Flyte workflows on behalf of many users. In the Flyte workflow, I would like to access my service and identify which user is making the request. I was looking at the secrets documentation and it's not clear if that's going to fit use cases where different secrets need to be passed in for different executions. E.g. if user A runs workflow A, and user B runs workflow B, we want each workflow to receive a different token. I see that I can probably pass a token as an input to the Flyte task, but then I am concerned that the input would be stored in Flyte or accidentally logged. Any recommendations?

Resolved I am running into this issue as well: https://discuss.flyte.org/t/5158547/Hello-I-was-exploring-on-Kubernetes-Spark-job-and-i-tried-to. I could use some help debugging next week 🙂

Failed to convert return value for var o0 for function src.workflow.t1 with error <class 'py4j.protocol.Py4JJavaError'>: An error occurred while calling o49.parquet.
: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"

Hello. Thanks to the Flyte team for their work on this framework. Real quick, using Flyte-Binary as a helm chart onto AWS EKS. Trying to set up OIDC and/or OAuth2 via Auth0 and have been messing around with some configs but have been unsuccessful getting the

/console

endpoint to query the browser for login with variations off of the following values.yaml config:

auth: 
      enabled: true
      enableAuthServer: false
      oidc:
        baseUrl: https://{DOMAIN}.<http://auth0.com/.well-known/openid-configuration|auth0.com/.well-known/openid-configuration>
        clientId: {CLIENT ID}
        clientSecret: {CLIENT SECRET}
      authorizedUris:
        - https://{LOAD BALANCED ADDRESS}.com
        - https://{LOAD BALANCED ADDRESS}.com

Bit unsure how to transpose the documentation from the official site into

flyte-binary

as the documentation seems to only go over

flyte-core

or

flyte

. The documentation doesn't seem to really exist for flyte-binary and there's scant discussion surrounding the topic unfortunately. I understand on some level that there are three components to the Flyte deployment but how does one access those configurations through the Helm chart within the

flyte-binary

deployment? Specifically, how are you supposed to affect the various OIDC/auth config values when the templates for binary don't seem to match the documentation? Also, we wish to use Auth0 as our external authorization server and don't need the internal server, but there have been conflicting resources suggesting that internal still needs to be defined. Any help in the matter would be most appreciated.

Hi there! I was reading this feature request about decorating tasks in python OOP classes. @Ketan (kumare3) you mentioned

ClassTaskResolver

, but the example is not super clear on how I should use the resolver. Can you elaborate more on how to use it? Thanks!

Is there a quick way of determining the

LiteralType

of a

Literal

?

hey, I’m running into issues, could you help me clarify the reasons and approach to fix them? I’m running in AWS on r5d.2xlarge machine (Ubuntu 20.04.3 LTS) I’ve started with examples from intro, so: 1. when I launch

pyflyte run …

I receive KeyRingError for my user, had to switch to root (any way to fix it?) 2. launching

--remote

ends up with OOM killed get_data container and warnings from MatPlotLib about non-writeable directory, had to set limits with memory=500mi not sure what am i doing wrong 🙂

One question about the scalability of Flyte, what’s the recommended max number of concurrent tasks running in a cluster? and how to set it (or do I need to set it)? I’m trying to run this distributed ML system on flyte. It’s a single-task workflow, It needs to run at least 500 workflows or tasks (each task needs 30G RAM, 4 CPU, 1GPU) in parallel in addition to other misc workflows. Just wanted to make sure flyte can handle that scalability (I believe it can)?

do you have any examples of class-based tasks/workflows instead of functions? or any examples of huge configs (like 100 options) with overload? usually we do it with hydra, but I wonder if there are any other options another question: how is the data passed between tasks, if I have huge dataset to supply for model? (in luigi we usually pass filepaths even if the file is in s3, any best practices for flyte?)

set up a reminder “🔴 Community meeting is starting now - Info: https://www.addevent.com/event/EA7823958” in this channel at 9AM every other Tuesday (next occurrence is today), Pacific Daylight Time.

I keep getting new errors from examples:

ValueError: Cannot launch an execution with missing project/domain/name ResolvedIdentifiers(project=None, domain=None, name='workflows.example_wf', version=None) for entity type <class 'flytekit.core.workflow.PythonFunctionWorkflow'>. Specify them in the execute method or when intializing FlyteRemote

Hello guys We are trying to use a map task in a workflow, but no matter how we increase the pod memory, it keeps giving OOMKilled with error 137... By looking at the pods creation, in a successfull execution (with less inputs), we can verify that the resources setted on the task are not the same that are requested in GKE... The image shows the pods resources, when we setted the task with 6Gi memory and 2CPU...

Does anyone have any experience with

flytekit.current_context().stats

I cannot find docs online about what the base stats are so that I can call

taggable_stats = flytekit.current_context().stats
stat = taggable_stats.get_stats(param)

and start to extract some information to log on each task run

Reminder: 🔴 Community meeting is starting now - Info: https://www.addevent.com/event/EA7823958

Running into issues with flytepropeller and flytescheduler when trying to setup okta authentication using OpenId https://docs.flyte.org/en/latest/deployment/configuration/auth_setup.html#openid-connect flytepropeller complaining

{"json":{"exec_id":"avjrcs5qwjzgnwz7cxfs","ns":"flytetester-development","res_ver":"76272166","routine":"worker-1","wf":"flytetester:development:<http://turbine_flyte.example.wf|turbine_flyte.example.wf>"},"level":"warning","msg":"Event recording failed. Error [EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}]]","ts":"2023-05-02T18:07:36Z"}

saw this thread https://flyte-org.slack.com/archives/CP2HDHKE1/p1676910082719799 but the PR is closed

Is there a way to override the task pod labels from task level (like, using

with_overrides()

), instead of the execution level? I’d like to attach different labels to different tasks in the same workflow for cost tracking purpose

I have a question related to automatic downloads of FlyteFiles/FlyteFolders. I have a large amount of data (~1TB) spread across ~15,000 files stored in GCS. I have a Flyte workflow that, as a first step, downloads these files to be operated on locally. I noticed that the download speed is around 20 MB/s and uses about 1-2 CPU cores. I did some testing, and when I manually invoke

gcloud storage cp -r …

inside of a Flyte workflow, I am able to download the same exact files at approximately 1.7GB/s, and I see usage of ~20 CPU cores (the workflow requests 32 cpu cores). I observe the same behavior when uploading files to GCS as well. How can I still make use of FlyteFiles/FlyteFolders and the automatic download/upload functionality, but get faster download/upload speed?

Hello. New user, just getting started. Are there any guides on how to integrate a jupyter notebook server into Flyte? Like jupyterhub or the Kubeflow notebook operator?

Does Flyte have a multi tenancy isolation or run all workflows in a single namespace?

This message was deleted.

I've deployed "Flyte the hard way" in my dev AWS account, which is all working now. I've deployed to prod and I'm trying to push a wf. When I do port forwarding and open the console I get the attached grpc error When I run a wf, the command seems happy

FLYTE_CONFIG=~/.flyte/config.yaml pyflyte run --remote workflows/greeting.py wf --name Sam

Go to <http://localhost:8089/console/projects/flytesnacks/domains/development/executions/fd724a646bd6342cab6c> to see execution in the console.

And when I go to the console, it again shows a grpc error. Any idea what I'm doing wrong. Looks like some connection issue

is there an easy way to mount some dir from execution space to container running task, or the only option is to fully build image, publish and then pull into k8s? I want fast iterations 🙂

I'm looking to interact with the flyte REST API and admin database to be able to query for execution input / output data. I have 2 questions: 1. How to query input / output data from the database? I can find references to proto input files, but can't find any reference to where the outputs exist. I also wonder if there is an example of decoding the downloaded proto files. 2. How to authenticate the REST API? I found the API swagger docs, but I neither see examples of using the API or how to authenticate. I can generate JWT's easily enough against our IdP, but passing that as a standard Authorization header doesn't appear sufficient.

what's the best way to query bigquery from aws? it seems the plugin only supports querying from GCP

Is there any way to run a workflow within another workflow such that both can be viewed under the same

execution_id

in the Flyte Console and such that the pods for pod workflows inherit this execution id? or is it best practice to not have nested workflows like this in flyte?

#ask-the-community Hello All, I am trying to install custom packages into the flyte container. In my build job prior to flyte package I am installing numpy which is 1.24.3. But after packaging when i trigger the workflow the version which is shown to me in workflow output is 1.20.0 Another example is I am installing transformers python package but in the workflow the same says module not found. I saw that in kubeflow I can install custom packages using

add_op = create_component_from_func(

with package_to_install option. Is there a Flyte way of doing this