Hi there! I'm interested in the ability to run flyte workflows with priorities so that workflows with a higher priority are executed before those with a lower one when they are running at the same time. I found some information about this in the documentation but did not understand how exactly you can start the workflow by specifying its priority as a parameter. Another question is - how can resources be managed? Let's say I have a CPU-intensive workflow, and it was experimentally revealed that my cluster is capable of processing up to 10 such workflows at the same time, what will happen if we run 20 workflows?

Hi people! I have not found a lot of documentation about the Prometheus metrics made available by Flyte. Are there metrics about tasks such as individual task completion times, durations, etc? Much like we have for workflows with metrics like "flytepropellerallworkflowsuccess_duration_ms_count". These informations are on Flyte console so I wonder if they are also available by metrics or if at least there is a plan to implement that. Also, are there metrics about CPU and memory usage by workflow/task?

Does Flyte sets any environment variables on the pods it runs which we could use? We’re interested in

flytekit.current_context().execution_id.domain

in particular. It’s a bit of a chore that now our libraries have to include 300+mb of deps of

flytekit

to just get the runtime env.

How do you change imagePullPolicy for existing deployments? By default, its ifnotpresent, but I would like to change it to always for all of my potential docker images/containers. Do I have to use https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#alwayspullimages?

Also, where can I set a default node selector for all the workflow pods to execute on? I need to ensure every pod in flyte and the workflows it kicks off are assigned to a specific node. I assume somewhere in the flyte_propeller configMap

Hey folks, trying to debug an issue. We’re seeing various tasks fail with

terminated in the background, manually

. I took a look at the code here and see that this occurs when the pod get deleted. I don’t see a delete request being sent to admin so don’t think there is anything there. Propeller logs haven’t shown anything either. So I’ve been looking at kube logs to try to see what is triggering the pod deleted but there’s a lot of noise in the logs so taking some time to through them. Meanwhile, I figured I’d ask here if anyone has run into something similar before

Resolved I am running into this issue as well: https://discuss.flyte.org/t/5158547/Hello-I-was-exploring-on-Kubernetes-Spark-job-and-i-tried-to. I could use some help debugging next week 🙂

Failed to convert return value for var o0 for function src.workflow.t1 with error <class 'py4j.protocol.Py4JJavaError'>: An error occurred while calling o49.parquet.
: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"

I'm having some trouble configuring a new installation of flyte, but I think I'm close. I installed flyte with opta using the documentation here https://docs.flyte.org/en/latest/deployment/aws/opta.html#deployment-aws-opta I'm using an external-ssl-cert and am able to access the flyte console. My two problems: 1. I don't think I am using the correct endpoint for flytectl. I thought it should be the subdomain I access the console through but that didn't work. After trying a few things I was able to get it working by pointing flytectl directly to the flyteadmin service's load balancer on port 81. I used 

kubectl -n flyte get services flyteadmin

 to find it. Is this the correct way to do it? 2. I am having trouble configuring authentication with google cloud. Using https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#deployment-cluster-config-auth-setup I did the following + Setup my google cloud OAuth2 Client Credential + Ran 

kubectl edit secret -n flyte flyte-admin-secrets

 and added the client secret + Ran 

kubectl edit configmap -n flyte flyte-admin-config

 updated the config according to the docs + Restarted flyteadmin with 

kubectl rollout restart deployment/flyteadmin -n flyte

I didn't get everything wrong because when I visited the flyte console it redirected me to google to login before going to the dashboard. However when I tried to run a workflow the new execution just hung with status unknown. I also was unable to connect with flytectl no matter what I tried. I'm not sure what I'm doing wrong here. Any help is much appreciated.

Hey flyte community, been doing some autoscaling stuff with Karpenter and EKS with flyte and performance is really good and fast. Just one question, is there any sort of inbuilt logic within Flyte to allocate tasks/pods to certain provisioners/nodegroups automatically based on CPU/MEM request sizes (instead of using nodeselectors,taints), or is this custom logic we would have to build out ourselves? The reason I ask is because the end-user for our use-case would be a non-developer/non-programmer and so we would like to abstract away as much of the configuration from them as possible. Piping in the pod-templates and pod specs manually into the task defintions or using the with_overrides method works just fine, but unfortunately, even something like this may prove too much for the end-user. Obviously this is probably not the intended workflow for Flyte, but was wondering if anybody had any insight into this. Or if the custom logic is the answer, then that's fine too (I am assuming some sort of wrapper function to convert our users' request sizes into pod specs/taints/tolerations to pipe into the task invocations)

On to the next question. I am setting up Flyte to be used by many different teams. My idea was use the same service account for each of the projects i.e., use the https://docs.flyte.org/en/latest/deployment/aws/manual.html#flyte-user-role the pods are assuming and pass IAM roles in the launch plan that the service account assumes during execution, I remembered that there were some problems with this a while back. Is this working or is there a better way to achieve project specific permissions?