Would there be any way to set up gang scheduling within a project/domain currently? I've seen something similar for Kubeflow's training operators using https://github.com/volcano-sh/volcano and was wondering if it'd be possible to achieve for Flyte workflows. For context, we'd like to create user-specific projects and enforce resource quotas on them in K8s; however, if we do this, we'd like to try to avoid multiple queued workflows contending for resources and let pods associated w/ a specific workflow get scheduled together. Another example of this would be https://github.com/palantir/k8s-spark-scheduler

I have an offline model pipeline that works 100% in spark: preprocessing, training LightGBM (synapse ml), and predictions. I am curious if this could fit into the UnionML framework. This would allow all the jobs to scale really nicely. I can see moving the preprocessing into the Dataset and then swapping out the DataFrame for a spark dataframe. There would need to be some config to have the tasks create spark jobs. Anyways let me know if this doesn’t sound crazy haha

Hey team.. so we have our custom UI that uses the REST API to trigger workflows in the Flyte backend. Now every time we bump the version of the workflows, the REST APIs invoked by the UI need to change as well to use the latest version. Is there a "latest" tag or something like that that can be used to alleviate this problem?

Hi, I'd like to ask a question of flyte type. I am using custom class module, Kirin with flyte. Flyte does not return any type error before when I run sample code of flyte, but I have no idea why flyte make this error.

ValueError: Type <class 'kirin.api.Kirin'> not supported currently in Flytekit. Please register a new transformer

The above exception was the direct cause of the following exception:

would you give any idea to solve this happening?

Hi - I noticed helm treated our

accountNumber

as a float in all annotation related lines in values-eks.yaml

<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::{{ .Values.userSettings.accountNumber }}:role/iam-role-flyte

I changed the lines to explicitly define the arn and can now schedule executions to run

remotely

But the console shows no actual executions and the role assigned is the

default

rather than 1 of the 2 roles created (

iam-role-flyte

,

flyte-user-role

) Doing a describe on the flytepropeller shows

Environment:
      POD_NAME:                     flytepropeller-7d67df4c85-7sf4v (v1:metadata.name)
      POD_NAMESPACE:                flyte (v1:metadata.namespace)
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           us-east-1
      AWS_REGION:                   us-east-1
      AWS_ROLE_ARN:                 arn:aws:iam::1.AWS_ACCTe+11:role/iam-role-flyte
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Any idea where the

AWS_ROLE_ARN

is being mounted from?

Hi! Is there a way to get current image FQN (in case of a remote execution) from within a task? Specifically I would be interested in having that information in

PythonTask.pre_execute()

, so

user_params: ExecutionParameters

would also be available if that helps.

Hi, is there some documentation on running a part of the workflow? If there are 5 tasks in a workflow for example and I’d like to run the first 3 tasks to view the output of task #3, how could I do that? Also, where are the intermediate outputs for each task saved? I’m trying to visualize the intermediate task outputs and not sure if it’ll be performant enough to be near real-time for large datasets?

Hello! I'm starting with flyte, I'm trying to understand how flyte isolates resources, especially how to define the domains and their resourceQuotas, can I change it on the fly? For now, I believe this is defined in the helm values and cannot be changed without changing the helm, is this true? or is there another way?

Hey folks. I was wondering if Flyte has support for writing spark jobs in Scala. I see a lot of examples for pyspark using the python flyte api, but I want to use flyte-scala to write scala spark pipelines, and documentation seems kinda sparse on this topic.

@Yuvraj Propeller workload pod is still crashing. Annotation with a namespace is done. Also, the actual problem statement is still there. As I need to access external IP for UI services. @Yee