How do you configure node selectors with plugins such as pyspark and ray? For example add non-spot node selector to spark driver and spot node selector to spark executors or non-spot for ray head node and spot for worker nodes?

Hi there! Recently we have updated our Flyte configuration to flyte-binary setup and faced issue with usage of default-pod-template-name configuration. flyte-template(located in flyte namespace) is listed in ConfigMap of flyte-binary(), but these template is not applied on running jobs. How to properly configure default-template in recent release?

Hello, I also have a question about scaling but on a different axis -- what is the expected way to scale Flyte servers horizontally. I can see that scaling the workers can be done by having multiple K8S clusters, but if I have millions or billions of workflows and the primary database and server get overwhelmed, is there a way to shard them? Or do I simply need to have multiple Flyte deployments at that point?

We are trying to execute SparkJobs written in scala. One strategy we considered is to run ContainerTasks and spark-submit pointing to a jar file. This hasn’t worked out yet. I see that support for scala is coming soon (https://docs.flyte.org/projects/cookbook/en/stable/auto/integrations/kubernetes/k8s_spark/pyspark_pi.html). We were wondering if anyone uses flyte with spark written in scala and what their setup looks like? Maybe we could use the Java Flytekit and annotate the spark jobs directly? (We just set up deployment on GKE using helm chart and are able to successfully run various flytesnacks examples - thanks for all the help with setup on this channel!)

Hi, does anyone know why the huggingface plugin has a dependency version restriction set to

flytekit < 1.2.0

? https://github.com/flyteorg/flytekit/blob/9e4069058845f018f410703ec7c4e3971a9ee85a/plugins/flytekit-huggingface/setup.py#L8

Hi flyte folks, was there any significant performance change in Flyte console? we recently swtiched to flyte 1.2 milestone version and its correspodning flyte backend. Unfortunately now its taking a lot of time display the workflow tasks as opposed to what previously and from the browser log `

vendor-a9fbc36b.js:2          GET https://.../fizlxcwy-n4-0-dn1-0/n1?limit=10000 net::ERR_INSUFFICIENT_RESOURCES

it might have been showing previously as well but never seem to be noticable. it takes solid min or 2 to even show what tasks were executed. previous version was flyteconsole

1.1.6

, propeller

1.1.15

and admin on

1.1.29

any idea on what have changed

Hi, unionml community! I'm trying to run the MNIST: Digits Classification Tutorial in my local Flyte cluster, but I'm running into issues. Specifically, I hope to train, run batch inference, and serve the model via FastAPI within my local Flyte Sandbox. I followed these steps: 1. I set up a local sandbox environment by running

flytectl demo start

, 2. I ensured my

~/.flyte/config.yaml

is pointed at my local environment. 3. I started a unionml project by running

unionml init mnist

and navigating into the directory. 4. I ran

unionml deploy app:model

to deploy a Model object named "model" in the app.py file within the current working directory. 5. I ran

unionml train app:model

to train the model. When I deploy the model, I see these logs:

[unionml] deploying app:model
2022-09-04 19:09:54,211 unionml INFO Building docker container in flyte demo cluster.
2022-09-04 19:09:54,862 unionml INFO unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /root/Dockerfile: no such file or directory
2022-09-04 19:09:54,895 unionml INFO Deploying workflow digits_classifier.train
2022-09-04 19:09:55,230 unionml INFO Deploying workflow digits_classifier.predict
2022-09-04 19:09:55,294 unionml INFO Deploying workflow digits_classifier.predict_from_features

When I train the model, I see that a workflow is triggered in the sandbox Flyte Console, but the workflow fails on the

dataset_task

node. The logs for that task are:

[1/1] currentAttempt done. Last Error: USER::containers with unready status: [ffa8038a9524b4920a61-n0-0]|Back-off pulling image "unionml:digits-classifier-a31c8bb4d7f235f31b25a3449e3dcc0c1f82e9be"

It looks like I'm getting an error with finding pulling an image, but I'm not sure what I'm doing wrong 😅. Could you advise on how I can get this working?

hello, flyte team. I have a question about workflow register process. I am currently automating ML process through flyte with github CI, and I have problem, launchplan deactivation. The workflow successfully registered in flyte, but it does not show DAG and users cannot start "launch" in console. where do I check log? I hope to get some advice. thanks

flytekit q:

execution = flyte_remote.fetch_execution(name = execution_id)
print (execution.node_executions)

is

execution.node_executions

expected to be populated? If not, how do I get the node execution details? I'm looking for level of details similar to returned by

flytectl get execution execid --details --nodeID n0 -o yaml

Hi, is it possible to get at some underlying task execution data from the flytekit python sdk? Specifically, say I create a

SagemakerBuiltinAlgorithmsTask

and I would like to retrieve the

TrainingJobName

, which the backend plugin has access to.