Hi, when is Flyte deleting Pods? I just had a failed task (due to image pull backoff, since some pull creds were missing in that namespace), but the Pod was not removed. Once I added the pull creds, my collegue relaunched the task, but now there were two Pods running! The "old" one which was marked as failed in Flyte and the new relaunched one. Had to manually remove the Pod that initially failed, but which in k8s started once the pull secret was there...

Hi everyone, Having some doubts about this: https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/multi_images.html Basically, there are 2 containers (for trainer and predictor). How the workflow should run to run the task images?

Hello Flyte team, I am trying to build a Flyte container with both k8s_spark and my custom python packages. I started the Dockerfile from this according to your suggestions (thanks!) https://github.com/flyteorg/flytesnacks/blob/master/cookbook/integrations/kubernetes/k8s_spark/Dockerfile However, this container requires changing the root user like this. This causes permission issues for other flyte tasks that access AWS etc. that are setup for the default user only.

ARG spark_uid=1001

# Set /root user and group
RUN chown -R ${spark_uid}:${spark_uid} /root

USER ${spark_uid}

Do we have to change the default USER for spark tasks like above?

Another issue this causes is when I run the workflow that has spark tasks, I have to supply --service-account spark e.g.

pyflyte run --remote --service-account spark --image spark_image ...

But this will cause other flyte tasks to have permission issues. How can we provide the --service-account spark param in the code for the spark task to avoid this? Will the following work?

@task(
    container_image=".../flyte-pyspark:latest",
    task_config=Spark(
        spark_conf={...
        },
        service_account: "spark",
    ),
)

Hi there - I’m standing up Flyte for (evaluation/exploratory purposes for my company) inside EKS. With the 1.3 release should I be `helm install`ing flyte-binary or flyte-core? These docs seem to conflict with these release notes

Hi everyone, I have question related to workflows that have many tasks that are executed in a different system (for example, hive or spark), Does flyte create one pod for each task? I am trying to study the performance impact of this situation when we have high fanout workflows of this kind. any pointers will be helpful.

I have another question. Let's say I want to create a plugin that is to execute Go tasks (it might be already there, i am still exploring Flyte). I am trying to figure out the changes i will have to make to realize this. any reference would be helpful.

Hi everyone…

trying to run a remote workflow with a local image but it never ends… this is the command that I’m using: pyflyte --config /Users/ematus/.flyte/config-sandbox.yaml run --remote --image test_image:latest automation/load_person_profile_workflow.py wf

running without the remote options takes 2 seconds to complete

Hello all, We are in the midst of evaluating open source ML pipeline and workflow orchestration tools w/in our organization. I received lots of great recommendations pointing me towards Flyte and am interested in doing a POC at some point. I'm still learning more and more about Flyte by watching some webinars. I did have some questions about multi-tenancy because that seems to be the biggest pain point when adopting new tools

Maybe it's just me, but when I followed https://docs.flyte.org/projects/cookbook/en/latest/auto/larger_apps/larger_apps_deploy.html

flytectl demo  exec -- docker build . --tag "whatever:v1"

failed with following message: OCI runtime exec failed: exec failed: unable to start container process: exec: "docker": executable file not found in $PATH: unknown It seems like the sandbox doesn't contain docker command. Am I missing something here? I'm using Mac.

Hello all, anyone using the Azure SSO with Flyte? I am able to get the SSO working on the Flyte UI but the cli auth fails with an error:

The 'redirect_uri' parameter does not match any of the OAuth 2.0 Client's pre-registered redirect urls

I wonder if someone else hit this issue? (note: I already verified that the redirect URI is registered in the Azure app)

Hi team, While executing map tasks in the order of 1000s, some of the tasks fail. My current task behavior is if one of the tasks fail, rest of the map task execution is aborted. Could I configure in such a way, that failed tasks are okay, lets proceed with rest of the execution? Thank you.

Does the Databricks integration released in 1.3 require usage of Databricks Connect and thus Databricks interactive clusters (vs automated clusters)? Does Flyte introspect into the spark cluster itself, or just rely on the web API?

We are setting up the databricks plugin and have it partially working! It works when the databricks PAT is passed into the task config:

@task(
    task_config=Databricks(
        databricks_token="token"
))

For some reason it isn’t working with the default token, which is identical to the PAT that I pass in https://docs.flyte.org/en/latest/deployment/plugin_setup/webapi/databricks.html. Would love to get some help debugging or general advice on what to look for.

Hi Flyte team, Looking at the flytekit plugins for KF jobs: Pytorch, Tensorflow and MPI. None of them seems to provide resource configs for workers/launcher separately. Wondering why is this the case? Does Flyte has some internal logic to “smartly” assign resources across different type of pods?

Has anyone figured out how to get sphinx-autodoc to work with Flyte task/workflow decorators? I found that creating a Sphinx extension (similar to what Celery did here) worked, but am curious if anyone else has dealt with this

Can anyone help me debug why my task is stuck in Queueing state? When I look at the status of the pods it says they have Completed - including the pod corresponding to the task that is stuck Queueing in the dashboard - but the next tasks do not kick off.

Hi, Flyte Team! I wonder flyte support Grafana Logging with Loki in Flyte console. My team currently is using Kibana (EFK), but we consider it to replace Loki.

Is it possible to somehow parametrize the generated execution and pod names?

I have ContainerTasks without any outputs, but it still creates the copilot uploader sidecar, which then exits immediately marking the pod as not ready... Would be better if the sidecar is not even created in the first place, and there is a check for that in https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/pluginmachinery/flytek8s/copilot.go#L243 Seems like a bug, any pointers to debug this further?

Hello ^_^ Before trying this out myself, has there been any attempts/thoughts on enabling nvidia-acceleration for the single container demo cluster?

Hi, FYI our corporate firewall is preventing access to

<http://blog.flyte.org|blog.flyte.org>

(

<http://flyte.org|flyte.org>

is okay) because the IP hosting the blog has some malware-related results in AlienVault: https://otx.alienvault.com/indicator/ip/76.76.21.21 It's probably not related to the Flyte blog itself, but I figured it's worth mentioning because we're likely not the only people blocked from accessing it.

This message was deleted.

Hello All, I am trying to use flyteremote to connect to a flyte cluster. I am able to run flytectl get projects from my terminal without any error. However when I run the flyteremote as given in the example page https://docs.flyte.org/projects/cookbook/en/stable/auto/remote_access/remote_workflow.html, I get SSL verify error E0123 142454.984053964 2564353 ssl_transport_security.cc:1501] Handshake failed with fatal error SSL_ERROR_SSL: error1000007dSSL routinesOPENSSL internalCERTIFICATE_VERIFY_FAILED. {"asctime": "2023-01-23 142454,985", "name": "flytekit.cli", "levelname": "ERROR", "message": "Non-auth RPC error <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNAVAILABLE\n\tdetails = \"failed to connect to all addresses; last error: UNKNOWN: ipv410.183.180.0443: Ssl handshake failed: SSL_ERROR_SSL: error1000007dSSL routinesOPENSSL internalCERTIFICATE_VERIFY_FAILED\"\n\tdebug_error_string = \"UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv410.183.180.0443: Ssl handshake failed: SSL_ERROR_SSL: error1000007dSSL routinesOPENSSL internalCERTIFICATE_VERIFY_FAILED {grpc_status:14, created_time:\"2023-01-23T142454.98558765+02:00\"}\"\n>, sleeping 200ms and retrying"} Kindly help #announcements #flytekit

Hi team, we want to upgrade our EKS to

1.22

that removes support for quite a few beta APIs (link). Shall we expect any issues with Flyte (e.g does it depend on any of those beta APIs)? datacatalog_version = "v1.0.1" flyteadmin_version = "v1.1.21" flyteconsole_version = "v1.1.0" flytecopilot_version = "v0.0.26" flytepropeller_version = "v1.1.33"

Hello all.. so I have a flyte setup where logs go to splunk and it works properly. However for map tasks it looks like the logging is not setup correctly and shows a non working k8s link in the flyte UI. Is there some additional config required?

it’s not possible to use conditions with named outputs?