Does Flyte sets any environment variables on the pods it runs which we could use? We’re interested in

flytekit.current_context().execution_id.domain

in particular. It’s a bit of a chore that now our libraries have to include 300+mb of deps of

flytekit

to just get the runtime env.

Also, where can I set a default node selector for all the workflow pods to execute on? I need to ensure every pod in flyte and the workflows it kicks off are assigned to a specific node. I assume somewhere in the flyte_propeller configMap

Hey folks, trying to debug an issue. We’re seeing various tasks fail with

terminated in the background, manually

. I took a look at the code here and see that this occurs when the pod get deleted. I don’t see a delete request being sent to admin so don’t think there is anything there. Propeller logs haven’t shown anything either. So I’ve been looking at kube logs to try to see what is triggering the pod deleted but there’s a lot of noise in the logs so taking some time to through them. Meanwhile, I figured I’d ask here if anyone has run into something similar before

Resolved I am running into this issue as well: https://discuss.flyte.org/t/5158547/Hello-I-was-exploring-on-Kubernetes-Spark-job-and-i-tried-to. I could use some help debugging next week 🙂

Failed to convert return value for var o0 for function src.workflow.t1 with error <class 'py4j.protocol.Py4JJavaError'>: An error occurred while calling o49.parquet.
: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"

[1/1] currentAttempt done. Last Error: USER::containers with unready status: [primary]|context deadline exceeded

I’ve seen a few times such error. What would be the root cause of it?

Hey flyte community, been doing some autoscaling stuff with Karpenter and EKS with flyte and performance is really good and fast. Just one question, is there any sort of inbuilt logic within Flyte to allocate tasks/pods to certain provisioners/nodegroups automatically based on CPU/MEM request sizes (instead of using nodeselectors,taints), or is this custom logic we would have to build out ourselves? The reason I ask is because the end-user for our use-case would be a non-developer/non-programmer and so we would like to abstract away as much of the configuration from them as possible. Piping in the pod-templates and pod specs manually into the task defintions or using the with_overrides method works just fine, but unfortunately, even something like this may prove too much for the end-user. Obviously this is probably not the intended workflow for Flyte, but was wondering if anybody had any insight into this. Or if the custom logic is the answer, then that's fine too (I am assuming some sort of wrapper function to convert our users' request sizes into pod specs/taints/tolerations to pipe into the task invocations)

Is there a way to locally execute a task or workflow using the inputs from a remote execution, ideally diretly using the Json string from the UI? The dispatch_execute (https://github.com/flyteorg/flytekit/blob/master/flytekit/bin/entrypoint.py#L63) function seems to do what I am looking for, but expects a pb as input. Does this work at all (either with pb or with json)? If yes, is there a way to directly use the Json from the UI or how can I easily find the inputs.pb for a remote task execution?

On to the next question. I am setting up Flyte to be used by many different teams. My idea was use the same service account for each of the projects i.e., use the https://docs.flyte.org/en/latest/deployment/aws/manual.html#flyte-user-role the pods are assuming and pass IAM roles in the launch plan that the service account assumes during execution, I remembered that there were some problems with this a while back. Is this working or is there a better way to achieve project specific permissions?

My code:

from flytekit import task, workflow, ImageSpec

image_spec = ImageSpec(packages=["numpy", "tensorflow"], base_image="<http://container-image-registry.corp.linkedin.com:8083/lps-image/linkedin/mldev-workspace/mldev-base-image:0.0.16|container-image-registry.corp.linkedin.com:8083/lps-image/linkedin/mldev-workspace/mldev-base-image:0.0.16>", pip_index="<https://lerna.tools.corp.linkedin.com/pypi/simple/>", registry="localhost:30000")

@task(container_image=image_spec)
def t1() -> str:
    import numpy as np
    print("task 1")
    print(np.array([1,2,3]))

    import tensorflow as tf
    print("task 2")
    print(tf.constant(8))
    return("blah1")

@workflow
def wf():
    t1()

Then I run:

pyflyte run --remote example.py wf

Error:

Failed with Unknown Exception <class 'Exception'> Reason: failed to build the imageSpec at /var/folders/9v/3dl8rh2x0nx9ccbv3xkd708h002_42/T/flyte-mktmp10q/sandbox/local_flytekit/4e7f726c4f5a035e23addb662302e8e1/build.envd with error b'error: failed to do request: Head "<http://localhost:30000/v2/flytekit/blobs/sha256:a0e77e7122fc5edfccf28327cc28f2c15e34ee2ec76fa0c7486a6679c014a134>": dial tcp 127.0.0.1:30000: connect: connection refused\n'

hi, is there any option to configure deletion policy for k8s pods that are on status Error? (we use flyte-core helm chart on gke)