Hey, we are using Spark task and encountered an implicit retry without us configuring it explicitly. When we run

kubectl get sparkapplication ...

we see

FLYTE_MAX_ATTEMPTS: "1"
      FLYTE_ATTEMPT_NUMBER: "3"

Is this some default bahavior with spark tasks or a potential bug?

Hi flyte community, happy monday 😆 So I am trying to get some clarification on some task related question: I have a

regular python task(T1)

that calls another

regular python task(T2)

without it being dynamic; T2 is a very simple function that runs quickly. I did not want to make T1 dynamic since i didnt wanna add overhead of launching pod for T2. Also I am using T2 in a workflow as a regular python task. My question is

is it a correct way to use it?

or is it fine to call regular task from another regular task. i could create a regular python function and call that from T1 but then i need to create another wrapper for T2 which just calls the regular function and i wanted to avoid that extra stuff. From this page https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/task.html#sphx-glr-auto-core-flyte-basics-task-py i read that

You can execute a Flyte task as any normal function.

maybe i took this statement a bit too liberally

One question about the scalability of Flyte, what’s the recommended max number of concurrent tasks running in a cluster? and how to set it (or do I need to set it)? I’m trying to run this distributed ML system on flyte. It’s a single-task workflow, It needs to run at least 500 workflows or tasks (each task needs 30G RAM, 4 CPU, 1GPU) in parallel in addition to other misc workflows. Just wanted to make sure flyte can handle that scalability (I believe it can)?

Has anyone deployed flyte with AWS Cognito for authentication? I'm having some trouble with it.

hi team, what’s the recommended way to pass transformers models and tokenizers between tasks? eg. i load using

model=transformers.AutoModelForCausalLM.from_pretrained(...)

, but i cannot directly return model and pass it to the next tasks because the model cannot be pickled

cannot pickle 'torch._C._distributed_c10d.ProcessGroup' object

. Do i need to save the model with transformers’

save_pretrained

to nfs and load it when i need to use it in the next task?

is there ci/cd workflows in flyte, if not did you implement workflows for ci/cd

Hey, as an extra layer of security I want to authenticate users in the ALB by adding oidc Ingress annotations. How would I configure `flytectl`to still be able to authenticate users when the OIDC flow happens in the ALB rather than in the Flyte Console? On another note, is it possible to deploy Flyte internally and still be able to authenticate users/deploy workflows in CI/CD? I guess the topics are quite different, but it would be interesting to see what choice you prefer and how that is implemented in practice. Thanks 🙂

I have a failed workflow, that I for some reason had an error that just said “unknown” I stopped the workflow so i could restart it and it now doesnt even have the relaunch recover options.

Maybe it's just me, but when I followed https://docs.flyte.org/projects/cookbook/en/latest/auto/larger_apps/larger_apps_deploy.html

flytectl demo  exec -- docker build . --tag "whatever:v1"

failed with following message: OCI runtime exec failed: exec failed: unable to start container process: exec: "docker": executable file not found in $PATH: unknown It seems like the sandbox doesn't contain docker command. Am I missing something here? I'm using Mac.

I follow the docs to deploy the flyte. But after I run the hello_world.py in remote, I got “Original exception: Unable to locate credentials” error. I’ve tried to search docs and issues, but found nothing about it.