Next, I'm guessing the issue you filed is regarding this question, is that correct? IIUC what you want is for a ContainerTask to be able to set Pod-specific configuration. So right now this is implemented in the backend - the Pod and Container Tasks are initialized and executing using the same plugin.The work on integrating this through to flytekit has been pending for some time and is being tracked in this issue. Can you confirm this issue is a duplicate of your proposal?

RE: ^^^ I'm assuming that since you prefer to set secrets manually on Pods (using the PodTask) that is the driver behind supporting manual secret injection using ContainerTask?

Let me explain what I want to run to give you some context: We have existing containerized "tasks" that run various things for our ML pipeline: data generation, training, evaluation, etc... and we are now at a point where we need to schedule all these on our multiple GPU machines. So I setup kubernetes and wanted to use flyte to define the workflows and let k8s take care of running the tasks on the appropriate nodes, schedule them, etc. Most tasks right now are not python tasks, so I need to use ContainerTask in Flyte. In the future we want to transition some to use the nice flyte native Python tasks, but first I need to get the current things running (and some will never be python tasks). And for most of these tasks I need to specify credentials as ENV vars (e.g. AWS_ACCESS_KEY, etc) and also place some files with secret information at a specified location in the filesystem. I can do that with plain k8s and inject the secret as env vars for the S3 credentials and I can use a projected volume for my file with a secret that needs to be placed in

/etc

ok sure, so the ENV var naming is a restriction because your existing tasks use the specific naming and those tasks can not be updated? So you want Flyte to inject secrets without the group prefix so your existing containers can use them without any modifications?

and I want to inject the secret with the same env name (or filepath/name) regardless of the group name (secret name), so I can easily switch credentials by specifying a different secret

Which I all could do if I could specify a full k8s pod spec per task

That would even allow me to set the runtimeClassName to nvidia only for some containers that need the GPU, which I currently also cannot (and I right now specified in the default pod template as a temporary workaround)

as long as all the things I specify in the pod spec won't be overwritten again, which at least for the env seems to be the case right now

cc @Yee @Eduardo Apolinario (eapolinario) - need to look into accelerating the work on adding Pod configuration to ContainerTask and python tasks.

The example above should output "yay" and not "nay"... And

env_from

and using a projected volume should also work, I can also try to create minimal examples for those

This should be correct I believe:

@Dan Rammer (hamersaw) did you get a chance to look at this? Anything I can help out with? I tried to add passing a pod spec as task_config to ContainterTask, but my naive changes in flytekit didn't do the trick...

Actually, just registering our own

PodFunctionTask

task & adding the appropriate fix was simpler to unblock short-term. This was the simplest fix for our usage, but heavy caveat that I'm not sure if this breaks other cases/fully captures all the intended behavior:

container.env = [V1EnvVar(name=key, value=val) for key, val in sdk_default_container.env.items()] + container.env

Otherwise I think I'll need to go for e.g. argo workflows instead to do this.....

Seems like env_from and so on cannot even be serialized to grpc/protobuf.... :

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INTERNAL
        details = "json: cannot unmarshal object into Go struct field Container.containers.envFrom of type []v1.EnvFromSource"
        debug_error_string = "{"created":"@1668631993.197113078","description":"Error received from peer ipv4:10.0.0.101:443","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"json: cannot unmarshal object into Go struct field Container.containers.envFrom of type []v1.EnvFromSource","grpc_status":13}"

@Eduardo Apolinario (eapolinario) I've been digging a bit on how to pass a pod spec to ContainerTask, but it seems a bit backwards since the pod plugin returns a PythonFunctionTask... and just copying the stuff from the pod plugin to ContainerTask

get_k8s_pod

doesn't seem the way to go here...