https://flyte.org
Join Slack
Hi all, could the pod plugin be extended to allow using secrets as env vars and files? Right now it ...
# flyte-support
q
Hi all, could the pod plugin be extended to allow using secrets as env vars and files? Right now it seems the env specified in pod spec is overwritten again... I cannot use flyte until I can create container tasks with some env vars and some files mounted from secrets.
👍 1
h
@quaint-diamond-37493 just clarifying a few things to make sure I have the right context. From a previous issue it sounds like you would prefer secrets are injected as ENV vars without the prefixed group, as you mentioned it sounds like this is one way to handle them with raw k8s. Is this just a preference for secret names or is there a specific use-case that will not work with the existing group prefixed ENV vars?
Next, I'm guessing the issue you filed is regarding this question, is that correct? IIUC what you want is for a ContainerTask to be able to set Pod-specific configuration. So right now this is implemented in the backend - the Pod and Container Tasks are initialized and executing using the same plugin.The work on integrating this through to flytekit has been pending for some time and is being tracked in this issue. Can you confirm this issue is a duplicate of your proposal?
RE: ^^^ I'm assuming that since you prefer to set secrets manually on Pods (using the PodTask) that is the driver behind supporting manual secret injection using ContainerTask?
q
yeah, so the issue is nearly the same, I just need it for ContainerTask and not a python task (via task decorator).
Let me explain what I want to run to give you some context: We have existing containerized "tasks" that run various things for our ML pipeline: data generation, training, evaluation, etc... and we are now at a point where we need to schedule all these on our multiple GPU machines. So I setup kubernetes and wanted to use flyte to define the workflows and let k8s take care of running the tasks on the appropriate nodes, schedule them, etc. Most tasks right now are not python tasks, so I need to use ContainerTask in Flyte. In the future we want to transition some to use the nice flyte native Python tasks, but first I need to get the current things running (and some will never be python tasks). And for most of these tasks I need to specify credentials as ENV vars (e.g. AWS_ACCESS_KEY, etc) and also place some files with secret information at a specified location in the filesystem. I can do that with plain k8s and inject the secret as env vars for the S3 credentials and I can use a projected volume for my file with a secret that needs to be placed in 
/etc
h
ah ok, i'm going to add a little bit more description to the issue if that's OK with you. So the 
@task
decorator and 
ContainerTask
use the same backend plugin - adding the functionality you request should be VERY similar for both, really just a flytekit side update.
ok sure, so the ENV var naming is a restriction because your existing tasks use the specific naming and those tasks can not be updated? So you want Flyte to inject secrets without the group prefix so your existing containers can use them without any modifications?
q
yeah, I cannot change all env var names since basically all third party libs (like s3 libs like boto3, or aws-cli) expect specific env names
and I want to inject the secret with the same env name (or filepath/name) regardless of the group name (secret name), so I can easily switch credentials by specifying a different secret
Which I all could do if I could specify a full k8s pod spec per task
That would even allow me to set the runtimeClassName to nvidia only for some containers that need the GPU, which I currently also cannot (and I right now specified in the default pod template as a temporary workaround)
h
alright i think i understand much better now, thanks for the explanation! so if we add support for pod configuration in the ContainerTask (and @task decorated python tasks) that is the best option - it allows manual secret injection and the GPU configuration you mentioned? Then we don't really need to support configuration of env var names within Flyte managed secrets (but maybe we still should).
q
yeah, that is my conclusion so far
as long as all the things I specify in the pod spec won't be overwritten again, which at least for the env seems to be the case right now
h
right - can you provide an example for that? Like in the PodSpec for a PodTask you're setting k8s volume mounts for secrets? These certainly shouldn't be overridden, I can dive through this an see what the issue is.
cc @thankful-minister-83577 @high-accountant-32689 - need to look into accelerating the work on adding Pod configuration to ContainerTask and python tasks.
q
minimal repro for just env var:
env_test.py
The example above should output "yay" and not "nay"... And 
env_from
and using a projected volume should also work, I can also try to create minimal examples for those
h
perfect, thanks! i agree, these should be maintained. i suspect this is a relatively easy fix on the backend, just overridding where we shouldn't be. i will take a look into it later today.
👍 1
q
I haven't use the kubernetes python client so far, will check if my invocation is correct..
This should be correct I believe:
env_test.py
@hallowed-mouse-14616 did you get a chance to look at this? Anything I can help out with? I tried to add passing a pod spec as task_config to ContainterTask, but my naive changes in flytekit didn't do the trick...
h
@quaint-diamond-37493,this is on my todo list for today. I'll reply by EOD, ok?
q
awesome, thanks! I don't want to rush you guys, but feel free to reach out if I can be of help.
h
Of course. Thank you.
q
any updates so far?
h
Oh, hey, yeah, sorry for not updating here. I'm working on a PR to allow for that. I'll be out later today and tomorrow, but expect to have this out for review on Monday, ok?
q
nice!
e
Also just encountered this, glad to see this is already being tracked. Would be happy to also help put up a fix for this / test out a new version of flytekit. For context, we need to use the downward API to set an env var corresponding to the cpu requests in order to correctly set 
num_threads
in LightGBM (o/w LGBM will incorrectly use the number of cpus on the node to set its internal thread count, as opposed to the cpu allocation for that particular pod). In this particular case, we can likely work around this by setting a Kyverno policy to add those env vars to each pod created in one of the 
*-development, *-staging, *-production
namespaces, but in general big +1 to addressing this.
👍 1
Actually, just registering our own 
PodFunctionTask
task & adding the appropriate fix was simpler to unblock short-term. This was the simplest fix for our usage, but heavy caveat that I'm not sure if this breaks other cases/fully captures all the intended behavior:
Copy code
container.env = [V1EnvVar(name=key, value=val) for key, val in sdk_default_container.env.items()] + container.env
q
@high-accountant-32689 any chance for a PR to review/test soon?
h
q
That is already good for the python tasks, but I actually need this for ContainerTask, see https://github.com/flyteorg/flyte/issues/3055 I tried to pass the 
task_config
in container_task.py, but that is not enough apparently....
h
got it, sorry, I was under the impression that doing this for python tasks was enough for your case. I can work on adding support for this in container tasks later this week.
q
yeah, I actually also need to be able to inject secrets (as env vars and and as files) into the container as mentioned in another thread... I will check if I can work around that via pod templates for now...
Otherwise I think I'll need to go for e.g. argo workflows instead to do this.....
Seems like env_from and so on cannot even be serialized to grpc/protobuf.... :
Copy code
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INTERNAL
        details = "json: cannot unmarshal object into Go struct field Container.containers.envFrom of type []v1.EnvFromSource"
        debug_error_string = "{"created":"@1668631993.197113078","description":"Error received from peer ipv4:10.0.0.101:443","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"json: cannot unmarshal object into Go struct field Container.containers.envFrom of type []v1.EnvFromSource","grpc_status":13}"
h
Interesting. Can you share the pod spec you're using? In any case, we'll get working on exposing pod configuration in container tasks asap.
👍 1
q
here is a simple test with a task using pod plugin with env_from
env_from_test.py
thx 1
e
Could you change the pod spec to mount the secret as a volume instead so you can read it from a file?
q
So for ContainerTask the task_config/pod_spec is currently completely ignored... For python tasks using 
@task
decorator and passing pod spec via task_config it seems only some things can be passed/serialized... in my case I would need to mount the secret as file as projected volume to /etc/my_file_name...
h
@quaint-diamond-37493, in your example, 
env_from
should take a list of 
V1EnvFromSource
q
awesome! thanks @high-accountant-32689
👍 1
@high-accountant-32689 I've been digging a bit on how to pass a pod spec to ContainerTask, but it seems a bit backwards since the pod plugin returns a PythonFunctionTask... and just copying the stuff from the pod plugin to ContainerTask 
get_k8s_pod
doesn't seem the way to go here...
162 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by