https://flyte.org
Join Slack
Hello - i'm testing a flow using `ContainerTask`. can i inject credentials directly into `flyte-copi...
# flyte-support
p
Hello - i'm testing a flow using 
ContainerTask
. can i inject credentials directly into 
flyte-copilot-downloader
(or any other initContainer)? i tried adding to the task 
PodTemplate
but that creates a conflict: 
failed to execute handle for plugin [container]: [Invalid] failed to create resource, caused by: Pod "{podname}" is invalid: spec.initContainers[1].name: Duplicate value: "flyte-copilot-downloader"
t
would you mind elaborating on your use case?
p
certainly - i have a a 
ContainerTask
defined that is used by a workflow:
Copy code
transforminput = ContainerTask(
    name="flowtest",
    input_data_dir="/app/data/inputs",
    output_data_dir="/app/data/outputs",
    inputs=kwtypes(inputpath=str, outputpath=str),
    image="{imagelocation}/flowtest:latest",
    pod_template_name="default-pod-template",
    command=[{stuff}],
)
creating and using a 
ContainerTask
in a remote wf appears to also create an `InitContainer`: 
flyte-copilot-downloader
that init container attempts to pull data from our s3 metadata bucket. I see this in the task pod details:
Copy code
Init Containers:
  flyte-copilot-downloader:
    Container ID:  <containerd://561fe38310bb2f5af11e8e145816d9d4a983613924beea4223c6e237780a667>9
    Image:         <http://cr.flyte.org/flyteorg/flytecopilot-release:v1.10.0|cr.flyte.org/flyteorg/flytecopilot-release:v1.10.0>
    Image ID:      <http://cr.flyte.org/flyteorg/flytecopilot-release@sha256:8f1ae57f51c5c0f68c0a6ddf4986967d106c596497b86c8eb333cca27203ed52|cr.flyte.org/flyteorg/flytecopilot-release@sha256:8f1ae57f51c5c0f68c0a6ddf4986967d106c596497b86c8eb333cca27203ed52>
  Args:
      download
      --from-remote
      s3://{containerName}/n0/data/inputs.pb
however, it fails because the credentials injected into the credentials injected via env vars into the task container are not merged with the spawned initcontainer
so the initcontainer fails
I tried adding this block to the 
PodTemplate
, but then a name collision pops:
Copy code
initContainers:
    - envFrom:
      - secretRef:
          name: default-aws-creds  
      name: flyte-copilot-downloader
      image: <http://cr.flyte.org/flyteorg/flytecopilot-release:v1.10.0|cr.flyte.org/flyteorg/flytecopilot-release:v1.10.0>
apparently some mechanism is trying to create the initContainer twice rather than merge the podtemplate config.
Copy code
failed to execute handle for plugin [container]: [Invalid] failed to create resource, caused by: Pod "fcaf1c64bb68945f7b67-n0-0" is invalid: spec.initContainers[1].name: Duplicate value: "flyte-copilot-downloader"
t
are you providing credentials (secrets) via your container task? cc @average-finland-92144
p
I am providing a 
PodTemplate
to the 
ContainerTask
, and that template contains a reference to the secret. but i don't know how to pass credentials specifically. can you point me towards documentation for that use case?
t
you should be able to provide 
secret_requests
directly to the 
ContainerTask
https://docs.flyte.org/en/latest/flytesnacks/examples/productionizing/use_secrets.html#secrets
👀 1
p
@tall-lock-23197 - i tried using 
secret_requests
a few different ways but ran into a couple problems 1. the secrets get injected into the 
InitContainer
but have a prepended 
FLYTE_SECRETS_ENV_PREFIX
which is set to 
_FSEC_
. a. The init for the stow container doesn't seem to be configurable to look for envvars with a specific prefix 2. the documentation suggests I can set 
FLYTE_SECRETS_ENV_PREFIX
to customize the behavior - that is true for 
Containers
but that configuration doesn't seem to get propagated to 
InitContainers
any other ideas or suggested workarounds?
t
@high-accountant-32689 would you mind chiming in?
p
Hello. We switched to using Azure Blob Storage with a Managed Identity because those credentials are injected at the pod level. Then Terence noticed a different exception in the InitContainer, which ended up being a permissions issue. The identity injected into the inicontainer wasn't able to read from azure blob storage
t
is it working now?
p
yes, injecting a managed identity works. we have not yet found a way to inject envvars (sans 
_FSEC_
prefix) into the 
InitContainer
. that would be helpful for the future but we are not currently blocked
👍 1
19 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by