Hi, I am struggling to inject secrets at execution...
# ask-the-community
Hi, I am struggling to inject secrets at execution time into tasks and workflows via the flytekit api. This is my task taken from the mnist training example
Copy code
    requests=Resources(gpu=gpu, mem=mem, storage=storage),
    limits=Resources(gpu=gpu, mem=mem, storage=storage),
    secret_requests=[Secret(group="wandb-secrets", key="API_KEY")],
def pytorch_mnist_task(hp: Hyperparameters) -> TrainingOutputs:
    secrets = current_context().secrets
    wandb_api_key = secrets.get(group="wandb-secrets", key="API_KEY")
And I would like to call a remote execution and inject the secret at the same time like this:
Copy code
current_config = Config.auto()

remote = FlyteRemote(config=current_config)
flyte_workflow = remote.fetch_workflow(name=workflow_name, version="v1", project="flytesnacks", domain="development",)
workflow_execution = remote.execute(entity=flyte_workflow, inputs={"hp": Hyperparameters(epochs=2, batch_size=128)}, project="flytesnacks", domain="development",)
But after checking out the docs, I am confused about where to put the secret and the format it should be in. I noticed the SecretsConfig, but it isn't clear to me about how to utilize that here since it appears to require a file. Can I not specific an execution time environment variable?
Hi Tarmily, we got your question, please hang tight!
cc @Kevin Su
Hi @Tarmily Wen, did you create a k8s secret in namespace
? You have to add a secret in k8s, and then the task will read the credential from that file. https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/use_secrets.html#configuring-a-secret-mana[…]nt-system-plugin-into-use
Hello, I created a kube secret and received this error upon execution
Copy code
Workflow[flytesnacks:development:workflows.ml.mnist_training.pytorch_training_wf] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook "<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>": Post "<https://flyte-pod-webhook.flyte.svc:30090/mutate--v1-pod?timeout=10s>": dial tcp: lookup host.docker.internal: no such host
As a sanity check, I commented out the secret lines and hardcoded the api key, and that ran properly.
it looks like api server failed to connect to webhook, is your webhook running in your cluster?
Copy code
kubectl -n flyte get service
NAME                         TYPE           CLUSTER-IP     EXTERNAL-IP            PORT(S)                         AGE
flyte-pod-webhook            ExternalName   <none>         host.docker.internal   <none>                          144m
minio                        NodePort    <none>                 9000:30084/TCP,9001:30088/TCP   144m
redoc                        ClusterIP   <none>                 87/TCP                          144m
flyte-kubernetes-dashboard   NodePort   <none>                 30082:30082/TCP                 144m
postgres                     NodePort   <none>                 5432:30089/TCP                  144m
It is running but it wasn't given a cluster IP
are you using flytectl demo?
ah, i see. Maybe there are some issues in demo, let me dig into it.
What os are you using? ubuntu?
cc @Dan Rammer (hamersaw) did you run into this issue before? api server failed to resolve host name
. I just tested it on macos, it’s working for me.
@Tarmily Wen so sounds like secrets are only working in our sandbox environment on mac os. we're looking into this, should have a fix soon.
@Tarmily Wen small update - on Friday we got secrets working in flytectl demo with this PR, just need to add a little more testing.