https://flyte.org logo
Join Slack
Powered by
Hi, I am struggling to inject secrets at execution...
# ask-the-community
t
Hi, I am struggling to inject secrets at execution time into tasks and workflows via the flytekit api. This is my task taken from the mnist training example
Copy code
@task(
    retries=2,
    cache=True,
    cache_version="1.0",
    requests=Resources(gpu=gpu, mem=mem, storage=storage),
    limits=Resources(gpu=gpu, mem=mem, storage=storage),
    secret_requests=[Secret(group="wandb-secrets", key="API_KEY")],
)
def pytorch_mnist_task(hp: Hyperparameters) -> TrainingOutputs:
    secrets = current_context().secrets
    wandb_api_key = secrets.get(group="wandb-secrets", key="API_KEY")
And I would like to call a remote execution and inject the secret at the same time like this:
Copy code
current_config = Config.auto()

remote = FlyteRemote(config=current_config)
flyte_workflow = remote.fetch_workflow(name=workflow_name, version="v1", project="flytesnacks", domain="development",)
workflow_execution = remote.execute(entity=flyte_workflow, inputs={"hp": Hyperparameters(epochs=2, batch_size=128)}, project="flytesnacks", domain="development",)
But after checking out the docs, I am confused about where to put the secret and the format it should be in. I noticed the SecretsConfig, but it isn't clear to me about how to utilize that here since it appears to require a file. Can I not specific an execution time environment variable?
t
Hi Tarmily, we got your question, please hang tight!
s
cc @Kevin Su
k
Hi @Tarmily Wen, did you create a k8s secret in namespace 
flytesnacks-development
? You have to add a secret in k8s, and then the task will read the credential from that file. https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/use_secrets.html#configuring-a-secret-mana[…]nt-system-plugin-into-use
t
Hello, I created a kube secret and received this error upon execution
Copy code
Workflow[flytesnacks:development:workflows.ml.mnist_training.pytorch_training_wf] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook "<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>": Post "<https://flyte-pod-webhook.flyte.svc:30090/mutate--v1-pod?timeout=10s>": dial tcp: lookup host.docker.internal: no such host
As a sanity check, I commented out the secret lines and hardcoded the api key, and that ran properly.
k
it looks like api server failed to connect to webhook, is your webhook running in your cluster?
t
Copy code
kubectl -n flyte get service
NAME                         TYPE           CLUSTER-IP     EXTERNAL-IP            PORT(S)                         AGE
flyte-pod-webhook            ExternalName   <none>         host.docker.internal   <none>                          144m
minio                        NodePort       10.43.51.49    <none>                 9000:30084/TCP,9001:30088/TCP   144m
redoc                        ClusterIP      10.43.156.52   <none>                 87/TCP                          144m
flyte-kubernetes-dashboard   NodePort       10.43.59.224   <none>                 30082:30082/TCP                 144m
postgres                     NodePort       10.43.27.129   <none>                 5432:30089/TCP                  144m
It is running but it wasn't given a cluster IP
k
are you using flytectl demo?
t
yes
k
ah, i see. Maybe there are some issues in demo, let me dig into it.
What os are you using? ubuntu?
t
yes
k
cc @Dan Rammer (hamersaw) did you run into this issue before? api server failed to resolve host name 
host.docker.internal
. I just tested it on macos, it’s working for me.
d
@Tarmily Wen so sounds like secrets are only working in our sandbox environment on mac os. we're looking into this, should have a fix soon.
@Tarmily Wen small update - on Friday we got secrets working in flytectl demo with this PR, just need to add a little more testing.
217 Views
Open in Slack
PreviousNext
Powered by