https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
t

Tarmily Wen

10/31/2022, 3:48 PM
Hi, I am struggling to inject secrets at execution time into tasks and workflows via the flytekit api. This is my task taken from the mnist training example
Copy code
@task(
    retries=2,
    cache=True,
    cache_version="1.0",
    requests=Resources(gpu=gpu, mem=mem, storage=storage),
    limits=Resources(gpu=gpu, mem=mem, storage=storage),
    secret_requests=[Secret(group="wandb-secrets", key="API_KEY")],
)
def pytorch_mnist_task(hp: Hyperparameters) -> TrainingOutputs:
    secrets = current_context().secrets
    wandb_api_key = secrets.get(group="wandb-secrets", key="API_KEY")
And I would like to call a remote execution and inject the secret at the same time like this:
Copy code
current_config = Config.auto()

remote = FlyteRemote(config=current_config)
flyte_workflow = remote.fetch_workflow(name=workflow_name, version="v1", project="flytesnacks", domain="development",)
workflow_execution = remote.execute(entity=flyte_workflow, inputs={"hp": Hyperparameters(epochs=2, batch_size=128)}, project="flytesnacks", domain="development",)
But after checking out the docs, I am confused about where to put the secret and the format it should be in. I noticed the SecretsConfig, but it isn't clear to me about how to utilize that here since it appears to require a file. Can I not specific an execution time environment variable?
t

Tyler Su

10/31/2022, 5:09 PM
Hi Tarmily, we got your question, please hang tight!
s

Shivay Lamba

10/31/2022, 5:34 PM
cc @Kevin Su
k

Kevin Su

10/31/2022, 6:09 PM
Hi @Tarmily Wen, did you create a k8s secret in namespace 
flytesnacks-development
? You have to add a secret in k8s, and then the task will read the credential from that file. https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/use_secrets.html#configuring-a-secret-mana[…]nt-system-plugin-into-use
t

Tarmily Wen

11/03/2022, 8:54 PM
Hello, I created a kube secret and received this error upon execution
Copy code
Workflow[flytesnacks:development:workflows.ml.mnist_training.pytorch_training_wf] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook "<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>": Post "<https://flyte-pod-webhook.flyte.svc:30090/mutate--v1-pod?timeout=10s>": dial tcp: lookup host.docker.internal: no such host
As a sanity check, I commented out the secret lines and hardcoded the api key, and that ran properly.
k

Kevin Su

11/03/2022, 8:57 PM
it looks like api server failed to connect to webhook, is your webhook running in your cluster?
t

Tarmily Wen

11/03/2022, 9:02 PM
Copy code
kubectl -n flyte get service
NAME                         TYPE           CLUSTER-IP     EXTERNAL-IP            PORT(S)                         AGE
flyte-pod-webhook            ExternalName   <none>         host.docker.internal   <none>                          144m
minio                        NodePort       10.43.51.49    <none>                 9000:30084/TCP,9001:30088/TCP   144m
redoc                        ClusterIP      10.43.156.52   <none>                 87/TCP                          144m
flyte-kubernetes-dashboard   NodePort       10.43.59.224   <none>                 30082:30082/TCP                 144m
postgres                     NodePort       10.43.27.129   <none>                 5432:30089/TCP                  144m
It is running but it wasn't given a cluster IP
k

Kevin Su

11/03/2022, 9:04 PM
are you using flytectl demo?
t

Tarmily Wen

11/03/2022, 9:04 PM
yes
k

Kevin Su

11/03/2022, 9:05 PM
ah, i see. Maybe there are some issues in demo, let me dig into it.
What os are you using? ubuntu?
t

Tarmily Wen

11/03/2022, 10:03 PM
yes
k

Kevin Su

11/03/2022, 10:06 PM
cc @Dan Rammer (hamersaw) did you run into this issue before? api server failed to resolve host name 
host.docker.internal
. I just tested it on macos, it’s working for me.
d

Dan Rammer (hamersaw)

11/04/2022, 11:35 PM
@Tarmily Wen so sounds like secrets are only working in our sandbox environment on mac os. we're looking into this, should have a fix soon.
@Tarmily Wen small update - on Friday we got secrets working in flytectl demo with this PR, just need to add a little more testing.
36 Views
Powered by