https://flyte.org
Join Slack
Hi, I am struggling to inject secrets at execution time into tasks and workflows via the flytekit ap...
# flyte-support
d
Hi, I am struggling to inject secrets at execution time into tasks and workflows via the flytekit api. This is my task taken from the mnist training example
Copy code
@task(
    retries=2,
    cache=True,
    cache_version="1.0",
    requests=Resources(gpu=gpu, mem=mem, storage=storage),
    limits=Resources(gpu=gpu, mem=mem, storage=storage),
    secret_requests=[Secret(group="wandb-secrets", key="API_KEY")],
)
def pytorch_mnist_task(hp: Hyperparameters) -> TrainingOutputs:
    secrets = current_context().secrets
    wandb_api_key = secrets.get(group="wandb-secrets", key="API_KEY")
And I would like to call a remote execution and inject the secret at the same time like this:
Copy code
current_config = Config.auto()

remote = FlyteRemote(config=current_config)
flyte_workflow = remote.fetch_workflow(name=workflow_name, version="v1", project="flytesnacks", domain="development",)
workflow_execution = remote.execute(entity=flyte_workflow, inputs={"hp": Hyperparameters(epochs=2, batch_size=128)}, project="flytesnacks", domain="development",)
But after checking out the docs, I am confused about where to put the secret and the format it should be in. I noticed the SecretsConfig, but it isn't clear to me about how to utilize that here since it appears to require a file. Can I not specific an execution time environment variable?
w
Hi Tarmily, we got your question, please hang tight!
v
cc @glamorous-carpet-83516
g
Hi @delightful-computer-49028, did you create a k8s secret in namespace 
flytesnacks-development
? You have to add a secret in k8s, and then the task will read the credential from that file. https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/use_secrets.html#configuring-a-secret-mana[…]nt-system-plugin-into-use
d
Hello, I created a kube secret and received this error upon execution
Copy code
Workflow[flytesnacks:development:workflows.ml.mnist_training.pytorch_training_wf] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook "<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>": Post "<https://flyte-pod-webhook.flyte.svc:30090/mutate--v1-pod?timeout=10s>": dial tcp: lookup host.docker.internal: no such host
As a sanity check, I commented out the secret lines and hardcoded the api key, and that ran properly.
g
it looks like api server failed to connect to webhook, is your webhook running in your cluster?
d
Copy code
kubectl -n flyte get service
NAME                         TYPE           CLUSTER-IP     EXTERNAL-IP            PORT(S)                         AGE
flyte-pod-webhook            ExternalName   <none>         host.docker.internal   <none>                          144m
minio                        NodePort       10.43.51.49    <none>                 9000:30084/TCP,9001:30088/TCP   144m
redoc                        ClusterIP      10.43.156.52   <none>                 87/TCP                          144m
flyte-kubernetes-dashboard   NodePort       10.43.59.224   <none>                 30082:30082/TCP                 144m
postgres                     NodePort       10.43.27.129   <none>                 5432:30089/TCP                  144m
It is running but it wasn't given a cluster IP
g
are you using flytectl demo?
d
yes
g
ah, i see. Maybe there are some issues in demo, let me dig into it.
What os are you using? ubuntu?
d
yes
g
cc @hallowed-mouse-14616 did you run into this issue before? api server failed to resolve host name 
host.docker.internal
. I just tested it on macos, it’s working for me.
h
@delightful-computer-49028 so sounds like secrets are only working in our sandbox environment on mac os. we're looking into this, should have a fix soon.
@delightful-computer-49028 small update - on Friday we got secrets working in flytectl demo with this PR, just need to add a little more testing.
262 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by