delightful-computer-49028
10/31/2022, 3:48 PM@task(
retries=2,
cache=True,
cache_version="1.0",
requests=Resources(gpu=gpu, mem=mem, storage=storage),
limits=Resources(gpu=gpu, mem=mem, storage=storage),
secret_requests=[Secret(group="wandb-secrets", key="API_KEY")],
)
def pytorch_mnist_task(hp: Hyperparameters) -> TrainingOutputs:
secrets = current_context().secrets
wandb_api_key = secrets.get(group="wandb-secrets", key="API_KEY")
And I would like to call a remote execution and inject the secret at the same time like this:
current_config = Config.auto()
remote = FlyteRemote(config=current_config)
flyte_workflow = remote.fetch_workflow(name=workflow_name, version="v1", project="flytesnacks", domain="development",)
workflow_execution = remote.execute(entity=flyte_workflow, inputs={"hp": Hyperparameters(epochs=2, batch_size=128)}, project="flytesnacks", domain="development",)
But after checking out the docs, I am confused about where to put the secret and the format it should be in. I noticed the SecretsConfig, but it isn't clear to me about how to utilize that here since it appears to require a file. Can I not specific an execution time environment variable?wonderful-afternoon-77766
10/31/2022, 5:09 PMvictorious-park-53030
10/31/2022, 5:34 PMglamorous-carpet-83516
10/31/2022, 6:09 PMflytesnacks-development
? You have to add a secret in k8s, and then the task will read the credential from that file. https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/use_secrets.html#configuring-a-secret-mana[…]nt-system-plugin-into-usedelightful-computer-49028
11/03/2022, 8:54 PMWorkflow[flytesnacks:development:workflows.ml.mnist_training.pytorch_training_wf] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook "<http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>": Post "<https://flyte-pod-webhook.flyte.svc:30090/mutate--v1-pod?timeout=10s>": dial tcp: lookup host.docker.internal: no such host
As a sanity check, I commented out the secret lines and hardcoded the api key, and that ran properly.glamorous-carpet-83516
11/03/2022, 8:57 PMdelightful-computer-49028
11/03/2022, 9:02 PMkubectl -n flyte get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
flyte-pod-webhook ExternalName <none> host.docker.internal <none> 144m
minio NodePort 10.43.51.49 <none> 9000:30084/TCP,9001:30088/TCP 144m
redoc ClusterIP 10.43.156.52 <none> 87/TCP 144m
flyte-kubernetes-dashboard NodePort 10.43.59.224 <none> 30082:30082/TCP 144m
postgres NodePort 10.43.27.129 <none> 5432:30089/TCP 144m
delightful-computer-49028
11/03/2022, 9:03 PMglamorous-carpet-83516
11/03/2022, 9:04 PMdelightful-computer-49028
11/03/2022, 9:04 PMglamorous-carpet-83516
11/03/2022, 9:05 PMglamorous-carpet-83516
11/03/2022, 10:03 PMdelightful-computer-49028
11/03/2022, 10:03 PMglamorous-carpet-83516
11/03/2022, 10:06 PMhost.docker.internal
. I just tested it on macos, it’s working for me.hallowed-mouse-14616
11/04/2022, 11:35 PMhallowed-mouse-14616
11/07/2022, 7:25 PM