#3963 [BUG] Secrets don't work for non python-tasks Issue created by cosmicBboy Describe the bug Secret requests cannot be found for

ContainerTask

and other plugins like

Elastic

, which create task types that are not the vanilla python task. Given a task configuration like:

@flytekit.task(
    ...
    task_config=Elastic(
        nnodes=2,
        nproc_per_node=8,
        rdzv_configs={"timeout": 1200, "join_timeout": 900}
    ),
    ...
)

An error like the following is raised:

Root Cause (first observed failure):
[0]:
  time      : 2023-08-14_15:54:34
  host      : f9944ebe99b184ee293a-n1-0-worker-0
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 254)
  error_file: /tmp/torchelastic_xc62kj37/f9944ebe99b184ee293a_78vmhc5o/attempt_0/0/error.json
  traceback : Traceback (most recent call last):
    File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
      return f(*args, **kwargs)
    File "/opt/conda/lib/python3.10/site-packages/flytekitplugins/kfpytorch/task.py", line 233, in spawn_helper
      return_val = fn(**kwargs)
    File "/root/fine_tuning/llm_fine_tuning.py", line 422, in train
      os.environ["WANDB_API_KEY"] = flytekit.current_context().secrets.get(
    File "/opt/conda/lib/python3.10/site-packages/flytekit/core/context_manager.py", line 365, in get
      raise ValueError(
  ValueError: Unable to find secret for key wandb_api_key-5t1ZwJ in group arn:aws:secretsmanager:us-east-2:590375264460:secret: in Env Var:_FSEC_ARN:AWS:SECRETSMANAGER:US-EAST-2:590375264460:SECRET:_WANDB_API_KEY-5T1ZWJ and FilePath: /etc/secrets/arn:aws:secretsmanager:us-east-2:590375264460:secret:/wandb_api_key-5t1zwj
  
============================================================

User error.

Expected behavior Secrets should be mounted as expected. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte