Hey all. I'm having trouble using a ShellTask with secrets. In the following task I would expect an environment variable possibly named

AWS-ACCESS-KEY_AWS_ACCESS_KEY_ID

to exist but it doesn't. What am I doing wrong here?

task_with_secrets = ShellTask(
    debug=True,
    name="task_with_secrets",
    script='''
    set -ex
    env > out_env
    ''',
    inputs=kwtypes(source=str),
    output_locs=[
        OutputLocation(var="x", var_type=FlyteFile, location="out_env")
    ],
    secret_requests=[
        Secret(
            group='aws-access-key',
            key='AWS_ACCESS_KEY_ID',
            mount_requirement=Secret.MountType.ENV_VAR
        ),
        Secret(
            group='aws-access-key',
            key='AWS_SECRET_ACCESS_KEY',
            mount_requirement=Secret.MountType.ENV_VAR
        ),
    ]
)

ohhh i think we are not injecting this env variable

Is there a way to access the context var in the ShellTask?

sadly we dont - oversight - https://github.com/flyteorg/flytekit/blob/master/flytekit/extras/tasks/shell.py#L176

Sure that would be awesome.

also the shell task is updated now - Zach from Zymergen contributed it.

So I am able to run

export X='{ctx.execution_id}'

, but this fails

export y="{ctx.secrets.get('aws-access-key', 'AWS_ACCESS_KEY_ID')}"

with

[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):

      File "/venv/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/venv/lib/python3.10/site-packages/flytekit/core/base_task.py", line 481, in dispatch_execute
        raise e
      File "/venv/lib/python3.10/site-packages/flytekit/core/base_task.py", line 478, in dispatch_execute
        native_outputs = self.execute(**native_inputs)
      File "/venv/lib/python3.10/site-packages/flytekit/extras/tasks/shell.py", line 191, in execute
        gen_script = self._interpolizer.interpolate(self._script, inputs=kwargs, outputs=outputs)
      File "/venv/lib/python3.10/site-packages/flytekit/extras/tasks/shell.py", line 82, in interpolate
        return self._Formatter().format(tmpl, **consolidated_args)
      File "/usr/local/lib/python3.10/string.py", line 161, in format
        return self.vformat(format_string, args, kwargs)
      File "/usr/local/lib/python3.10/string.py", line 165, in vformat
        result, _ = self._vformat(format_string, args, kwargs, used_args, 2)
      File "/usr/local/lib/python3.10/string.py", line 205, in _vformat
        obj, arg_used = self.get_field(field_name, args, kwargs)
      File "/usr/local/lib/python3.10/string.py", line 276, in get_field
        obj = getattr(obj, i)

Message:

    'SecretsManager' object has no attribute 'get('aws-access-key', 'AWS_ACCESS_KEY_ID')'

SYSTEM ERROR! Contact platform administrators.

ya, so this is what I was telling you. since the shell task was alpha, we were working through a couple different syntax options

Ok cool, no problem. One issue with the new syntax could be collisions in input names, ctx, and output variable names. Who wins if I have an input and an output named ctx ?

it will not allow you to do that 😞

I'd vote to namespace both inputs and outputs so there is no chance for collisions. But that's just me.

hmm