https://flyte.org logo
Title
j

JP Kosymna

01/26/2022, 11:19 PM
Hey all. I'm having trouble using a ShellTask with secrets. In the following task I would expect an environment variable possibly named
AWS-ACCESS-KEY_AWS_ACCESS_KEY_ID
to exist but it doesn't. What am I doing wrong here?
task_with_secrets = ShellTask(
    debug=True,
    name="task_with_secrets",
    script='''
    set -ex
    env > out_env
    ''',
    inputs=kwtypes(source=str),
    output_locs=[
        OutputLocation(var="x", var_type=FlyteFile, location="out_env")
    ],
    secret_requests=[
        Secret(
            group='aws-access-key',
            key='AWS_ACCESS_KEY_ID',
            mount_requirement=Secret.MountType.ENV_VAR
        ),
        Secret(
            group='aws-access-key',
            key='AWS_SECRET_ACCESS_KEY',
            mount_requirement=Secret.MountType.ENV_VAR
        ),
    ]
)
k

Ketan (kumare3)

01/26/2022, 11:31 PM
ohhh i think we are not injecting this env variable
i think it should be available as the context var
ohh thank you for trying the shell task
j

JP Kosymna

01/26/2022, 11:41 PM
Is there a way to access the context var in the ShellTask?
k

Ketan (kumare3)

01/26/2022, 11:49 PM
but should be a trivial change
do you mind opening an issue?
i can push in a change soon if you would like
j

JP Kosymna

01/26/2022, 11:51 PM
Sure that would be awesome.
k

Ketan (kumare3)

01/26/2022, 11:53 PM
also the shell task is updated now - Zach from Zymergen contributed it.
Hi JP, once you create an issue, I can check this in - https://github.com/flyteorg/flytekit/pull/832
Also let me know if this works
j

JP Kosymna

01/27/2022, 8:59 PM
So I am able to run
export X='{ctx.execution_id}'
, but this fails
export y="{ctx.secrets.get('aws-access-key', 'AWS_ACCESS_KEY_ID')}"
with
[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):

      File "/venv/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/venv/lib/python3.10/site-packages/flytekit/core/base_task.py", line 481, in dispatch_execute
        raise e
      File "/venv/lib/python3.10/site-packages/flytekit/core/base_task.py", line 478, in dispatch_execute
        native_outputs = self.execute(**native_inputs)
      File "/venv/lib/python3.10/site-packages/flytekit/extras/tasks/shell.py", line 191, in execute
        gen_script = self._interpolizer.interpolate(self._script, inputs=kwargs, outputs=outputs)
      File "/venv/lib/python3.10/site-packages/flytekit/extras/tasks/shell.py", line 82, in interpolate
        return self._Formatter().format(tmpl, **consolidated_args)
      File "/usr/local/lib/python3.10/string.py", line 161, in format
        return self.vformat(format_string, args, kwargs)
      File "/usr/local/lib/python3.10/string.py", line 165, in vformat
        result, _ = self._vformat(format_string, args, kwargs, used_args, 2)
      File "/usr/local/lib/python3.10/string.py", line 205, in _vformat
        obj, arg_used = self.get_field(field_name, args, kwargs)
      File "/usr/local/lib/python3.10/string.py", line 276, in get_field
        obj = getattr(obj, i)

Message:

    'SecretsManager' object has no attribute 'get('aws-access-key', 'AWS_ACCESS_KEY_ID')'

SYSTEM ERROR! Contact platform administrators.
Also it looks like the template syntax has changed flytekit v0.26.0
export AWS_SECRET_ACCESS_KEY='{{.inputs.AWS_SECRET_ACCESS_KEY}}'
this branch
export AWS_ACCESS_KEY_ID='{AWS_ACCESS_KEY_ID}'
is this correct?
k

Ketan (kumare3)

01/27/2022, 9:41 PM
ya, so this is what I was telling you. since the shell task was alpha, we were working through a couple different syntax options
and the new syntax is in the latest beta
it is simpler and makes using variables like using format strings in python
this change was contributed by zymergen and we did this, after some feedback from them and another user
Extremely sorry if this broke you, but we had no idea that you were using it already 🙂
seems like a usage issue - let me see. I think it does not
eval
the code -
'SecretsManager' object has no attribute 'get('aws-access-key', 'AWS_ACCESS_KEY_ID')’
I think i will need to implement a
attribute getter
to support this lookup
j

JP Kosymna

01/27/2022, 9:47 PM
Ok cool, no problem. One issue with the new syntax could be collisions in input names, ctx, and output variable names. Who wins if I have an input and an output named ctx ?
I have a work around now by having a different task return the secrets and then the workflow passes them in using inputs. So this is not a blocker for me.
k

Ketan (kumare3)

01/27/2022, 9:59 PM
it will not allow you to do that 😞
j

JP Kosymna

01/27/2022, 10:15 PM
I'd vote to namespace both inputs and outputs so there is no chance for collisions. But that's just me.
k

Ketan (kumare3)

01/27/2022, 11:07 PM
hmm
sure