JP Kosymna

    JP Kosymna

    7 months ago
    Hey all. I'm having trouble using a ShellTask with secrets. In the following task I would expect an environment variable possibly named
    AWS-ACCESS-KEY_AWS_ACCESS_KEY_ID
    to exist but it doesn't. What am I doing wrong here?
    task_with_secrets = ShellTask(
        debug=True,
        name="task_with_secrets",
        script='''
        set -ex
        env > out_env
        ''',
        inputs=kwtypes(source=str),
        output_locs=[
            OutputLocation(var="x", var_type=FlyteFile, location="out_env")
        ],
        secret_requests=[
            Secret(
                group='aws-access-key',
                key='AWS_ACCESS_KEY_ID',
                mount_requirement=Secret.MountType.ENV_VAR
            ),
            Secret(
                group='aws-access-key',
                key='AWS_SECRET_ACCESS_KEY',
                mount_requirement=Secret.MountType.ENV_VAR
            ),
        ]
    )
    Ketan (kumare3)

    Ketan (kumare3)

    7 months ago
    ohhh i think we are not injecting this env variable
    i think it should be available as the context var
    ohh thank you for trying the shell task
    JP Kosymna

    JP Kosymna

    7 months ago
    Is there a way to access the context var in the ShellTask?
    Ketan (kumare3)

    Ketan (kumare3)

    7 months ago
    but should be a trivial change
    do you mind opening an issue?
    i can push in a change soon if you would like
    JP Kosymna

    JP Kosymna

    7 months ago
    Sure that would be awesome.
    Ketan (kumare3)

    Ketan (kumare3)

    7 months ago
    also the shell task is updated now - Zach from Zymergen contributed it.
    Hi JP, once you create an issue, I can check this in - https://github.com/flyteorg/flytekit/pull/832
    Also let me know if this works
    JP Kosymna

    JP Kosymna

    7 months ago
    So I am able to run
    export X='{ctx.execution_id}'
    , but this fails
    export y="{ctx.secrets.get('aws-access-key', 'AWS_ACCESS_KEY_ID')}"
    with
    [3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):
    
          File "/venv/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
            return wrapped(*args, **kwargs)
          File "/venv/lib/python3.10/site-packages/flytekit/core/base_task.py", line 481, in dispatch_execute
            raise e
          File "/venv/lib/python3.10/site-packages/flytekit/core/base_task.py", line 478, in dispatch_execute
            native_outputs = self.execute(**native_inputs)
          File "/venv/lib/python3.10/site-packages/flytekit/extras/tasks/shell.py", line 191, in execute
            gen_script = self._interpolizer.interpolate(self._script, inputs=kwargs, outputs=outputs)
          File "/venv/lib/python3.10/site-packages/flytekit/extras/tasks/shell.py", line 82, in interpolate
            return self._Formatter().format(tmpl, **consolidated_args)
          File "/usr/local/lib/python3.10/string.py", line 161, in format
            return self.vformat(format_string, args, kwargs)
          File "/usr/local/lib/python3.10/string.py", line 165, in vformat
            result, _ = self._vformat(format_string, args, kwargs, used_args, 2)
          File "/usr/local/lib/python3.10/string.py", line 205, in _vformat
            obj, arg_used = self.get_field(field_name, args, kwargs)
          File "/usr/local/lib/python3.10/string.py", line 276, in get_field
            obj = getattr(obj, i)
    
    Message:
    
        'SecretsManager' object has no attribute 'get('aws-access-key', 'AWS_ACCESS_KEY_ID')'
    
    SYSTEM ERROR! Contact platform administrators.
    Also it looks like the template syntax has changed flytekit v0.26.0
    export AWS_SECRET_ACCESS_KEY='{{.inputs.AWS_SECRET_ACCESS_KEY}}'
    this branch
    export AWS_ACCESS_KEY_ID='{AWS_ACCESS_KEY_ID}'
    is this correct?
    Ketan (kumare3)

    Ketan (kumare3)

    7 months ago
    ya, so this is what I was telling you. since the shell task was alpha, we were working through a couple different syntax options
    and the new syntax is in the latest beta
    it is simpler and makes using variables like using format strings in python
    this change was contributed by zymergen and we did this, after some feedback from them and another user
    Extremely sorry if this broke you, but we had no idea that you were using it already 🙂
    seems like a usage issue - let me see. I think it does not
    eval
    the code -
    'SecretsManager' object has no attribute 'get('aws-access-key', 'AWS_ACCESS_KEY_ID')’
    I think i will need to implement a
    attribute getter
    to support this lookup
    JP Kosymna

    JP Kosymna

    7 months ago
    Ok cool, no problem. One issue with the new syntax could be collisions in input names, ctx, and output variable names. Who wins if I have an input and an output named ctx ?
    I have a work around now by having a different task return the secrets and then the workflow passes them in using inputs. So this is not a blocker for me.
    Ketan (kumare3)

    Ketan (kumare3)

    7 months ago
    it will not allow you to do that 😞
    JP Kosymna

    JP Kosymna

    7 months ago
    I'd vote to namespace both inputs and outputs so there is no chance for collisions. But that's just me.
    Ketan (kumare3)

    Ketan (kumare3)

    7 months ago
    hmm
    sure