#2272 [BUG] [flytekit] ShellTask example not working Issue created by myz540 Describe the bug Name: flytekit Version: 0.31.0 macOS Big Sur 11.6.2 Python 3.7.6 Following the example shown here, we encounter the following error when running locally.

==============================================


    set -ex
    echo "Hey there! Let's run some bash scripts using Flyte's ShellTask."
    echo "Showcasing Flyte's Shell Task." >> /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt
    if grep "Flyte" /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt
    then
        echo "Found it!" >> /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt
    else
        echo "Not found!"
    fi


==============================================

+ echo 'Hey there! Let'\''s run some bash scripts using Flyte'\''s ShellTask.'
Hey there! Let's run some bash scripts using Flyte's ShellTask.
+ echo 'Showcasing Flyte'\''s Shell Task.'
+ grep Flyte /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt
Showcasing Flyte's Shell Task.
+ echo 'Found it!'
Traceback (most recent call last):
  File "shell.py", line 98, in <module>
    print(f"Running wf() {wf()}")
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/workflow.py", line 237, in __call__
    return flyte_entity_call_handler(self, *args, **input_kwargs)
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/promise.py", line 991, in flyte_entity_call_handler
    result = cast(LocallyExecutable, entity).local_execute(child_ctx, **kwargs)
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/workflow.py", line 252, in local_execute
    function_outputs = self.execute(**kwargs)
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/workflow.py", line 685, in execute
    return exception_scopes.user_entry_point(self._workflow_function)(**kwargs)
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/exceptions/scopes.py", line 198, in user_entry_point
    return wrapped(*args, **kwargs)
  File "shell.py", line 91, in wf
    t1_out = t1(x=x)
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/base_task.py", line 288, in __call__
    return flyte_entity_call_handler(self, *args, **kwargs)
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/promise.py", line 984, in flyte_entity_call_handler
    return cast(LocallyExecutable, entity).local_execute(ctx, **kwargs)
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/base_task.py", line 269, in local_execute
    outputs_literal_map = self.dispatch_execute(ctx, input_literal_map)
  File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/base_task.py", line 520, in dispatch_execute
    raise TypeError(f"Output({k}) in task{self.name} received a tuple {v}, instead of {py_type}")
TypeError: Output(i) in tasktask_1 received a tuple (/tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt, /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt), instead of <class 'flytekit.types.file.file.FlyteFile'>

Task 1 is apparently returning a tuple of length 2, with the two items being identical (the same file). We expect a single output from this shell task. We have already investigated this issue briefly and believe the cause is here If we add print statements, we see that

self._output_locs

has length 1 as expected and we can see that

self._output_locs

contains:

[OutputLocation(var='i', var_type=<class 'flytekit.types.file.file.FlyteFile'>, location='{inputs.x}')]
<class 'flytekit.extras.tasks.shell.OutputLocation'>

However, the two if statements both resolve as

True

. I believe this a result of the metaprogramming being used and the metaclasses that

FlyteFile

and

FlyteDirectory

are inheriting from. I am new to the code base so it is not immediately obvious what the solution would be but using

issubclass()

is not sufficient for determining if the

var_type

is

FlyteFile

or

FlyteDirectory

. Perhaps a simple solution would be:

for v in self._output_locs:
            if v.var_type is FlyteFile:
                final_outputs.append(FlyteFile(outputs[v.var]))
            elif v.var_type is FlyteDirectory:
                final_outputs.append(FlyteDirectory(outputs[v.var]))

EDIT UPDATE: Discovered this issue occurs on python 3.7.6 as well as 3.8.0 but not in 3.8.5 3.8.0

Python 3.8.0 (default, Nov  6 2019, 15:49:01)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.1.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from flytekit.types.directory import FlyteDirectory
   ...:
   ...: from flytekit.types.file import FlyteFile

In [2]: issubclass(FlyteDirectory, FlyteFile)
Out[2]: True

In [3]: issubclass(FlyteFile, FlyteDirectory)
Out[3]: True

3.8.5

Python 3.8.5 (default, Sep  4 2020, 02:22:02)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.1.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from flytekit.types.directory import FlyteDirectory
   ...:
   ...: from flytekit.types.file import FlyteFile

In [2]: issubclass(FlyteDirectory, FlyteFile)
Out[2]: False

In [3]: issubclass(FlyteFile, FlyteDirectory)
Out[3]: False

EDIT UPDATE 2 Issue looks to have been raised and patched in 3.7.7 and 3.8.2 but restricts usage of

ShellTask

(in its current state) to those versions https://bugs.python.org/issue38878 Expected behavior The expected behavior is for the example

ShellTasks

and

workflow

to actually execute and run to completion Additional context to reproduce 1. Install python 3.7.6 2. Copy the example shell tasks and workflow into a file:

shell.py

3. Run locally using

python shell.py

Screenshots ``` ============================================== set -ex echo "Hey there! Let's run some bash scripts using Flyte's ShellTask." echo "Showcasing Flyte's Shell Task." >> /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt if grep "Flyte" /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt then echo "Found it!" >> /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt else echo "Not found!" fi ============================================== + echo 'Hey there! Let'\''s run some bash scripts using Flyte'\''s ShellTask.' Hey there! Let's run some bash scripts using Flyte's ShellTask. + echo 'Showcasing Flyte'\''s Shell Task.' + grep Flyte /tmp/flyte/20220317_112528/raw/cfa6f2d613e427d0e0ff8396bd830539/test.txt Showcasing Flyte's Shell Task. + echo 'Found it!' Traceback (most recent call last): File "shell.py", line 98, in <module> print(f"Running wf() {wf()}") File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/workflow.py", line 237, in call return flyte_entity_call_handler(self, *args, **input_kwargs) File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/promise.py", line 991, in flyte_entity_call_handler result = cast(LocallyExecutable, entity).local_execute(child_ctx, **kwargs) File "/Users/mikezhong/opt/anaconda3/envs/pipeline/lib/python3.7/site-packages/flytekit/core/wo… flyteorg/flyte