https://flyte.org logo
Join Slack
Powered by
Is there a plan to enable running ContainerTask lo...
# ask-the-community
j
Is there a plan to enable running ContainerTask locally any time soon? It would be very valuable to if Flyte local and remote mode become more streamlined
k
cc @L godlike
l
Yes, upcoming!
Will prioritize it
j
Great to hear, looking forward to it!
l
Hi, @Jonatan Eriksson Here's the implementation Could you help me try it and leave the screentshots? https://github.com/flyteorg/flytekit/pull/2258
You can direct go to 
how to setup
and test your python raw container task
thank you!
j
Nice, I'll see if I can make it run with my code
l
Thanks, if there’s error please give me your code and I will fix it
@Eric Chang can you help try it too?
e
@L godlike I'll try it tonight, thanks a lot !
l
Thank you both!
e
@L godlike It works, but the first time I ran this example, it takes about 2 min ? or above to finish, this is strange
l
Yes because you need to download the image
Can you test it in other cases?
For example using javascript
e
Maybe we can add some log about downloading image to improve UX.
l
It seems like that’s my example
Ok, I will do it
e
ok I'll test JS
j
I get the following error: Error encountered while executing 'local_test': [Errno 2] No such file or directory: '/tmp/flyte-_60x095u/sandbox/local_flytekit/59ccd24a2257005c4e14a69ed29e0a41/out' This is my code: 
from flytekit import ContainerTask, dynamic, kwtypes
from flytekit.types.file import FlyteFile
task_1 = ContainerTask(
name="task1",
input_data_dir="/var/inputs",
output_data_dir="/var/outputs",
inputs=kwtypes(infile=FlyteFile, param1=str, param2=float),
outputs=kwtypes(out=FlyteFile),
image="my_image1:0.0.1",
command=[
"python",
"test_script1.py",
"/var/inputs/infile",
"/var/outputs/out",
"{{.inputs.param1}}",
"{{.inputs.param2}}",
],
)
task_2 = ContainerTask(
name="task2",
input_data_dir="/var/inputs",
output_data_dir="/var/outputs",
inputs=kwtypes(infile=FlyteFile, param1=str, param2=float),
outputs=kwtypes(out=FlyteFile),
image="my_image2:0.0.1",
command=[
"python",
"test_script2.py",
"/var/inputs/infile",
"/var/outputs/out",
"{{.inputs.param1}}",
"{{.inputs.param2}}",
],
)
@dynamic
def local_test(input_file: FlyteFile) -> FlyteFile:
param1 = "some random string"
param2 = 42.0
file1 = task_1(infile=input_file, param1=param1, param2=param2)
file2 = task_2(infile=file1, param1=param1, param2=param2)
return file2
The examples with calculate_ellipse_area works though!
l
I will try to support blob type
Than you
Thank you!
Can you also provide the dockerfile of your image?
oh I saw the example
you don't need to provide docker image, I know how to do that
j
Great, let me know when to try again!
l
It seems that for List and Dict, Raw Container doesn't support it as input
but I've already supported these types 
int, str, float, bool, datetime.datetime, FlyteDirectory and FlyteFile
Can you help me take a look, and if looks good, approve it?
@Eric Chang @Jonatan Eriksson
Thank you both ❤️
j
The example you provided works. However, the following example, which works for remote, does not work locally for me:
Copy code
from flytekit import ContainerTask, kwtypes, workflow
from flytekit.types.file import FlyteFile

python_return_same_values = ContainerTask(
    name="python_return_same_values",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    outputs=kwtypes(out=FlyteFile),
    image="myimage:1",
    command=[
        "python",
        "myscript.py",
        "/var/outputs/out",
    ],
)

@workflow
def wf() -> FlyteFile:
    return python_return_same_values()


if __name__ == "__main__":
    print(wf())
and myscript.py
Copy code
import sys
from pathlib import Path


def write_output(out_path: Path):
    out_path.write_text("dgklsjljs")


if __name__ == "__main__":
    write_output(Path(sys.argv[1]))
And Dockerfile:
Copy code
FROM python:3.9-slim-bullseye

WORKDIR /root
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root

# Virtual environment
ENV VENV /opt/venv
RUN python3 -m venv ${VENV}
ENV PATH="${VENV}/bin:$PATH"


COPY myscript.py /root/myscript.py

# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows, and launch plans
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag
l
Thank you, will fix it
I think copilot has bug when we use 
Flytefile
as input
your case i've fixed, will support 
FlyteDirectroy
too and push it
Hi, I've supported the case you want
thank you very much ❤️
j
Hmm my last example works now. But my original one still doesn't. It's basically the same except that it runs two container tasks in sequence where the output file from the first is the input to the second
l
can you provied 
test_script1.py
and 
test_script2.py
?
will fix it tomorrow
j
Copy code
from flytekit import ContainerTask, kwtypes, workflow
from flytekit.types.file import FlyteFile

task1 = ContainerTask(
    name="task1",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    inputs=kwtypes(infile=FlyteFile, str_param=str, float_param=float),
    outputs=kwtypes(out=FlyteFile),
    image="docker:1",
    command=[
        "python",
        "script1.py",
        "/var/inputs/infile",
        "/var/outputs/out",
        "{{.inputs.str_param}}",
        "{{.inputs.float_param}}",
    ],
)

task2 = ContainerTask(
    name="task2",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    inputs=kwtypes(infile=FlyteFile, str_param=str, float_param=float),
    outputs=kwtypes(out=FlyteFile),
    image="docker:2",
    command=[
        "python",
        "script2.py",
        "/var/inputs/infile",
        "/var/outputs/out",
        "{{.inputs.str_param}}",
        "{{.inputs.float_param}}",
    ],
)


@workflow
def wf(input_file: FlyteFile) -> FlyteFile:
    str_param = "..."
    float_param = 25.0

    file1 = task1(infile=input_file, str_param=str_param, float_param=float_param)
    file2 = task2(infile=file1, str_param=str_param, float_param=float_param)

    return xml_file  # type: ignore
l
Thank you appreciate it 🙏🏻
Currenlty, raw container task will trim string with white space as an input for example, input like 
ss ss
, will be 
ss
. does this cause the bug?
can you also provide 
script1.py
and 
script2.py
? Since I've tested 
flytefile
as input and output and both case works for me, so I think there's a little bit chance that the example is wrong, or please confirm that it works remotely
j
This example and the previous one work remotely. The problem seems to be that I can't access the input file properly inside the container task. 
pyflyte run test_wf.py run_test --file test_wf.py
produces the following output: 
input file exists: False
test_wf.py:
Copy code
from pathlib import Path

from flytekit import ContainerTask, dynamic, kwtypes, task
from flytekit.types.file import FlyteFile

test_ct = ContainerTask(
    name="ct1",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    inputs=kwtypes(infile=FlyteFile),
    outputs=kwtypes(out=FlyteFile),
    image="myimage:0.1",
    command=[
        "python",
        "script1.py",
        "/var/inputs/infile",
        "/var/outputs/out",
    ],
)


@task
def to_str(file: FlyteFile) -> str:
    f = Path(file.download())
    return f.read_text()


@dynamic
def run_test(file: FlyteFile) -> str:
    out_file = test_ct(infile=file)
    return to_str(file=out_file)
script1.py:
Copy code
from pathlib import Path
from sys import argv


def check_exists(input_path: Path, output_path: Path) -> None:
    output_path.write_text(f"input file exists: {input_path.exists()}")


if __name__ == "__main__":
    check_exists(Path(argv[1]), Path(argv[2]))
l
just fix it!
thank you for catching this scneario
I forgot to support 
direct file paths
, but only support 
template-style references
Can you help me try again? Here's my screen shot
image.png
image.png
j
It works now! And my original example works too! Great job!
l
Love it
Really happy this PR helps you
and really nice catch
thank you very very much!
k
great job @L godlike, I’ll ship it ASAP
j
Fantastic!
3 Views
Open in Slack
PreviousNext
Powered by