https://flyte.org logo
#ask-the-community
Title
# ask-the-community
j

Jonatan Eriksson

03/12/2024, 9:07 AM
Is there a plan to enable running ContainerTask locally any time soon? It would be very valuable to if Flyte local and remote mode become more streamlined
k

Kevin Su

03/12/2024, 9:20 AM
cc @L godlike
l

L godlike

03/12/2024, 9:22 AM
Yes, upcoming!
Will prioritize it
j

Jonatan Eriksson

03/12/2024, 11:37 AM
Great to hear, looking forward to it!
l

L godlike

03/13/2024, 10:46 AM
Hi, @Jonatan Eriksson Here's the implementation Could you help me try it and leave the screentshots? https://github.com/flyteorg/flytekit/pull/2258
You can direct go to
how to setup
and test your python raw container task
thank you!
j

Jonatan Eriksson

03/13/2024, 11:27 AM
Nice, I'll see if I can make it run with my code
l

L godlike

03/13/2024, 11:34 AM
Thanks, if there’s error please give me your code and I will fix it
@Eric Chang can you help try it too?
e

Eric Chang

03/14/2024, 3:23 AM
@L godlike I'll try it tonight, thanks a lot !
l

L godlike

03/14/2024, 12:33 PM
Thank you both!
e

Eric Chang

03/15/2024, 2:59 AM
@L godlike It works, but the first time I ran this example, it takes about 2 min ? or above to finish, this is strange
l

L godlike

03/15/2024, 2:59 AM
Yes because you need to download the image
Can you test it in other cases?
For example using javascript
e

Eric Chang

03/15/2024, 3:00 AM
Maybe we can add some log about downloading image to improve UX.
l

L godlike

03/15/2024, 3:00 AM
It seems like that’s my example
Ok, I will do it
e

Eric Chang

03/15/2024, 3:01 AM
ok I'll test JS
j

Jonatan Eriksson

03/15/2024, 3:02 PM
I get the following error: Error encountered while executing 'local_test': [Errno 2] No such file or directory: '/tmp/flyte-_60x095u/sandbox/local_flytekit/59ccd24a2257005c4e14a69ed29e0a41/out' This is my code:
from flytekit import ContainerTask, dynamic, kwtypes
from flytekit.types.file import FlyteFile
task_1 = ContainerTask(
name="task1",
input_data_dir="/var/inputs",
output_data_dir="/var/outputs",
inputs=kwtypes(infile=FlyteFile, param1=str, param2=float),
outputs=kwtypes(out=FlyteFile),
image="my_image1:0.0.1",
command=[
"python",
"test_script1.py",
"/var/inputs/infile",
"/var/outputs/out",
"{{.inputs.param1}}",
"{{.inputs.param2}}",
],
)
task_2 = ContainerTask(
name="task2",
input_data_dir="/var/inputs",
output_data_dir="/var/outputs",
inputs=kwtypes(infile=FlyteFile, param1=str, param2=float),
outputs=kwtypes(out=FlyteFile),
image="my_image2:0.0.1",
command=[
"python",
"test_script2.py",
"/var/inputs/infile",
"/var/outputs/out",
"{{.inputs.param1}}",
"{{.inputs.param2}}",
],
)
@dynamic
def local_test(input_file: FlyteFile) -> FlyteFile:
param1 = "some random string"
param2 = 42.0
file1 = task_1(infile=input_file, param1=param1, param2=param2)
file2 = task_2(infile=file1, param1=param1, param2=param2)
return file2
The examples with calculate_ellipse_area works though!
l

L godlike

03/16/2024, 12:31 AM
I will try to support blob type
Than you
Thank you!
Can you also provide the dockerfile of your image?
oh I saw the example
you don't need to provide docker image, I know how to do that
j

Jonatan Eriksson

03/16/2024, 7:53 AM
Great, let me know when to try again!
l

L godlike

03/18/2024, 11:36 AM
It seems that for List and Dict, Raw Container doesn't support it as input
but I've already supported these types
int, str, float, bool, datetime.datetime, FlyteDirectory and FlyteFile
Can you help me take a look, and if looks good, approve it?
@Eric Chang @Jonatan Eriksson
Thank you both ❤️
j

Jonatan Eriksson

03/19/2024, 1:30 PM
The example you provided works. However, the following example, which works for remote, does not work locally for me:
Copy code
from flytekit import ContainerTask, kwtypes, workflow
from flytekit.types.file import FlyteFile

python_return_same_values = ContainerTask(
    name="python_return_same_values",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    outputs=kwtypes(out=FlyteFile),
    image="myimage:1",
    command=[
        "python",
        "myscript.py",
        "/var/outputs/out",
    ],
)

@workflow
def wf() -> FlyteFile:
    return python_return_same_values()


if __name__ == "__main__":
    print(wf())
and myscript.py
Copy code
import sys
from pathlib import Path


def write_output(out_path: Path):
    out_path.write_text("dgklsjljs")


if __name__ == "__main__":
    write_output(Path(sys.argv[1]))
And Dockerfile:
Copy code
FROM python:3.9-slim-bullseye

WORKDIR /root
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root

# Virtual environment
ENV VENV /opt/venv
RUN python3 -m venv ${VENV}
ENV PATH="${VENV}/bin:$PATH"


COPY myscript.py /root/myscript.py

# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows, and launch plans
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag
l

L godlike

03/20/2024, 1:09 AM
Thank you, will fix it
I think copilot has bug when we use
Flytefile
as input
your case i've fixed, will support
FlyteDirectroy
too and push it
Hi, I've supported the case you want
thank you very much ❤️
j

Jonatan Eriksson

03/21/2024, 4:12 PM
Hmm my last example works now. But my original one still doesn't. It's basically the same except that it runs two container tasks in sequence where the output file from the first is the input to the second
l

L godlike

03/21/2024, 4:17 PM
can you provied
test_script1.py
and
test_script2.py
?
will fix it tomorrow
j

Jonatan Eriksson

03/21/2024, 4:22 PM
Copy code
from flytekit import ContainerTask, kwtypes, workflow
from flytekit.types.file import FlyteFile

task1 = ContainerTask(
    name="task1",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    inputs=kwtypes(infile=FlyteFile, str_param=str, float_param=float),
    outputs=kwtypes(out=FlyteFile),
    image="docker:1",
    command=[
        "python",
        "script1.py",
        "/var/inputs/infile",
        "/var/outputs/out",
        "{{.inputs.str_param}}",
        "{{.inputs.float_param}}",
    ],
)

task2 = ContainerTask(
    name="task2",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    inputs=kwtypes(infile=FlyteFile, str_param=str, float_param=float),
    outputs=kwtypes(out=FlyteFile),
    image="docker:2",
    command=[
        "python",
        "script2.py",
        "/var/inputs/infile",
        "/var/outputs/out",
        "{{.inputs.str_param}}",
        "{{.inputs.float_param}}",
    ],
)


@workflow
def wf(input_file: FlyteFile) -> FlyteFile:
    str_param = "..."
    float_param = 25.0

    file1 = task1(infile=input_file, str_param=str_param, float_param=float_param)
    file2 = task2(infile=file1, str_param=str_param, float_param=float_param)

    return xml_file  # type: ignore
l

L godlike

03/21/2024, 4:23 PM
Thank you appreciate it 🙏🏻
Currenlty, raw container task will trim string with white space as an input for example, input like
ss ss
, will be
ss
. does this cause the bug?
can you also provide
script1.py
and
script2.py
? Since I've tested
flytefile
as input and output and both case works for me, so I think there's a little bit chance that the example is wrong, or please confirm that it works remotely
j

Jonatan Eriksson

03/25/2024, 5:19 PM
This example and the previous one work remotely. The problem seems to be that I can't access the input file properly inside the container task.
pyflyte run test_wf.py run_test --file test_wf.py
produces the following output:
input file exists: False
test_wf.py:
Copy code
from pathlib import Path

from flytekit import ContainerTask, dynamic, kwtypes, task
from flytekit.types.file import FlyteFile

test_ct = ContainerTask(
    name="ct1",
    input_data_dir="/var/inputs",
    output_data_dir="/var/outputs",
    inputs=kwtypes(infile=FlyteFile),
    outputs=kwtypes(out=FlyteFile),
    image="myimage:0.1",
    command=[
        "python",
        "script1.py",
        "/var/inputs/infile",
        "/var/outputs/out",
    ],
)


@task
def to_str(file: FlyteFile) -> str:
    f = Path(file.download())
    return f.read_text()


@dynamic
def run_test(file: FlyteFile) -> str:
    out_file = test_ct(infile=file)
    return to_str(file=out_file)
script1.py:
Copy code
from pathlib import Path
from sys import argv


def check_exists(input_path: Path, output_path: Path) -> None:
    output_path.write_text(f"input file exists: {input_path.exists()}")


if __name__ == "__main__":
    check_exists(Path(argv[1]), Path(argv[2]))
l

L godlike

03/27/2024, 1:44 AM
just fix it!
thank you for catching this scneario
I forgot to support
direct file paths
, but only support
template-style references
Can you help me try again? Here's my screen shot
image.png
image.png
j

Jonatan Eriksson

03/27/2024, 8:34 AM
It works now! And my original example works too! Great job!
l

L godlike

03/27/2024, 8:51 AM
Love it
Really happy this PR helps you
and really nice catch
thank you very very much!
k

Kevin Su

03/27/2024, 4:48 PM
great job @L godlike, I’ll ship it ASAP
j

Jonatan Eriksson

04/01/2024, 8:08 AM
Fantastic!
2 Views