sticky-angle-28419
11/23/2022, 2:16 PMhigh-accountant-32689
11/23/2022, 10:57 PMimport typing
from flytekit import task, workflow, dynamic
my_tuple = typing.NamedTuple("A", b=str, c=int)
@task
def t1() -> my_tuple:
return my_tuple(b="hello world", c=42)
@task
def t3(b: str, c: int):
print(f"{b} - {c}")
@workflow
def wf_valid():
res = t1()
t3(b=res.b, c=res.c)
Notice how we have to access the values separately in the invocation of the downstream t3
. In other words, we couldn't have NamedTuple
as an input of t3
.
In your example, you're probably using the result of calling train_task
as an input to another task, right? Can you share how that's happening in your case?sticky-angle-28419
11/23/2022, 11:01 PMsticky-angle-28419
11/23/2022, 11:02 PM_wf_outputs=typing.NamedTuple("WfOutputs",train_task_o0=flytekit.types.file.file.FlyteFile)
@workflow
def mnist(_wf_args:Hyperparameters)->_wf_outputs:
train_task_o0_=train_task(hp=_wf_args)
return _wf_outputs(train_task_o0_)
high-accountant-32689
11/23/2022, 11:12 PM@workflow
def mnist(_wf_args:Hyperparameters)->_wf_outputs:
return train_task(hp=_wf_args)
high-accountant-32689
11/23/2022, 11:57 PMsticky-angle-28419
11/24/2022, 12:05 AMsticky-angle-28419
11/24/2022, 12:05 AMhigh-accountant-32689
11/24/2022, 12:05 AMtrain_task
already returns a named tuplesticky-angle-28419
11/24/2022, 12:06 AMsticky-angle-28419
11/24/2022, 12:06 AMhigh-accountant-32689
11/24/2022, 12:07 AM_wf_outputs=typing.NamedTuple("WfOutputs",train_task_o0=flytekit.types.file.file.FlyteFile)
@workflow
def mnist(_wf_args:Hyperparameters)->_wf_outputs:
x = train_task(hp=_wf_args)
return _wf_outputs(hp=x.hp)
high-accountant-32689
11/24/2022, 12:08 AMhigh-accountant-32689
11/24/2022, 12:08 AMhigh-accountant-32689
11/24/2022, 12:09 AMtrain_task
you return a dataclass)sticky-angle-28419
11/24/2022, 12:11 AMsticky-angle-28419
11/24/2022, 12:11 AMhigh-accountant-32689
11/24/2022, 12:12 AMsticky-angle-28419
11/24/2022, 12:12 AMhigh-accountant-32689
11/24/2022, 12:13 AMtrain_task
work?sticky-angle-28419
11/24/2022, 12:14 AMsticky-angle-28419
11/24/2022, 12:14 AMsticky-angle-28419
11/26/2022, 2:53 AMsticky-angle-28419
11/26/2022, 2:54 AMsticky-angle-28419
11/26/2022, 2:54 AMtorch>=1.13.0
torchvision>=0.14.0
pytorch_lightning>=1.8.1
flytekit>=1.2.3
matplotlib>=3.6.2
sticky-angle-28419
11/26/2022, 2:56 AMsticky-angle-28419
11/26/2022, 2:56 AMsticky-angle-28419
11/26/2022, 2:59 AMsticky-angle-28419
11/26/2022, 3:00 AMFROM python:3.8-slim-buster
WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root
RUN apt-get update && apt-get install -y build-essential curl
# Install pip-tools
RUN pip3 install pip-tools
# Install the AWS cli separately to prevent issues with boto being written over
RUN pip3 install awscli
# Install flytectl
RUN curl -sL <https://ctl.flyte.org/install> | bash
ENV PATH="/root/bin:$PATH"
ENV VENV /opt/venv
# Virtual environment
RUN python3 -m venv ${VENV}
ENV PATH="${VENV}/bin:$PATH"
# Compile source dependencies (i.e. <http://requirements.in|requirements.in>) to requirements.txt and then use that to install Python dependencies
COPY ./requirements.in /root
RUN pip-compile --output-file=/root/requirements.txt /root/requirements.in
# --no-cache-dir to prevent OOMKilled
RUN pip install --no-cache-dir -r /root/requirements.txt
# Copy the actual code
COPY . /root
# Init flytectl to use the correct remote host
RUN flytectl config init --host=<https://flyte.sidetrek.com>
# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows, and launch plans
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag
high-accountant-32689
11/29/2022, 4:39 AMdocker history <image_tag>
to have a sense of which step is taking space? I have a feeling that all the pip operations you're running while building the image are the culprit.