seunggs
11/23/2022, 2:16 PMEduardo Apolinario (eapolinario)
11/23/2022, 10:57 PMimport typing
from flytekit import task, workflow, dynamic
my_tuple = typing.NamedTuple("A", b=str, c=int)
@task
def t1() -> my_tuple:
return my_tuple(b="hello world", c=42)
@task
def t3(b: str, c: int):
print(f"{b} - {c}")
@workflow
def wf_valid():
res = t1()
t3(b=res.b, c=res.c)
Notice how we have to access the values separately in the invocation of the downstream t3
. In other words, we couldn't have NamedTuple
as an input of t3
.
In your example, you're probably using the result of calling train_task
as an input to another task, right? Can you share how that's happening in your case?seunggs
11/23/2022, 11:01 PM_wf_outputs=typing.NamedTuple("WfOutputs",train_task_o0=flytekit.types.file.file.FlyteFile)
@workflow
def mnist(_wf_args:Hyperparameters)->_wf_outputs:
train_task_o0_=train_task(hp=_wf_args)
return _wf_outputs(train_task_o0_)
Eduardo Apolinario (eapolinario)
11/23/2022, 11:12 PM@workflow
def mnist(_wf_args:Hyperparameters)->_wf_outputs:
return train_task(hp=_wf_args)
seunggs
11/24/2022, 12:05 AMEduardo Apolinario (eapolinario)
11/24/2022, 12:05 AMtrain_task
already returns a named tupleseunggs
11/24/2022, 12:06 AMEduardo Apolinario (eapolinario)
11/24/2022, 12:07 AM_wf_outputs=typing.NamedTuple("WfOutputs",train_task_o0=flytekit.types.file.file.FlyteFile)
@workflow
def mnist(_wf_args:Hyperparameters)->_wf_outputs:
x = train_task(hp=_wf_args)
return _wf_outputs(hp=x.hp)
train_task
you return a dataclass)seunggs
11/24/2022, 12:11 AMEduardo Apolinario (eapolinario)
11/24/2022, 12:12 AMseunggs
11/24/2022, 12:12 AMEduardo Apolinario (eapolinario)
11/24/2022, 12:13 AMtrain_task
work?seunggs
11/24/2022, 12:14 AMtorch>=1.13.0
torchvision>=0.14.0
pytorch_lightning>=1.8.1
flytekit>=1.2.3
matplotlib>=3.6.2
FROM python:3.8-slim-buster
WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root
RUN apt-get update && apt-get install -y build-essential curl
# Install pip-tools
RUN pip3 install pip-tools
# Install the AWS cli separately to prevent issues with boto being written over
RUN pip3 install awscli
# Install flytectl
RUN curl -sL <https://ctl.flyte.org/install> | bash
ENV PATH="/root/bin:$PATH"
ENV VENV /opt/venv
# Virtual environment
RUN python3 -m venv ${VENV}
ENV PATH="${VENV}/bin:$PATH"
# Compile source dependencies (i.e. <http://requirements.in|requirements.in>) to requirements.txt and then use that to install Python dependencies
COPY ./requirements.in /root
RUN pip-compile --output-file=/root/requirements.txt /root/requirements.in
# --no-cache-dir to prevent OOMKilled
RUN pip install --no-cache-dir -r /root/requirements.txt
# Copy the actual code
COPY . /root
# Init flytectl to use the correct remote host
RUN flytectl config init --host=<https://flyte.sidetrek.com>
# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows, and launch plans
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag
Eduardo Apolinario (eapolinario)
11/29/2022, 4:39 AMdocker history <image_tag>
to have a sense of which step is taking space? I have a feeling that all the pip operations you're running while building the image are the culprit.