Hello 👋 I'm trying to get my first remote Flyte run to complete in a custom Docker containers. In the container, I've installed flytekit using Poetry, but when the task runs I'm getting this error:

[ah9fgw2d68t8zpflfkvb-n0-0] terminated with exit code (2). Reason [Error]. Message: 
Usage: pyflyte-fast-execute [OPTIONS] [TASK_EXECUTE_CMD]...
Try 'pyflyte-fast-execute --help' for help.

Error: No such option: --inputs
.

Reading through the entrypoint code here, it looks like

--inputs

is intended for the subsequent call to

pyflyte-execute

, but the first

pyflyte-fast-execute

call is raising an error before it gets that far due to the unknown option. According to the click docs,

ignore_unknown_options=True

should be set to allow passing through unknown options, but I don't see that being set anywhere. On the other hand, when I run using the default Flyte Docker container, the same command runs totally fine. Do I need to make sure to install a particular version of

click

in my container, or do some other setup, in order to make it work?

Why would you install

flytekit

with poetry instead of

uv

or

pip

?

We've been using Poetry for a while, certainly looking to move to

uv

nowadays, it looks great

I can share some brief snippets of what has worked for me to run with custom containers: Workflow example

import pandas as pd
from flytekit import task, workflow

image_spec = "{{.image.example_image.fqn}}:{{.image.example_image.version}}"


@task(container_image=image_spec)
def count_rows(data: pd.DataFrame) -> int:
    """Count rows in DataFrame."""
    return data.shape[0]


@workflow
def pipeline(data: pd.DataFrame) -> int:
    """Pipeline."""
    rows_ct = count_rows(data)

    return rows_ct

With a Dockerfile

# Use an official miniconda3 as a parent image
FROM continuumio/miniconda3

WORKDIR /root

# Copy requirements into the container
COPY requirements.txt /root/requirements.txt

# Install requirements
RUN pip install uv==0.4.18
RUN uv pip install -r /root/requirements.txt --system --no-cache-dir

With requirements

flytekit==1.14.1
pandas==2.2.1
pyarrow==18.1.0
fastparquet==2024.11.0
numpy==1.26.4

And configuration

admin:
  ...
images:
  example_image: localhost:30000/example-image:latest

Running with:

pyflyte --config ~/.flyte/config.yaml run --remote --copy all -p <PROJECT> -d <DOMAIN> workflows/<SCRIPT>.py pipeline --data s3://<INPUT>

@stale-artist-94777, thank you for figuring it out. We should document it. I opened https://github.com/flyteorg/flyte/issues/6119 to track this. Feel free to submit a PR.

@thankful-minister-83577 can you help file this to make sure it works with image builder

@stale-artist-94777 mind sending me your original dockerfile (redacted however you need)?

Sure, this is working, slightly cut down:

FROM pytorch/pytorch:2.1.2-cuda11.8-cudnn8-devel

# Install system dependencies
RUN apt-get update --fix-missing && \
    apt-get install -y git python3-pip && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Install Python dependencies before copying the rest of the code
WORKDIR /app
COPY poetry.lock pyproject.toml /app/
RUN pip --disable-pip-version-check install "poetry==1.5.1" awscli && \
    poetry config virtualenvs.create true && \
    poetry config virtualenvs.in-project true && \
    poetry config virtualenvs.options.always-copy true && \
    poetry install --no-interaction --no-ansi

# Copy the rest of the code
COPY . /app

# Set up a generic entrypoint so we can run arbitrary commands in the poetry environment
# Pass an argument separator so that poetry doesn't consume any further argument separators
ENTRYPOINT ["poetry", "run", "--"]

I've also ran into #4853, as the code in the container was being used instead of the local code when I tried a remote

map_tasks

, and an issue where passing Pydantic objects from an

@dynamic

workflow to a

@task

fails on remote with an error like:

(type [simple:BINARY]) doesn't match expected type [simple:STRUCT ...]

(works locally, I've not found a ticket for that yet) So a few things work out before I get into really migrating my workflow over!

mind if i copy this into the issue as well? just for posterity/others.