Hi Team, We are trying to automate the Flyte deplo...
# flyte-support
h
Hi Team, We are trying to automate the Flyte deployment using CI/CD and what we have noticed is when we run the or flyte workflow in one of env we are getting following error:
Copy code
RunContainerError: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: \"pyflyte-execute\": executable file not found in $PATH: unknown" logger="UnhandledError"
But when we try to run the same image from local we can see that
pyflyte-execute
is available but when we run in dev server we get error. For ref I have attached our docker file :
Copy code
FROM <http://dockerhubproxy.schibsted.io/python:3.9.18-slim|dockerhubproxy.schibsted.io/python:3.9.18-slim>

# The installer requires curl (and certificates) to download the release archive
RUN apt-get update && apt-get install -y --no-install-recommends curl ca-certificates

# Download the latest installer
ADD <https://astral.sh/uv/install.sh> /uv-installer.sh
# TODO: Specify a version
#ADD <https://astral.sh/uv/0.5.#/install.sh> /uv-installer.sh

# Run the installer then remove it
RUN sh /uv-installer.sh && rm /uv-installer.sh

# Ensure the installed binary is on the `PATH`
ENV PATH="/root/.local/bin/:$PATH"

COPY /flyte /flyte
COPY /tests /tests

ENV WORKDIR=/app
WORKDIR $WORKDIR
ENV PYTHONPATH="$WORKDIR:$WORKDIR/flyte"

ENV PATH="/app/.venv/bin:$PATH"

COPY pyproject.toml /pyproject.toml
COPY create_config.sh /create_config.sh

COPY .github/scripts /.github/scripts
RUN chmod +x /.github/scripts/build_and_push_image.sh

# Run the shell script to create the config-remote.yaml file with an argument
# dev is the default value. Can be overridden during the build process using --build-arg.
ARG FLYTE_ENVIRONMENT=dev
RUN chmod +x /create_config.sh && /create_config.sh ${FLYTE_ENVIRONMENT}

# Install a specific version of flytectl
ENV FLYTECTL_VERSION=v0.9.2
RUN curl -sLo flytectl_Linux_x86_64.tar.gz <https://github.com/flyteorg/flyte/releases/download/flytectl/${FLYTECTL_VERSION}/flytectl_Linux_x86_64.tar.gz> && \
	tar -xzf flytectl_Linux_x86_64.tar.gz && \
	install flytectl /usr/local/bin/ && \
	# Clean up downloaded and extracted files
	rm -rf flytectl_Linux_x86_64.tar.gz flytectl

RUN uv sync --inexact --no-cache-dir
So can anyone help us on how to address this issue. Your input and help will be much appreciated
t
it may have to do with the pythonpath. are you sure it's the right path?
h
We are installing everything using UV so it should be in correct path, we checked the image from local and was able to execute the
pyflyte-execute --help
command from anywhere in the image
This is our pyproject.toml file
Copy code
[project]
name = "my-flyte-workflows"
version = "0.1.0"
description = "My workflow"
readme = "README.md"
requires-python = "~=3.9.17"
dependencies = [
    "flytekit~=1.8.1",
    "kubernetes~=28.1.0",
    "pandas~=1.5.3",
    "numpy~=1.26.4",
    "toml~=0.10.2",
    "datadog~=0.48.0",
    "ddtrace~=2.5.0",
    "pytest>=8.3.3",
    "fire>=0.7.0",
    "ruff>=0.9.0",
]

[project.optional-dependencies]
dbt = [
    "flytekitplugins-dbt~=1.12.0",
    "dbt-core~=1.8.7",
    "dbt-snowflake~=1.8.3",
]
optuna = [
    "psycopg2-binary~=2.9.9"
]
redis = [
    "redis[hiredis]~=5.0.4"
]
s3 = [
    "s3fs~=2024.6.1",
    "boto3~=1.34.70"
]
snowflake = [
    "snowflake-connector-python[pandas]~=3.12.2",
    "PySocks~=1.7.1",
]
visualization = [
    "plotly~=5.20.0",
    "flytekitplugins-deck-standard~=1.11.0",
]

[tool.uv]
dev-dependencies = [
    "jupyterlab>=4.3.1",
]
t
where's pyflyte-execute installed? what's the path?
h
I think that get installed with the flytekit?
t
yes, but we could work backwards by first understanding where it's being installed.
it definitely is weird though that it works locally.
h
Ok let me check
When check in docker image these are the path which i get
Copy code
root@fc2cd75f1d7c:/app# which pyflyte-execute
/app/.venv/bin/pyflyte-execute
root@fc2cd75f1d7c:/app# which flytectl
/usr/local/bin/flytectl
root@fc2cd75f1d7c:/app#
thats a bit strange for me now
/usr/local/bin/flytectl
in here, while
/app/.venv/bin/pyflyte-execute
is in different path
t
that should be okay as pyflyte-execute gets installed with flytekit.
looks like
/app/.venv/bin
is in PATH.
h
yes looks like :
PATH=/root/.local/bin:/app/.venv/bin:/root/.local/bin/:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
t
the same image being used for CI/CD works locally, right?
h
Yes
b
Samhita, what command does flyte execute when running the tasks in a pod? Because, we’re able to pyflyte commands when we tests the docker container locally. Is there a specific
ENTRYPOINT
or another command it does? Also, I thought that the containers themselves do not need to have
pyflyte
installed? I’ve tried to search for docs about this, but have not found anything relevant unfortunately.
Btw, I’m a colleague of Vipul 👋 .
Here is an example from a local
docker run
command with the relevant image:
Copy code
docker run -it uv-test-arm64 pyflyte-execute --help
Usage: pyflyte-execute [OPTIONS] [RESOLVER_ARGS]...

Options:
  --inputs TEXT                  [required]
  --output-prefix TEXT           [required]
  --raw-output-data-prefix TEXT
  --checkpoint-path TEXT
  --prev-checkpoint TEXT
  --test
  --dynamic-addl-distro TEXT
  --dynamic-dest-dir TEXT
  --resolver TEXT
  --help                         Show this message and exit.
I thought that flyte executed something similar when running a
@task
in a pod?
a
> what command does flyte execute when running the tasks in a pod? seems like it's `pyflyte-execute`(ref)
if the container image has flytekit on it, and assuming correct PATH,
pyflyte-execute
should be found
I'm curious, why
flytekit~=1.8.1
?
b
We figured out the issue. A bit difficult to explain, but in short terms, there was an issue with how we set the
WORKDIR
in our
Dockerfile
. Yup, we need to upgrade to a higher
flytekit
version, most def 😊 .
a
Oh thanks for sharing