https://flyte.org logo
Join Slack
Powered by
hello everyone, new to flyte here. I'm running int...
# flyte-support
c
hello everyone, new to flyte here. I'm running into the problem trying to execute a remote workflow. Here's the error
Copy code
[1/1] currentAttempt done. Last Error: USER::
[afgfvc5lc9mlpgrqkwst-n0-0] terminated with exit code (255). Reason [Error]. Message: 
exec /opt/venv/bin/pyflyte-fast-execute: exec format error
.
The workflow is a simple hello world with a bs4 import to test that the dependencies are available to the flyte job. We have a docker image that we built and pushed to an ACR and the flyte job is able to pull from it. What I've found in searching has been to ensure that the architeture of the kubernetes nodes matches the docker image, which i verified with 
docker inspect
and 
kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.architecture}'
to find that they're both amd64. I have tried running this on a separate docker image with flytekit installed that i know is being used in prod and running into the same issue, so my concern is that it something with my local environment or workflow that's causing it. Here's my entire workflow script
Copy code
import bs4
from flytekit import workflow, task


@task
def dummy_task():
    return "Hello, world!"

@workflow
def dummy_workflow():
    result = dummy_task()
    return result
a
@cold-teacher-46250 yes seems a mismatch in base OS arch. Have you considered ImageSpec? You can specify the target arch (example)
c
happy to give that a shot, but that's just another way of determining the image architecture right? What should i be trying to match up the architecture with? running 
kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.architecture}'
produces 
amd64 amd64 amd64 amd64
and running 
docker inspect <image>
produces 
"Architecture": "amd64",
as a field just for sanity's sake I also rebuilt the docker image specifying the architecture using 
docker build --platform linux/amd64
to no avail so what would i try to be matching the target arch with?
So I gave ImageSpec a try
Copy code
image_spec = ImageSpec(
    name="say-hello-image",
    packages=["beautifulsoup4 >= 4.10.0"],
    platform="linux/amd64",
    registry=registry,
)
and the workflow succeeded - any ideas why the ImageSpec would work when building the docker image from a dockerfile does not? DockerFile in question
Copy code
FROM continuumio/miniconda3:latest as condabase
COPY conda_env.yaml .

RUN --mount=type=secret,id=pip_url,target=/.pip_url \
    pip config set --global global.index-url "$(cat /.pip_url)" &&\
    conda env update -f conda_env.yaml -n base  &&\
    conda clean -afy &&\
    pip cache purge &&\
    pip config unset --global global.index-url


FROM condabase as runtime
COPY src src

WORKDIR src
Ah further discoveries for anyone else who's running into the same issue - the behavior with 
docker build --platform linux/amd64
is only present when the image to the registry where the image:tag already exists. If I were to define a new tag, the image created does not run into the same error. As a sanity check - I rebuilt the docker images with and without the python dependencies, then ran them locally and was able to verify the image with the same tag was in fact updating. However flyte must be doing some sort of caching behind the scenes as the image:tag that had a successful run, when updated to include none of the dependencies, still ran successfully
5 Views
Open in Slack
PreviousNext
Powered by