KS Tarun
09/29/2022, 6:44 AMModuleNotFoundError
, though that module was installed when creating the docker image.
And this is getting resolved when I pip install
that module in the instance.
So, please clarify whether on running a workflow, would flyte propeller look for the dependencies in the specified docker container or it tries to use the dependencies from the server it is running in ?Samhita Alla
09/29/2022, 7:13 AMKS Tarun
09/29/2022, 7:16 AMfrom ubuntu:focal
WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root
RUN : \
&& apt-get update \
&& apt install -y software-properties-common \
&& add-apt-repository ppa:deadsnakes/ppa
RUN : \
&& apt-get update \
&& apt-get install -y python3.8 python3-pip python3-venv make build-essential libssl-dev curl vim
RUN apt-get -y update
RUN apt-get -y install git
# This is necessary for opencv to work
RUN apt-get update && apt-get install -y libsm6 libxext6 libxrender-dev ffmpeg
# Install the AWS cli separately to prevent issues with boto being written over
RUN pip3 install awscli
WORKDIR /opt
RUN curl <https://sdk.cloud.google.com> > install.sh
RUN bash /opt/install.sh --install-dir=/opt
ENV PATH $PATH:/opt/google-cloud-sdk/bin
WORKDIR /root
# Virtual environment
ENV VENV /opt/venv
RUN python3 -m venv ${VENV}
ENV PATH="${VENV}/bin:$PATH"
# Install Python dependencies
COPY requirements.txt /root
RUN ${VENV}/bin/pip install -r /root/requirements.txt
# Copy the makefile targets to expose on the container. This makes it easier to register.
# COPY <http://in_container.mk|in_container.mk> /root/Makefile
COPY sandbox.config /root
# Copy over the helper script that the SDK relies on
RUN cp ${VENV}/bin/flytekit_venv /usr/local/bin/
RUN chmod a+x /usr/local/bin/flytekit_venv
# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows, and launch plans
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag
requirements.txt:
flytekit
awscli
pymysql
scikit-learn
xgboost
sqlalchemy
Samhita Alla
09/29/2022, 7:17 AMKS Tarun
09/29/2022, 7:20 AMModuleNotFoundError
Samhita Alla
09/29/2022, 7:29 AMKS Tarun
09/29/2022, 7:30 AMSamhita Alla
09/29/2022, 7:34 AMKS Tarun
09/29/2022, 7:38 AMSamhita Alla
10/03/2022, 11:58 AMKS Tarun
10/03/2022, 12:00 PMpyflyte --config ~/.flyte/config-sandbox.yaml register --image fibo:1.0 Test1.py
The image was created inside the flyte-sandbox.Samhita Alla
10/03/2022, 12:25 PMKS Tarun
10/05/2022, 9:27 AMKevin Su
10/05/2022, 2:21 PMRUN ${VENV}/bin/pip install -r /root/requirements.txt
to pip install -r /root/requirements.txt
Samhita Alla
10/06/2022, 6:16 AM