Mücahit
05/10/2023, 9:46 PMpyflyte package
it fails because it does not have all the dependencies --> flytekitplugins-spark + tensorflow.
I thought by specifying container_image
differently for those tasks and importing those modules within the task context was magically solving this problem but not? Do we need to install all python source code dependencies when we run pyflyte package
command?Kevin Su
05/10/2023, 10:09 PMMücahit
05/10/2023, 10:19 PMKevin Su
05/10/2023, 10:24 PMpyflyte run
, it will build the image first if image not found.
• it uses docker under the hood, so I think it should support private pip as well. I haven’t try it, but let me know if you run into any issue.
• yes, you can specify package version. like`pandas==1,4.0`Mücahit
05/10/2023, 10:25 PMKevin Su
05/12/2023, 4:48 PMMücahit
05/12/2023, 8:20 PMImage flytekit:_CB4mqOFLHJIgtJSHe9hyQ.. not found. Building...
Run command: envd build --path /var/folders/l5/b5c4dzc12n3cd4pbyrhkwxd40000gp/T/flyte-3wc4a3dt/sandbox/local_flytekit/833f8fa328235df82b6a50a4958bb4fd
Kevin Su
05/12/2023, 8:30 PMMücahit
05/12/2023, 9:10 PM1.6.0b4
and
tensorflow_image_spec=ImageSpec(
base_image="tensorflow/tensorflow:2.12.0-gpu",
python_version="3.9"
)
if image_spec.registry:
command += f" --output type=image,name={image_spec.image_name()},push=true"
I’ll add it and see