#CP2HDHKE1 Hello All, I am trying to install custom packages into the flyte container. In my build job prior to flyte package I am installing numpy which is 1.24.3. But after packaging when i trigger the workflow the version which is shown to me in workflow output is 1.20.0 Another example is I am installing transformers python package but in the workflow the same says module not found. I saw that in kubeflow I can install custom packages using

add_op = create_component_from_func(

with package_to_install option. Is there a Flyte way of doing this

The

ImageSpec

feature which we'll ship in flytekit 1.6 should let you specify the packages in flytekit code, and you don't have to think about building and pushing docker images.

Another example is I am installing transformers python package but in the workflow the same says module not found.

Have you specified the package in your dockerfile?

We have a docker process which basically builds the images with the requirements.txt which has the custom packages. Once the image is built then flytectl pkg option pushes them to flyte and then I trigger from UI

Are you running

pyflyte --pkgs

and

flytectl register

?

yes exactly

It should work! Can you try running the commands manually?

I am using First generate the project docker image then, shell.run_command( f"export PYTHONPATH=/workspace/{project_dir}:/workspace/{project_dir}/{project_dir} && " f"{pyflyte_bin} " f"-c ~/.flyte/config " f"--pkgs {project_dir} " f"package " f"--image {DOCKER_REGISTRY_NAME}/{container_name}:{workflow_version} " f"-f" ) and then shell.run_command( f"flytectl register files " f"-p {project} -d {domain} " f"--archive flyte-package.tgz " f"--version {workflow_version} && " f"rm flyte-package.tgz" )

btw, @tall-lock-23197 is there any way to test ImageSpec already? nightly builds/etc? 🙂

@stocky-notebook-88311, can you double check if you're sending the right image to the package command? @miniature-zebra-33903, release: https://github.com/flyteorg/flytekit/releases/tag/v1.6.0b0; docs: https://flyte--988.org.readthedocs.build/projects/cookbook/en/988/auto/core/containerization/image_sepc_example.html

The right image is being sent to package.....there are other code pieces also which are coming fine in the new image, issue is only with the packages... Do the packages have to be installed in the environment from which we are issuing the package command by the way???

Yeah. It won't work otherwise.

This is the first time we are trying with not having the packages in the environment where the packaging command is working.

Um, it should error out. Not sure why it's working.