https://flyte.org logo
#ask-the-community
Title
# ask-the-community
s

Sujith Samuel

05/04/2023, 7:24 AM
#ask-the-community Hello All, I am trying to install custom packages into the flyte container. In my build job prior to flyte package I am installing numpy which is 1.24.3. But after packaging when i trigger the workflow the version which is shown to me in workflow output is 1.20.0 Another example is I am installing transformers python package but in the workflow the same says module not found. I saw that in kubeflow I can install custom packages using
Copy code
add_op = create_component_from_func(
with package_to_install option. Is there a Flyte way of doing this
s

Samhita Alla

05/04/2023, 10:13 AM
The
ImageSpec
feature which we'll ship in flytekit 1.6 should let you specify the packages in flytekit code, and you don't have to think about building and pushing docker images.
Another example is I am installing transformers python package but in the workflow the same says module not found.
Have you specified the package in your dockerfile?
s

Sujith Samuel

05/04/2023, 11:14 AM
We have a docker process which basically builds the images with the requirements.txt which has the custom packages. Once the image is built then flytectl pkg option pushes them to flyte and then I trigger from UI
What is happening is that the workflow doesnt reflect the updated packages
s

Samhita Alla

05/04/2023, 11:18 AM
Are you running
pyflyte --pkgs
and
flytectl register
?
s

Sujith Samuel

05/04/2023, 11:20 AM
yes exactly
s

Samhita Alla

05/04/2023, 11:21 AM
It should work! Can you try running the commands manually?
s

Sujith Samuel

05/04/2023, 11:27 AM
I am using First generate the project docker image then, shell.run_command( f"export PYTHONPATH=/workspace/{project_dir}:/workspace/{project_dir}/{project_dir} && " f"{pyflyte_bin} " f"-c ~/.flyte/config " f"--pkgs {project_dir} " f"package " f"--image {DOCKER_REGISTRY_NAME}/{container_name}:{workflow_version} " f"-f" ) and then shell.run_command( f"flytectl register files " f"-p {project} -d {domain} " f"--archive flyte-package.tgz " f"--version {workflow_version} && " f"rm flyte-package.tgz" )
a

Anatolii Bardukov

05/04/2023, 11:28 AM
btw, @Samhita Alla is there any way to test ImageSpec already? nightly builds/etc? 🙂
s

Samhita Alla

05/04/2023, 11:43 AM
@Sujith Samuel, can you double check if you're sending the right image to the package command? @Anatolii Bardukov, release: https://github.com/flyteorg/flytekit/releases/tag/v1.6.0b0; docs: https://flyte--988.org.readthedocs.build/projects/cookbook/en/988/auto/core/containerization/image_sepc_example.html
s

Sujith Samuel

05/04/2023, 11:49 AM
The right image is being sent to package.....there are other code pieces also which are coming fine in the new image, issue is only with the packages... Do the packages have to be installed in the environment from which we are issuing the package command by the way???
s

Samhita Alla

05/04/2023, 12:03 PM
Yeah. It won't work otherwise.
Unless you send an argument to it. I'm not sure what it is.
Not an argument I believe. It's when you package from within the docker container. Anyway, if you're able to package successfully, that means the command's working fine.
Also, has this worked before?
And do you see the correct image in the task details?
s

Sujith Samuel

05/04/2023, 1:10 PM
This is the first time we are trying with not having the packages in the environment where the packaging command is working.
s

Samhita Alla

05/05/2023, 1:13 PM
Um, it should error out. Not sure why it's working.
87 Views