<#CP2HDHKE1|ask-the-community> Hello All, I am try...
# ask-the-community
#ask-the-community Hello All, I am trying to install custom packages into the flyte container. In my build job prior to flyte package I am installing numpy which is 1.24.3. But after packaging when i trigger the workflow the version which is shown to me in workflow output is 1.20.0 Another example is I am installing transformers python package but in the workflow the same says module not found. I saw that in kubeflow I can install custom packages using
Copy code
add_op = create_component_from_func(
with package_to_install option. Is there a Flyte way of doing this
feature which we'll ship in flytekit 1.6 should let you specify the packages in flytekit code, and you don't have to think about building and pushing docker images.
Another example is I am installing transformers python package but in the workflow the same says module not found.
Have you specified the package in your dockerfile?
We have a docker process which basically builds the images with the requirements.txt which has the custom packages. Once the image is built then flytectl pkg option pushes them to flyte and then I trigger from UI
What is happening is that the workflow doesnt reflect the updated packages
Are you running
pyflyte --pkgs
flytectl register
yes exactly
It should work! Can you try running the commands manually?
I am using First generate the project docker image then, shell.run_command( f"export PYTHONPATH=/workspace/{project_dir}:/workspace/{project_dir}/{project_dir} && " f"{pyflyte_bin} " f"-c ~/.flyte/config " f"--pkgs {project_dir} " f"package " f"--image {DOCKER_REGISTRY_NAME}/{container_name}:{workflow_version} " f"-f" ) and then shell.run_command( f"flytectl register files " f"-p {project} -d {domain} " f"--archive flyte-package.tgz " f"--version {workflow_version} && " f"rm flyte-package.tgz" )
btw, @Samhita Alla is there any way to test ImageSpec already? nightly builds/etc? 🙂
@Sujith Samuel, can you double check if you're sending the right image to the package command? @Anatolii Bardukov, release: https://github.com/flyteorg/flytekit/releases/tag/v1.6.0b0; docs: https://flyte--988.org.readthedocs.build/projects/cookbook/en/988/auto/core/containerization/image_sepc_example.html
The right image is being sent to package.....there are other code pieces also which are coming fine in the new image, issue is only with the packages... Do the packages have to be installed in the environment from which we are issuing the package command by the way???
Yeah. It won't work otherwise.
Unless you send an argument to it. I'm not sure what it is.
Not an argument I believe. It's when you package from within the docker container. Anyway, if you're able to package successfully, that means the command's working fine.
Also, has this worked before?
And do you see the correct image in the task details?
This is the first time we are trying with not having the packages in the environment where the packaging command is working.
Um, it should error out. Not sure why it's working.