Hello It happens to me from time to time that the command py

Question

Hello It happens to me from time to time that the command pyflyte register either takes a long time to complete ```pyflyte register project examples image train train wf py``` And when it finally failed with the error ```OverflowError string longer than 2147483647 bytes flytekit exceptions user FlyteAssertion Failed to put data from var folders xn j7gcmr5j12b7jy0nm2kfykhm0000gp T tmp58m18z2p fast53e0b9edd2101668d22c8cd5fe99d0b8 tar gz to recursive=False Original exception string longer than 2147483647 bytes``` 2147483647 bytes is 2000+ MB I did du h and the total size of my project folder is only 380+ KB Why does flyte fast register needs to send 2000+ MB data to S3

Kevin Su · Answer

because flytekit will serialize workflow to workflow proto and upload it to s3

Kevin Su · Answer

how many tasks are in your workflow

Frank Shen · Answer

just one

Frank Shen · Answer

I extracted the var folders xn j7gcmr5j12b7jy0nm2kfykhm0000gp T tmp58m18z2p fast53e0b9edd2101668d22c8cd5fe99d0b8 tar gz file and it packaged all the files from the parent folder of where I executed the pyflyte register command

Frank Shen · Answer

This has never happened before

Kevin Su · Answer

hmm pyflyte register will always package all the files from the parent folder

Frank Shen · Answer

I executed pyflyte at the root folder of my workflow project

Kevin Su · Answer

which flytekit version did you use before

Frank Shen · Answer

1 2 4

Frank Shen · Answer

But it has always worked before

Kevin Su · Answer

so there is a regression which flytekit version are you using now

Frank Shen · Answer

I have always used 1 2 4 before and now

Kevin Su · Answer

did you upgrade flyteadmin server

Frank Shen · Answer

Say a b train is my folder structure b is my workflow project root In b I executed command ```pyflyte register project examples train train wf py```

Frank Shen · Answer

And it packaged everything in a

Frank Shen · Answer

Is this expected

Kevin Su · Answer

if b has ` init ` file it will package a

Frank Shen · Answer

I see It does and I have removed the ` init ` file And that fixed the issue

Frank Shen · Answer

Thanks < Kevin Su>

Frank Shen · Answer

< Kevin Su> What if I have a venv install folder under b Will it be packaged

Frank Shen · Answer

I don t want the installed dependencies packaged

Kevin Su · Answer

for now it will package venv folder you could add a VenvIgnore <https github com flyteorg flytekit blob c8433ea2d2ecf362815b1ee63554549ced9acd75 flytekit tools fast registration py L32|here>

Kevin Su · Answer

how about adding a new flag in pyflyte register something like ```pyflyte register ignore venv wf```

Frank Shen · Answer

Yes that will work

Frank Shen · Answer

< Kevin Su> Since I removed the init py file at the root now I got ``` File Users fshen hbo code dai mle paid renewal train train wf py line 22 in lt module gt from sql load input data import load input data ValueError attempted relative import beyond top level package```

Frank Shen · Answer

folder train and sql are both under b

Kevin Su · Answer

you have to update the relative import to ` b sql loal input data import `

Frank Shen · Answer

< Kevin Su> for that to work I need to have a init py file in b I am in a catch 22 situation

Frank Shen · Answer

Thanks a lot