https://flyte.org logo
Join Slack
Powered by
Hello, It happens to me from time to time that the...
# ask-the-community
f
Hello, It happens to me from time to time that the command pyflyte register …. either takes a long time to complete.
Copy code
pyflyte register --project examples --image <http://xxx.dkr.ecr.us-east-1.amazonaws.com/yyy:latest|xxx.dkr.ecr.us-east-1.amazonaws.com/yyy:latest> train/train_wf.py
And when it finally failed with the error:
Copy code
OverflowError: string longer than 2147483647 bytes
...
flytekit.exceptions.user.FlyteAssertion: Failed to put data from /var/folders/xn/j7gcmr5j12b7jy0nm2kfykhm0000gp/T/tmp58m18z2p/fast53e0b9edd2101668d22c8cd5fe99d0b8.tar.gz to <https://dev-wm-max-flyte-us-east-1.s3.amazonaws.com/examples/development/>....... (recursive=False).

Original exception: string longer than 2147483647 bytes
2147483647 bytes is 2000+ MB. I did du -h . and the total size of my project folder is only 380+ KB. Why does flyte fast register needs to send 2000+ MB data to S3?
k
because flytekit will serialize workflow to workflow.proto, and upload it to s3.
how many tasks are in your workflow
f
just one
I extracted the /var/folders/xn/j7gcmr5j12b7jy0nm2kfykhm0000gp/T/tmp58m18z2p/fast53e0b9edd2101668d22c8cd5fe99d0b8.tar.gz file and it packaged all the files from the parent folder of where I executed the pyflyte register command.
This has never happened before.
k
hmm, pyflyte register will always package all the files from the parent folder.
f
I executed pyflyte at the root folder of my workflow project.
k
which flytekit version did you use before?
f
1.2.4
But it has always worked before.
k
so there is a regression, which flytekit version are you using now?
f
I have always used 1.2.4, before and now.
k
did you upgrade flyteadmin server?
f
Say a/b/train is my folder structure. /b is my workflow project root. In /b, I executed command:
Copy code
pyflyte register --project examples train/train_wf.py
And it packaged everything in /a.
Is this expected?
k
if b has _`__init__`_ file, it will package /a
f
I see. It does and I have removed the _`__init__`_ file. And that fixed the issue.
Thanks @Kevin Su!
@Kevin Su, What if I have a venv install folder under /b? Will it be packaged?
I don’t want the installed dependencies packaged.
k
for now it will package venv folder. you could add a VenvIgnore here
how about adding a new flag in pyflyte register, something like
Copy code
pyflyte register --ignore venv/ wf
f
Yes, that will work.
@Kevin Su Since I removed the init py file at the root, now I got:
Copy code
File "/Users/fshen/hbo-code/dai-mle-paid-renewal/train/train_wf.py", line 22, in <module>
    from ..sql.load_input_data import load_input_data
ValueError: attempted relative import beyond top-level package
folder /train and /sql are both under /b.
k
you have to update the relative import to 
.b.sql.loal_input_data import ...
f
@Kevin Su, for that to work, I need to have a init py file in /b. I am in a catch 22 situation.
Thanks a lot!
151 Views
Open in Slack
PreviousNext
Powered by