https://flyte.org logo
Join Slack
Powered by
Hello, It happens to me from time to time that the...
# flyte-support
s
Hello, It happens to me from time to time that the command pyflyte register …. either takes a long time to complete.
Copy code
pyflyte register --project examples --image <http://xxx.dkr.ecr.us-east-1.amazonaws.com/yyy:latest|xxx.dkr.ecr.us-east-1.amazonaws.com/yyy:latest> train/train_wf.py
And when it finally failed with the error:
Copy code
OverflowError: string longer than 2147483647 bytes
...
flytekit.exceptions.user.FlyteAssertion: Failed to put data from /var/folders/xn/j7gcmr5j12b7jy0nm2kfykhm0000gp/T/tmp58m18z2p/fast53e0b9edd2101668d22c8cd5fe99d0b8.tar.gz to <https://dev-wm-max-flyte-us-east-1.s3.amazonaws.com/examples/development/>....... (recursive=False).

Original exception: string longer than 2147483647 bytes
2147483647 bytes is 2000+ MB. I did du -h . and the total size of my project folder is only 380+ KB. Why does flyte fast register needs to send 2000+ MB data to S3?
g
because flytekit will serialize workflow to workflow.proto, and upload it to s3.
how many tasks are in your workflow
s
just one
I extracted the /var/folders/xn/j7gcmr5j12b7jy0nm2kfykhm0000gp/T/tmp58m18z2p/fast53e0b9edd2101668d22c8cd5fe99d0b8.tar.gz file and it packaged all the files from the parent folder of where I executed the pyflyte register command.
This has never happened before.
g
hmm, pyflyte register will always package all the files from the parent folder.
s
I executed pyflyte at the root folder of my workflow project.
g
which flytekit version did you use before?
s
1.2.4
But it has always worked before.
g
so there is a regression, which flytekit version are you using now?
s
I have always used 1.2.4, before and now.
g
did you upgrade flyteadmin server?
s
Say a/b/train is my folder structure. /b is my workflow project root. In /b, I executed command:
Copy code
pyflyte register --project examples train/train_wf.py
And it packaged everything in /a.
Is this expected?
g
if b has _`__init__`_ file, it will package /a
s
I see. It does and I have removed the _`__init__`_ file. And that fixed the issue.
Thanks @glamorous-carpet-83516!
@glamorous-carpet-83516, What if I have a venv install folder under /b? Will it be packaged?
I don’t want the installed dependencies packaged.
g
for now it will package venv folder. you could add a VenvIgnore here
how about adding a new flag in pyflyte register, something like
Copy code
pyflyte register --ignore venv/ wf
s
Yes, that will work.
@glamorous-carpet-83516 Since I removed the init py file at the root, now I got:
Copy code
File "/Users/fshen/hbo-code/dai-mle-paid-renewal/train/train_wf.py", line 22, in <module>
    from ..sql.load_input_data import load_input_data
ValueError: attempted relative import beyond top-level package
folder /train and /sql are both under /b.
g
you have to update the relative import to 
.b.sql.loal_input_data import ...
s
@glamorous-carpet-83516, for that to work, I need to have a init py file in /b. I am in a catch 22 situation.
Thanks a lot!
153 Views
Open in Slack
PreviousNext
Powered by