https://flyte.org logo
#ask-the-community
Title
# ask-the-community
b

Broder Peters

01/24/2024, 10:27 AM
Hello, are there memory recommendations for using
pyflyte register
?
We've been running into the following issue while executing it in AWS codebuild with 3GB of memory and 2vCPUs We switched from doing
pyflyte package
and
flytectl register files
in two independent steps to the combined
pyflyte register
but the environment resources where the command is executed stayed the same.
Copy code
Running command pyflyte register -p myproject -d mydomain --image myimage:latest --activate-launchplans launch_plan.py
Running pyflyte register from /codebuild/[...] with images ImageConfig(default_image=Image(name='default', fqn='myimage', tag='latest'), images=[Image(name='default', fqn='myimage', tag='latest')]) and image destination folder /root on 1 package(s) ('/codebuild/[...]/launch_plan.py',)
Registering against flyte.something.com
Detected Root /codebuild/[...], using this to create deployable package...
No output path provided, using a temporary directory at /tmp/tmpxnksaw06 instead
Failed with Unknown Exception <class 'MemoryError'> Reason:
Using
pyflyte register
locally has also "attacked" my local memory quite aggressively (total of 16GB, running on linux for that matter) causing my laptop to slow down.
Or is that all due to the fast register mechanism? > This command runs "fast" register by default. This means that a zip is created from the detected root of the packages given and uploaded. Just like with ``pyflyte run``, tasks registered from this command will download and unzip that code package before running. Any ideas or experiences?
Running with
non-fast
helps in my case. I should read up on this to understand the fast pattern better
Ahh nevermind... It was due a not cleaned up .git directory that was probably included during the register process. We had jupyter notebooks in the repository before, removed them, but git wasn't cleaned up. The repo was locally 1.8GB big which most likely was the cause for the memory error.
2 Views