https://flyte.org logo
#ask-the-community
Title
# ask-the-community
l

Laura Lin

10/21/2022, 10:26 PM
how does fast registration work? When I try giving it an s3 output like the docs
pyflyte register -p project_name -d domain_name --output <s3://my-s3-bucket/raw_data>
<-- subbing in my own bucket, I get
FileNotFoundError: [Errno 2] No such file or directory: '<GIT REPO PATH>/s3:/my-s3-bucket/fast-serialize/fast3e198a8e9dd654e746828c0ae929fce3.tar.gz'
and when I don't feed in a
--output
, it creates a new folder inside
my-s3-bucket
. When I run it from the UI, I get
tar: development/fast3e198a8e9dd654e746828c0ae929fce3.tar.gz: Cannot open: Not a directory
y

Yee

10/22/2022, 12:14 AM
Because users update code a lot more frequently than they update the image, Flyte has this notion of fast register. Think of it as a patch to your image before running. in practice, flytekit will zip up your code from what it detects is the source, upload thar archive to the blob store configured with Admin by requesting a signed url for upload via the data proxy service, and then prepends a command to the task’s normal command. That command, at run time, will download that archive back from the flyte configured blob store and unzip it to the root of your container’s image, thus saving the user a build cycle. But generally you should think of the act of registration as a two step process. The two steps are compilation (which is the act of parsing the Python code and creating IDL protobuf-based objects that represent them) and registration (the act of shipping those proto objects to a Flyte backend). In fact there are still separate commands for the two steps. Instead of running
pyflyte register
you can still run
pyflyte package
and then
flytectl register
. Here’s a short doc that describes the difference between them.
cc @Ankit Goyal if this helps?
the
--output
switch you mention is meant for local use. that’s where the proto artifacts are saved in between those two steps.
regardless of whether or not you specify an output, it should create a new s3 file holding the contents of the archive.
l

Laura Lin

10/24/2022, 3:16 PM
why do I get this error tho
tar: development/fast3e198a8e9dd654e746828c0ae929fce3.tar.gz: Cannot open: Not a directory
when I try running a task after doing pyflyte register
Copy code
subprocess.CalledProcessError: Command '['tar', '-xvf', 'development/fast3e198a8e9dd654e746828c0ae929fce3.tar.gz', '-C', 'development']' returned non-zero exit status 2.
wait, taking another look.
-d
is mapped to domain and destination-dir?
ok, I got past the errors and it seems to run now but the code that was run isn't changing but the version did get bumped. And I can see that the container args do call
pyflyte-fast-execute
Copy code
@task # original docker task
def fast_serialize_test_task(test_int: int) -> str:
    return str(test_int + 1)

@task # fast serialized 
def fast_serialize_test_task(test_int: int) -> str:
    return str(test_int + 1)
When I look at the file inside the tar.gz, I see the new version of the task
y

Yee

10/24/2022, 5:58 PM
can you download the tar file locally and see if it indeed has the updated code?
l

Laura Lin

10/24/2022, 5:58 PM
yea, I did and it does.
y

Yee

10/24/2022, 5:59 PM
is it unpacking to the wrong place?
l

Laura Lin

10/24/2022, 6:01 PM
where is it supposed to unpack to?
in my docker, I have the code copied over to
/<repo>/src
so I set the destination dir to
/<repo>/src
also
y

Yee

10/24/2022, 7:47 PM
@Eduardo Apolinario (eapolinario)
e

Eduardo Apolinario (eapolinario)

10/26/2022, 5:11 PM
@Laura Lin, one thing to confirm here is what you're setting
WORKDIR
in your dockerfile.
l

Laura Lin

10/26/2022, 5:13 PM
yes its set in the docker
e

Eduardo Apolinario (eapolinario)

10/26/2022, 5:15 PM
Cool, thanks for confirming. Did you have a chance to use the latest update in that branch to kick off an execution?
l

Laura Lin

10/26/2022, 5:16 PM
I'll do it ~10 min.
71 Views