https://flyte.org logo
Join Slack
Powered by
how does fast registration work? When I try giving...
# ask-the-community
l
how does fast registration work? When I try giving it an s3 output like the docs 
pyflyte register -p project_name -d domain_name --output <s3://my-s3-bucket/raw_data>
<-- subbing in my own bucket, I get 
FileNotFoundError: [Errno 2] No such file or directory: '<GIT REPO PATH>/s3:/my-s3-bucket/fast-serialize/fast3e198a8e9dd654e746828c0ae929fce3.tar.gz'
and when I don't feed in a 
--output
, it creates a new folder inside 
my-s3-bucket
. When I run it from the UI, I get 
tar: development/fast3e198a8e9dd654e746828c0ae929fce3.tar.gz: Cannot open: Not a directory
y
Because users update code a lot more frequently than they update the image, Flyte has this notion of fast register. Think of it as a patch to your image before running. in practice, flytekit will zip up your code from what it detects is the source, upload thar archive to the blob store configured with Admin by requesting a signed url for upload via the data proxy service, and then prepends a command to the task’s normal command. That command, at run time, will download that archive back from the flyte configured blob store and unzip it to the root of your container’s image, thus saving the user a build cycle. But generally you should think of the act of registration as a two step process. The two steps are compilation (which is the act of parsing the Python code and creating IDL protobuf-based objects that represent them) and registration (the act of shipping those proto objects to a Flyte backend). In fact there are still separate commands for the two steps. Instead of running 
pyflyte register
you can still run 
pyflyte package
and then 
flytectl register
. Here’s a short doc that describes the difference between them.
cc @Ankit Goyal if this helps?
the 
--output
switch you mention is meant for local use. that’s where the proto artifacts are saved in between those two steps.
regardless of whether or not you specify an output, it should create a new s3 file holding the contents of the archive.
l
why do I get this error tho 
tar: development/fast3e198a8e9dd654e746828c0ae929fce3.tar.gz: Cannot open: Not a directory
when I try running a task after doing pyflyte register
Copy code
subprocess.CalledProcessError: Command '['tar', '-xvf', 'development/fast3e198a8e9dd654e746828c0ae929fce3.tar.gz', '-C', 'development']' returned non-zero exit status 2.
wait, taking another look. 
-d
is mapped to domain and destination-dir?
ok, I got past the errors and it seems to run now but the code that was run isn't changing but the version did get bumped. And I can see that the container args do call 
pyflyte-fast-execute
Copy code
@task # original docker task
def fast_serialize_test_task(test_int: int) -> str:
    return str(test_int + 1)

@task # fast serialized 
def fast_serialize_test_task(test_int: int) -> str:
    return str(test_int + 1)
When I look at the file inside the tar.gz, I see the new version of the task
y
can you download the tar file locally and see if it indeed has the updated code?
l
yea, I did and it does.
y
is it unpacking to the wrong place?
l
where is it supposed to unpack to?
in my docker, I have the code copied over to 
/<repo>/src
so I set the destination dir to 
/<repo>/src
also
y
@Eduardo Apolinario (eapolinario)
e
@Laura Lin, one thing to confirm here is what you're setting 
WORKDIR
in your dockerfile.
l
yes its set in the docker
e
Cool, thanks for confirming. Did you have a chance to use the latest update in that branch to kick off an execution?
l
I'll do it ~10 min.
157 Views
Open in Slack
PreviousNext
Powered by