I'm having issues when running a fast-registered workflow. It looks like flytekit (0.30.3) can't find the code archive:

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.

[vej152kn4b-n1-0] terminated with exit code (1). Reason [Error]. Message: 
tar: /root/8143d4634b6d53c26072284ce429e69af1278102-fast1-fast32b90af686df921bd623ae6df68f5c48.tar.gz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
Traceback (most recent call last):
  File "/opt/venv/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.9/site-packages/flytekit/bin/entrypoint.py", line 467, in fast_execute_task_cmd
    _download_distribution(additional_distribution, dest_dir)
  File "/opt/venv/lib/python3.9/site-packages/flytekit/tools/fast_registration.py", line 112, in download_distribution
    result.check_returncode()
  File "/usr/lib/python3.9/subprocess.py", line 460, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['tar', '-xvf', '/root/8143d4634b6d53c26072284ce429e69af1278102-fast1-fast32b90af686df921bd623ae6df68f5c48.tar.gz', '-C', '/root']' returned non-zero exit status 2.
.

Here's what's executing from the pod spec

spec:
  containers:
  - args:
    - pyflyte-fast-execute
    - --additional-distribution
    - <gs://my-bucket/fast/8143d4634b6d53c26072284ce429e69af1278102-fast1-fast32b90af686df921bd623ae6df68f5c48.tar.gz>
...

I checked the bucket and the object exists so uploading during registration seems to be working correctly. Perhaps something wrong with the download or the path it is copied to in the container. Any ideas?

cc @Yee?

is there a chance you can get the pod logs?

@Yee I checked but this actually is the full log. Perhaps I have to increase the loglevel?

sorry are you trying to run this on sandbox?

No this is on GCP

ohh you should not

are you using the fsspec plugin?

Yes. It is all working fine if I do normal registration. Here's what my poetry deps look like:

[tool.poetry.dependencies]
python = "^3.9"
flytekit = "^0.30.3"
flytekitplugins-data-fsspec = "^0.30.3"
gcsfs = "^2022.1.0"
...

so I checked, it is using the persistence layer

having trouble replicating this locally…

Good idea I'll try that tomorrow.

oh so the issue is in flytekitplugins-data-fsspec?

Looks like. Could also be in fsspec/gcsfs of course. I get the same error if I run pyflyte-fast-execute locally. Seems like it chooses the right plugin, but skips the actual download. Couldn't find the root cause yet.

i need to dig up my gcp account, will do that and try to debug this