Hi there! I'm new to Flyte! It had always been a project that's been on my radar and recently wanted to test out a deployment of Flyte OSS v1 to get a feel for it. I'm using the Single Cluster Simple Cloud Deployment, on GCP with Cloud SQL and GCS storage, Workload Identity Federation, etc. The deployment itself appears to be fine (

pyflyte info

and

flytectl get projects

both work fine) and I can submit a remote job (

pyflyte run --remote basics/hello_world.py hello_world_wf

), but the workflow/task fails with the following error in the logs:

[1/1] currentAttempt done. Last Error: USER::
[a64w5v2dwzz48xl2vdv7-n0-0] terminated with exit code (1). Reason [Error]. Message: 
in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/micromamba/envs/runtime/lib/python3.12/site-packages/flytekit/bin/entrypoint.py", line 773, in fast_execute_task_cmd
    _download_distribution(additional_distribution, dest_dir)
  File "/opt/micromamba/envs/runtime/lib/python3.12/site-packages/flytekit/core/utils.py", line 309, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/micromamba/envs/runtime/lib/python3.12/site-packages/flytekit/tools/fast_registration.py", line 310, in download_distribution
    FlyteContextManager.current_context().file_access.get_data(
  File "/opt/micromamba/envs/runtime/lib/python3.12/site-packages/flytekit/utils/asyn.py", line 113, in wrapped
    return self.run_sync(coro_func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/micromamba/envs/runtime/lib/python3.12/site-packages/flytekit/utils/asyn.py", line 106, in run_sync
    return self._runner_map[name].run(coro)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/micromamba/envs/runtime/lib/python3.12/site-packages/flytekit/utils/asyn.py", line 85, in run
    res = fut.result(None)
          ^^^^^^^^^^^^^^^^
  File "/opt/micromamba/envs/runtime/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/micromamba/envs/runtime/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/opt/micromamba/envs/runtime/lib/python3.12/site-packages/flytekit/core/data_persistence.py", line 614, in async_get_data
    raise FlyteDownloadDataException(
flytekit.exceptions.system.FlyteDownloadDataException: SYSTEM:DownloadDataError: error=Failed to get data from <gs://glen-flyte-metadata/flytesnacks/development/B76XP4JOBARIUYDOE24QS5ZV3A======/faste5ffb4bd8f31b2d9690cb1cb8cd74f77.tar.gz> to ./ (recursive=False).

Original exception: the JSON object must be str, bytes or bytearray, not NoneType
.

I think the main thing that caught my eye was near the end where it had the

SYSTEM:DownloadDataError

. My role has storage.objectUser which includes read and write perms (and it can obviously write). What am I missing here? If i download the tarball, it does have my hello_world.py script in there which is just also taken from one of your examples:

from flytekit import task, workflow, ImageSpec

image_spec = ImageSpec(
    name="hello-world-image",
    apt_packages=["git", "build-essential", "curl"],
    packages=["flytekit==1.16.14"],
    registry="[REDACTED_MY_REGISTRY]"
)

@task(container_image=image_spec)
def say_hello() -> str:
    return "Hello, World!"

@workflow
def hello_world_wf() -> str:
    res = say_hello()
    return res


if __name__ == "__main__":
    print(f"Running hello_world_wf() {hello_world_wf()}")

I've tried specifying an image with ImageSpec() as well as no image provided and just using

@task()

I feel like I'm missing something silly here, can someone point me in the right direction?

Did you set the service account? I'm guessing fsspec tried to read the credential and it's empty. Maybe this docs can help? https://github.com/unionai-oss/deploy-flyte/blob/main/environments/gcp/flyte-core/README.md

The one I deployed is: https://docs-legacy.flyte.org/en/latest/deployment/deployment/cloud_simple.html, which is just a single binary/pod. The k8s SA which the deployment is using has workload identity setup, which is how it's able to connect to the Cloud SQL db and the GCS storage. If I were to contain the

ImageSpec()

in the hello_world job, it builds, pushes the image as well as pulls is in the workflow logs.