https://flyte.org logo
#flyte-on-gcp
Title
# flyte-on-gcp
j

Jake Dodd

12/06/2023, 4:41 PM
Hi, I am running into an error when trying to run the BigQuery Query example. The
no_io_wf
workflow is running fine, however, the
full_bigquery_wf
is failing on the
Python Task
with the following error:
Copy code
ModuleNotFoundError: No module named 'flytekitplugins.bigquery'
s

Samhita Alla

12/06/2023, 4:47 PM
hi @Jake Dodd, please install the bigquery plugin:
pip install flytekitplugins-bigquery
.
j

Jake Dodd

12/06/2023, 4:49 PM
Hi @Samhita Alla, I have added
flytekitplugins-bigquery
as a dependency to Poetry and am registering the workflow using the command:
Copy code
poetry run pyflyte run --remote demo/bigquery.py full_bigquery_wf --version 1
s

Samhita Alla

12/06/2023, 4:59 PM
this will work locally. but for it to work on the cluster, you need to build an image or use image spec to specify the dependency. here's an example:
Copy code
custom_image = ImageSpec(
    name="ray-flyte-plugin",
    registry="<your-registry>",
    packages=["flytekitplugins-ray"],
)

@task(
    task_config=ray_config,
    requests=Resources(mem="2Gi", cpu="2"),
    container_image=custom_image,
)
def ray_task(n: int) -> typing.List[int]:
    futures = [f.remote(i) for i in range(n)]
    return ray.get(futures)
j

Jake Dodd

12/06/2023, 5:00 PM
ah ok, thanks @Samhita Alla, I’ll give it a go with
ImageSpec