sparse-window-1536
07/29/2022, 6:23 PMTraceback (most recent call last):
...
File "/home/flyte/.venv/lib/python3.8/site-packages/flytekit/types/structured/basic_dfs.py", line 63, in decode
ctx.file_access.get_data(path, local_dir, is_multipart=True)
File "/home/flyte/.venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 427, in get_data
raise FlyteAssertion(
Message:
Failed to get data from to /tmp/flyteltak17mx/local_flytekit/85e6844a8f8283f8841994437be73626 (recursive=True).
Original exception: cannot copy tree '': not a directory
User error.
No path for the resulting query file. Could it be that there was an error executing the task? Flyte says the task succeeded, but I can't check on the GCP console (I don't have the necessary permissions).glamorous-carpet-83516
07/29/2022, 6:41 PMsparse-window-1536
07/29/2022, 6:46 PMGET_OPERATION_FLIGHT_OPTIONS_QUERY = """
SELECT status_name, status_id, count(*) as qty
FROM `...` # Redacted because I don't think I can't share the table name
WHERE status_name <> "{{ .inputs.ignore }}"
GROUP BY status_name, status_id
""".strip()
# Annotate a StructuredDataset type to conform with Flyte's strong typing
FlightOptionsCountDataset = Annotated[
StructuredDataset, kwtypes(status_name=str, status_id=int, qty=int)
]
# Define the BigQuery task
get_operation_flight_options = BigQueryTask(
name="tasks.bigquery.get_operation_flight_options",
inputs=kwtypes(ignore=str),
output_structured_dataset_type=FlightOptionsCountDataset,
query_template=GET_OPERATION_FLIGHT_OPTIONS_QUERY,
task_config=BIG_QUERY_CONFIG,
)
sparse-window-1536
07/29/2022, 6:46 PMsparse-window-1536
07/29/2022, 6:50 PMsparse-window-1536
07/29/2022, 6:51 PM{{ .inputs.<input_name> }}
, like what happens in this interpolate_query function https://github.com/flyteorg/flytekit/blob/386626356ff2cb810733a29a51cf3d6351c8424d/flytekit/core/base_sql_task.pysparse-window-1536
07/29/2022, 6:51 PMglamorous-carpet-83516
07/29/2022, 6:54 PMglamorous-carpet-83516
07/29/2022, 6:54 PMsparse-window-1536
07/29/2022, 6:54 PMsparse-window-1536
07/29/2022, 6:55 PMsparse-window-1536
07/29/2022, 6:55 PMglamorous-carpet-83516
07/29/2022, 6:57 PMglamorous-carpet-83516
07/29/2022, 6:57 PMsparse-window-1536
07/29/2022, 7:08 PMglamorous-carpet-83516
07/29/2022, 7:08 PMsparse-window-1536
07/29/2022, 7:08 PMsparse-window-1536
07/29/2022, 7:08 PMsparse-window-1536
07/29/2022, 7:09 PMsparse-window-1536
07/29/2022, 7:09 PMsparse-window-1536
07/29/2022, 7:09 PM@task
def convert_query_to_dataframe(dataset: StructuredDataset) -> pd.DataFrame:
"""Convert a BigQuery table to a pandas DataFrame."""
return dataset.open(pd.DataFrame).all()
sparse-window-1536
07/29/2022, 7:09 PMsparse-window-1536
07/29/2022, 7:10 PM@workflow
def test_bigquery_task() -> None:
"""Test the BigQuery task plugin."""
bq_dataset = tasks.bigquery.get_operation_flight_options(ignore="Rascunho")
dataframe = tasks.bigquery.convert_query_to_dataframe(dataset=bq_dataset)
sparse-window-1536
07/29/2022, 7:10 PMglamorous-carpet-83516
07/29/2022, 7:10 PMsparse-window-1536
07/29/2022, 7:11 PMsparse-window-1536
07/29/2022, 7:11 PMsparse-window-1536
07/29/2022, 7:12 PMtag: v1.1.15
sparse-window-1536
07/29/2022, 7:14 PMSELECT status_name, status_id, count(*) as qty
FROM `...`
WHERE status_name <> @ignore
GROUP BY status_name, status_id
this is how the query is appearing on the bq console. should @ignore
be present?glamorous-carpet-83516
07/29/2022, 7:16 PMglamorous-carpet-83516
07/29/2022, 7:16 PMsparse-window-1536
07/29/2022, 7:21 PMsparse-window-1536
07/29/2022, 7:21 PM@ignore
parameter was indeed passedsparse-window-1536
07/29/2022, 7:37 PMsparse-window-1536
07/29/2022, 7:41 PMStructuredDataset
, not FlightOptionsCountDataset
sparse-window-1536
07/29/2022, 7:42 PMsparse-window-1536
07/29/2022, 10:18 PM{
"dataset": {
"format": "parquet",
"columns": [
{
"status_name": "string"
},
{
"status_id": "integer"
},
{
"qty": "integer"
}
]
}
}
sparse-window-1536
07/29/2022, 10:19 PMthankful-minister-83577
thankful-minister-83577
sparse-window-1536
08/01/2022, 2:49 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
sparse-window-1536
08/04/2022, 3:05 PMsparse-window-1536
08/04/2022, 3:06 PMsparse-window-1536
08/04/2022, 3:06 PMsparse-window-1536
08/04/2022, 3:07 PMsparse-window-1536
08/04/2022, 3:09 PM"type": "parquet"
.thankful-minister-83577
thankful-minister-83577
"full_outputs": {
"literals": {
"results": {
"scalar": {
"structured_dataset": {
"metadata": {
"structured_dataset_type": {
"columns": [{
"name": "status_name",
"literal_type": {
"simple": "STRING"
}
}, {
"name": "status_id",
"literal_type": {
"simple": "INTEGER"
}
}, {
"name": "qty",
"literal_type": {
"simple": "INTEGER"
}
}],
"format": "parquet"
}
}
}
}
}
}
}
thankful-minister-83577
/api/v1/data/node_executions/
endpoint for n0thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
glamorous-carpet-83516
08/05/2022, 2:21 PMglamorous-carpet-83516
08/05/2022, 2:21 PMsparse-window-1536
08/05/2022, 2:21 PMsparse-window-1536
08/05/2022, 2:24 PM