Also where can i get some info on how the results ...
# flyte-supportm
magnificent-teacher-86590
08/01/2023, 5:30 PMAlso where can i get some info on how the results are getting stored in datacatalog, i kept getting this error
Copy code
errorsyncing'my-development/azcrpvdss4xngltqgpd6': FailedtocheckCatalogforpreviousresults: unexpectedartifactData: [trimmed_fastqs]
type:
[collection_type: <map_value_type: <union_type: <variants: <blob: <>>variants: <union_type: <variants: <blob: <>structure: <tag: "FlyteFilePath">>>>>>>]
does not match any task output type:
[collection_type: <map_value_type: <union_type: <variants: <blob: <>structure: <tag: "FlyteFilePath">>variants: <simple: NONEstructure: <tag: "none">>>>>]g
glamorous-carpet-83516
08/01/2023, 6:13 PMseems like a bug in the type checker. could you share the task code? I can take a look into it
h
hallowed-mouse-14616
08/01/2023, 6:21 PM@magnificent-teacher-86590 can you share a minimal repro of this? I think it's actually a part of a greater issue we're debugging right now.
m
magnificent-teacher-86590
08/01/2023, 6:56 PMyeah the most annoying thing is i just cannot replicate it in a sandbox, but here is the task signature
Copy code
@BASE_TASK_WORKER.run_dynamic_task_with_args()
def process_chunked_fastqs_to_raw_bams_task(
in_sample_name: str,
in_spec_path: FlyteFile,
in_fastq_pairs: list[dict[str, Optional[FlyteFile]]],
in_expected_spikein: str,
out_gcs_dir: str,
) -> NamedTuple( # type: ignore
"process_chunked_fastqs_to_raw_bams_outputs", # noqa
trimmed_fastqs=list[dict[str, Optional[FlyteFile]]],
ipc_fastqs=list[dict[str, Optional[FlyteFile]]],
raw_bams_to_merge=list[FlyteFile],
raw_adapter_trim_metrics=list[Optional[FlyteFile]],
raw_umi_metrics=list[dict[str, Optional[FlyteFile]]],
spike_in_metrics=list[Optional[FlyteFile]],
):magnificent-teacher-86590
08/01/2023, 6:58 PMso above task is executed as part of a large workflow and that workflow crashes whenever i try to relaunch or recover.
if things wont work, maybe i should just do
list[dict[str, FlyteFile]]magnificent-teacher-86590
08/01/2023, 7:58 PM@hallowed-mouse-14616 do you have a github issue with more info regarding your big issue 😅
magnificent-teacher-86590
08/01/2023, 10:25 PMoh i was jjust reading through the release note, noticed this
Copy code
Union | optional return types supported by @kumare3 in #1703v1.7.1b1h
hallowed-mouse-14616
08/02/2023, 10:03 PM@magnificent-teacher-86590 here is the github issue. I have a fix coming shortly - it looks to be the exact same problem you're facing, namely the double
unionm
magnificent-teacher-86590
08/02/2023, 10:24 PMthank you very much, yeah the issue is exactly the same. maybe the way it was triggered is different, in our case its triggered on relaunch/recover operation. i will look forward to the fix and test it out.
6 Views