Nicholas LoFaso
12/10/2021, 2:33 AM@dynamic
task from a map_task
and am seeing a strange error. Is there a limitation I should be aware of?
[1/1] currentAttempt done. Last Error: UNKNOWN::failed to read data from dataDir [<gs://mybucket/metadata/propeller/nickflyte-dev-fc4174c9bef1a4c2caa5/n0/data/0/0/outputs.pb>]. Error: path:<gs://mybucket/metadata/propeller/nickflyte-dev-fc4174c9bef1a4c2caa5/n0/data/0/0/outputs.pb>: not found
The attached code fails on our GCS cluster. Other map_tasks that don’t call a dynamic task work fine@dataclass_json
@dataclass
class BundledData:
arg1: str
arg2: int
@task
def t1(input: str) -> str:
print(input)
return "hello"
@task t2(input: int, msg: str):
print(input)
@dynamic
def dyn(input: BundledData):
o = t1(input=input.arg1)
t2(input=input.arg2, msg=o)
@workflow
def wf(inputs: List[BundledData]):
map_task(input=inputs)
Ketan (kumare3)
Nicholas LoFaso
12/10/2021, 2:10 PMKetan (kumare3)
Nicholas LoFaso
12/10/2021, 2:48 PM@dynamic
def dyn(inputs: List[str]):
for input in inputs:
task1(input=input)
If my inputs were [“apple”, “banana”, “cherry”]. `task1(input=“banana”) wouldn’t start until task1(input=“apple”) finished?
Can I use standard python multithreading to make it happen in parallel?Ketan (kumare3)
Nicholas LoFaso
12/10/2021, 4:01 PMKetan (kumare3)
Nicholas LoFaso
12/10/2021, 4:03 PMtask1
will be created at the same time for apple, banana, and cherry?Ketan (kumare3)
Nicholas LoFaso
12/10/2021, 4:05 PMKetan (kumare3)
Nicholas LoFaso
12/10/2021, 4:06 PMKetan (kumare3)
Nicholas LoFaso
12/10/2021, 4:12 PM@dyanmic
Ketan (kumare3)
Nicholas LoFaso
12/10/2021, 4:14 PM