https://flyte.org
Join Slack
Hello all Can someone help me with a dynamic workflow situation? In local tests it runs OK, but when...
# flyte-support
d
Hello all Can someone help me with a dynamic workflow situation? In local tests it runs OK, but when I upload it to Flyte in my cluster, it queue the tasks but never run them... The picture shows what's happening.
m
whats your flytepropeller log say
d
Copy code
1: 0: 0: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
1: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
2: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
3: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
4: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
5: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
6: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
7: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
8: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
9: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
10: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
11: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
12: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
13: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
14: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
15: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
16: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
m
missing project ?
maybe registration problem 🤔
how are you calling your workflow in the dynamic task
d
Copy code
@workflow
def main_workflow()-> pd.DataFrame:

    ...

    return dynamic workflow
In local it works just fine
m
i think local you dont need that project parameter, maybe its an issue on the back end, you can see your queued workflow registered in the console right
d
Yup, but it never starts Other workflows (without dynamic) works just right
m
we saw a similar issue when updating flyte backend
i think we saw it when updating to flytekit 1.3 could be something similar
g
what version of propeller and flytekit you are using? mind sharing the code snippet
d
We've recently upgraded my flyte environment to 1.3... Now we are using everything in v1.3.0. I've shared the code in this thread, earlier
If you need more info, I'm here 🙂
If you notice in image, the type is showing as 
unknown
... perhaps there is a relation...?
Copy code
{6 items
config:{}0 items
container:{7 items
args:[18 items
0:"pyflyte-execute"
1:"--inputs"
2:"{{.input}}"
3:"--output-prefix"
4:"{{.outputPrefix}}"
5:"--raw-output-data-prefix"
6:"{{.rawOutputDataPrefix}}"
7:"--checkpoint-path"
8:"{{.checkpointOutputPrefix}}"
9:"--prev-checkpoint"
10:"{{.prevCheckpointPrefix}}"
11:"--resolver"
12:"flytekit.core.python_auto_container.default_task_resolver"
13:"--"
14:"task-module"
15:"src.features.workflow.workflow"
16:"task-name"
17:"features_workflow"
]
command:[]0 items
config:[]0 items
env:[1 item
0:{2 items
key:"_F_SS_C"
value:""
}
]
image:""
ports:[]0 items
resources:{2 items
limits:[2 items
0:{2 items
name:1
value:"1"
}
1:{2 items
name:3
value:"2Gi"
}
]
requests:[]0 items
}
}
id:{5 items
domain:"development"
name:"src.features.workflow.workflow.features_workflow"
project:"flyte-tasks"
resourceType:1
version:"a1df825"
}
interface:{2 items
inputs:{1 item
variables:{2 items
activities_dataset:{2 items
description:"activities_dataset"
type:{1 item
structuredDatasetType:{1 item
columns:[]0 items
}
}
}
dataset:{2 items
description:"dataset"
type:{1 item
structuredDatasetType:{1 item
columns:[]0 items
}
}
}
}
}
outputs:{1 item
variables:{1 item
o0:{2 items
description:"o0"
type:{1 item
structuredDatasetType:{1 item
columns:[]0 items
}
}
}
}
}
}
metadata:{2 items
retries:{}0 items
runtime:{3 items
flavor:"python"
type:1
version:"1.5.0"
}
}
type:"python-task"
}
This is the parameters flyte console is giving to me. I've omitted some parts for legal purposes (if any)
h
@dry-ability-69144, flytekit 1.3.0 (and 1.3.1) had an issue with dynamic workflows. This was fixed in 1.3.2 and beyond. More details about the issue can be found here.
🎉 1
d
Those are great news! Thanks @high-accountant-32689
@high-accountant-32689 I've updated to Flyte 1.8.0 using flyte-core for kubernetes, and the problem persists... Is there something that I need to do to make it work?
I'm using flytekit 1.6.1 to register tasks
h
@dry-ability-69144, I'm confused, can you clarify what you mean by Flyte 1.8.0? I can hop on a call to debug this
d
@high-accountant-32689 I'm using the version available on GitHub (Flyteconsole 1.8.0, and the other components in the versions in 
values.yaml
available on GH) About the call, it would be great! When you're available, we can schedule this
150 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by