Hello,
Our CICD consists of workflows, registered tasks and registered sub-workflows. The idea was to pre-register reusable tasks to be used by different workflows within a project. An example: a standardized QC task that can be referenced in many different workflows.
The current
documentation on registration seems to limit reference workflows to
1. bind to a specific version
2. Use the macro and package/register the task with the workflow at the same time.
This seems to limit the power of reference tasks.
• 1. knowing the version is not conducive to our CICD. The CICD registers the tasks first, then workflows. The workflows with reference tasks should be able to fetch the latest task versions. "
Tasks are fully independent units of execution and first-class entities of Flyte"
• 2. the version macro requires the 2 step pkg + register instead of fast registration. if you want to pull in the latest reference task when you register your workflow you need to register the reference task using the same version you are about to register the workflow with. which sends us back to square one
• 2. Limiting reference tasks to be packaged with workflows seems to negate the power of pre-registration and references tasks. Consider the scenario where a task is referenced in 2 different workflows and we want to ensure everyone is using the same task version.
commands used
pyflyte --pkgs workflows package --copy all -f
flytectl register files --archive flyte-packag.tgz --project mypyoj --domain development --version test-0
Are we missing something here or were reference tasks never meant to be used in the way we planned?