New RunLLM response generated: >>>*# Question* Hello, Our CICD consists of workflows, registered tasks and registered sub-workflows. The idea was to pre-register reusable tasks to be used by different workflows within a project. An example: a standardized QC task that can be referenced in many different workflows. The current documentation on registration seems to limit reference workflows to 1. bind to a specific version 2. Use the macro and package/register the task with the workflow at the same time. This seems to limit the power of reference tasks. • 1. knowing the version is not conducive to our CICD. The CICD registers the tasks first, then workflows. The workflows with reference tasks should be able to fetch the latest task versions. "Tasks are fully independent units of execution and first-class entities of Flyte" • 2. the version macro requires the 2 step pkg + register instead of fast registration. if you want to pull in the latest reference task when you register your workflow you need to register the reference task using the same version you are about to register the workflow with. which sends us back to square one • 2. Limiting reference tasks to be packaged with workflows seems to negate the power of pre-registration and references tasks. Consider the scenario where a task is referenced in 2 different workflows and we want to ensure everyone is using the same task version. commands used

pyflyte --pkgs workflows package --copy all -f

flytectl register files --archive flyte-packag.tgz --project mypyoj --domain development --version test-0

Are we missing something here or were reference tasks never meant to be used in the way we planned? # Answer Currently, Flyte requires specifying a version for reference tasks, which can complicate CICD processes that aim to use the latest version automatically. According to this GitHub issue, there is a request to make the version field optional, allowing the latest version to be fetched at runtime. However, this feature is not yet implemented, and the current workaround involves using

FlyteRemote

to fetch the latest version, which adds complexity. The limitation of needing to package and register tasks with workflows using the same version is acknowledged in the Flyte documentation. This approach can indeed limit the flexibility of pre-registering tasks for reuse across multiple workflows. The current system does not fully support the seamless integration of pre-registered tasks into workflows without version coupling. Would you like more detail on potential workarounds or further insights into Flyte's task versioning? Sources: