acoustic-carpenter-78188
03/09/2023, 11:40 PMpyflyte package
, we get an error like:
Multiple definitions of the following tasks were found: ['ShiftDatetimeByDurationTaskV0', 'LookupFixedRangeBigQueryTaskV1', 'LookupBigQueryTaskV1']
where those tasks are remote tasks used within a workflow.
The error is introduced In flytekit versions after 1.2.7 (probably 1.2.8) and is maybe due to that _should_register_with_admin
function no longer filters out remote tasks entities as FlyteTask superclass changed from:
class FlyteTask(hash_mixin.HashOnReferenceMixin, RemoteEntity, _task_model.TaskTemplate):
to
class FlyteTask(hash_mixin.HashOnReferenceMixin, RemoteEntity, TaskSpec):
def _should_register_with_admin(entity) -> bool:
"""
This is used in the code below. The translator.py module produces lots of objects (namely nodes and BranchNodes)
that do not/should not be written to .pb file to send to admin. This function filters them out.
"""
return isinstance(
entity, (task_models.TaskSpec, _launch_plan_models.LaunchPlan, admin_workflow_models.WorkflowSpec)
)
Expected behavior
Should not get an error of duplicate tasks defined for remote tasks when running pyflyte package
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteacoustic-carpenter-78188
03/09/2023, 11:40 PM