@ancient-wolf-19325 Once you register a task using flyte remote API, trying to register the related workflow results in error "TASK version not found". How can I debug the issue?

This error typically means the workflow references a task version that is not registered or not discoverable by FlyteAdmin. When registering workflows with FlyteRemote, ensure all referenced tasks are registered first and with the same project, domain, and version as the workflow. If the workflow references a task that is missing or registered under a different version, FlyteAdmin will return a "TASK version not found" error during workflow registration. This is a common issue when using FlyteRemote for dynamic or multi-entity registration, as seen in Flyte issue #3639 and Flyte issue #5989. To debug, verify that all tasks used in the workflow are registered and visible in the Flyte UI or via FlyteRemote/flytectl before registering the workflow. Also, ensure that the registration order is correct: register tasks first, then workflows. If using dynamic tasks or node dependency hints, be aware of known issues where registration order or entity resolution can cause this error (Flyte issue #5989). Would you like more detail on how to check registration status or resolve version mismatches? Sources: - Flyte issue #3639 - Flyte issue #5989 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 I can see that the task was registered successfully from FLyte UI, and yet the workflow registration fails. How can I debug the issue

If the task appears in the Flyte UI but workflow registration still fails with "TASK version not found," it often means the workflow is referencing a different project, domain, or version than the registered task. Double-check that the task's project, domain, and version exactly match those referenced in the workflow. This mismatch is a common cause, especially when registering entities programmatically or across environments, as described in Flyte issue #3639 and Flyte issue #5989. To debug further, inspect the workflow's serialized spec (or registration logs) to confirm the referenced task identifiers. Also, ensure there are no typos or discrepancies in naming. If using FlyteRemote, try registering both task and workflow in the same script/session to avoid context mismatches. Would you like more detail on how to inspect workflow specs or troubleshoot registration with FlyteRemote? Sources: - Flyte issue #5989 - Flyte issue #3639 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 yes

Thank you for confirming! Here’s how you can further debug the issue when your task appears in the Flyte UI, but workflow registration still fails with "TASK version not found": *1. Check Task Identifiers* - Project, Domain, Name, Version: In the Flyte UI, note the exact

project

,

domain

,

name

, and

version

of the registered task. - Workflow Reference: In your workflow code or the serialized workflow spec, ensure that the referenced task uses the exact same identifiers. *2. Inspect the Workflow Spec* - If you are using

pyflyte register

or

FlyteRemote.register_script

, you can inspect the generated protobuf or logs to see which task identifier the workflow is referencing. - Look for lines like:

missing entity of type TASK with identifier project:"myproject" domain:"development" name:"my_module.my_task" version:"abc123"

Compare these values to what you see in the UI. *3. Registration Order and Context* - Order: Always register tasks before workflows. If you register a workflow that references a task not yet registered (or with a different version), you’ll get this error. - Context: If you use different environments, containers, or registration methods, ensure the context (project/domain/version) is consistent. *4. Module and Path Issues* - Sometimes, the task name includes the full Python module path (e.g.,

my_package.my_module.my_task

). If you move files or change package structure, the name may change. Make sure the workflow references the correct fully qualified task name. *5. FlyteRemote Specifics* - If using

FlyteRemote

, register both the task and workflow in the same session/script to avoid discrepancies. - Use

FlyteRemote.fetch_task()

to confirm the task is discoverable with the expected identifiers. *6. Check for Known Issues* - There are known issues with dynamic tasks, node dependency hints, and registration order (see this issue). If you’re using advanced features, check the Flyte GitHub issues for workarounds. --- Example Debugging Steps: 1. Register your task:

remote.register_script(my_task)

2. Confirm in UI: Project =

myproject

, Domain =

development

, Name =

my_module.my_task

, Version =

abc123

3. Register your workflow:

remote.register_script(my_workflow)

4. If it fails, check the error message for the missing task identifier and compare it to the UI. --- If you want, you can share the error message and the identifiers you see in the UI and in your code, and I can help you pinpoint the mismatch! Would you like instructions on how to inspect the serialized workflow spec or logs for task references? Or do you want help with a specific FlyteRemote registration script? 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 Here is the task queried using flyte remote API

TASK:data-normalization-pipelines:development:src.workflows.dummy_test.test_workflow.dummy_task:9928cc3cf316ccc98e6d2f50da397c0e1fb1d545ac27528bd250ab56d1b3215c

and the error that happens when I try to register the workflow

debug_error_string = "UNKNOWN:Error received from peer  
{grpc_message:"failed to compile workflow for [resource_type:WORKFLOW 
project:\"data-normalization-pipelines\" domain:\"development\" 
name:\"src.workflows.dummy_test.test_workflow.dummy_wf\" 
version:\"9928cc3cf316ccc98e6d2f50da397c0e1fb1d545ac27528bd250ab56d1b3215c\"] 
with err failed to compile workflow with err Collected Errors: 1\n\tError 0: 
Code: TaskReferenceNotFound, Node Id: start-node, Description: Referenced Task 
[resource_type:TASK name:\"src.workflows.dummy_test.test_workflow.dummy_task\" 
version:\"9928cc3cf316ccc98e6d2f50da397c0e1fb1d545ac27528bd250ab56d1b3215c\"] 
not found.\n", grpc_status:3, created_time:"2025-07-10T14:07:14.433696-07:00"}

Your task and workflow are both in project

data-normalization-pipelines

, domain

development

, with the same name and version, but the workflow registration fails with

TaskReferenceNotFound

. This usually means that, although the task appears registered, FlyteAdmin cannot resolve it at workflow compilation—often due to registration context, serialization, or environment mismatches (Flyte issue #3639, Flyte issue #5989). Common causes include: - The task was registered in a different environment or with a different serialization context than the workflow. - The workflow references a task name that differs in its fully qualified Python path from the registered task (e.g., due to running as

__main__

or moving files). - The registration order or method (e.g., using different scripts or sessions) led to context mismatches. To debug: 1. Ensure both task and workflow are registered in the same environment/session, using the same FlyteRemote instance. 2. Check that the fully qualified task name in the workflow matches exactly what is registered (including module path). 3. Try re-registering both task and workflow together in a single script to avoid context drift. 4. If using dynamic tasks or node dependency hints, be aware of known issues with entity resolution (Flyte issue #5989). Would you like a code example for registering both task and workflow together, or instructions for inspecting the serialized workflow spec? Sources: - Flyte issue #3639 - Flyte issue #5989 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 can you give me examples for example for registering both task and workflow together,

Certainly! To avoid context or version mismatches, register both the task and workflow in the same script/session using the same FlyteRemote instance. Here’s a minimal example using FlyteRemote:

import flytekit as fl
from flytekit.remote import FlyteRemote

# Define your task and workflow
@fl.task
def dummy_task(x: int) -> int:
    return x + 1

@fl.workflow
def dummy_wf(x: int) -> int:
    return dummy_task(x=x)

# Create a FlyteRemote instance
remote = FlyteRemote()

# Register the task first, then the workflow, using the same version
version = "my_version_001"
remote.register_task(dummy_task, version=version)
remote.register_workflow(dummy_wf, version=version)

This ensures both entities are registered with the same identifiers and context, preventing the "TaskReferenceNotFound" error (FlyteRemote API reference). You can also use

register_script

to register both at once if they are in the same file. Would you like an example using

register_script

or more details on versioning? Sources: - FlyteRemote API reference 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 yes, I would like example using

register_script

or more details on versioning?

To register both a task and workflow together using

register_script

, place them in the same Python file and use

FlyteRemote.register_script

, specifying the version. This ensures both entities are registered with the same context and version, avoiding reference errors (FlyteRemote API reference). Example:

# my_entities.py
import flytekit as fl

@fl.task
def dummy_task(x: int) -> int:
    return x + 1

@fl.workflow
def dummy_wf(x: int) -> int:
    return dummy_task(x=x)

Then, in your registration script:

from flytekit.remote import FlyteRemote
from pathlib import Path

remote = FlyteRemote()
version = "my_version_001"
source_path = Path("my_entities.py").parent.resolve()
from my_entities import dummy_wf

remote.register_script(
    entity=dummy_wf,
    source_path=source_path,
    version=version,
    copy_all=True,
)

This registers both the task and workflow with the same version and context. For versioning, always use the same version string for all related entities to ensure correct references (Flyte registration docs). Would you like more detail on how to structure your project for versioning or on using

copy_all

? Sources: - Flyte registration docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 can I use custom image with register_script

Yes, you can use a custom image with

register_script

by providing an

ImageConfig

to the

image_config

parameter. This allows you to specify the Docker image that Flyte will use for your tasks and workflows (FlyteRemote API reference). Example:

from flytekit.remote import FlyteRemote
from flytekit.configuration import ImageConfig
from my_entities import dummy_wf

remote = FlyteRemote()
image_cfg = ImageConfig.from_images("my_image=ghcr.io/myorg/myimage:latest")
remote.register_script(
    entity=dummy_wf,
    image_config=image_cfg,
    version="my_version_001",
    source_path=".",
)

This ensures your registered entities use the specified custom image. Would you like more detail on image configuration or best practices? Sources: - FlyteRemote API reference 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 yes, I would like more detail on image configuration or best practices

To configure custom images for your Flyte tasks and workflows, use the

ImageConfig

object and pass it to the

image_config

parameter in

register_script

. You can specify one or more images, and assign them by name to tasks using the

container_image

parameter or by default for all tasks (Flyte registration docs, Multiple images in a workflow). Best practices: - Use a consistent image version/tag for all related entities to ensure reproducibility. - If multiple tasks require different dependencies, define multiple images in

ImageConfig

and assign them by name. - For public images, ensure the registry is accessible to your Flyte deployment; for private images, configure authentication as needed. - Use the same

ImageConfig

for both task and workflow registration to avoid mismatches. Example:

from flytekit.configuration import ImageConfig
from flytekit import task, workflow

image_cfg = ImageConfig.from_images(
    "default=ghcr.io/myorg/base:latest",
    "gpu=ghcr.io/myorg/gpu:latest"
)

@task(container_image="gpu")
def gpu_task(x: int) -> int:
    ...

@workflow
def my_wf(x: int) -> int:
    return gpu_task(x=x)

Would you like more detail on handling private registries or multi-image workflows? Sources: - Flyte registration docs - Multiple images in a workflow 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.