I am trying to register and run a workflow on a remote flyte cluster using an image in ecr, however I keep getting error message. Does anyone how to get around this

img = ImageConfig.from_images(
    "6***********.<http://dkr.ecr.us-east-1.amazonaws.com/feathr-demo:latest|dkr.ecr.us-east-1.amazonaws.com/feathr-demo:latest>"
)

flyte_workflow = remote.register_workflow(
    entity=wf_bb,
    serialization_settings=SerializationSettings(image_config=img),
    version="v1.0",
)

Error

[1/1] currentAttempt done. Last Error: USER::containers with unready status: [ax9m22wmc8xwpmhqmllt-n0-0]|Back-off pulling image "6*********.<http://dkr.ecr.us-east-1.amazonaws.com/feathr-demo:latest|dkr.ecr.us-east-1.amazonaws.com/feathr-demo:latest>"

Hi I am trying to integrate jupyter notebook with flyte. When I execute the script in local I am able to get the output but when I execute in remote I am getting error.

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[ff43e19ed1617402a8c6-n0-0] terminated with exit code (1). Reason [Error]. Message: 
ine 286, in _handle_annotated_task
    _dispatch_execute(ctx, task_def, inputs, output_prefix)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 76, in _dispatch_execute
    logger.debug(f"Starting _dispatch_execute for {task_def.name}")
AttributeError: 'function' object has no attribute 'name'
Traceback (most recent call last):
  File "/opt/venv/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in _call_
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 502, in fast_execute_task_cmd
    subprocess.run(cmd, check=True)
  File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pyflyte-execute', '--inputs', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ff43e19ed1617402a8c6/n0/data/inputs.pb>', '--output-prefix', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-ff43e19ed1617402a8c6/n0/data/0>', '--raw-output-data-prefix', '<s3://my-s3-bucket/qf/ff43e19ed1617402a8c6-n0-0>', '--checkpoint-path', '<s3://my-s3-bucket/qf/ff43e19ed1617402a8c6-n0-0/_flytecheckpoints>', '--prev-checkpoint', '""', '--dynamic-addl-distro', '<s3://my-s3-bucket/72/flytesnacks/development/LXFFQS6IOSM2OAH57AOZKM2SZA======/scriptmode.tar.gz>', '--dynamic-dest-dir', '/root', '--resolver', 'flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'flytekitplugins.papermill.task', 'task-name', '_dummy_task_func']' returned non-zero exit status 1.

Hello, i am extending flytekit to create a custom plugin and i want to define some custom code when the pod gets terminated. are there any handlers that needs to be defined? 🧵

Hi all! My team is preparing to provide Pulumi to our researchers. We want to set a timezone that is familiar to researchers Can I set the default timezone? It's installed using helm chart on bare metal k8s

Hi, everyone. I have a naive question: How do I load a file from a GCS bucket inside a spark_task? Is there some idiomatic ways to do it in Flyte? (Normally I would add some maven coordinates to my spark config, but I noticed that this does not work for Flyte in a cluster where spark images has to be pre-built.)

HI all, so i am trying to use FlyteRemote to aggregate failed task executions of a given workflow execution. So it looks like this

remote = FlyteRemote(
        default_project=project,
        default_domain=domain,
        flyte_admin_url=flyte_admin_url,
        insecure=True,
    )
    wf_exec = remote.fetch_workflow_execution(name=execution_id)
    synced_wf_exec = remote.sync(wf_exec, sync_nodes=True)

However it fails to sync nodes because of this error

File ~/.virtualenvs/balrog/lib/python3.9/site-packages/flytekit/remote/executions.py:114, in FlyteWorkflowExecution.outputs(self)
    110     raise _user_exceptions.FlyteAssertion(
    111         "Please wait until the node execution has completed before requesting the outputs."
    112     )
    113 if self.error:
--> 114     raise _user_exceptions.FlyteAssertion("Outputs could not be found because the execution ended in failure.")
    115 return self._outputs

FlyteAssertion: Outputs could not be found because the execution ended in failure.

is there anyway to get all the failed node/task executions? Looks like it only syncs if the all node execution was successful, it is failing here in

flytekit/remote/remote.py

if launched_exec.is_complete:
                # The synced underlying execution should've had these populated.
                execution._inputs = launched_exec.inputs
                execution._outputs = launched_exec.outputs

👋 I’m hoping to get a better understanding of when a workflow is considered “executed”. I have a series of workflows that are structured as follows: • Workflow A: A prerequisite workflow that does some initial data processing • Workflow B: A downstream consumer of WF A, that invokes WF A as a subworkflow in execution • Workflow C: A downstream consumer of WF A, that invokes WF A as a subworkflow in execution - has no relationship to Workflow B other than the shared dependency to A I launched Workflows B and C sequentially, and as expected: • A was invoked from B, and the Flyte console indicates that A’s output was written to cache • A was then invoked from C, and the Flyte console indicates that A’s output was read from cache However, when I look at the Flyte console for WF A, it indicates that no executions can be found. That leads me to believe that a workflow is only considered executed if it is explicitly launched by a user, and that sub-workflow executions are not assigned their own execution IDs. Is my understanding correct?

Hi all, I’m seeing retries 3 and counting on a workflow that’s throwing

RuntimeError

. According to the docs, I thought this would be considered non-recoverable and thus not retried? I’l put the stack trace in the thread.

Hey all, I’m seeing some weird behavior with some flyte tasks where they seem to be stuck in a “Running” state. The “Running” status has the additional information “[ContainersNotReady|ContainerCreating]: containers with unready status…“. From looking at the k8s pod for the task, the task seems to have completed and the execution shows as succeeded in the flyte ui. Has anyone seen something like this?

#CP2HDHKE1 I am having long running flyteworkflows. I could see that in the FLyteUI i get timeout in NODE. Is there a configuration in FLyte where i can tell flyte to continue the job infinitely till it completes. I am dealing with ML workflows which could run upto 7 days.