https://flyte.org logo
#ask-the-community
Title
# ask-the-community
x

Xinzhou Liu

06/06/2023, 6:10 PM
Hi all! Can anyone tell me the difference between a sidecar task and python task? I can’t seem to find much doc about it. Thanks! My issue is that I don’t know how they are configured
k

Kevin Su

06/06/2023, 7:41 PM
x

Xinzhou Liu

06/06/2023, 7:55 PM
I see. And the sidecar task seems to only use the
latest
image. Is there a way to specify the image tag the sidecar task container?
k

Kevin Su

06/06/2023, 7:59 PM
you could do that by
Copy code
pod_spec=V1PodSpec(
            containers=[
                V1Container(
                    name="primary",
                    image=<image_name>
x

Xinzhou Liu

06/06/2023, 8:00 PM
got it, thanks
@Kevin Su have you seen this issue? It seems to only happen to sidecar tasks
k

Kevin Su

06/06/2023, 9:03 PM
did you push the image to the registry? it seems like propeller can’t pull the image
x

Xinzhou Liu

06/06/2023, 9:05 PM
I’m pretty sure I did, the other python tasks using the same image in the same workflow succeeded
k

Kevin Su

06/06/2023, 9:10 PM
could you also share the propeller logs
x

Xinzhou Liu

06/06/2023, 9:11 PM
How do I check the propeller logs? I’m not managing the infrastructure, so I’m not familiar with the setup
k

Kevin Su

06/06/2023, 9:16 PM
kubectl logs <flytepropeller-pod-name> -n flyte. or go to kubernetes dashboard to see the logs
x

Xinzhou Liu

06/06/2023, 9:42 PM
thanks! these are relevant logs
Copy code
Failed to record taskEvent, error [EventAlreadyInTerminalStateError: conflicting events; destination: ABORTED, caused by [rpc error: code = FailedPrecondition desc = invalid phase change from FAILED to ABORTED for task execution {resource_type:TASK project:"melody" domain:"staging" name:"atomwise.atomnet.flyte.tasks.train" version:"root-a187ff5188044973986b250c791f6b8e-cVO2gVPDzJ-3f9oE-wCwCA=="  node_id:"n11-1-dn2" execution_id:<project:"melody" domain:"staging" name:"root-i85xkgsuskuxsl1quizkjw" >  0 {} [] 0}]]. Trying to record state: ABORTED. Ignoring this error!

Workflow namespace[project-staging]/name[root-i85xkgsuskuxsl1quizkjw] has already been terminated.

Trying to abort a node in state [Failed]
It seems a grpc issue. Is there a timeout setting we can increase?
k

Kevin Su

06/06/2023, 9:52 PM
change from FAILED to ABORTED
how did you know it’s grpc issue from this log? do you have any other logs in propeller
x

Xinzhou Liu

06/06/2023, 9:52 PM
i just guessed because
rpc error:
9 Views