Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-on-gcp
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
integrations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
show-and-tell
sig-large-models
torch-elastic
workflow-building-ui-proj
writing-w-sfloris
Title
d
Dan Rammer (hamersaw)
10/05/2022, 2:39 PMHi, @Jacob Wang - can you provide a little more context? What does the error message say? Can you use the ID to check if the job is still actually running?
j
Jacob Wang
10/05/2022, 2:45 PMHey, the error is just a aws batch job runtime error and the job goes to the failed state. So it is not running. But on the flyte console side the shows task is still running so I cannot know until I click the task detail in the flyte console
d
Dan Rammer (hamersaw)
10/05/2022, 2:50 PMInteresting, it seems that Flyte is failing to report the error. Do mind filing an issue on Github? it helps to be as thorough as possible in the description (ie. exact error message, etc). We will pick this us ASAP.
cc @Kevin Su I know you've done some work on AWS batch - do you have any initial ideas?
k
Kevin Su
10/05/2022, 2:57 PMDoes this task rerun again? (task will rerun 30 times by default if propeller got retryable failure). if the task is rerunning, the status will be “running”
j
Jacob Wang
10/06/2022, 7:56 AMHey, no the task didn’t rerun, ok I will fill this issue on GitHub
Sorry for the late reply since we are on different time zone
Ok it seems that the aws batch job is actually in SUCCEEDED state, but there’s exception captured by flyte and logged in cloudwatch. And also flyte console knows it is failed. So I guess this is related to flyte captured the exception so no exception raised in aws batch job container runtime. But flyte knows there’s an exception…
@Dan Rammer (hamersaw) @Kevin Su
Do you think this is a bug?
If so, would be good if you send me an guideline about filling an issue in flyte’s github repo, if there’s any
k
Kevin Su
10/06/2022, 6:10 PMJust file a bug at here. btw, mind sharing the code snippet, that will help debug.