numerous-hamburger-7178
12/06/2022, 12:14 AMhigh-accountant-32689
12/06/2022, 12:44 AMnumerous-hamburger-7178
12/06/2022, 1:12 AMflyte-core-v1.2.0
magnificent-teacher-86590
12/06/2022, 1:32 AMhallowed-mouse-14616
12/06/2022, 2:12 AMnumerous-hamburger-7178
12/06/2022, 2:13 AMhallowed-mouse-14616
12/06/2022, 2:16 AMhallowed-mouse-14616
12/06/2022, 2:17 AMnumerous-hamburger-7178
12/06/2022, 2:17 AMhallowed-mouse-14616
12/06/2022, 2:18 AMnumerous-hamburger-7178
12/06/2022, 2:23 AM(combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "29dc9ef6819b16e613927333fbc2a069b819c5346d69628d580e817e9b1cf8d0": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
it happens sometimes when karpenter spins up new nodes, haven't quite figured it out yet either.
so it gets stuck on container creatinghallowed-mouse-14616
12/06/2022, 2:29 AMnumerous-hamburger-7178
12/06/2022, 2:46 AMcode:"UnexpectedObjectDeletion" message:"object
or something similar. I don't remember exactly but it was some kind of deleted object errorhallowed-mouse-14616
12/06/2022, 2:47 AMSYSTEM::object [flytesnacks-development/f0632d6d37de845a7937-n1-3] terminated in the background, manually
Does that look familiar?hallowed-mouse-14616
12/06/2022, 2:47 AMnumerous-hamburger-7178
12/06/2022, 3:47 AMhallowed-mouse-14616
12/06/2022, 3:37 PM@task(retries=3)
def foo:
# omitted
@workflow
def bar:
mapped_out = map_task(foo)(a=a).with_overrides(retries=3)
# omitted
high-accountant-32689
12/06/2022, 6:35 PMnumerous-hamburger-7178
12/06/2022, 6:52 PMRecoverable vs. Non-Recoverable failures: Recoverable failures will be retried and counted against the task's retry count. Non-recoverable failures will just fail, i.e., the task isn't retried irrespective of user/system retry configurations. All user exceptions are considered non-recoverable unless the exception is a subclass of FlyteRecoverableException.
hallowed-mouse-14616
12/06/2022, 6:53 PMaverage-finland-92144
01/19/2023, 4:28 PMnumerous-hamburger-7178
01/19/2023, 4:31 PM