Hey Flyte folks, just a quick question about execu...
# flyte-support
m
Hey Flyte folks, just a quick question about execution times. Is there a way to know how long a task has been in queue and how long it was actively run, i assume the execution time shown in console combines the 2 right?
h
Hey Jay, I'll preface by saying that observability in to performance is a big portion of this RFC. Right now I have the telemetry exporting from Flyte-side done and I'm just starting work on filling out exactly what you're looking for.
That being said, we currently report both Node and Task runtime timestamps (
created_at
,
started_at
,
updated_at
) that are stored in the admin DB. The precision of these is not always 100% reliable, as the timestamp often coincides with the Flyte event rather than the k8s operation. So task
created_at
will be when Flyte creates the Pod,
started_at
will be when the Pod enters the
Running
phase, and
updated_at
is the last update. So if the pod succeeds, this is the final timestamp.
There are a few caveats that will be ironed out. For example, the
started_at
timestamp is when the Pod transitions to
RUNNING
but this still requires container image pull times, etc. It would be VERY usefully to have start / end times for single container runtimes. Then users know exactly how long their tasks ran, how much overhead attributes to k8s ops, and how much overhead attributes to flyte ops.
m
whoa this looks awesome, thank you Dan. Really looking forward to this 😄. yeah im trying to see how much time my tasks are waiting in queue, im submitting quite a lot of jobs. when the additional dev work tasks that i am trying to optimize is waiting in queue a bit and that is misleading my eye haha
h
🤣 yeah, i think a lot of Flyte users feel your pain! So we hear you guys and are spending a significant bit of time here to iron this out. Understanding this is so important for improving workflow speed and scaling up!
🙏 2
So right now your best option is to look into the admin DB at the Node and Task timestamps. The differences between
created_at
and
started_at
on a task should be when Flyte creates the k8s Pod and when the Pod transitions to
RUNNING
. IIUC this is the best we can currently do to capture the queued time you're looking for.
m
gotcha yeah that works
155 Views