Hey Flyte folks, just a quick question about execu...
# ask-the-community
j
Hey Flyte folks, just a quick question about execution times. Is there a way to know how long a task has been in queue and how long it was actively run, i assume the execution time shown in console combines the 2 right?
d
Hey Jay, I'll preface by saying that observability in to performance is a big portion of this RFC. Right now I have the telemetry exporting from Flyte-side done and I'm just starting work on filling out exactly what you're looking for.
That being said, we currently report both Node and Task runtime timestamps (
created_at
,
started_at
,
updated_at
) that are stored in the admin DB. The precision of these is not always 100% reliable, as the timestamp often coincides with the Flyte event rather than the k8s operation. So task
created_at
will be when Flyte creates the Pod,
started_at
will be when the Pod enters the
Running
phase, and
updated_at
is the last update. So if the pod succeeds, this is the final timestamp.
There are a few caveats that will be ironed out. For example, the
started_at
timestamp is when the Pod transitions to
RUNNING
but this still requires container image pull times, etc. It would be VERY usefully to have start / end times for single container runtimes. Then users know exactly how long their tasks ran, how much overhead attributes to k8s ops, and how much overhead attributes to flyte ops.
j
whoa this looks awesome, thank you Dan. Really looking forward to this 😄. yeah im trying to see how much time my tasks are waiting in queue, im submitting quite a lot of jobs. when the additional dev work tasks that i am trying to optimize is waiting in queue a bit and that is misleading my eye haha
d
🤣 yeah, i think a lot of Flyte users feel your pain! So we hear you guys and are spending a significant bit of time here to iron this out. Understanding this is so important for improving workflow speed and scaling up!
So right now your best option is to look into the admin DB at the Node and Task timestamps. The differences between
created_at
and
started_at
on a task should be when Flyte creates the k8s Pod and when the Pod transitions to
RUNNING
. IIUC this is the best we can currently do to capture the queued time you're looking for.
j
gotcha yeah that works
128 Views