Hi everyone,
I've encountered an intriguing patte...
# ask-the-communityp
Péter Kun
02/23/2024, 1:16 PMHi everyone,
I've encountered an intriguing pattern while working with a dynamic workflow containing over 10 tasks. Within this workflow, the node IDs exhibit an interesting behavior: they follow a predictable sequence up to dn9 (e.g., dn0, dn1, dn2, ..., dn9), but starting from dn10, they switch to seemingly random strings like "fhfyukpa". This pattern is reflected both in the Flyte dashboard and when I attempt to access the inputs and outputs of these nodes during cluster runs.
What is the reason behind it and by what logic these ID-s are generated? This would be useful for me to reach the inputs and outputs in the cluster runs.
Péter Kun
02/23/2024, 3:39 PMTo make it more clear, I've attached a screenshot illustrating the issue using a simplified example. Up to dn9, the node naming follows a predictable pattern, but beyond that point, UUIDs are used as node names. And it would be important for me to anticipate the node IDs to retrieve their inputs and outputs. How can I query the node id-s in these cases, and what drives the shift in the naming pattern after the 10th node?
d
Dan Rammer (hamersaw)
02/23/2024, 4:14 PMDo you have a deeply nested structure here? Node IDs are hashed if they exceed a certain number of characters, which is what is happening in this scenario.
p
Péter Kun
02/23/2024, 4:24 PMOh, it makes sense. Could you tell me what the hash function is or where I can find it?
Péter Kun
02/23/2024, 4:34 PMBecause having access to the hashed ID allows me to query the node execution data, but based on the pattern I could build only the unhashed ID.
d
Dan Rammer (hamersaw)
02/24/2024, 2:07 PMHey @Péter Kun, code is located here. It may make sense to make this value configurable, it's relatively small to reduce storage requirements in etcd (workflow execution status) and admin db.
p
Péter Kun
02/24/2024, 7:55 PMGreat, thanks a lot!