Greg Gydush

    Greg Gydush

    4 months ago
    Jeev helped me find a very interesting bug this morning when using dictionaries on Flyte. The order of the keys seems to be non-deterministic, issue is here!
    Ketan (kumare3)

    Ketan (kumare3)

    4 months ago
    cc ohh no, this means that the caching fails?
    cc @Eduardo Apolinario (eapolinario)
    j

    jeev

    4 months ago
    damn didn't even think about the effect on caching.
    Ketan (kumare3)

    Ketan (kumare3)

    4 months ago
    then what was the problem?
    j

    jeev

    4 months ago
    the output (xml doc) depended on the order of the keys of the input dict, so was not as expected.
    Ketan (kumare3)

    Ketan (kumare3)

    4 months ago
    hmm i think caching actually already sorts maps
    i do not know if it sorts recursively
    Eduardo Apolinario (eapolinario)

    Eduardo Apolinario (eapolinario)

    4 months ago
    Jeev, this is not an easy problem to solve. There are some languages, like python, that guarantee the order of keys in a dictionary in the same context (e.g. in the case of python: the runtime environment), but that's not the execution model that Flyte uses, right? Data is communicated between tasks using Flyte types, which under the covers uses protobufs. We map python dictionaries to protobuf maps and that is not guaranteed to be stable:
    Wire format ordering and map iteration ordering of map values are undefined, so you cannot rely on your map items being in a particular order.
    Is there any way you can specify the order separately?
    Ketan (kumare3)

    Ketan (kumare3)

    4 months ago
    Like use list. Orderedmap is not supported
    j

    jeev

    4 months ago
    the interesting thing is that the order looks right in the printed LiteralMap after reading it back in from the protobuf. is that just my eyes playing tricks on me? 😛
    but that's good to know @Eduardo Apolinario (eapolinario) . @Greg Gydush fyi.