https://flyte.org logo
Join Slack
Powered by
:wave: We have some users that created some custo...
# flyte-support
b
👋 We have some users that created some custom task annotations to work around some Flyte constructs (eg., positional args, and default params). As such they also had to define their own 
DefaultTaskResolver
. This implementation was working on an older version of flytekit (1.3.2), but after updating to 1.9.1, workflows using this annotation are now failing pyflyte workflow serialization with 
Could not find LHS for PythonFunctionTask
. Debugging a bit, it appears it is failing this validation, I believe because it's using the default_task_resolver during serialization. Is there any way to pass in, and or register, their custom task resolver as part of pyflyte serialization? Obviously, one solve is to stop using their custom task annotation, but a lot of their workflows are currently built around this and would help avoid a lot of toil on their part.
g
are you able to share the code snippet? that will be really helpful
👍 1
btw, I have fixed some issues in tracker.py recently. could you install flytekit from master branch, and run it again?
f
Also @brash-london-45337 what were the issues that folks were working around. Actually positional args is something we want to use as well, if you have things, please upstream and we would love to make it part of the core
b
So the annotation implementation is pretty hairy with a lot of wrapping. I believe the crux of it is in the task resolver though as they load the task by setting the 
PythonFunctionTask
as part of a function attribute:
Copy code
class FITaskResolver(DefaultTaskResolver):
    def name(self) -> str:
        return "FITaskResolver"

    def load_task(  # pyright: ignore[reportIncompatibleMethodOverride]
        self, loader_args: List[str]
    ) -> PythonAutoContainerTask:
        return super().load_task(loader_args).task_wrapped
@freezing-airport-6809 There's some internal conversations about best practices and is something I wanted to loop union.ai in on as well. Some thoughts from a user (danbrown — but I see that he's not in this community slack) : > ◦ Allow default values for task inputs (flyte doesn't support it, iiuc because protobuf) > ◦ Allow positional args (flyte doesn't support it, iiuc because protobuf) > ◦ Allow `*args`/`**kwargs` (flyte doesn't support it, iiuc because protobuf) > ◦ Allow any input/output serdes via e.g. jsonpickle, instead of restricting to flyte's limited set of types it can map to protobuf + 
@dataclass_json
which only supports things that can 
json.dumps
> ◦ Allow any input/output types that your python tools can typecheck, instead of restricting to flyte's very limited type checking (no generics is a big one, plus lots of issues and surprises with stuff like Optional/Union/etc.) > ◦ Basically, I wish flyte designed its python ergonomics around what its python users want to do, not what its protobuf backends want to do. The more I use flyte the more I feel this is completely upside down.
g
we support cloud pickle. here is an example. https://docs.flyte.org/en/latest/user_guide/data_types_and_io/pickle_type.html it will allow to use any type in the task.
f
• Allow positional args (flyte doesn't support it, iiuc because protobuf)
This is just abundance of caution on our part TBH. We could easily fix this and maybe the fix is limited to only this part. Help with this would be appreciated.
◦ Allow `*args`/`**kwargs` (flyte doesn't support it, iiuc because protobuf)
Related to ^
◦ Allow any input/output serdes via e.g. jsonpickle, instead of restricting to flyte's limited set of types it can map to protobuf + 
@dataclass_json
which only supports things that can 
json.dumps
Supported - pickle. Just use Any or do not use a dataclass. All unknown types are supported
◦ Allow any input/output types that your python tools can typecheck, instead of restricting to flyte's very limited type checking (no generics is a big one, plus lots of issues and surprises with stuff like Optional/Union/etc.)
This is supported through pickle. Maybe you are using an old version
◦ Basically, I wish flyte designed its python ergonomics around what its python users want to do, not what its protobuf backends want to do. The more I use flyte the more I feel this is completely upside down.
Fair ^ We are working on this Can we have a deep dive with these users. We would love to understand examples cc @brash-london-45337 ^
🙏 1
cc @thankful-minister-83577 / @high-accountant-32689
@brash-london-45337 here is a solution we have are thinking of implementing for Json in Flyte - https://github.com/flyteorg/flyte/issues/5318
🙏 1
b
Thanks for the heads up! Forwarded to the team
f
Also we are working on some of the other ones too
🙌 1
@brash-london-45337 here are the issues - https://github.com/flyteorg/flyte/issues/5320
6 Views
Open in Slack
PreviousNext
Powered by