https://flyte.org logo
Title
b

Broder Peters

05/26/2023, 11:56 AM
Spoiler: I'm pretty much a Python noob. I've started writing a plugin, deriving my plugin Task from
PythonCustomizedContainerTask[T]
and
PythonTask[PluginConfig]
. I then started using my plugin in the workflow, which works out fine. Now I would like to apply caching and other usual decorators like requests and limit. How can I achieve this, as I don't have the classic
Task(...)
decorator for my plugin?
class GlueTask(PythonCustomizedContainerTask[T], PythonTask[GlueConfig]):
    # following the usual init

class GlueTaskExecutor(ShimTaskExecutor[GlueTask]):
    def execute_from_model(self, tt: task_models.TaskTemplate, **kwargs) -> typing.Any:
        # my plugin execution code

# usage in workflow
GlueTask(name="testing", task_config=GlueConfig(crawler_name="test-crawler"), container_image="bla",cache=True, cache_version="1.0")()
That
cache
parameters don't throw errors, but also don't do caching.
Yes that worked 😄
GlueTask(name="testing", task_config=GlueConfig(crawler_name="test-crawler-delete-me"), container_image="bla", metadata=TaskMetadata(cache=True, cache_version="1.0"))()
y

Yee

05/28/2023, 12:29 AM
nice. welcome @Broder Peters
so the background on this "PythonCustomizedContainerTask" is that we wanted to offer users a way to run tasks where the task definition (ie everything you need to run the task, is all encapsulated in the task template). in these cases, you don't need to build different containers for each task, you can just have one.
there's a new style forthcoming as well that will do something similar - the interface for which should we think is a bit simpler. be on the lookout for information on the flyte 'agent service' and agent tasks if interested.
b

Broder Peters

05/30/2023, 9:00 AM
Cool, thanks for the input!