Hi all, wondering if anyone can check my thinking re the appropriate abstraction to use (ShellTask vs ContainerTask). In my mind, these have a lot of overlap in functionality, but the raw ContainerTask is more of a catch-all. I think I should be using regular @tasks for any python driven tasks, ShellTask for anything that bash can control, and ContainerTask for everything else - in that order. My logic being that ContainerTask has more overhead since the container isn't Flyte-aware. Am I way off base? Also, I feel like ShellTask should be able to do anything ContainerTask can do.. given the right image.

Hi @Pryce, I second your opinion that a raw ContainerTask is a catch-all.

I think I should be using regular @tasks for any python driven tasks, ShellTask for anything that bash can control, and ContainerTask for everything else - in that order.

+1

My logic being that ContainerTask has more overhead since the container isn't Flyte-aware.

That shouldn't be the case because every task runs in a container. A raw ContainerTask is useful when you want to run non-Python tasks.

Hey @Samhita Alla, sorry for the delay, but thank you for confirming my thinking! The "more overhead" comment came from ContainerTask requiring Flyte CoPilot, but maybe ShellTask and @task also use it? I suppose ShellTask should be able to run anything that ContainerTask can run, given a custom image and the right command. So reformulating the original question: is it safe to assume that execution will be more efficient in a container with flytekit installed, than one without? There will be more dev overhead due to creating a custom image, but the execution at least should be quicker because it cuts down on sidecars.

Hey @Yee, would like to hear your thoughts on the performance comparison between container tasks and shell tasks.