Hello! I'm having some issues with flyte when isti...
# ask-the-community
c
Hello! I'm having some issues with flyte when istio sidecars are present. When the flyte task container finishes, the pod does not terminate (because the istio sidecar is active) so it is stuck forever. Have you come across this? Do you have any tips or tricks? Thanks very much!
b
Hi! We're also using
istio
over here and we've chosen setup our namespace such that the sidecar does not get injected. You should be able to check if injection is enabled using:
kubectl get namespace -L istio-injection
You could also add the
<http://sidecar.istio.io/inject|sidecar.istio.io/inject>: "false"
annotation by default to the pod by adding it to the
configmap.k8s.plugins.k8s.default-annotations
config value. Also of this only works assuming that you don't need the sidecar as part of the pod's work of course.
👍 1
k
Thank you @Bernhard Stadlbauer for the suggestions To dive deeper into the problem, if a sidecar is injected k8s will prevent the pod from ending/stopping. And flyte container 🔌 j waits for the container to exit But, if you use Flyte pod task it will work correctly- because it will look at the primary container only https://docs.flyte.org/projects/cookbook/en/stable/auto/integrations/kubernetes/pod/index.html
👍 1
c
Thanks @Ketan (kumare3)! I take a look on how to use this
k
Ya you just have to set the confit
Config
c
Hello @Ketan (kumare3), I tried using a Flyte pod task. In the UI flyte shows the task as completed but the istio sidecar remains stuck. I used the code below, is there something I'm missing? Thanks
```import typing
import pandas as pd
import numpy as np
from flytekitplugins.pod import Pod
from kubernetes.client.models import V1Container, V1PodSpec
from flytekit import task, workflow
def generate_pod_spec_for_task():
# Primary containers do not require us to specify an image, the default image built for Flyte tasks will get used.
primary_container = V1Container(name="primary")
pod_spec = V1PodSpec(containers=[primary_container])
return pod_spec
@task(task_config=Pod(pod_spec=generate_pod_spec_for_task(), primary_container_name="primary"))
def generate_normal_df(n: int, mean: float, sigma: float) -> pd.DataFrame:
return pd.DataFrame({"numbers": np.random.normal(mean, sigma, size=n)})
@task(task_config=Pod(pod_spec=generate_pod_spec_for_task(), primary_container_name="primary"))
def compute_stats(df: pd.DataFrame) -> typing.Tuple[float, float]:
return float(df["numbers"].mean()), float(df["numbers"].std())
@workflow
def wf(n: int = 200, mean: float = 0.0, sigma: float = 1.0) -> typing.Tuple[float, float]:
return compute_stats(df=generate_normal_df(n=n, mean=mean, sigma=sigma))```
k
You will have to enable delete on finalize. Flyte otherwise keeps the pods around for a configurable gc interval to easily access logs
👍 1
c
Thanks very much, @Ketan (kumare3)! now flyte deletes the istio proxy sidecar and finishes the pipeline with success.
🦜 1
k
@caioau would you be open to writing a doc piece for this?
280 Views