Hey team good morning I m trying to expose Flyte logs from K Flyte #ask-the-community

Hey team, good morning! I’m trying to expose Flyte...

Franco Bocci

02/14/2023, 8:17 AM

Hey team, good morning! I’m trying to expose Flyte logs from K8s tasks to Datadog. I saw there was a question about this here and I could not find a work around for this. I was wondering: a) Is there a way to set a fixed name for the container running the task? b) Is there a way to set a volume for the pod running the task? I checked the k8s plugin config but could not find an answer for these. Thanks in advance!

Samhita Alla

02/14/2023, 9:08 AM

Hi Franco! a) You can set the node name, not the pod name: https://github.com/flyteorg/flytekit/blob/c3d54b1867e597c9a8846ee552d9b974f11ae0bd/tests/flytekit/unit/remote/test_calling.py#L56 b) https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/pod/pod.html#multiple-containers: does this help? I found a thread on datadog: https://discuss.flyte.org/t/1034/it-s-pretty-hacky-but-you-can-get-logs-assuming-it-s-scrapin. @Yee, would you be able to help Franco?

Franco Bocci

02/14/2023, 9:18 AM

Morning. Thanks for reaching out 🙂 . Okay, so a) would not work for this. Let me take a look at b) and the linked discussion

Franco Bocci

02/14/2023, 9:20 AM

So, this seems to be for Datadog metrics, which is working fine for us. The problem for the time being is to integrate logs

Franco Bocci

02/14/2023, 9:23 AM

Would you know if there is a way of setting

primary_container_name

from the propeller config, so that every launched pod has that container name? That would be useful

Samhita Alla

02/14/2023, 9:27 AM

I believe this should help you: https://docs.flyte.org/en/latest/deployment/configuration/general.html#using-default-k8s-podtemplates

Franco Bocci

02/14/2023, 9:28 AM

Uuuuh, interesting. I’ve missed this one

Franco Bocci

02/14/2023, 11:53 AM

Hey! So I’ve been checking on this one. The container name is overriden. I was thinking on setting an init-container to update the pods label before launching the task itself. Would you have any advice against this?

Samhita Alla

02/14/2023, 12:51 PM

It should be possible to specify init containers in the spec, right?

Franco Bocci

02/14/2023, 12:52 PM

Yes, it’s possible and that is what I am going to try! I was wondering if you have any advice against doing so 🙂

Samhita Alla

02/14/2023, 12:54 PM

Ah, gotcha. I think it can be done but I'll let @Dan Rammer (hamersaw) confirm.

Bernhard Stadlbauer

02/14/2023, 2:35 PM

@Franco Bocci We’re using datadog to collect logs from out Flyte tasks. I don’t remember having to do anything special to get this going - we’re also creating log-links that reference the pod_name (and not the container), which has been working well for us. As to how to mount a volume - you should be able to do that using PodTemplates (docs)

Franco Bocci

02/14/2023, 2:36 PM

Hey @Bernhard Stadlbauer! Thanks for reaching out. So… how did you set up the DD integration? It’s a broad question, so feel free to reply at a super high level

Bernhard Stadlbauer

02/14/2023, 4:36 PM

Sorry for the late reply here. I have not deploy our Datadog stack, so I might be wrong here (@Derek Yu feel free to correct me). I think we’re using DatadogAgents (a k8s Daemon set) to scrape all logs automatically. So after that, they should be available in the Logs explorer tab of your datadog instance. With that being available, we’ve created a custom URL that we configured in the Flyte config. The only thing we couldn’t get to work is to set the correct time in the URL, as datadog works in milliseconds but Flyte only offers seconds (or vice versa)

Franco Bocci

02/14/2023, 4:37 PM

Perfect! We have the DD Daemon’s Set working, but I’m getting stuck here when it asks to configure something using the container identifier

Bernhard Stadlbauer

02/14/2023, 4:39 PM

Do you need a special configuration per container? 🤔

Franco Bocci

02/14/2023, 4:40 PM

Nope, but just setting: •

DD_LOGS_ENABLED

•

DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL

Is not tracking logs

Franco Bocci

02/14/2023, 4:42 PM

Maybe I’m missing this part? But I did not found this one in the other tutorial

Bernhard Stadlbauer

02/14/2023, 4:48 PM

Let me ask our infra team real quick

Bernhard Stadlbauer

02/14/2023, 5:12 PM

We’re deploying datadog using their helm chart and set the following config:

Copy code

logs:
      # datadog.logs.enabled -- Enables this to activate Datadog Agent log collection
      ## ref: <https://docs.datadoghq.com/agent/basic_agent_usage/kubernetes/#log-collection-setup>
      enabled: true

      # datadog.logs.containerCollectAll -- Enable this to allow log collection for all containers
      ## ref: <https://docs.datadoghq.com/agent/basic_agent_usage/kubernetes/#log-collection-setup>
      containerCollectAll: true

      # datadog.logs.containerCollectUsingFiles -- Collect logs from files in /var/log/pods instead of using container runtime API
      ## It's usually the most efficient way of collecting logs.
      ## ref: <https://docs.datadoghq.com/agent/basic_agent_usage/kubernetes/#log-collection-setup>
      containerCollectUsingFiles: true

Franco Bocci

02/14/2023, 5:29 PM

Thanks a lot for this. Will try this out tomorrow and come back

Franco Bocci

02/14/2023, 5:29 PM

Appreciate the effort, really 🙂

Franco Bocci

02/15/2023, 3:13 PM

Hey! So, I tried this out and it was definitely of help. What worked for me was: From DD’s Agent side: • Setting the config @Bernhard Stadlbauer shared • Mapping the volumes defined here . I thought this was optional and skipped it, but there is one volume that is to avoid loosing logs on container restart, and two volumes that are required for this (

logpodpath

and

logcontainerpath

) From Flyte’s side: • Adding these tags and envs through the propeller config, I did not need to use a PodTemplate

Franco Bocci

02/15/2023, 3:13 PM

Thanks a lot to everyone, you’ve been more than helpful 🙂

Bernhard Stadlbauer

02/15/2023, 3:19 PM

That’s great to hear! 🎉

Samhita Alla

02/16/2023, 4:32 AM

Awesome! @Franco Bocci would you be open to documenting this?

Franco Bocci

02/16/2023, 8:55 AM

Hey! Yes, sure. I haven’t contributed to this so if you could orient me or tell me where are the guidelines I should follow, I’m in 🙂

Samhita Alla

02/16/2023, 9:53 AM

Awesome, thank you! 🙂 You can add a section to https://github.com/flyteorg/flytesnacks/blob/master/cookbook/deployment/configure_logging_links.py page. It uses rst format.

Franco Bocci

02/22/2023, 12:10 PM

Hey! Working on this one here. As I said, never did this before so suggestions are more than welcome, even if I have to redo everything 🙂

Samhita Alla

02/22/2023, 1:34 PM

Thanks much, Franco! We really appreciate it. 🙂 Will review the PR very soon.

Franco Bocci

02/22/2023, 1:35 PM

No worries, take your time and let me know

Samhita Alla

03/10/2023, 6:58 AM

Hey @Franco Bocci! Left a couple of comments on the PR. Could you take a look at them?

Franco Bocci

03/17/2023, 11:42 AM

Hey! Updated this

Franco Bocci

03/17/2023, 11:43 AM

Will be more active, changed my email in the github account to a one I use actively, I had one I stopped using this year so did not get notifications 😞

Samhita Alla

03/17/2023, 12:26 PM

No problem. Thanks!

248 Views

Open in Slack

Previous Next