https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
f

Franco Bocci

02/14/2023, 8:17 AM
Hey team, good morning! I’m trying to expose Flyte logs from K8s tasks to Datadog. I saw there was a question about this here and I could not find a work around for this. I was wondering: a) Is there a way to set a fixed name for the container running the task? b) Is there a way to set a volume for the pod running the task? I checked the k8s plugin config but could not find an answer for these. Thanks in advance!
s

Samhita Alla

02/14/2023, 9:08 AM
f

Franco Bocci

02/14/2023, 9:18 AM
Morning. Thanks for reaching out 🙂 . Okay, so a) would not work for this. Let me take a look at b) and the linked discussion
So, this seems to be for Datadog metrics, which is working fine for us. The problem for the time being is to integrate logs
Would you know if there is a way of setting 
primary_container_name
from the propeller config, so that every launched pod has that container name? That would be useful
s

Samhita Alla

02/14/2023, 9:27 AM
f

Franco Bocci

02/14/2023, 9:28 AM
Uuuuh, interesting. I’ve missed this one
Hey! So I’ve been checking on this one. The container name is overriden. I was thinking on setting an init-container to update the pods label before launching the task itself. Would you have any advice against this?
s

Samhita Alla

02/14/2023, 12:51 PM
It should be possible to specify init containers in the spec, right?
f

Franco Bocci

02/14/2023, 12:52 PM
Yes, it’s possible and that is what I am going to try! I was wondering if you have any advice against doing so 🙂
s

Samhita Alla

02/14/2023, 12:54 PM
Ah, gotcha. I think it can be done but I'll let @Dan Rammer (hamersaw) confirm.
b

Bernhard Stadlbauer

02/14/2023, 2:35 PM
@Franco Bocci We’re using datadog to collect logs from out Flyte tasks. I don’t remember having to do anything special to get this going - we’re also creating log-links that reference the pod_name (and not the container), which has been working well for us. As to how to mount a volume - you should be able to do that using PodTemplates (docs)
f

Franco Bocci

02/14/2023, 2:36 PM
Hey @Bernhard Stadlbauer! Thanks for reaching out. So… how did you set up the DD integration? It’s a broad question, so feel free to reply at a super high level
b

Bernhard Stadlbauer

02/14/2023, 4:36 PM
Sorry for the late reply here. I have not deploy our Datadog stack, so I might be wrong here (@Derek Yu feel free to correct me). I think we’re using DatadogAgents (a k8s Daemon set) to scrape all logs automatically. So after that, they should be available in the Logs explorer tab of your datadog instance. With that being available, we’ve created a custom URL that we configured in the Flyte config. The only thing we couldn’t get to work is to set the correct time in the URL, as datadog works in milliseconds but Flyte only offers seconds (or vice versa)
f

Franco Bocci

02/14/2023, 4:37 PM
Perfect! We have the DD Daemon’s Set working, but I’m getting stuck here when it asks to configure something using the container identifier
b

Bernhard Stadlbauer

02/14/2023, 4:39 PM
Do you need a special configuration per container? 🤔
f

Franco Bocci

02/14/2023, 4:40 PM
Nope, but just setting: • 
DD_LOGS_ENABLED
• 
DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL
Is not tracking logs
Maybe I’m missing this part? But I did not found this one in the other tutorial
b

Bernhard Stadlbauer

02/14/2023, 4:48 PM
Let me ask our infra team real quick
We’re deploying datadog using their helm chart and set the following config:
Copy code
logs:
      # datadog.logs.enabled -- Enables this to activate Datadog Agent log collection
      ## ref: <https://docs.datadoghq.com/agent/basic_agent_usage/kubernetes/#log-collection-setup>
      enabled: true

      # datadog.logs.containerCollectAll -- Enable this to allow log collection for all containers
      ## ref: <https://docs.datadoghq.com/agent/basic_agent_usage/kubernetes/#log-collection-setup>
      containerCollectAll: true

      # datadog.logs.containerCollectUsingFiles -- Collect logs from files in /var/log/pods instead of using container runtime API
      ## It's usually the most efficient way of collecting logs.
      ## ref: <https://docs.datadoghq.com/agent/basic_agent_usage/kubernetes/#log-collection-setup>
      containerCollectUsingFiles: true
f

Franco Bocci

02/14/2023, 5:29 PM
Thanks a lot for this. Will try this out tomorrow and come back
Appreciate the effort, really 🙂
Hey! So, I tried this out and it was definitely of help. What worked for me was: From DD’s Agent side: • Setting the config @Bernhard Stadlbauer shared • Mapping the volumes defined here . I thought this was optional and skipped it, but there is one volume that is to avoid loosing logs on container restart, and two volumes that are required for this (
logpodpath
and 
logcontainerpath
) From Flyte’s side: • Adding these tags and envs through the propeller config, I did not need to use a PodTemplate
Thanks a lot to everyone, you’ve been more than helpful 🙂
b

Bernhard Stadlbauer

02/15/2023, 3:19 PM
That’s great to hear! 🎉
s

Samhita Alla

02/16/2023, 4:32 AM
Awesome! @Franco Bocci would you be open to documenting this?
f

Franco Bocci

02/16/2023, 8:55 AM
Hey! Yes, sure. I haven’t contributed to this so if you could orient me or tell me where are the guidelines I should follow, I’m in 🙂
s

Samhita Alla

02/16/2023, 9:53 AM
Awesome, thank you! 🙂 You can add a section to https://github.com/flyteorg/flytesnacks/blob/master/cookbook/deployment/configure_logging_links.py page. It uses rst format.
f

Franco Bocci

02/22/2023, 12:10 PM
Hey! Working on this one here. As I said, never did this before so suggestions are more than welcome, even if I have to redo everything 🙂
s

Samhita Alla

02/22/2023, 1:34 PM
Thanks much, Franco! We really appreciate it. 🙂 Will review the PR very soon.
f

Franco Bocci

02/22/2023, 1:35 PM
No worries, take your time and let me know
s

Samhita Alla

03/10/2023, 6:58 AM
Hey @Franco Bocci! Left a couple of comments on the PR. Could you take a look at them?
f

Franco Bocci

03/17/2023, 11:42 AM
Hey! Updated this
Will be more active, changed my email in the github account to a one I use actively, I had one I stopped using this year so did not get notifications 😞
s

Samhita Alla

03/17/2023, 12:26 PM
No problem. Thanks!
36 Views
Powered by