https://flyte.org
Join Slack
Hey team, good morning! I’m trying to expose Flyte logs from K8s tasks to Datadog. I saw there was a...
# flyte-support
t
Hey team, good morning! I’m trying to expose Flyte logs from K8s tasks to Datadog. I saw there was a question about this here and I could not find a work around for this. I was wondering: a) Is there a way to set a fixed name for the container running the task? b) Is there a way to set a volume for the pod running the task? I checked the k8s plugin config but could not find an answer for these. Thanks in advance!
t
t
Morning. Thanks for reaching out 🙂 . Okay, so a) would not work for this. Let me take a look at b) and the linked discussion
So, this seems to be for Datadog metrics, which is working fine for us. The problem for the time being is to integrate logs
Would you know if there is a way of setting 
primary_container_name
from the propeller config, so that every launched pod has that container name? That would be useful
t
t
Uuuuh, interesting. I’ve missed this one
Hey! So I’ve been checking on this one. The container name is overriden. I was thinking on setting an init-container to update the pods label before launching the task itself. Would you have any advice against this?
t
It should be possible to specify init containers in the spec, right?
t
Yes, it’s possible and that is what I am going to try! I was wondering if you have any advice against doing so 🙂
t
Ah, gotcha. I think it can be done but I'll let @hallowed-mouse-14616 confirm.
a
@thankful-tailor-28399 We’re using datadog to collect logs from out Flyte tasks. I don’t remember having to do anything special to get this going - we’re also creating log-links that reference the pod_name (and not the container), which has been working well for us. As to how to mount a volume - you should be able to do that using PodTemplates (docs)
thx 1
t
Hey @agreeable-kitchen-44189! Thanks for reaching out. So… how did you set up the DD integration? It’s a broad question, so feel free to reply at a super high level
a
Sorry for the late reply here. I have not deploy our Datadog stack, so I might be wrong here (@straight-businessperson-54649 feel free to correct me). I think we’re using DatadogAgents (a k8s Daemon set) to scrape all logs automatically. So after that, they should be available in the Logs explorer tab of your datadog instance. With that being available, we’ve created a custom URL that we configured in the Flyte config. The only thing we couldn’t get to work is to set the correct time in the URL, as datadog works in milliseconds but Flyte only offers seconds (or vice versa)
t
Perfect! We have the DD Daemon’s Set working, but I’m getting stuck here when it asks to configure something using the container identifier
a
Do you need a special configuration per container? 🤔
t
Nope, but just setting: • 
DD_LOGS_ENABLED
• 
DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL
Is not tracking logs
Maybe I’m missing this part? But I did not found this one in the other tutorial
a
Let me ask our infra team real quick
gratitude thank you 1
We’re deploying datadog using their helm chart and set the following config:
Copy code
logs:
      # datadog.logs.enabled -- Enables this to activate Datadog Agent log collection
      ## ref: <https://docs.datadoghq.com/agent/basic_agent_usage/kubernetes/#log-collection-setup>
      enabled: true

      # datadog.logs.containerCollectAll -- Enable this to allow log collection for all containers
      ## ref: <https://docs.datadoghq.com/agent/basic_agent_usage/kubernetes/#log-collection-setup>
      containerCollectAll: true

      # datadog.logs.containerCollectUsingFiles -- Collect logs from files in /var/log/pods instead of using container runtime API
      ## It's usually the most efficient way of collecting logs.
      ## ref: <https://docs.datadoghq.com/agent/basic_agent_usage/kubernetes/#log-collection-setup>
      containerCollectUsingFiles: true
➕ 1
t
Thanks a lot for this. Will try this out tomorrow and come back
Appreciate the effort, really 🙂
Hey! So, I tried this out and it was definitely of help. What worked for me was: From DD’s Agent side: • Setting the config @agreeable-kitchen-44189 shared • Mapping the volumes defined here . I thought this was optional and skipped it, but there is one volume that is to avoid loosing logs on container restart, and two volumes that are required for this (
logpodpath
and 
logcontainerpath
) From Flyte’s side: • Adding these tags and envs through the propeller config, I did not need to use a PodTemplate
Thanks a lot to everyone, you’ve been more than helpful 🙂
a
That’s great to hear! 🎉
🙌 1
t
Awesome! @thankful-tailor-28399 would you be open to documenting this?
t
Hey! Yes, sure. I haven’t contributed to this so if you could orient me or tell me where are the guidelines I should follow, I’m in 🙂
t
Awesome, thank you! 🙂 You can add a section to https://github.com/flyteorg/flytesnacks/blob/master/cookbook/deployment/configure_logging_links.py page. It uses rst format.
t
Hey! Working on this one here. As I said, never did this before so suggestions are more than welcome, even if I have to redo everything 🙂
🎉 1
t
Thanks much, Franco! We really appreciate it. 🙂 Will review the PR very soon.
t
No worries, take your time and let me know
🙂 1
t
Hey @thankful-tailor-28399! Left a couple of comments on the PR. Could you take a look at them?
👀 1
t
Hey! Updated this
Will be more active, changed my email in the github account to a one I use actively, I had one I stopped using this year so did not get notifications 😞
t
No problem. Thanks!
310 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by