https://flyte.org
Join Slack
Hey Flyte community :heart: We are making progress in integrating Flyte into our Kubernetes cluster...
# flyte-support
s
Hey Flyte community ❤️ We are making progress in integrating Flyte into our Kubernetes cluster. I assume you are aware of the absence of logs in the tasks. In the UI we only see "*No Logs Found*" In order to overcome that challenge we want to use Grafana Loki. We have installed it into our cluster, and Loki is perfectly accessible. Now we want to let Flyte know where to find these logs. Into our 
values.yaml
we added the following segment, but with no success. (I am awere that we want to log the entire namespace)
Copy code
task_logs:
    plugins:
      logs:
        templates:
          - displayName: Grafana Loki
            templateUris:
              - '<http://grafana-loki.loki.svc.cluster.local:3100/loki/api/v1/query_range?start=1692873712&end=1692880912&query={namespace=>"our-namespace"}' 
            messageFormat: 2   
            scheme: TaskExecution
What do we need to do in order to see the task logs in the UI.
t
Copy code
plugins:
      logs:
        kubernetes-enabled: false
        cloudwatch-enabled: true
        cloudwatch-template-uri: <https://your-region.console.aws.amazon.com/cloudwatch/home?region=us-east-2#logsV2:log-groups/log-group/$252Faws$252Feks$252Fmy-eks-cluster$252Fworkloads/log-events/fluentbit-kube.var.log.containers.{{.podName}}_{{.namespace}}_{{.containerName}}-{{.containerID}}.log>
        stackdriver-enabled: false
is what we have internally.
if it’s not showing up, i would check that the configuration is actually making it in there.
i feel like it’s easy to set this stuff at the wrong level. that should be all you need though.
s
Hey @thankful-minister-83577 , thanks for your quick response. we are not using AWS or Cloudwatch, but an on-premise solution with Grafana Loki as our log source. Is it even possible to show the logs in the UI this way?
@thankful-minister-83577 I double checked my backend and the task pods YAML description. In none of them the grafana was visible
t
turn that on and point it to loki instead of kubernetes
i think it’s just a template, there’s nothing forcing a real k8s endpoint.
s
Hey @thankful-minister-83577 I hope I get it right. I changed my 
values.yaml
like this:. Still no logs in the UI
Copy code
task_logs:
    plugins:
      logs:
        kubernetes-enabled: true
        cloudwatch-enabled: false
        stackdriver-enabled: false
        templates:
          - displayName: Grafana Loki
            templateUris:
              - '<http://grafana-loki.loki.svc.cluster.local:3100/loki/api/v1/query_range?start=1692873712&end=1692880912&query={namespace=>"our-namespace"}' 
            messageFormat: 2   
            scheme: TaskExecution
Can you please tell me what lines of configuration I need to put into my 
values.yaml
for my 
helm
installation?
t
i think the config is still incorrect no?
you need to specify KubernetesTemplateURI
the config interface changed a year back i think. it used to be nested under 
templates
but no longer is
s
Hi @thankful-minister-83577, I think I found the corresponding configuration. Can you have a look at this? Is there still something missing? I still dont see any logs in the UI
Copy code
task_logs:
    plugins:
      logs:
        kubernetes-enabled: true
        cloudwatch-enabled: false
        stackdriver-enabled: false
        kubernetes-template-uri: '<http://grafana-loki.loki.svc.cluster.local:3100/loki/api/v1/query_range?start=1692873712&end=1692880912&query={namespace=>"our-namespace"}'
t
are you sure this is getting picked up?
i think that’s right
s
I am not sure whether they are picked up. I am using the following flyte-binary chart: https://artifacthub.io/packages/helm/flyte/flyte-binary Is that the reason why my above solution is not working? If logs dont work that would be a show stopper for us and we would uninstall Flyte again 😞
👀 1
t
can you check the start of the logs of the pod?
when the flyte pod starts up, it should print out the configs that it uses.
the container image also comes with a shell so you can attach to it and ls/cat files mounted into the pod.
f
hi @strong-carpet-69168 I am sorry you are going through so much trouble
let us try and help here, give me time to read through it. cc @average-finland-92144
h
@strong-carpet-69168, I can assure you log links do work, our customers in Union Cloud use them day in and day out 🙂 ... What is the task type you are running here? Is it just a
Copy code
@task()
def my_task():
  ...
Or is it a plugin (e.g. Spark task)? because those are configured in a different section
Can you also confirm the root section of the config? it should be:
Copy code
configmap:
  task_logs:
    plugins:
      logs:
        ...
f
@strong-carpet-69168 I added the grafana loki link to my sandbox deployment and it worked. Look at what @high-park-82026 ^ recommended
🙇🏽 1
👀 1
s
Hello @average-finland-92144, @high-park-82026, @freezing-airport-6809 thanks for your support 🙂 I am on vacation now but @bulky-machine-98870 will contact you on how to proceed with our log issue.
b
Hi @average-finland-92144, @high-park-82026, @freezing-airport-6809, thanks for your support! 🙏🎉👍 We finally managed to get the task logs picked up by flyte. 🙌 I think we missed the 
configmap
key and had 
configuration.logging.plugins.kubernetes.enabled: false
. I was wondering, if these details were documented somewhere and we just didn't find the right resource? Anyway, huge thanks again 😉 Greets from Germany Manuel cc: @strong-carpet-69168
🙇🏽 1
🔥 3
🙇 2
h
Great to hear, @bulky-machine-98870! I don't think this is properly documented... the closest thing is the automatic documentation generated for helm chart values files here: https://artifacthub.io/packages/helm/flyte/flyte pointing to: 
flyte.configmap.task_logs
Would you be open to writing up a doc on configuring log links perhaps? we can include that as part of docs.flyte.org so the next person wouldn't have to go through the same struggle 🙂
b
Hi @high-park-82026, we're still working on some details about the logs. Once we have finalized the setup, I'll have a chat with @strong-carpet-69168 about your suggestion! Tanks again and greets! Manuel
❤️ 1
42 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by