https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
c

Carsten Klaus

08/24/2023, 6:01 PM
Hey Flyte community ❤️ We are making progress in integrating Flyte into our Kubernetes cluster. I assume you are aware of the absence of logs in the tasks. In the UI we only see "*No Logs Found*" In order to overcome that challenge we want to use Grafana Loki. We have installed it into our cluster, and Loki is perfectly accessible. Now we want to let Flyte know where to find these logs. Into our 
values.yaml
we added the following segment, but with no success. (I am awere that we want to log the entire namespace)
Copy code
task_logs:
    plugins:
      logs:
        templates:
          - displayName: Grafana Loki
            templateUris:
              - '<http://grafana-loki.loki.svc.cluster.local:3100/loki/api/v1/query_range?start=1692873712&end=1692880912&query={namespace=>"our-namespace"}' 
            messageFormat: 2   
            scheme: TaskExecution
What do we need to do in order to see the task logs in the UI.
y

Yee

08/24/2023, 7:01 PM
Copy code
plugins:
      logs:
        kubernetes-enabled: false
        cloudwatch-enabled: true
        cloudwatch-template-uri: <https://your-region.console.aws.amazon.com/cloudwatch/home?region=us-east-2#logsV2:log-groups/log-group/$252Faws$252Feks$252Fmy-eks-cluster$252Fworkloads/log-events/fluentbit-kube.var.log.containers.{{.podName}}_{{.namespace}}_{{.containerName}}-{{.containerID}}.log>
        stackdriver-enabled: false
is what we have internally.
if it’s not showing up, i would check that the configuration is actually making it in there.
i feel like it’s easy to set this stuff at the wrong level. that should be all you need though.
c

Carsten Klaus

08/24/2023, 8:20 PM
Hey @Yee , thanks for your quick response. we are not using AWS or Cloudwatch, but an on-premise solution with Grafana Loki as our log source. Is it even possible to show the logs in the UI this way?
@Yee I double checked my backend and the task pods YAML description. In none of them the grafana was visible
y

Yee

08/24/2023, 8:38 PM
turn that on and point it to loki instead of kubernetes
i think it’s just a template, there’s nothing forcing a real k8s endpoint.
c

Carsten Klaus

08/25/2023, 6:35 AM
Hey @Yee I hope I get it right. I changed my 
values.yaml
like this:. Still no logs in the UI
Copy code
task_logs:
    plugins:
      logs:
        kubernetes-enabled: true
        cloudwatch-enabled: false
        stackdriver-enabled: false
        templates:
          - displayName: Grafana Loki
            templateUris:
              - '<http://grafana-loki.loki.svc.cluster.local:3100/loki/api/v1/query_range?start=1692873712&end=1692880912&query={namespace=>"our-namespace"}' 
            messageFormat: 2   
            scheme: TaskExecution
Can you please tell me what lines of configuration I need to put into my 
values.yaml
for my 
helm
installation?
y

Yee

08/25/2023, 7:42 PM
i think the config is still incorrect no?
you need to specify KubernetesTemplateURI
the config interface changed a year back i think. it used to be nested under 
templates
but no longer is
c

Carsten Klaus

08/25/2023, 9:32 PM
Hi @Yee, I think I found the corresponding configuration. Can you have a look at this? Is there still something missing? I still dont see any logs in the UI
Copy code
task_logs:
    plugins:
      logs:
        kubernetes-enabled: true
        cloudwatch-enabled: false
        stackdriver-enabled: false
        kubernetes-template-uri: '<http://grafana-loki.loki.svc.cluster.local:3100/loki/api/v1/query_range?start=1692873712&end=1692880912&query={namespace=>"our-namespace"}'
y

Yee

08/25/2023, 9:33 PM
are you sure this is getting picked up?
i think that’s right
c

Carsten Klaus

08/27/2023, 7:32 PM
I am not sure whether they are picked up. I am using the following flyte-binary chart: https://artifacthub.io/packages/helm/flyte/flyte-binary Is that the reason why my above solution is not working? If logs dont work that would be a show stopper for us and we would uninstall Flyte again 😞
y

Yee

08/28/2023, 9:33 PM
can you check the start of the logs of the pod?
when the flyte pod starts up, it should print out the configs that it uses.
the container image also comes with a shell so you can attach to it and ls/cat files mounted into the pod.
k

Ketan (kumare3)

08/29/2023, 5:07 PM
hi @Carsten Klaus I am sorry you are going through so much trouble
let us try and help here, give me time to read through it. cc @David Espejo (he/him)
h

Haytham Abuelfutuh

08/29/2023, 6:10 PM
@Carsten Klaus, I can assure you log links do work, our customers in Union Cloud use them day in and day out 🙂 ... What is the task type you are running here? Is it just a
Copy code
@task()
def my_task():
  ...
Or is it a plugin (e.g. Spark task)? because those are configured in a different section
Can you also confirm the root section of the config? it should be:
Copy code
configmap:
  task_logs:
    plugins:
      logs:
        ...
k

Ketan (kumare3)

08/29/2023, 6:17 PM
@Carsten Klaus I added the grafana loki link to my sandbox deployment and it worked. Look at what @Haytham Abuelfutuh ^ recommended
c

Carsten Klaus

08/30/2023, 10:49 PM
Hello @David Espejo (he/him), @Haytham Abuelfutuh, @Ketan (kumare3) thanks for your support 🙂 I am on vacation now but @Manuel Hoffmann will contact you on how to proceed with our log issue.
m

Manuel Hoffmann

08/31/2023, 12:35 PM
Hi @David Espejo (he/him), @Haytham Abuelfutuh, @Ketan (kumare3), thanks for your support! 🙏🎉👍 We finally managed to get the task logs picked up by flyte. 🙌 I think we missed the 
configmap
key and had 
configuration.logging.plugins.kubernetes.enabled: false
. I was wondering, if these details were documented somewhere and we just didn't find the right resource? Anyway, huge thanks again 😉 Greets from Germany Manuel cc: @Carsten Klaus
h

Haytham Abuelfutuh

08/31/2023, 5:42 PM
Great to hear, @Manuel Hoffmann! I don't think this is properly documented... the closest thing is the automatic documentation generated for helm chart values files here: https://artifacthub.io/packages/helm/flyte/flyte pointing to: 
flyte.configmap.task_logs
Would you be open to writing up a doc on configuring log links perhaps? we can include that as part of docs.flyte.org so the next person wouldn't have to go through the same struggle 🙂
m

Manuel Hoffmann

09/04/2023, 6:21 AM
Hi @Haytham Abuelfutuh, we're still working on some details about the logs. Once we have finalized the setup, I'll have a chat with @Carsten Klaus about your suggestion! Tanks again and greets! Manuel
3 Views
Powered by