https://flyte.org logo
Join Slack
Powered by
I have set up logging using stackdriver: ``` t...
# flyte-support
a
I have set up logging using stackdriver:
Copy code
task_logs:
      plugins:
        logs:
          kubernetes-enabled: false
          # Enable GCP stackdriver integration for log display
          stackdriver-enabled: true
          stackdriver-logresourcename: k8s_container
        k8s-array:
          logs:
            config:
              stackdriver-enabled: true
              stackdriver-logresourcename: k8s_container
However it is not sending any logs to GCP, is there something I am missing? Is it providing a link to a cloud logging query, but there are no logs (even though the workflow does output logs)
a
@aloof-magazine-44547 I think this not well documented unfortunately. According to the code, the block needs the following:
Copy code
task_logs:
  plugins:
    logs: 
      stackdriver-enabled: true
      gcp-project: <name of the GCP project>
      stackdriver-logresourcename: <logresource name in stackdriver>
      stackdriver-template-uri: "<template_uri>"
as long as the template URI, I haven't tried this yet but a structure like the following should work:
Copy code
<https://console.cloud.google.com/logs/query;query=resource.labels.namespace_name%3D%22{{>`{{.namespace}}`}}%22%0Aresource.labels.pod_name%3D%22{{`{{.podName}}`}}%22%0Aresource.labels.container_name%3D%22{{`{{.containerName}}`}}%22;startTime={{`{{.podRFC3339StartTime}}`}};endTime={{`{{.podRFC3339FinishTime}}`}}?project=<your_GCP_project_ID>&angularJsUrl=%2Flogs%2Fviewer%3Fproject%3D<your_GCP_project_ID>"
could you try and report back the result? 🙇🏽
I think 
k8s-array
is not needed unless you're using Map Tasks
a
Thanks will try!
Unfortunately that did not work
a
uh, any logs in 
flyteadmin
?
a
No logs, could it be anything with the permissions? I added log writer
But those template uri are just for displaying a link to the logs right? The problem is those logs are not send to gcp logging in the first place
I have added a unique id to be able to search over all my logs for that id, but it doesn't find any match even though I see the logs in the pod itself
a
No logs,
could you get the logs from the flyteadmin Pod? I mean 
kubectl logs <flyteadmin-pod-name> -n flyte
The problem is those logs are not send to gcp logging in the first place
Yeah, it could be either that flyteadmin is not sending anything (flyte config issue), or is sending it to the wrong location (flyte config issue) or that sends logs to the right location but it's not allowed to write on the log resource (perms issue which could be part GCP/flyte config)
but if it's 2nd or 3rd case, we should see something in the logs
I mean, the flyteadmin Pod logs
a
actually the issue was with GKE and GCP, so the GKE cluster need to have workloads added for logging and the logging.googleapis.com needs to be enabled, my mistake
a
oh, so does it work now?
a
I am testing now
yes it works
thank you for looking into it!
a
this is great, I'll make sure to add this to the docs/reference implementation
a
In the end i only specified templateuri not gcp project and logresource
a
oh, interesting
9 Views
Open in Slack
PreviousNext
Powered by