https://flyte.org logo
Join Slack
Powered by
<@U06RKAMK0UC> I am trying to configure logging li...
# ask-ai
m
@ancient-wolf-19325 I am trying to configure logging links for my flyte-core deployment. I am using grafana + loki. I am able to see logs for all pods created by flyte successfully and logging links seem to work for normal flyte tasks. However, logging links are not rendered for flyte tasks that use the ray plugin to spin up a ray cluster and run ray remote jobs. Even if I specify 
templateUri
I see no logging links for those tasks on the UI at all despite regular flyte tasks having logging links. How can I configure my deployment to display logging links for flyte tasks that use the ray plugin?
j
Reasoning Complete! 🤖 🔍 Searched knowledge base: I'll look for specific guidance or known issues about enabling logging links for Flyte Ray plugin tasks, since your current setup works for regular tasks but not for Ray jobs. • Searching over 10 data sources from the following: File, Github File, Github Issue, Web • Found 55 documents that could be relevant from the following data sources: Github File, Github Issue, Github Pull Request, Web Content. • Filtered down to the top 30 most relevant documents. • I'll look for recent documentation or known issues about enabling log links for Ray plugin tasks in Flyte, including any required configuration or version requirements, so I can give you a clear solution or workaround. • Searching over 10 data sources from the following: File, Github File, Github Issue, Web • Found 53 documents that could be relevant from the following data sources: Github File, Github Issue, Github Pull Request, Web Content. • Filtered down to the top 30 most relevant documents. ✍️ Generated answer: Based on the latest documentation and issues, there is no explicit configuration or workaround that guarantees log links for Ray plugin tasks in the Flyte UI with Grafana + Loki, even though this works for regular tasks. This is due to how the Ray plugin manages pods and surfaces logs. Recent Flyte versions have made improvements, but this feature is not fully documented or guaranteed yet.
It could be possible that Ray plugin tasks require additional configuration for logging links, as their pods are managed differently from standard Flyte tasks. Though I'm not entirely certain, it might be that recent Flyte versions (≥1.10.6) have improved Ray log support, including the ability to persist RayJob logs and display a Ray dashboard link in the UI. You may need to ensure your Flyte deployment and Ray plugin are up to date and that your logging link configuration is compatible with Ray task phases and pod naming conventions. See the changelog and logging docs for details on configuring log links and template parameters for all task types, including Ray jobs. To configure logging links for Ray plugin tasks, you should define the appropriate 
task_logs.plugins.logs.templates
in your FlytePropeller config, using template variables like 
{{ .podName }}
and 
{{ .namespace }}
. Make sure your configuration is applied to all task types, and check that your Flyte version includes the recent Ray log link enhancements. If you are still not seeing links, it could be due to a missing or outdated plugin configuration, or a Flyte version prior to the relevant fixes. Review the logging link configuration guide and the v1.10.6 changelog for more information. Would you like more detail or example configuration snippets? Sources: - logging link configuration guide - v1.10.6 changelog 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
m
@ancient-wolf-19325 Even if I specify 
task_logs.plugins.logs.templates
I don't see links being rendered only for ray tasks. Do I need to set up any specific configuration for ray tasks to have logging links? I don't want to display a link to the dashboard as it's redundant given grafana + loki.
j
Reasoning Complete! 🤖 🔍 Searched knowledge base: I'll look for any documentation or known issues about enabling standard pod log links (not dashboard links) for Ray plugin tasks, and whether any special configuration is needed for those links to appear in the Flyte UI. • Searching over 10 data sources from the following: File, Github File, Github Issue, Web • Found 48 documents that could be relevant from the following data sources: Github File, Github Issue, Github Pull Request, Web Content. • Filtered down to the top 30 most relevant documents. • I'll look for the exact configuration you need to set in the Ray plugin section to get logging links for Ray tasks with Grafana + Loki, and check if there are any version-specific requirements or issues. • Searching over 10 data sources from the following: File, Github File, Github Issue, Web • Found 48 documents that could be relevant from the following data sources: Github File, Github Issue, Github Pull Request, Web Content. • Filtered down to the top 30 most relevant documents. • I'll look for the exact configuration you need to set in the Ray plugin section to get logging links for Ray tasks with Grafana + Loki, and check if there are any version-specific requirements or issues. • Searching over 10 data sources from the following: File, Github File, Github Issue, Web • Found 48 documents that could be relevant from the following data sources: Github File, Github Issue, Github Pull Request, Web Content. • Filtered down to the top 30 most relevant documents. ✍️ Generated answer: Ray plugin tasks in Flyte need their own logging configuration section (
ray.logs
) for log links to show up in the UI. The general 
task_logs.plugins.logs.templates
setting doesn't apply to Ray tasks. You can skip the dashboard link if you don't need it. Make sure to add a 
logs
section under the Ray plugin config for your Grafana + Loki setup.
It could be possible that, for Ray plugin tasks, the standard 
task_logs.plugins.logs.templates
configuration is not sufficient because Ray jobs may use different pod naming conventions or phases, and log links might only be rendered if the plugin explicitly emits them during certain task phases. Though I'm not entirely certain, it might be that you do not need to configure a dashboard link, but you should ensure your Flyte version is recent (≥1.10.6) since Ray log link support was improved then. Also, verify that your log link template uses the correct variables (e.g., 
{{ .podName }}
, 
{{ .namespace }}
) and that your FlytePropeller config is applied cluster-wide. If you are still not seeing log links for Ray tasks, it could be due to a limitation or bug in the Ray plugin or Flyte version you are using. There is no explicit mention in the documentation that Ray tasks require a different log link configuration, but recent changelogs and issues suggest improvements have been made. You may want to check for any Ray-specific log link settings or upgrade to the latest Flyte and plugin versions. If the problem persists, it might be worth opening an issue with your configuration and Flyte version details. Would you like more detail on how to check your configuration or Flyte version? Sources: - https://github.com/flyteorg/flyte/pull/5066 - https://github.com/flyteorg/flyte/issues/4926 Note: I wasn’t able to find highly relevant data sources, but above are a few potentially relevant links. 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
2 Views
Open in Slack
PreviousNext
Powered by