https://flyte.org
Join Slack
I am using the terraform helm_release to deploy flyte to k8s. I need to set certain labels for the d...
# flyte-deployment
g
I am using the terraform helm_release to deploy flyte to k8s. I need to set certain labels for the deployment to be allowed on our cluster. I found the 'commonLabels' value that I am trying to populate using 'set' in the helm_release but I get an error on apply saying that it accepts only a string. How do I set multiple labels using this value? The other one is "deployment.labels". Both say they accept 'object' but assigning an object as a value in 'set' does not work. I tried this as well, but I am getting this error - 
error converting YAML to JSON: yaml: line 32: mapping values are not allowed in this context
Copy code
dynamic "set" {
  for_each = module.labels.tags

  content {
    name = "commonLabels.${set.key}"
    value = set.value
  }
}
Slack Conversation
t
Can you change the YAML file directly?
g
I cannot because the yaml is generated by terraform. I want to do this in the terraform helm_release resource somehow.
t
@average-finland-92144, you've any idea how to resolve this issue?
g
Tried this another way with values.yaml but still getting the same issue. I have this in my values.yaml
Copy code
commonLabels:
    team: ${team}
    project: ${project}
And then in my helm_release, I use the templatefile function to pass in the variables -
Copy code
values = [templatefile("values.yaml", {
  team                  = var.team,
  project               = var.project,
})]
I get this error on apply -
Copy code
│ Error: YAML parse error on flyte-binary/templates/deployment.yaml: error converting YAML to JSON: yaml: line 38: mapping values are not allowed in this context
│ 
│   with helm_release.flyte-backend,
│   on <http://flyte-backend-k8s.tf|flyte-backend-k8s.tf> line 1, in resource "helm_release" "flyte-backend":
│    1: resource "helm_release" "flyte-backend" {
How do I debug this issue? All I can find online is that this is due to a yaml formatting / indentation issue, however I have checked my values.yaml file with multiple yaml checkers / validators and all say it is valid.
a
Hi @gorgeous-beach-23305 and sorry for the delay. I've been investigating this -despite the fact I'm not an advanced Terraform user- but wondering: 1. Could you share the content/structure of the 
tags
variable? 2. How would you use the output of your code (
commonLabels.team = team1
,etc) to change the actual values.yaml content? Sorry if I'm asking dumb questions, just trying to get more info. This is not really the domain of Flyte itself, but we want to help as much as possible
g
Hi @average-finland-92144, 1. The tags variable is a map of key-value pairs -
Copy code
{
      'team' = 'my_team',
      'Project' = 'my_project'
}
But, ideally we want to use a custom values.yaml to override the default values.yaml so you can disregard the question regarding assigning the tags variable. 2. We will not use the commonLabels.team. This is a label that will be applied to all kubernetes resources that are created by this helm chart (flyte-binary). Basically we want to apply labels to the resources our team creates on kubernetes for auditing and metering purposes. commonLabels is here in flyte-binary's values.yaml here and we will just override this empty value as shown in my latest example. Our custom values.yaml is well formatted and validated, so that is not a problem. Let me know if you need more information.
t
@gorgeous-beach-23305, this error may crop up if linting isn't done right. Can you check if your YAML is formatted properly?
g
@tall-lock-23197 I have run the values.Yamil through multiple formatters/linters/validators and confirmed that it is formatted correctly and is a valid yaml file.
t
@freezing-boots-56761 need your help 😅
g
Yes.
I have templatized it to inject values from terraform as shown in my previous example -
Copy code
configuration:
  database:
    username: '${db_username}'
    password: '${db_password}'
    host: '${db_host}'
    dbname: '${db_name}'
In the terraform helm_resource, I set the values like this -
Copy code
values = [templatefile("values.yaml", {
  db_username             = var.db_username,
  db_host               = var.db_host,
})]
f
ah its a bug @gorgeous-beach-23305
🤦
Once this is merged, it should push a nightly helm chart to the OCI registry here: https://github.com/flyteorg/flyte/pkgs/container/helm-charts%2Fflyte-binary
g
@freezing-boots-56761 Ah ok! Any ETA on when that might be merged and will be available in the OCI registry?
f
first thing in the morning PDT
in 9 hours or so
g
Perfect! Thanks!
f
you can also use my branch to test via a local chart path now
g
@freezing-boots-56761 I am using your branch to test, I think the master is still not updated, I am getting a chart.yaml not found error when I try to use that repo. But about the commonLabels, does it apply the labels to all resources? Because it looks like the labels are not being applied to some pods. Even deployment.podLabels doesn't seem to be working, I mean applying labels to all pods. For some the labels are applied -
Copy code
38m         Normal    PolicyApplied             clusterpolicy/require-labels                           Deployment mlops-platform-infrastructure-flyte-namespace/flyte-backend-flyte-binary: pass
56m         Normal    PolicyApplied             clusterpolicy/require-labels                           Pod mlops-platform-infrastructure-flyte-namespace/flyte-backend-flyte-binary-647f4d875c-fhdd7: pass
38m         Normal    PolicyApplied             clusterpolicy/require-labels                           Pod mlops-platform-infrastructure-flyte-namespace/flyte-backend-flyte-binary-647f4d875c-zpvnq: pass
30m         Normal    PolicyApplied             clusterpolicy/require-labels                           Pod mlops-platform-infrastructure-flyte-namespace/flyte-backend-flyte-binary-647f4d875c-s7728: pass
And for some they are not -
Copy code
10m         Warning   PolicyViolation           clusterpolicy/require-labels                           Pod flyte/flyte-flyte-binary-6bfd5b44c-x6bpn: [environment-label] fail (blocked)
10m         Warning   PolicyViolation           clusterpolicy/require-labels                           Pod flyte/flyte-flyte-binary-6bfd5b44c-x6bpn: [ba-label] fail (blocked)
f
@gorgeous-beach-23305 that looks odd. that’s the same deployment basically but looks like from a different helm release name. is it possible it’s left over from an old release without labels applied? it should be a single deployment/pod only.
they also look like they’re from different namespaces: mlops-platform-infrastructure-flyte-namespace and flyte respectively
i’ll look into why the chart isn’t publishing
i'm able to access the chart fine @gorgeous-beach-23305. can you paste a snippet of your terraform code?
g
This issue is resolved. I can access the updated chart on the OCI registry and the labels are also being applied. On to the next issue 🙂 Thanks @freezing-boots-56761
🙌🏽 1
🙌 2
1064 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by