Nuno Martins
01/23/2024, 4:27 PMShahwar Saleem
01/23/2024, 6:18 PMAlex Beach
01/23/2024, 11:13 PMkubectl get certificaterequest -n flyte
No resources found in flyte namespace.
Alex Beach
01/24/2024, 1:07 AMuseAuth: true
per docs for flyte-core, but the UI does not prompt for a login.
https://docs.flyte.org/en/latest/deployment/configuration/auth_setup.htmlAlex Beach
01/24/2024, 1:17 AMMichael Tinsley
01/25/2024, 5:12 PMdefault-pod-template-name
on a flyte-core v1.10.6 deployment…
I’m currently deploying it with this
configmap:
...
k8s:
plugins:
default-cpus: 500m
default-memory: 512Mi
default-pod-template-name: "flyte-default"
inject-finalizer: true
However, whilst propeller successfully registers the template, it isn’t being used as a default. I’m pretty sure this is the correct place to define it as per here. And the template works when explicitly requested within a task. Any pointers?Griffin Brome
02/02/2024, 11:32 PMValentin
02/03/2024, 2:37 PMGuy Arad
02/04/2024, 11:05 AM...
configuration
...
inline:
plugins:
k8s:
inject-finalizer: false
default-env-vars:
FLYTE_AWS_ACCESS_KEY_ID: "foo"
FLYTE_AWS_SECRET_ACCESS_KEY: "secret-bar"
FLYTE_AWS_ENDPOINT: "custom-endpoint"
I logged into the binary pod and located the inline config: /etc/flyte/config.d/100-inline-config.yaml
. I can clearly see that the default-env-vars
is not lowercased.
I tried a little to locate the relevant Go code but didn't find anything that lower-case the env var names.
Thanks for your help!Devendra Johari
02/05/2024, 6:38 AMAlex Beach
02/07/2024, 11:20 PMGriffin Brome
02/09/2024, 7:21 PMflyte-core
on GCP using helm, and I seem to be running into an issue when injecting k8s secrets via the flytepropeller webhook.
RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook "flyte-pod-webhook.flyte.org": failed to call webhook: Post "<https://flyte-backend-flyte-binary-webhook.flyte.svc:443/mutate--v1-pod?timeout=10s>": service "flyte-backend-flyte-binary-webhook" not found
It seems that the service endpoint is not correct, as there is no service with that name. I’ve tried creating a new service, however it still cannot be accessed, since the certs are not valid.
failed Execute for node. Error: failed at Node[g<my-task>]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook \"flyte-pod-webhook.flyte.org\": failed to call webhook: Post \"<https://flyte-backend-flyte-binary-webhook.flyte.svc:443/mutate--v1-pod?timeout=10s>\": tls: failed to verify certificate: x509: certificate is valid for flyte-pod-webhook, flyte-pod-webhook.flyte, flyte-pod-webhook.flyte.svc, not flyte-backend-flyte-binary-webhook.flyte.svc","ts":"2024-02-09T19:15:22Z"}
Ideally I would like to just have flytepropeller call the service: flyte-pod-webhook
that is already created by the helm installation. Can anyone help me out with this?Guy Harel
02/12/2024, 1:23 PMRamsey Natour
02/14/2024, 8:28 PMnode-selector
on the k8s-array plugin is? Does this affect the pods that are spun up to fulfill a map_task
or something else?Daniel Howard
02/16/2024, 11:09 PMDebajyoti Chatterjee
02/19/2024, 9:13 AM{"json":{"exec_id":"atsr9qglqwgmsfjc4fzx","node":"n1"},"level":"error","msg":"Failed to read from the raw store [<s3://flyte-dataplane/metadata/propeller/flytesnacks-development-atsr9qglqwgmsfjc4fzx/n1/data/inputs.pb>] Error: Conf container:flyte-controlplane != Passed Container:flyte-dataplane. Dynamic loading is disabled: not found","ts":"2024-02-19T08:34:35Z"}
{"json":{"exec_id":"awg84z5mvwbmql2qrc9n","node":"n0"},"level":"warning","msg":"Failed to read inputs from URI [<s3://flyte-dataplane/metadata/propeller/flytesnacks-development-awg84z5mvwbmql2qrc9n/n0/data/inputs.pb>] with err: path:<s3://flyte-dataplane/metadata/propeller/flytesnacks-development-awg84z5mvwbmql2qrc9n/n0/data/inputs.pb>: Conf container:flyte-controlplane != Passed Container:flyte-dataplane. Dynamic loading is disabled: not found","ts":"2024-02-19T08:46:10Z"}
{"json":{"exec_id":"awg84z5mvwbmql2qrc9n","node":"n0"},"level":"error","msg":"Failed to read from the raw store [<s3://flyte-dataplane/metadata/propeller/flytesnacks-development-awg84z5mvwbmql2qrc9n/n0/data/0/outputs.pb>] Error: Conf container:flyte-controlplane != Passed Container:flyte-dataplane. Dynamic loading is disabled: not found","ts":"2024-02-19T08:46:10Z"}
If the control plane uses a different bucket from the Propeller, it refuses to load anything. Can we circumvent this behaviour? I was not expecting the "Recover" feature to also stop working; if I understand correctly, the control plane is trying to "bind" the outputs of the previously successful workflow nodes while recovering, and it is not proceeding because of the mismatching Bucket name.Debajyoti Chatterjee
02/19/2024, 9:40 AMinputs:
literals:
tokenizer:
scalar:
primitive:
stringValue: a
Is there any way we can ask the control plane components to not keep the actual values in the control plane bucket?Yini Gao
02/20/2024, 5:04 PMError: template: flyte-core/charts/flyteagent/templates/agent/deployment.yaml:17:17: executing "flyte-core/charts/flyteagent/templates/agent/deployment.yaml" at <include "flyteagent.podLabels" .>: error calling include: template: flyte-core/templates/_helpers.tpl:122:16: executing "flyteagent.podLabels" at <.Values.flyteagent.podLabels>: nil pointer evaluating interface {}.podLabels
The context is that in the value.yaml
, I set
flyteagent:
enabled: true
While if I remove the flyteagent
section in value.yaml
, it got successfully deployed.
I’m wondering if it’s relevant to the podLabels
is not configured in the default values here: https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values.yaml#L274-L281Julian
02/21/2024, 6:57 AMDenis Skakalski
02/21/2024, 2:58 PMDevin McCabe
02/23/2024, 4:12 PMhelm_release.flyte-core: Still creating... [5m10s elapsed]
╷
│ Warning: Helm release "flyte-core" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
│
│ with helm_release.flyte-core,
│ on <http://flyte.tf|flyte.tf> line 6, in resource "helm_release" "flyte-core":
│ 6: resource "helm_release" "flyte-core" {
│
╵
╷
│ Error: context deadline exceeded
I'm not sure how to "use the helm
command to investigate". I do wonder whether this entry in Log Explorer is related:
"panic: failed to connect to `host=10.99.0.3 user=flyteadmin database=flyteadmin`: failed SASL auth (FATAL: password authentication failed for user "flyteadmin" (SQLSTATE 28P01))
goroutine 1 [running]:
<http://github.com/flyteorg/flyte/datacatalog/pkg/repositories.Migrate({0x26dd960|github.com/flyteorg/flyte/datacatalog/pkg/repositories.Migrate({0x26dd960>, 0x38a2e00})
/go/src/github.com/flyteorg/datacatalog/pkg/repositories/initialize.go:63 +0x786"
This is a fresh install, so I wonder if maybe flyte-deploy and flyte-core have gotten out of sync somehow?Chris Schafer
02/27/2024, 11:11 PMBroder Peters
02/29/2024, 8:34 AMflytectl demo start
I assume that there are plenty of arguments and commands used to start the container (also by looking veeery briefly into the golang code behind the command).
Does a docker run [...]
command for the demo/sandbox image exists that I could use? I would love to keep the setup of hosting very simple and not bother with installing flytectl first (as AWS App Runner just takes docker images and runs them)Alex Beach
02/29/2024, 11:52 PMJonas G
03/05/2024, 2:32 PM{"json":{"src":"controller.go:602"},"level":"warning","msg":"Failed to create cluster resources for namespace [flytesnacks-production] with err: Failed to create kubernetes object from config template [002_serviceaccount.yaml] for namespace [flytesnacks-production] with err: serviceaccounts is forbidden: User \"sys
tem:serviceaccount:flyte:flyte-backend-flyte-binary\" cannot create resource \"serviceaccounts\" in API group \"\" in the namespace \"flytesnacks-production\"","ts":"2024-03-05T14:11:44Z"}
my current helm release looks like:
inline:
tasks:
task-plugins:
enabled-plugins:
- container
- sidecar
- K8S-ARRAY
default-for-task-types:
- container: container
- container_array: k8s-array
cluster_resources:
customData:
- production:
- gsa:
value: <mailto:cool-gcp-sa@gke-foo-bar.iam.gserviceaccount.com|cool-gcp-sa@gke-foo-bar.iam.gserviceaccount.com>
- staging:
- gsa:
value: <mailto:cool-gcp-sa@gke-foo-bar.iam.gserviceaccount.com|cool-gcp-sa@gke-foo-bar.iam.gserviceaccount.com>
- development:
- gsa:
value: <mailto:cool-gcp-sa@gke-foo-bar.iam.gserviceaccount.com|cool-gcp-sa@gke-foo-bar.iam.gserviceaccount.com>
clusterResourceTemplates:
inline:
001_namespace.yaml: |
apiVersion: v1
kind: Namespace
metadata:
name: '{{ namespace }}'
002_serviceaccount.yaml: |
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
namespace: '{{ namespace }}'
annotations:
<http://iam.gke.io/gcp-service-account|iam.gke.io/gcp-service-account>: '{{ gsa }}'
Help would be much appreciated, thanks in advance.Cornelis Boon
03/06/2024, 5:27 PMeks-starter.yaml
mentioned in the docs applies to deploying AWS's EKS. Is there a GCP/GKE equivalent (don't see it in repo) or should I just edit the values.yaml
?
• I used to use gcp's cloudsql-proxy with kubeflow. Can I use this with flyte too or is there a better way to set things up (e.g. VPC)?Jegadesh Thirumeni
03/07/2024, 10:06 AMCornelis Boon
03/08/2024, 11:03 AMgke-starter.yaml
to try a flyte-binary
install on GKE.
Looking at eks-starter.yaml
I see there's the field configuration.inline.flyteadmin.roleNameKey
. Is there a correct value for this for a deployment on GCP? Seems like all the examples have <http://iam.amazonaws.com/role|iam.amazonaws.com/role>
and the underlying Go code doesn't to do much with either. Could I just leave it out?Cornelis Boon
03/08/2024, 10:23 PM"Container [<bucket-name>] lookup failed. Error googleapi: Error 403: Caller does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., forbidden"
I gave the service account that's the default SA all the necessary permissions though. Feel that once this is resolved, I've got something going 🙂Yini Gao
03/14/2024, 3:43 PMflyte-secret-auth
secret.
Our setup is Values.secrets.adminOauthClientCredentials.enabled
as false and managing the flyte-secret-auth
with secret manager ourselves (reference: flyte doc on auth setup). After doing so, it seems the auth
is not mounted as expected due to this condition.
We propose enhancing the configuration in values.yaml by introducing an additional key to indicate whether flyte-secret-auth
to create. This enhancement would enable conditional mounting of flyte-secret-auth
based on its creation status.