Durgai Vel
09/11/2023, 2:48 PM<http://docs.flyte.org|docs.flyte.org>
, I could only find Multi Cluster or SingleCluster Single Node examples...
If I already have a kubernetes cluster with multi node running, can I just start the flyte sandbox
in the control plane node, if so will the flyte schedule the tasks on different nodes based on resource availability.
Thanks in advance, :)David Espejo (he/him)
09/11/2023, 5:52 PMlike optionally specifing node values for tasks, so it can be executed on that nodeYou can make use of PodTemplates, adding the
nodeAffinity
(K8s docs) config you want there and then calling the PodTemplate from your Task(s).
If your use case is about nodes with GPUs, then using taints and tolerations might be betterDurgai Vel
09/11/2023, 5:57 PMDavid Espejo (he/him)
09/11/2023, 6:03 PMDurgai Vel
09/11/2023, 6:16 PMDavid Espejo (he/him)
09/11/2023, 6:31 PMDurgai Vel
09/12/2023, 2:38 PM2. Review the manifest located here
, the here
points to a removed file location...
Is there any updated docs or guide on how to start the flyte kit with all of its components like flyteadmin, propeller, UI, minio, kubernetes-dashboard, docker registery
manually or using any yaml file
and kubectl
. I had a lot of issues in installing flyte on my local cluster by following the docs above, in need of help.David Espejo (he/him)
09/12/2023, 4:06 PMDoes flyte schedule tasks on worker nodes from the master node itself or should it be deployed on worker nodes tooI think this division of "master vs worker nodes" is more meaningful on K8s than on Flyte. Typically, a K8s master node is configured to prevent workloads to run there. With Flyte, a single or multi node cluster is treated both as control and data plane, meaning, workflows can run in all nodes (unlesss you need something different). Conversely, the "control and data plane" dialogue in Flyte is meaningful if you have multiple Kubernetes clusters (which doesn't seem to be your case) where you have control plane and data plane nodes but even there, you could run workflows in both. Not sure if this is completely clear for you, otherwise happy to have a call
Durgai Vel
09/13/2023, 6:26 AM6. Install Flyte using the values file provided here:
, kindly fix this too..
2. Would be more helpful if there is any straightforward yaml file, like the postgre and minio deploying yaml files, for flyte's admin, propeller, registery etc.
3. From your statement I think, from a single cluster multi node point, all the flyte deployments should be deployed in the master node, and the workloads will be scheduled and executed on the worker nodes (by flyte scheduler or kubernetes scheduler). If this is correct, thanks for the clarification, I'll let you know if I have any further doubts.David Espejo (he/him)
09/13/2023, 2:26 PMhelm uninstall...
instead of deleting resources manually, If you need YAML files you could render the Helm chart locally, but that's not typically needed
3. In a single Flyte cluster with multiple nodes, all nodes can be used to run workloads. If you have two servers with GPUs, I guess you want to use both to run workloads and that's the default behavior. Node placement is controlled by the K8s scheduler by default, unless you need something else. If you want to use your two nodes you just need to make sure that the K8s master node doesn't have a taint that prevents it from running Pods (this is the case by default with K3s). As long that's in place, Flyte will submit executions to the K8s API who will, in turn, pick the node where they will run as Pods. Propeller is a controller that helps K8s to translate a DAG definition to actual K8s resources and manages the execution lifecycle, but doesn't control node placement.Durgai Vel
09/13/2023, 5:19 PMDavid Espejo (he/him)
09/13/2023, 5:48 PMhost
here is in the format where the Flyte pod will use K8s service discovery to contact the postgres
service in the flyte
namespace inside the same cluster. So it's not something resolvable from your machineDurgai Vel
09/13/2023, 6:26 PMDavid Espejo (he/him)
09/13/2023, 7:40 PMAlso when using flytectl demo start, the flyte propeller, flyte admin, flyte docker registry etc. will get deployed as a separate pods
flyte-binary
package all those services on, err, a single binary (single Pod). The flyte-core
Helm chart deploys separate Pods but it's more geared towards multi K8s cluster deployments.Durgai Vel
09/14/2023, 6:23 AM5. save DB password in secret
, I haven't manually set any DB password to postgres DB also couldn't find any password field in the local-flyte-resources.yaml , so what should I specify in the secret?
2. At what port will the flyte docker registry be running, as of flytectl demo start
the registry will be running on port: 30000
, since now all the services were handled within a single pod.
3. Also in our case, by starting the cluster using k3s and then using helm to deploy flyte-binary, when I stopped the k3s cluster using k3s-uninstall.sh
and then again tried to start the cluster from the initial step, I couldn't find the registered tasks on my flyte UI, but that is not the case with flytectl demo teardown
. I can get the previously registered workflows by starting the flytectl demo cluster. So is it the expected behaviour or am I making any mistake (does it relate to postgres db connection, since I haven't specified any password when creating the secret)?David Espejo (he/him)
09/15/2023, 5:19 PMflyte-deps
chart. It allows local connections without requiring a password. Not great. Would you mind filing an issue for this? I'll need to test the alternative, more secure options for this. If you also want to help, that'd be awesome
2. I don't think flyte-binary ships with a Docker registry
3. I think this is expected behavior. If you uninstall the K3s cluster that holds Flyte and its dependencies (including DB), it won't be able to retrieve any pre-existing object. Sandbox is different because (not 100% sure of this) if you teardown
the cluster, the Docker daemon may still retain the volumes and then a new cluster will reuse those volumes. On the tutorial the DB survives Pod deletions/crashes but not a K3s cluster deletion, because even the PersistentVolumes will go away. We can find ways to persist that data outside the lifecycle of the K3s cluster, it just takes a bit more time and still won't meet every possible on-prem scenario but, feel free to file an issue
I hope some of this is helpful.
You can still use the sandbox on the two nodes, but that would be two separate clusters. Sandbox is not prepared for multicluster opsDurgai Vel
09/27/2023, 8:20 AM