https://flyte.org logo
#ask-ai
Title
# ask-ai
a

Andrew

02/08/2024, 8:39 PM
@Glime In the single-node guide for running flyte on-premises, it mentions a multi-node guide if you plan to add more physical nodes later on, but I can’t find that guide. Is that still in progress, or can I find that guide somewhere?
g

Glime

02/08/2024, 8:39 PM
Based on the provided context, there is no direct mention of a specific guide for transitioning from a single-node to a multi-node Flyte deployment on-premises. However, for scaling and productionizing your Flyte cluster, which would include adding more nodes to handle increased workload, you can refer to the "Productionize Your Flyte Cluster" guide. This guide, while not explicitly stated as a multi-node guide, covers aspects of scaling and securing a Flyte cluster for production use, which inherently involves considerations for multi-node setups. For detailed steps on how to productionize your Flyte cluster, please visit the following link: Productionize Your Flyte Cluster. If your specific need is to find documentation on adding physical nodes to an existing Flyte cluster, this might not be directly covered in the available documentation. It's recommended to explore the general deployment documentation for more insights on cluster management and to consider reaching out to the Flyte community for more specific guidance or to confirm if a dedicated guide for adding physical nodes is in progress or available elsewhere. --- This is an AI-generated response, take its recommendations with caution. Please help us improve the model accuracy by giving this response a thumbs up/thumbs down reaction. 👍 button 👎 button
a

Andrew

02/08/2024, 10:48 PM
@Glime I’m trying to run a launchplan using the remote execute function. If I do this, is there a way to check if the execution failed, and if so, get the error message? And can I get a link to the execution in the flyte console as well?
g

Glime

02/08/2024, 10:48 PM
Yes, you can check if the execution failed and get the error message when running a launch plan using the
remote.execute
function. Additionally, you can obtain a link to the execution in the Flyte console. To check if the execution failed and to get the error message, you can inspect the execution object returned by the
remote.execute
function. If an execution fails, attempting to access its outputs will raise an exception indicating that the execution ended in failure. For example:
Copy code
try:     print(execution.outputs) except Exception as e:     print(f"Execution failed with error: {e}")
This approach is based on the behavior described in the GitHub issue where it's mentioned that accessing outputs of a failed execution will raise an error indicating the failure (source). To get a link to the execution in the Flyte console, you can construct the URL using the execution's project, domain, and name. The URL format typically looks like this:
Copy code
https://<flyte-console-host>/console/projects/<project>/domains/<domain>/executions/<execution-name>
You can obtain the project, domain, and execution name from the execution object. Here's how you might do it:
Copy code
execution_project = execution.id.project execution_domain = execution.id.domain execution_name = execution.id.name console_link = f"https://<flyte-console-host>/console/projects/{execution_project}/domains/{execution_domain}/executions/{execution_name}" print(f"View execution in Flyte console: {console_link}")
Replace
<flyte-console-host>
with the actual host of your Flyte console. This information can usually be found in your Flyte deployment documentation or from your Flyte administrator.
d

David Espejo (he/him)

02/08/2024, 11:03 PM
@Andrew the multi=node on prem addition to the Flyte the Hard Hay guide is in progress
I'm currently stuck on some permissions Issues preventing the Task pods to download metadata from the minio bucket on top of an NFS share. Once I get past it, will make it public
Is running multiple K8s nodes with NFS shared storage a pattern useful for you?
a

Andrew

02/08/2024, 11:08 PM
Sounds good, that’s good to know. The main thing that would be useful for us is being able to utilize multiple on-prem machines, and be able to kick off pipelines on any of those machines. The shared storage would probably be GCS for us, so if the storage and database and what not could be on GCP, that would be awesome. But we’re just looking into that as an option to save on some cloud costs
d

David Espejo (he/him)

02/08/2024, 11:10 PM
sure I see that pattern of "hybrid" deployments coming up more frequently I see it feasible with any of the Helm charts. And probably easier than dealing with CSI drivers and NFS permissions
a

Andrew

02/08/2024, 11:27 PM
Gotcha, I see. So I could essentially just follow the flyte-core setup, but modify it for microk8s?
d

David Espejo (he/him)

02/08/2024, 11:36 PM
More or less. It won't give you persistency for the Pods, which is fine for the Flyte ones but not for Postgres (assuming the DB is also on prem) so you'll still need some form of shared storage and the K8s config to use it
a

Andrew

02/08/2024, 11:55 PM
Ok, I see. Would it be possible to use a database in GCP as well? If there’s any more info on this or a guide, that would be awesome, just to get an idea of what all would be involved. If not I could at least try to play around with it when I get the time
d

David Espejo (he/him)

02/09/2024, 12:05 AM
I think it's possible, there are some IAM and networking challenges involved but def doable
The thing with building the guides is the time it takes, especially when things don't work. For now, could you file issues for this on the Flyte the hard way repo? The idea is to capture these requests and eventually make it happen
a

Andrew

02/09/2024, 12:16 AM
For sure, I don’t want to make you add a bunch just for me, I definitely understand. For sure, I can add some issues, and when I get around to trying this out, I can see what I run into
d

David Espejo (he/him)

02/09/2024, 12:17 AM
No prob! I don't think it's only for you :) IMO this pattern will likely happen more often
2 Views