thousands-area-8239
08/09/2022, 6:16 PMefs
volume in my eks
cluster as a shared mount for ReadWriteMany
. I have looked at this example and see the V1Volume
being passed into the pod spec. I’m looking through these docs and see that you can create a V1Volume
from a persistent_volume_claim
. So in theory, I can just create a PV and PVC in my cluster, include them in my pod spec, and attach the spec to a flyte task. But I noticed that PVCs are namespace specific and flyte uses the project-domain
namespace for tasks/workflows that are executing. Two questions.
1. Are PVCs the right solution here and if so, how can I dynamically create PVCs for my project-domain
s? Is this something flyte could be configured to do for us or would we be responsible for ensuring any referenced PVCs and PVs exist.
2. What other options are available for mounting shared persistent volumes to my flyte tasks?gifted-house-14547
08/10/2022, 12:52 AMgreat-school-54368
08/10/2022, 2:04 AMcluster_resource_manager
, For creating PVC for each project-domain namespace like this https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values-eks.yaml#L360
@thousands-area-8239 There are two ways to mount volume in your flyte task,
• First is using pod plugin
• Second is propeller’s pod template https://docs.flyte.org/en/latest/deployment/cluster_config/general.html#using-default-k8s-podtemplatesfreezing-airport-6809
thousands-area-8239
08/10/2022, 1:51 PMthousands-area-8239
08/10/2022, 1:53 PMcluster_resource_manager
is that we either need to know up-front what all the projects are, or we will have to constantly update our flyte deployment with new templates as projects are added. Not sure if that is an improvement over just using CI/CD and shell scripts to manually create the namespace PVCs as neededfreezing-airport-6809
freezing-airport-6809
thousands-area-8239
08/10/2022, 8:48 PMcluster_resource_manager
block is optional, but if we define namespace PVCs there, we would need to add them as new projects are created and redeploy to keep things updated. I was able to seamlessly get EFS working with flyte tasks so that was nice, just trying to figure out the best way to grant and manage access to the PV from other projectsfreezing-airport-6809
thousands-area-8239
08/10/2022, 11:45 PMflyte
projects (-p
). Since the namespaces are {project}-{domain}
and PVCs exist per namespace, any project that wants to use an efs
backed PV would need the PVC created for its namespace. Please correct me if this is not correctfreezing-airport-6809
freezing-airport-6809