Hi! I'm trying to figure out if/how it's possible to setup flyte for multi-tenancy, ie. isolate tenants workloads in separate namespaces, without sharing/mounting any global secret, thus relying only on tenant-scoped secrets. Ideally tenant workloads would run under tenant namespace. While there seems to be a way to have propellers per tenants, thus enabling true parallelism, IIUC there doesn't seem to be any way to isolate metadata per tenant, since there's a single s3 configuration shared by admin and all propellers/task executions. Which means sharing the bucket secret with all tenants, which wouldn't fit our requirements. Has anybody any experience/recommendations to share? Thanks a lot

You can control namespaces executions end up with this: https://docs-legacy.flyte.org/en/v1.13.2/deployment/configuration/generated/flyteadmin_config.html#section-namespace-mapping by default its project-domain

There's only a single bucket that I can configure right?

Its kinda gross but you can configure buckets per project using flytectl. Again not really well documented but

flytectl update workflow-execution-config --attrFile ${cfg}

Tell me more plz 🙂

https://github.com/flyteorg/flyte/blob/master/flyteidl/protos/flyteidl/admin/matchable_resource.proto#L106-L138 We configure raw output data config

Thanks for the pointer. Does that mean I wouldn't have to share the global bucket config with execution pods?

There are two buckets. There is a metadata bucket which I believe is global and then there is the data plane bucket which can be project specific.

That's definitely good news if that means the data plane bucket is the only one used by the execution pod, it can user owned

S3 perms generally look like this for a tenant service account.

{
        # Allow reading and writing to specified S3 buckets.
        Effect = "Allow"
        Action = [
          "s3:AbortMultipartUpload",
          "s3:CreateMultipartUpload",
          "s3:CompleteMultipartUpload",
          "s3:GetObject",
          "s3:HeadObject",
          "s3:ListBucket",
          "s3:ListBucketMultipartUploads",
          "s3:ListParts",
          "s3:PutObject",
          "s3:UploadPart",
          "s3:UploadPartCopy",
        ],
        Resource = [
          "arn:aws:s3:::flyte-data-plane-<project>-<domain>",
          "arn:aws:s3:::flyte-data-plane-<project>-<domain>/*",
          "arn:aws:s3:::flyte-metadata-prod",
          "arn:aws:s3:::flyte-metadata-prod/*",
        ]
      },

I see, yet the tenant service account needs access to the global bucket, right?

Global metadata yeah

ok thank you so much for the info. I will dig into the links you provided tomorrow and if you don't mind will follow up with more questions 😉