I'm struggling with installing Flyte core on our p...
# ask-the-community
t
I'm struggling with installing Flyte core on our private Azure K8S (AKS) cluster using the official Helm Charts. Particularly, specifying
storage.type: custom
and
storage.custom: {}
to make Stow use Azure Blob Storage (non-S3 compatible). Does anyone have an example of using
storage.type: custom
correctly?
n
We're running all of our Flyte cluster in Azure using the following storage config:
Copy code
storage:
  type: custom
  bucketName: "<CONTAINER_NAME>"
  custom:
    container: "<CONTAINER_NAME>"
    enable-multicontainer: true
    connection: {}
    type: stow
    stow:
      kind: azure
      config:
        account: "<STORAGE_ACCOUNT_NAME>"
        key: "<STORAGE_ACCOUNT_KEY>"
that's been working fine so far with the official helm charts
t
Thanks. Were missing these:
Copy code
container: "<CONTAINER_NAME>"
    enable-multicontainer: true
    connection: {}
They made the deployment progress 😌
k
@Nick Müller (MorpheusXAUT) would it possible to upstream it
And also flytekit - fsspec instructions
n
@Ketan (kumare3) sure, although I'm not 100% sure what to upstream? do you mean creating another helm chart override for azure like there currently already is for gcp/aws? or just documentation in general somewhere?
t
Would be great if you could share your
values.yaml
for AKS. I tried (to the best of my ability) to adopt
values-eks.yaml
to Azure, but now I am stuck again, facing an issue where gRPC connections are forcely closed, like described here. I use an nginx ingress controller, but issuing
flytectl get projects --admin.insecureSkipVerify --logger.level=10
yields me with a
{"json":{"src":"client.go:183"},"level":"error","msg":"failed to initialize token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unavailable desc = connection closed","ts":"2022-12-19T13:08:42+01:00"}
😞 Any ideas what could be the cause, @Nick Müller (MorpheusXAUT)?
Adding
insecureSkipVerify: true
in
config.yaml
seems to have solved it for now.
n
you should be able to do without that as well 🤔 we've had to add a few different ingress routes though, although we're using traefik, not nginx as a ingress controller. I'll try to get you a
values.yaml
of our complete setup a bit later this afternoon, will take a bit to put together since we use some extra templating on top of the public helm chart. will let you know @Thomas Kobber Panum
t
@Nick Müller (MorpheusXAUT) Thank you so much for your help 🙏
k
@Nick Müller (MorpheusXAUT) - atleast 3-5 companies decided not to use Flyte as they thought azure is not supported. If you upstream and contribute azure docs, lots of companies could join in So, Deployment docs, Example flytekit Values
n
@Ketan (kumare3) sure, I can try to put something together. can't guarantee when I'll get to it (especially over the christmas/new year period), but I've put it down as a task in my backlog 🙂
@Thomas Kobber Panum looks like I might not be able to get back to you today anymore, another important issue came up that took longer than expected. will try to get the config to you tomorrow first thing in the morning, sorry about that!
t
@Nick Müller (MorpheusXAUT): Absolutely no worries 🙂
n
@Thomas Kobber Panum sent you a link to a gist with our current setup. since I couldn't add
/
in the filenames, they have two spaces now to show a folder/file structure
the way we've set it up: the
values.yaml
in that gist is used to overwrite the default
values.yaml
from flyte's helm chart. we have modified the chart a bit to our needs (e.g. adding imagepullsecrets since we're using private-built flyte images), I've tried to include everything that's relevant in that gist as well
the resource limits etc can obviously be ignored, they're just tuned to our dev cluster needs, so can safely be skipped, but I included them for completeness sake. we also use a split DB for flyteadmin/datacatalog as it was the case for older versions of flyte, so that can be ignored as well
as mentioned before, we're using traefik instead of nginx as our ingress controller (thus the separate
IngressRoute
, but that still allows us to connect to our clusters via SSL without having to set
insecureSkipVerify: true
. example flytectl config:
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///<FLYTE_FQDN>
  authType: Pkce
  insecure: false
logger:
  show-source: true
  level: 0
I will try to extract everything relevant from our internal setup into a more generic setup for Azure in the next days/new year and will also push some of the minor helm chart changes we've made internally upstream, should be less confusing than this, sorry about the quick mess 😅
Nothing so far yet, unfortunately, sorry 😕 I was planning on getting some documentation done over the christmas period but got quite busy with other tasks. the best I can offer at the moment is what's in this thread ^
s
got it! Thanks Nick. 🙂
n
I have a gist of the full helm chart I sent to Thomas in case you're interested in that, that's what we're using for the deployment
s
yes sure. Will be glad to take a look at that and see if its helpful for me.
k
Thank you Nick. I hop @Sidharth(Sid) can help put it in Flyte docs
s
Yes once I do it, coz im not working with a team as of now, I will inform and add it to docs.
n
@Sidharth (Sid) we’re currently revamping the deployment documentation, please hold tight as a lot of the current docs will change significantly! I’ll ping you when those changes have been merged
s
Okay great. @Niels Bantilan
v
Hi there, I'm trying out a deployment on Azure for my company, and wondering if you had any docs besides what I've seen here or in Slack? Even in-progress would be greatly appreciated! Cheers
196 Views