set the channel topic: If you have any questions about deploying Flyte, please don't hesitate to ask us here!

set the channel description: This channel is dedicated to discussions about deploying Flyte.

Hi Flyte community, When setting up the flyte-binary helm resources and attempting to run commands (register/run) through jflyte or pyflyte, I see the following error:

Max retries exceeded with url: /my-s3-bucket/flytesnacks/development... 'Connection to 10.110.44.3 timed out

I'm able to create a new project through flytectl and see it in the flyte console. I've installed the flyte-binary helm chart with the following overrides:

configuration:
  database:
    port: 5432
    dbname: flyteadmin
    host: 10.99.211.252
  storage:
    metadataContainer: my-s3-bucket
    userDataContainer: my-s3-bucket
    provider: s3
    providerConfig:
      s3:
        disableSSL: true
        v2Signing: false
        endpoint: <http://10.110.44.3:9000>
        authType: accesskey
        accessKey: minio
        secretKey: miniostorage
  plugins:
  # All k8s plugins default configuration
    k8s:
      inject-finalizer: true
      default-env-vars:
        - AWS_METADATA_SERVICE_TIMEOUT: 5
        - AWS_METADATA_SERVICE_NUM_ATTEMPTS: 20
        - FLYTE_AWS_ENDPOINT: "<http://10.110.44.3:9000>"
        - FLYTE_AWS_ACCESS_KEY_ID: minio
        - FLYTE_AWS_SECRET_ACCESS_KEY: miniostorage

I'm able to connect the minio client CLI to localhost:9000 after portforwarding the minio service (from 9000->9000). I'm also able to successfully curl

<http://10.110.44.3:9000>

from the flyte-binary pod using

kubectl exec

. Would folks happen to know what might be happening here or what I could do to move forward in this investigation?

While reading through the flyte helm chart I came across

k8s-array

as a named plugin, but I cannot find any docs on it. Anyone have an idea of what this is?

Something weird going on with our flytepropeller. Will probably try to fully redeploy.

Is it possible to configure other defaults than resource requests through the helm chart? For example default timeout for tasks.

@João Lobo Guerra Neto let’s continue here

If we're not running the propeller manager to shard workflows across propeller pods, is there any reason to run more than one propeller pod?

Faster recovery from crashes or during deployment

But otherwise propeller is fully self healing

Hi guys, I have a question. I would’t like to use auth0 in localhost/login I can to use an user and password and put this information in a secret? How I can do it? Help me please

Hey folks, deploying Flyte on a single EKS, is it possible to deploy Flyte's binary in an ondemand nodegroup, and then have tasks run in a spot nodegroup?

Hello 🙂 I am currently following the multi cluster deployment. At the moment, i have a problem when applying the values-override.yaml with the correct entries for my data-plane cluster on the control plane. From every deployment being healthy only flyte admin fails with the following logs:

{"json":{},"level":"warning","msg":"stow configuration section missing, defaulting to legacy s3/minio connection config","ts":"2023-03-01T21:25:00Z"}
{"json":{},"level":"fatal","msg":"caught panic: entries is empty [goroutine 1 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24

Can anyone point me in the right direction what be missing here?

@Dan Rammer (hamersaw) remind me again where we can control the max futures file size?

Hey all, I put together this repo with a bunch of tooling to get the multi-cluster deployment operational on EKS. https://github.com/alexifm/flyte-eks-deployment

Hey! I have a question regarding how the role permission setup is supposed to work on the multi cluster setup. What i would like to run is basically one control-plane with multiple data-planes in different AWS Accounts. Every data-plane should have a dedicated

flyte-user-roles

for accessing stuff in their account. What is currently happening when following the official documentation is that after exchanging secret, token and adjusting the cluster config on the control plane, the data-plane retrieves all the namespaces, quotas and Service Accounts from the control plane (which are created by the

cluster_resource_manager

i guess). This leaves me with default Service Accounts in the data-plane for all the projects/domains where the

flyte-user-role

of the control plane is annotated. Obviously i want the

flyte-user-roles

of the data-planes in there, which are completely unused so far in my setup. One way would be to just replace the default Service Account annotation with the correct flyte-user-role in the project/domains i need them. Is there a better or correct way of doing this?

Hey folks, Apologies if I missed the solution to this in the documentation somewhere. I’m trying to deploy Flyte onto AWS EKS and enable the AWS Batch plug-in. So far I’m using helm with a

values.yaml

listed below and I can’t seem to figure out how to get the right configuration into the flyte admin config. # Helm command

helm install flyteorg/flyte-binary \
    --generate-name \
    --kube-context=<context> \
    --namespace flyte \
    --values flyte-binary/flyte-binary-eks-values.yaml

# flyte-binary-eks-values.yaml

configuration:
  database:
    password:<RD Password>
    host: <DB Host URI>
    dbname: app

  storage:
    metadataContainer: <bucket>
    userDataContainer: <bucket>
    provider: s3
    providerConfig:
      s3:
        region: "us-west-2"
        authType: "iam"

  logging:
    level: 1
    plugins:
      cloudwatch:
        enabled: true
        templateUri: |-
          <https://console.aws.amazon.com/cloudwatch/home?region=us-west-2#logEventViewer:group=/eks/opta-development/cluster;stream=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}-{{ .containerId }}.log

  inline:
    plugins:
      aws:
        batch:
          roleAnnotationKey: <Redacted>
        region: us-west-2
    tasks:
      task-plugins:
        enabled-plugins:
          - container
          - sidecar
          - aws_array
        default-for-task-types:
          - container_array: aws_array
          - aws-batch: aws_array    
          - container: container

serviceAccount:
  create: true
  annotations:
    <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: <Redacted>

# Where should this go?
configMaps:
  adminServer:
    flyteadmin:
      roleNameKey: <Redacted>
      queues:
        executionQueues:
          - dynamic: <JobQueueName>
            attributes:
              - default
        workflowConfigs:
          - tags:
              - default

And when I try and run a workflow ith Batch tasks I get this error:

Workflow[flytesnacks:development:<http://workflows.example.wf|workflows.example.wf>] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [aws_array]: [BadTaskSpecification] config[dynamic_queue] is missing

Thanks for reading this far 🙂 🙏

This message was deleted.

Hello! Is it possible to run

flytectl demo start

on a VSI (virtual server instance) and be able to reach the UI from the public using the public ip of the server? For example:

<public_ip>:30080/console

. I’ve tried this and the demo is running from within the server but i cant seem to reach it from my own browser- do i need to setup a reverse proxy or do something else? I try to navigate to:

<public_ip>:30080/console

and nothing ends up loading despite the demo/sandbox running properly. I also curled the localhost:30089/console from inside the server and I get the html response back. Any sort of help or feedback would be greatly appreciated, thanks!! 🙏

hi guys, i’m having issues w flyte sandbox deployment. i’ve tried deploying the demo sandbox first on my mac, and when that failed, on a dev ec2 instance. both are exhibiting the same issues: on the flyte console page, i no longer get the standard page with the flytesnacks`development`,

staging

,

production

options (see screenshots). on clicking

login

, i get

Not Found

. i’ve been running a bunch of POCs over the past few weeks on a local cluster, and this hasn’t been an issue.

Hi Team, I am following the first example to launch a test workflow, I see the following error in the docker desktop dashboard console:

"Error syncing pod, skipping" err="failed to \"StartContainer\" for \"f154ee96e6c1a4fed852-n0-0\" with ImagePullBackOff: \"Back-off pulling image \\\"<http://cr.flyte.org/flyteorg/flytekit:py3.10-1.2.11\\\|cr.flyte.org/flyteorg/flytekit:py3.10-1.2.11\\\>"\"" pod="flytesnacks-development/f154ee96e6c1a4fed852-n0-0" podUID=5952591c-fde8-47bd-a9c4-2885a01915a7
E0317 22:24:01.121900      71 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"f154ee96e6c1a4fed852-n0-0\" with ImagePullBackOff: \"Back-off pulling image \\\"<http://cr.flyte.org/flyteorg/flytekit:py3.10-1.2.11\\\|cr.flyte.org/flyteorg/flytekit:py3.10-1.2.11\\\>"\"" pod="flytesnacks-development/f154ee96e6c1a4fed852-n0-0" podUID=5952591c-fde8-47bd-a9c4-2885a01915a7

How do I resolve this ? Posted in #ask-the-community

Hi Folks, We need to be able to provide different images for different tasks in a workflow, so I am testing the Multiple Container Images in a Single Workflow feature. I am using the Whylogs example. It won't work as is in a remote cluster. So, I added the flytecoobook whylogs container image to each task to be able to run the workflow successfully in a remote cluster with

pyflyte run --remote whylogs_example wf

@task(container_image="<http://ghcr.io/flyteorg/flytecookbook:whylogs_examples-latest|ghcr.io/flyteorg/flytecookbook:whylogs_examples-latest>")
def get_reference_data() -> pd.DataFrame:
    ...

@task(container_image="<http://ghcr.io/flyteorg/flytecookbook:whylogs_examples-latest|ghcr.io/flyteorg/flytecookbook:whylogs_examples-latest>")
def get_target_data() -> pd.DataFrame:
    ...

@task(container_image="<http://ghcr.io/flyteorg/flytecookbook:whylogs_examples-latest|ghcr.io/flyteorg/flytecookbook:whylogs_examples-latest>")
def create_profile_view(df: pd.DataFrame) -> DatasetProfileView:
    ...

@task(container_image="<http://ghcr.io/flyteorg/flytecookbook:whylogs_examples-latest|ghcr.io/flyteorg/flytecookbook:whylogs_examples-latest>")
def constraints_report(profile_view: DatasetProfileView) -> bool:
    ...

However, the

get_reference_data

and

get_target_data

should not need whylogs. They just work with pandas and scikit-learn. We should be able to run those tasks with the

@task(container_image="<http://ghcr.io/flyteorg/flytecookbook:core-latest|ghcr.io/flyteorg/flytecookbook:core-latest>")

image. I did try that but it fails, k8s logs say:

File "/opt/venv/lib/python3.8/site-packages/flytekit/core/python_auto_container.py", line 279, in load_task
    task_module = importlib.import_module(name=task_module)  # type: ignore
  ...
  File "/root/whylogs_example.py", line 17, in <module>
    import whylogs as why
ModuleNotFoundError: No module named 'whylogs'
Traceback (most recent call last):

Every task container is trying to parse the entire whyogs_example.py file and since

fytecookbook:core-latest

does not have whylogs it is failing. What is the best design pattern or strategy to be followed in such cases ? How can I make it work remotely ? I read the containerization/multi_images.html, that example has two methods

svm_trainer

and

svm_predictor

but both end up using the same image. All the examples I see in https://github.com/flyteorg/flytesnacks/tree/master/cookbook/case_studies/ml_training also have only one custom docker file. Is there a production grade example workflow with tasks taking different images which are significantly different with each other ? Looking for a reference complex workflow that talks about these nuances on how to organize the pieces together with different custom images for each task.

Hello everyone, I followed the installation document from this link https://docs.flyte.org/en/v1.0.0/deployment/aws/manual.html#prerequisites. All the outputs were as per the documentation, but on the final step where I try to login, "endpoint/console", I get 503 Service Temporarily Unavailable. Any help on this? thanks in advance

When using the

flyte-binary

helm chart is there a way to specify which project's you'd like automatically defined? I'm looking for something like

initialProjects

from the

flyte-core

helm chart

Hi everyone I've been recently going through the process of deploying Flyte on EKS and wanted to capture the learnings in a single place. For now it's an insecure non-production deployment but will be adding more features and documenting in the repo. Feel free to fork/improve/extend: https://github.com/davidmirror-ops/flyte-the-hard-way

Hello Flyte-Gang, does anyone have experience with deploying flyte on an ARM kubernetes cluster? I am currently trying to deploy flytepropeller via helm on this aws instancetype:

c6g.xlarge

but i keep running into

format exec

errors. Anything particular to watch out for? Thanks!

Hello, I am trying to install flyte in a local kubernetes cluster. Is there a helm chart available for the same. The helm charts in the documentation is for GKE/EKS cluster only

Hello, could somebody help me debug connection to the flyte cluster? I’ve just deployed flyte-binary following this guide. I’ve already run some workflows but now I’ve added ingress configuration, so I don’t have to run port-forward every time:

ingress:
  create: true
  commonAnnotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: internal
    <http://nginx.ingress.kubernetes.io/force-ssl-redirect|nginx.ingress.kubernetes.io/force-ssl-redirect>: "true"
  httpAnnotations:
    <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: /console
  grpcAnnotations:
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
  host: <http://flyte-dev.internal.qa-knowledge-base-coin-dev.z-dn.net|flyte-dev.internal.qa-knowledge-base-coin-dev.z-dn.net>

Console works just fine but I get these errors from flytectl:

$ flytectl config init --host <http://flyte-dev.internal.qa-knowledge-base-coin-dev.z-dn.net|flyte-dev.internal.qa-knowledge-base-coin-dev.z-dn.net>
...
$ flytectl get project
Error: Connection Info: [Endpoint: dns:///flyte-dev.internal.qa-knowledge-base-coin-dev.z-dn.net, InsecureConnection?: false, AuthMode: Pkce]: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 464 (); malformed header: missing HTTP content-type

OR
$ flytectl config init --host <http://flyte-dev.internal.qa-knowledge-base-coin-dev.z-dn.net|flyte-dev.internal.qa-knowledge-base-coin-dev.z-dn.net> --insecure
...
$ flytectl get project
Error: Connection Info: [Endpoint: dns:///flyte-dev.internal.qa-knowledge-base-coin-dev.z-dn.net, InsecureConnection?: true, AuthMode: Pkce]: rpc error: code = Unavailable desc = connection closed before server preface received

Any suggestions?

Hi, I spoke with the lead solution architect for Data on EKS at KubeCon EU today (https://awslabs.github.io/data-on-eks/docs/intro) regarding including Flyte in the AI/ML on EKS. Data on Amazon EKS(DoEKS) - A tool for building aws managed and self-managed scalable data platforms on Amazon EKS. Right now an integration for Airflow and Kubeflow exists on EKS. Since previous deployment for Flyte included steps such as setting up RDBMS, S3 and EKS, it might be worth trying out creating an integration for Flyte on Data on EKS. If the Flyte team is interested, can start a thread with the folks from Data on EKS. @David Espejo (he/him) @Ketan (kumare3) @Kevin Su

Hi! Is there a way to deploy a data plane node inside a k8s cluster that is inside a private network, while the control plane node is inside the cloud. The private network can connect to the cloud, but the cloud cannot connect to the private cluster. From what I understood from reading the docs, the control plane cluster must have direct access to the data plane cluster (via k8s api) to push jobs which would make this impossible, but is there a way to have a pull-based approach where the data plane cluster pulls the jobs it needs to run? A bit similar to how GitHub Actions self-hosted runner works where it's a component inside the data plane that fetches the jobs to run. Thanks in advance!