https://flyte.org logo
Join Slack
Powered by
Hey folks, Apologies if I missed the solution to ...
# flyte-deployment
h
Hey folks, Apologies if I missed the solution to this in the documentation somewhere. I’m trying to deploy Flyte onto AWS EKS and enable the AWS Batch plug-in. So far I’m using helm with a 
values.yaml
listed below and I can’t seem to figure out how to get the right configuration into the flyte admin config. # Helm command
Copy code
helm install flyteorg/flyte-binary \
    --generate-name \
    --kube-context=<context> \
    --namespace flyte \
    --values flyte-binary/flyte-binary-eks-values.yaml
# flyte-binary-eks-values.yaml
Copy code
configuration:
  database:
    password:<RD Password>
    host: <DB Host URI>
    dbname: app

  storage:
    metadataContainer: <bucket>
    userDataContainer: <bucket>
    provider: s3
    providerConfig:
      s3:
        region: "us-west-2"
        authType: "iam"

  logging:
    level: 1
    plugins:
      cloudwatch:
        enabled: true
        templateUri: |-
          <https://console.aws.amazon.com/cloudwatch/home?region=us-west-2#logEventViewer:group=/eks/opta-development/cluster;stream=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}-{{ .containerId }}.log

  inline:
    plugins:
      aws:
        batch:
          roleAnnotationKey: <Redacted>
        region: us-west-2
    tasks:
      task-plugins:
        enabled-plugins:
          - container
          - sidecar
          - aws_array
        default-for-task-types:
          - container_array: aws_array
          - aws-batch: aws_array    
          - container: container

serviceAccount:
  create: true
  annotations:
    <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: <Redacted>

# Where should this go?
configMaps:
  adminServer:
    flyteadmin:
      roleNameKey: <Redacted>
      queues:
        executionQueues:
          - dynamic: <JobQueueName>
            attributes:
              - default
        workflowConfigs:
          - tags:
              - default
And when I try and run a workflow ith Batch tasks I get this error:
Copy code
Workflow[flytesnacks:development:<http://workflows.example.wf|workflows.example.wf>] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [aws_array]: [BadTaskSpecification] config[dynamic_queue] is missing
Thanks for reading this far 🙂 🙏
Update. I was able get this to work by manually editing the configmap generated by helm but is there a better way?
s
y
what update did you make manually?
oh the bit in where should this go?
try under 
configuration
Copy code
inline:
  flyteadmin:
    roleNameKey: ...
    ...
h
Hey @Samhita Alla. Thanks for answering! I have looked at that AWS Batch Setup doc you linked. The helm blocks are helpful but I struggle to know what their proper context is. I ended up adapting flyte-binary/eks-production.yaml and mostly got it working. Once I get things working I’ll see if I can write up where I went wrong in a helpful way. Thanks, @Yee! I’ll give that a try 🙏
Adding that block to 
configuration.inline
doesn’t seem to solve it. This is the ConfigMap manifest I edited to solve the issue but I’d prefer to do it in helm.
Copy code
# Source: flyte-binary/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: flyte-binary-1678743094-config
  namespace: "flyte"
  labels:
    <http://helm.sh/chart|helm.sh/chart>: flyte-binary-v1.3.0
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: flyte-binary
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: flyte-binary-1678743094
    <http://app.kubernetes.io/version|app.kubernetes.io/version>: "1.16.0"
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
  annotations:
data:
  000-core.yaml: |
    admin:
      endpoint: localhost:8089
      insecure: true
    catalog-cache:
      endpoint: localhost:8081
      insecure: true
      type: datacatalog
    cluster_resources:
      standaloneDeployment: false
      templatePath: /etc/flyte/cluster-resource-templates
    logger:
      show-source: true
      level: 1
    propeller:
      create-flyteworkflow-crd: true
>>>> Added this      
    flyteadmin:
      roleNameKey: <REDACTED>
    queues:
      # A list of items, one per AWS Batch Job Queue.
      executionQueues:
        # The name of the job queue from AWS Batch
        - dynamic: "default_EC2_job_queue"
          # A list of tags/attributes that can be used to match workflows to this queue.
          attributes:
            - default
      # A list of configs to match project and/or domain and/or workflows to job queues using tags.
      workflowConfigs:
        # An empty rule to match any workflow to the queue tagged as "default"
        - tags:
            - default
<<<<
    webhook:
      certDir: /var/run/flyte/certs
      localCert: true
      secretName: flyte-binary-1678743094-webhook-secret
      serviceName: flyte-binary-1678743094-webhook
      servicePort: 443
  001-plugins.yaml: |
    tasks:
      task-plugins:
        enabled-plugins:
          - container
          - sidecar
          - K8S-ARRAY
        default-for-task-types:
          - container: container
          - container_array: K8S-ARRAY
    plugins:
      logs:
        kubernetes-enabled: false
        cloudwatch-enabled: true
        cloudwatch-template-uri: <https://console.aws.amazon.com/cloudwatch/home?region=us-west-2#logEventViewer:group=/eks/opta-development/cluster;stream=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}-{{ .containerId }}.log
        stackdriver-enabled: false
      k8s-array:
        logs:
          config:
            kubernetes-enabled: false
            cloudwatch-enabled: true
            cloudwatch-template-uri: <https://console.aws.amazon.com/cloudwatch/home?region=us-west-2#logEventViewer:group=/eks/opta-development/cluster;stream=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}-{{ .containerId }}.log
            stackdriver-enabled: false
  002-database.yaml: |
    database:
      postgres:
        username: postgres
        passwordPath: /var/run/secrets/flyte/db-pass
        host: <REDACTED>
        port: 5432
        dbname: app
        options: "sslmode=disable"
  003-storage.yaml: |
    propeller:
      rawoutput-prefix: <REDACTED>
    storage:
      type: stow
      stow:
        kind: s3
        config:
          region: us-west-2
          disable_ssl: false
          v2_signing: false
          auth_type: iam
      container: <REDACTED>
  010-inline-config.yaml: |
    plugins:
      aws:
        batch:
          roleAnnotationKey: <REDACTED>
        region: us-west-2
    tasks:
      task-plugins:
        default-for-task-types:
        - container_array: aws_array
        - aws-batch: aws_array
        - container: container
        enabled-plugins:
        - container
        - sidecar
        - aws_array
171 Views
Open in Slack
PreviousNext
Powered by