Hey, a few questions, 1. are there any examples of setting up CI/CD to register workflows using GitHub Actions when Flyte is deployed with OIDC authentication? I assume you need to generate a

~/.flyte/config.yaml

and configure backend to authenticate using OIDC flow with

clientId

&

clientSecret

but not sure if I am missing anything else? Basically what I want to achieve is something in line with what you described at Set up DevOps in this blogpost: https://mlops.community/mlops-with-flyte-the-convergence-of-workflows-between-machine-learning-and-engineering/ 2. How would you add tolerations to the Pod Specifications for a task/workflow to run on dedicated node pool such as e.g., nodes with GPU/dedicated CPU nodes?

Hi, @Hampus Rosvall! 1. We don’t really have code examples as such; however, https://github.com/unionai-oss/flytectl-setup-action should help smoothen interactions with the backend right from your GitHub Actions. @Yuvraj @Ketan (kumare3), do we need to work on an example CI/CD workflow for Flyte to replicate the Lyft’s way of handling Flyte code, as outlined in https://docs.flyte.org/en/latest/deployment/ideal_flow.html#case-study-mlops-at-lyft? 2. You can add resource tolerations as specified in https://docs.flyte.org/projects/cookbook/en/stable/auto/deployment/configure_use_gpus.html. Let me know if this isn’t what you’re looking for.

Hi again, thanks for your quick reply! 1. To authenticate against the backend I need to provide the credentials in the flytectl config file, right? So I can use that action to install flytectl and then write some custom step that populates the config with the correct values? 2. Ah okay, thanks. So there is no way to provide the tolerations on launch plan level? Will all the pods have those tolerations I add under plugins then? I will add the plugin as new field here then? https://github.com/flyteorg/flyte/blob/af5b94d1d7503e58e0301164229db4ef3d247567/charts/flyte-core/values.yaml#L642

1. Yeah, I believe so. @Prafulla Mahindrakar, is that the preferred technique? 2. All the pods should have the tolerations when you set it up under the k8s plugin section. I don’t think you’ll have to modify it in the values.yaml file you shared. It has to do with the deployment technique you opted for. If you’re using the flyte helm chart for instance, it ends up here. @Prafulla Mahindrakar, can tolerations be provided at launch plan level?

1. Yeah, I believe so. @Prafulla Mahindrakar, is that the preferred technique?

Yes that should be right way if you are leveraging github action . Also for reference config file we use in flyte CI for executing functional tests https://github.com/flyteorg/flytetools/blob/master/functional-tests/config.yaml

@Prafulla Mahindrakar, can tolerations be provided at launch plan level?

Currently no. These are only supported through plugin configuration

@Samhita Alla an example would be great

@Yuvraj is probably the best person to handle the CI/CD part but I can take a stab at it. Issue: https://github.com/flyteorg/flyte/issues/2772

Should be a tutorial

Oh what has been done at Lyft is exactly what I worked on last week at Wolt for the CI using Github Action, I could probably help if needed 😄

Oh thank you so much for all help guys. I will have a look at the tolerations plugin and let you know how it turns out once I allocate some time to it. @Stephen how did you configure flytectl to authenticate against your backend in your CI, given that you are using OIDC authetnication? Also looking to implementing the Lyft workflow, so I might reach out once I grasp my head around the very details of it 🙂 Did you set up the flytectl config as @Prafulla Mahindrakar suggested i.e.,

admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///development.uniondemo.run
  # Change insecure flag to ensure that you use the right setting for your environment
  insecure: false
  clientId: flytepropeller
  clientSecretLocation: /home/runner/secret_location
logger:
  # Logger settings to control logger output. Useful to debug logger:
  show-source: true
  level: 1

@Stephen, if you have the bandwidth, would love a contribution detailing how you set up the CI through GitHub Actions. We could add it to the deployment guide.

@Stephen even a partial contribution that we can complete can help a ton

@Hampus Rosvall For CI we use Github Secrets to be able to authenticate with Flytectl

admin:
  endpoint: dns:///console.flyte.example.com
  insecure: false
  authType: ClientSecret
  clientId: github-client
logger:
  show-source: true
  level: 1
storage:
  type: stow
  stow:
    kind: s3
    config:
      auth_type: iam
      region: eu-west-1
  container: whatever-name-container
Footer

Then during the step, we create a

/etc/secrets

directory like the following

docker run \
          -e TAG=$TAG \
          --entrypoint /bin/sh \
          ${{ needs.setup.outputs.path_builder }} \
          -c "\
            mkdir /etc/secrets && \
            echo ${{ secrets[format('{0}_FLYTE_CLIENT_SECRET', needs.setup.outputs.flyte_client_domain )] }} > /etc/secrets/client_secret && \
            pyflyte -c ci/cfg/flyte.config package --image ${{ needs.setup.outputs.path_runner }} -f && \
            flytectl register files --k8sServiceAccount xxx -p ${{ env.PROJECT }} -d ${{ needs.setup.outputs.domain_name }} --version ${{ env.VERSION }} --archive flyte-package.tgz

Depending on the domain we either get the secrets for our dev cluster or or production cluster.

Thank you! How about you add it to https://docs.flyte.org/en/latest/deployment/ideal_flow.html? I can thereby polish or add more content to it.

Also - this does not seem to be the standard GitHub action