Hey, a few questions, 1. are there any examples o...
# announcements
h
Hey, a few questions, 1. are there any examples of setting up CI/CD to register workflows using GitHub Actions when Flyte is deployed with OIDC authentication? I assume you need to generate a
~/.flyte/config.yaml
and configure backend to authenticate using OIDC flow with
clientId
&
clientSecret
but not sure if I am missing anything else? Basically what I want to achieve is something in line with what you described at Set up DevOps in this blogpost: https://mlops.community/mlops-with-flyte-the-convergence-of-workflows-between-machine-learning-and-engineering/ 2. How would you add tolerations to the Pod Specifications for a task/workflow to run on dedicated node pool such as e.g., nodes with GPU/dedicated CPU nodes?
s
Hi, @Hampus Rosvall! 1. We don’t really have code examples as such; however, https://github.com/unionai-oss/flytectl-setup-action should help smoothen interactions with the backend right from your GitHub Actions. @Yuvraj @Ketan (kumare3), do we need to work on an example CI/CD workflow for Flyte to replicate the Lyft’s way of handling Flyte code, as outlined in https://docs.flyte.org/en/latest/deployment/ideal_flow.html#case-study-mlops-at-lyft? 2. You can add resource tolerations as specified in https://docs.flyte.org/projects/cookbook/en/stable/auto/deployment/configure_use_gpus.html. Let me know if this isn’t what you’re looking for.
h
Hi again, thanks for your quick reply! 1. To authenticate against the backend I need to provide the credentials in the flytectl config file, right? So I can use that action to install flytectl and then write some custom step that populates the config with the correct values? 2. Ah okay, thanks. So there is no way to provide the tolerations on launch plan level? Will all the pods have those tolerations I add under plugins then? I will add the plugin as new field here then? https://github.com/flyteorg/flyte/blob/af5b94d1d7503e58e0301164229db4ef3d247567/charts/flyte-core/values.yaml#L642
s
1. Yeah, I believe so. @Prafulla Mahindrakar, is that the preferred technique? 2. All the pods should have the tolerations when you set it up under the k8s plugin section. I don’t think you’ll have to modify it in the values.yaml file you shared. It has to do with the deployment technique you opted for. If you’re using the flyte helm chart for instance, it ends up here. @Prafulla Mahindrakar, can tolerations be provided at launch plan level?
p
1. Yeah, I believe so. @Prafulla Mahindrakar, is that the preferred technique?
Yes that should be right way if you are leveraging github action . Also for reference config file we use in flyte CI for executing functional tests https://github.com/flyteorg/flytetools/blob/master/functional-tests/config.yaml
@Prafulla Mahindrakar, can tolerations be provided at launch plan level?
Currently no. These are only supported through plugin configuration
k
@Samhita Alla an example would be great
s
@Yuvraj is probably the best person to handle the CI/CD part but I can take a stab at it. Issue: https://github.com/flyteorg/flyte/issues/2772
k
Should be a tutorial
👍 2
s
Oh what has been done at Lyft is exactly what I worked on last week at Wolt for the CI using Github Action, I could probably help if needed 😄
🙏 1
h
Oh thank you so much for all help guys. I will have a look at the tolerations plugin and let you know how it turns out once I allocate some time to it. @Stephen how did you configure flytectl to authenticate against your backend in your CI, given that you are using OIDC authetnication? Also looking to implementing the Lyft workflow, so I might reach out once I grasp my head around the very details of it 🙂 Did you set up the flytectl config as @Prafulla Mahindrakar suggested i.e.,
Copy code
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///development.uniondemo.run
  # Change insecure flag to ensure that you use the right setting for your environment
  insecure: false
  clientId: flytepropeller
  clientSecretLocation: /home/runner/secret_location
logger:
  # Logger settings to control logger output. Useful to debug logger:
  show-source: true
  level: 1
Also, @Samhita Alla what do you mean with _I don’t think you’ll have to modify it in the values.yaml file you shared. It has to do with the deployment technique you opted for. If you’re using the flyte helm chart for instance, it ends up here_ - shouldn’t I modify the values to enable the resource_manager plugin to add toleraions? Not quite following where exactly I specify this for my deployment
s
@Stephen, if you have the bandwidth, would love a contribution detailing how you set up the CI through GitHub Actions. We could add it to the deployment guide.
@Hampus Rosvall, sorry if I wasn’t clear. You should indeed add tolerations for which you’ll have to edit the propeller configmap. And then restart your propeller instance using
kubectl -n flyte rollout restart deploy flytepropeller
. Reference thread: https://discuss.flyte.org/t/432335/Trying-to-use-GPUs-I-added-a-tolerations-section-as-describe.
❤️ 1
k
@Stephen even a partial contribution that we can complete can help a ton
s
@Hampus Rosvall For CI we use Github Secrets to be able to authenticate with Flytectl
Copy code
admin:
  endpoint: dns:///console.flyte.example.com
  insecure: false
  authType: ClientSecret
  clientId: github-client
logger:
  show-source: true
  level: 1
storage:
  type: stow
  stow:
    kind: s3
    config:
      auth_type: iam
      region: eu-west-1
  container: whatever-name-container
Footer
Then during the step, we create a
/etc/secrets
directory like the following
Copy code
docker run \
          -e TAG=$TAG \
          --entrypoint /bin/sh \
          ${{ needs.setup.outputs.path_builder }} \
          -c "\
            mkdir /etc/secrets && \
            echo ${{ secrets[format('{0}_FLYTE_CLIENT_SECRET', needs.setup.outputs.flyte_client_domain )] }} > /etc/secrets/client_secret && \
            pyflyte -c ci/cfg/flyte.config package --image ${{ needs.setup.outputs.path_runner }} -f && \
            flytectl register files --k8sServiceAccount xxx -p ${{ env.PROJECT }} -d ${{ needs.setup.outputs.domain_name }} --version ${{ env.VERSION }} --archive flyte-package.tgz
Depending on the domain we either get the secrets for our dev cluster or or production cluster.
@Samhita Alla @Ketan (kumare3) Sure thing, where could I write a little something to contribute to it?
s
Thank you! How about you add it to https://docs.flyte.org/en/latest/deployment/ideal_flow.html? I can thereby polish or add more content to it.
k
Also - this does not seem to be the standard GitHub action
161 Views