• Vijay Saravana

    Vijay Saravana

    1 month ago
    Hello Team, I want to specify a GPU pool for my map task. I went through this documentation and it is not very clear. Could you please help with this ?
  • k

    katrina

    1 month ago
  • varsha Parthasarathy

    varsha Parthasarathy

    1 month ago
    Hi @katrina - We use this spark config to specify in which pool to launch a spark task in
    "<http://spark.kubernetes.node.selector.l5.lyft.com/pool|spark.kubernetes.node.selector.l5.lyft.com/pool>": eks-pdx-pool-gpu,
    Now, @Vijay Saravana is trying to convert the spark task to a map task which requires GPU, we are wondering how to do that. Should we add it to task’s Resource?
  • k

    katrina

    1 month ago
    you should definitely add GPU to the task resource decorator but please go through the above documentation (specifically with regards to the configmap changes) in order for the map tasks to be scheduled on machines with gpus
  • varsha Parthasarathy

    varsha Parthasarathy

    1 month ago
    Sure @katrina = Thank you, we will look into that!
    you should definitely add GPU to the task resource decorator Can you show an example for this?
  • please go through the above documentation Where should we add that config? in propeller/admin ?
    Bear with me for all noob questions! 🙂
  • k

    katrina

    1 month ago
    no problem, we love questions 😄 (and finding ways to improve our documentation) you'll need to update the propeller configmap
  • for adding gpu to the map task instances, you can use the with_overrides syntax: https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.map_task.html#flytekit-map-task
  • varsha Parthasarathy

    varsha Parthasarathy

    1 month ago
    k8s:
            co-pilot:
              name: "flyte-copilot-"
              image: ${ecr_repository}:flytecopilot-${flytecopilot_version}
              start-timeout: "30s"
            scheduler-name: flyte-scheduler
            inject-finalizer: true
            default-annotations:
              - <http://flyte.lyft.com/deployment|flyte.lyft.com/deployment>: flyte-l5
            resource-tolerations:
              <http://nvidia.com/gpu|nvidia.com/gpu>:
                key: <http://lyft.com/gpu|lyft.com/gpu>
                value: dedicated
                operator: Equal
                effect: NoSchedule
  • Looks like we already have this added to our propeller config Map 🙂
  • k

    katrina

    1 month ago
    great
  • varsha Parthasarathy

    varsha Parthasarathy

    1 month ago
    That helped @katrina
  • We launched Map tasks on gpu successfully!
  • How can we edit/update flyte documentation?
  • Samhita Alla

    Samhita Alla

    1 month ago
    @varsha Parthasarathy, here’s a detailed guide to update docs. In a nutshell, • Fork the
    flytesnacks
    repo • Update the relevant Python file •
    pip install -r docs-requirements.txt
    in the
    cookbook
    folder • Run
    make html
    in the
    docs
    folder • Open HTML pages present in the
    docs/_build
    directory in the browser to verify if everything’s rendered correctly