https://flyte.org
Join Slack
I am kind of a newbee with Flyte. But I am trying to understand how one can make certain properties ...
# flyte-support
f
I am kind of a newbee with Flyte. But I am trying to understand how one can make certain properties such as resource constraints for tasks configurable from command line. I have a pipeline that is configured from command line and uses Flyte remote to launch a workflow. Say I want to change the resource configuration of one of my tasks, say number of CPUs or memory size. Do I need to use a @dynamic task for that and rely on with_overrides?
f
you mean you want to change the cpu/memory at execution time?
does the cli mean - you also want to change it from the ui?
if not, then you can write a wrapper to execute, which can set some constant values that are read prior to module load and then configure
for example a simple way could be an env var to use high mem
Copy code
MEM = "10Gi"
if os.getenv("HIGH_MEM", False):
   MEM = "100Gi"

@task(resources=Resources(mem=MEM))
def foo():
  ...
You can do this to set per domain too
if you are using 
pyflyte run
this will auto-register a new version
f
Ok, I'll try this out (using environment variables). Is there some plan to introduce a notion of global configs in Flyte? Might be useful (a proprietary orchestration system I'm familiar with added such a feature recently and it was widely adopted).
f
We do have attributes config https://docs.flyte.org/en/latest/deployment/configuration/customizable_resources.html#customizing-project-domain-an[…]rkflow-resources-with-flytectl? You can set attributes in project/project-domain/workflow levels. For example
Copy code
flytectl update task-resource-attribute --attrFile tra.yaml
Copy code
# tra.yaml
domain: development
project: flytesnacks
workflow: core.control_flow.merge_sort.merge_sort
defaults:
  cpu: "1"
  memory: "150Mi"
limits:
  cpu: "2"
  memory: "450Mi"
🙇🏽 1
4 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by