Hey, everyone! Quick question: is it possible to "...
# flytekit
m
Hey, everyone! Quick question: is it possible to "lazily evaluate" tasks when registering them? My use case: I have some tasks that will connect with Google Cloud and a remote MLflow server. To do so, we need to define some environment variables, such as
GOOGLE_APPLICATION_CREDENTIALS
and
MLFLOW_TRACKING_USERNAME
. To make things easier for the team, I defined a decorator called
@setup_task
, that receives strings specifying which integrations we want to use. For instance,
Copy code
@task(secret_requests=MLFLOW_SECRETS, ...)
@task_setup(integration_requests=['mflow'])
def hello_mlflow() -> str:
In the above case,
task_setup
is responsible for setting up the environment variables for MLflow. The problem is that these variables are defined as secrets on the Kubernetes cluster, so when I try to register the tasks, I receive errors such as
Copy code
ValueError: Unable to find secret for key MLFLOW_TRACKING_URI in group mlflow-credentials in Env Var:_FSEC_MLFLOW-CREDENTIALS_MLFLOW_TRACKING_URI and FilePath: /etc/secrets/mlflow-credentials/mlflow_tracking_uri
Which is (kinda) expected, since I don't have the secret locally. But my whole logic is that these tasks won't be running locally. Is there no way that I can avoid this? The only workaround I can think is transforming the decorator into a normal function, and calling it inside the task. Is this the only way?
167 Views