Hi team, I am looking for a way to pass extra parameters to a flyte agent. I noticed that there are two fields in the task_template - custom and config and the config one is meant for this. It accepts a json which it then transform to a dict. I wonder if the task_config field in the task decorator is actually passed as the config field in the task_template. If that’s the case then why we fetch the task_config data from the task_template.custom field in an agent ?
01/18/2024, 3:58 PM
Cc @Kevin Su / @Haytham Abuelfutuh we should send env vars and other run time metadata
task_config should send configuration data that is specific to the plugin that executes it, right ? In my case if I have a databricks job, I need to pass a databricks object filled with the configuration. I found a way using a class decorator, where I implemend the get_extra_config method and I can pass a dict to the config field in the task_metadata. I don’t know if this is recommended but it works. Forgive my ignorance and please correct me if I am wrong but if I am to use the task_config to pass let’s say custom data into a databricks task, then I could not add extra fields cause the fields names are strongly typed and it would cause a protobuf error (flyteidl,etc). I would have to pass data within one of the current fields like databricks_conf and extract it on the agent bit before I create the payload for databricks, is this correct?
what I would like to get from the agent would be the task identifiers project and domain but I think I can get that from the task id identifier (correct me if I am wrong)
I would also like to see if I can get the principal. I believe currently this is stored in the execution metadata and is not passed to the agent as via the task_template. I would have to construct a flyte client and make a call back to the admin to get this information, is this correct ?
02/09/2024, 5:42 PM
RE: Principal, that's correct... we are in the process of passing additional execution metadata to the agent though...