Hi team, I am looking for a way to pass extra parameters to a flyte agent. I noticed that there are two fields in the task_template - custom and config and the config one is meant for this. It accepts a json which it then transform to a dict. I wonder if the task_config field in the task decorator is actually passed as the config field in the task_template. If that’s the case then why we fetch the task_config data from the task_template.custom field in an agent ?

Cc @Kevin Su / @Haytham Abuelfutuh we should send env vars and other run time metadata

what parameters you want to pass to the agent?

task_config should send configuration data that is specific to the plugin that executes it, right ? In my case if I have a databricks job, I need to pass a databricks object filled with the configuration. I found a way using a class decorator, where I implemend the get_extra_config method and I can pass a dict to the config field in the task_metadata. I don’t know if this is recommended but it works. Forgive my ignorance and please correct me if I am wrong but if I am to use the task_config to pass let’s say custom data into a databricks task, then I could not add extra fields cause the fields names are strongly typed and it would cause a protobuf error (flyteidl,etc). I would have to pass data within one of the current fields like databricks_conf and extract it on the agent bit before I create the payload for databricks, is this correct?

RE: Principal, that's correct... we are in the process of passing additional execution metadata to the agent though...