Hi Community I am busy doing a poc to use flyte with apache spark. 🙂 So far its been great however i have one requirement i havent been able to solve and that is

To be able to set spark config at task runtime just before the spark session is created instead of at Flyte task creation or registration time

(The reason is we have a multi tenanted platform where we need to control which meta-store to talk to based on config passed to the flyte task at runtime) Does anybody have any suggestions or tips as to how to solve this issue?

You can set it in the spark context right. If it is only metastore config

so we are looking for a way to set the metastore at flyte job runtime. as the code that runs is generic and takes in a tenantIdentifier and that then fetches data from the correct tenant from their metastore. What you are suggesting means we need the exact same job for each tenant which doesnt scale from maintainability standpoint if we have 100 tenants its alot of maintenance if every single job is the same code with a diffrent runtime value hardcoded

@dry-tent-11199 registration in Flyte is lightweight, you don’t need multiple code copies

so you say manually register a new task per tenant with the spark conf hard coded? (we wanted to try avoid this as it makes knowing which tasks to run a problem further up). But its an option (a less desirable option) if there is no other clear way

You can use dynamic workflows to modify the config

Great ill look into this

Here's an example: https://discuss.flyte.org/t/15646251/hi-whats-the-best-way-to-pass-a-configurable-value-to-a-task#d37ed6c0-f36a-4b6f-be76-8dd100277e01

> And eventually we will support runtime overrides +1 for configuring and creating spark session at runtime. I'm pretty sure that will be useful for us in the future 🙂 (mostly for configs that must be set before starting the spark context).

Let’s make an issue and please chime in on it