Greg Linklater10/25/2023, 10:17 AM
contains workflows currently being worked on, including ad-hoc registrations resulting from users running
; these are run against corresponding development environments of other apps. 2.
pyflyte run --remote …
contains workflows that are versioned (incl. prerelease versions) for deployment against other corresponding staging environments. Essentially this would be a dedicated environment for QA without all the noise of development and to test rollout. 3.
has the same versioned workflows as
for deployment against the corresponding official production environment. The above is aspirational and I was really glad to see Flyte’s separation of domains per project; however I am missing some things to make this happen and would like to better understood how Flyte is intended to be used for this: 1. Development and Staging are fundamentally separate if all ways – they deploy separate code that is not “promoted” from one environment to the next. On the other hand, Production should run identical code without needing to rebuild/register the workflows after a particular version is cleared to be deployed to production. What is the appropriate way to deal with this? Or is it appropriate to simply reserialize and register the workflow again for the production domain? 2. As each environment is separate, there are separate secrets to be retrieved from an appropriate plugin; however I haven’t found a way to configure this at runtime, i.e. per domain without needing to hard-code the path to those secrets. Is there an idiomatic way to do this? Obviously hard-coding the secret path is not desirable as it would make it difficult to keep the code between staging and production identical (see previous point). Any advice appreciated
Lee Ning Jie Leon10/25/2023, 10:26 AM
Greg Linklater10/25/2023, 10:31 AM
? There isn’t an accessible python runtime for that so one can’t access
Lee Ning Jie Leon10/25/2023, 10:43 AM
works outside tasks if it doesn't there is an env var too, not sure if this is recommended though. https://flyte-org.slack.com/archives/CP2HDHKE1/p1691687747543199?thread_ts=1691687330.613679&cid=CP2HDHKE1
Greg Linklater10/25/2023, 11:01 AM
is the secrets are passed to the
annotation, which AFAICS is finalized at build-time… I guess there isn’t a way to specify different secrets based on the domain which a workflow is deployed to? Now that I think about it, even getting secrets from
still requires specifying those secret requests on the
beforehand anyway. I would need to load secrets for all environments into the task to have a switch based on
Lee Ning Jie Leon10/25/2023, 11:19 AM
Greg Linklater10/26/2023, 7:14 AM
secrets, however I am using Hashicorp Vault secrets which means that either the
will need to change between environments because they are not automatically separated per namespace. I was just about to experiment with using something like the Vault Operator/Injector to create secrets automatically in each namespace and then reference those secrets from within Flyte as
secrets… I’m sure that will work but it is unfortunate that I cannot seem to use the Vault plugin out of the box. I have also had other difficulties with it.
Dan Rammer (hamersaw)10/26/2023, 1:04 PM
Greg Linklater10/26/2023, 1:23 PM