Richard Bellon
09/12/2023, 11:25 AMdomains
are intended for.
I guess that it is an abstract term and I can use it for whatever I want. However, all the examples I have seen so far use domains
to separate environments (e.g. dev
, staging
, prod
). If that is the main purpose then I'm confused, I really don't see the point of Flyte taking the responsibility of managing these environments. I would rather use a CI/CD tool to deploy the workflows to different Kubernetes environments that we maintain. I would like to know if I'm missing something here.
The second question that I have is partly derives from the above one. Due to the high number of ML dependencies that our modules depend on I'm trying to utilize reference workflows
in order to decouple some of those thus allowing us to build docker images with smaller sizes. It seems to be the perfect feature to achieve this, however, to define a reference workflow it is unavoidable to define the target workflow's domain. It is confusing to me because tasks and workflows become tied to a domain only when they are registered. As for reference workflows I have to pass the domain in the workflow definition, corrupting our source code with information that relates to an abstraction level that is not even present at that point. I would like to know if there is a way to register reference workflows the same transparent way as tasks and workflows, i.e. without having to pass the domain in the workflow definition, only at the time of registration. (Of course in that case no actual registration would take place, it would only verify that the target workflow exists in the specified domain.)
Thanks in advance!Ketan (kumare3)
Mike Olasz
09/12/2023, 2:41 PMKetan (kumare3)
Mike Olasz
09/12/2023, 8:16 PMKetan (kumare3)
Samhita Alla
@reference_task(
project="flytesnacks",
domain="{{ registration.domain }}",
name="advanced_composition.files.normalize_columns",
version="ef84Eg5pvPJd3IFtJ-o0Bg==",
)
def normalize_columns(
csv_url: FlyteFile,
column_names: List[str],
columns_to_normalize: List[str],
output_location: str,
) -> FlyteFile:
...
The following are the commands I ran to package and register the code:
pyflyte --pkgs productionizing.reference_task package -f
flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version v2
Richard Bellon
09/13/2023, 7:05 AMSamhita Alla