https://flyte.org logo
#ask-the-community
Title
# ask-the-community
b

Brian Tang

12/19/2023, 11:22 PM
šŸ‘‹ We're in the process of updating our flyte components to support
@eager
and I had a few questions: ā€¢ It seems
client_secret_group
and
client_secret_key
are required ā€” but we use a different authentication strategy with
authType: ExternalCommand
that passes an access token that is validated on a proxy that sits in front of flyte-admin. Are these secrets truly required in our auth scenario? ie., we do not use secrets in our
flytectl
usage ā€¢ For all
@eager
usage that will execute on the backend, will users need to provide the
FlyteRemote
configuration for all of them? This feels pretty toilsome:
Copy code
@eager(
    remote=FlyteRemote(
        config=Config.auto(config_file="config.yaml"),
        default_project="flytesnacks",
        default_domain="development",
    ),
    client_secret_group="<my_client_secret_group>",
    client_secret_key="<my_client_secret_key>",
)
ā—¦ We likely can provide predefined configs, but were also curious if there were recommendations from the team on how to better manage this
k

Ketan (kumare3)

12/20/2023, 12:23 AM
there is an issue to automatically pass the auth down
b

Brian Tang

12/20/2023, 4:59 PM
Ah ā€” I see. Would you be able to share more specifics or if there's already an open issue you could link me to
k

Ketan (kumare3)

12/21/2023, 5:47 AM
b

Brian Tang

01/05/2024, 10:06 PM
@John Votta @Ketan (kumare3) Feedback about
@eager
from our user:
from Limitations: https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/advanced_composition/eager_workflows.html#limitations
ā€¢ "You cannot invoke dynamic workflows, map tasks, or launch plans inside an eager workflow."
ā—¦ šŸŸ¢/šŸŸ” this seems ~fine i think? no real need for
@dynamic
or
@workflow
if you're adopting
@eager
ā€¢ "Context managers will only work on locally executed functions within the eager workflow, i.e. using a context manager to modify the behavior of a task or subworkflow will not work because they are executed on a completely different pod."
ā—¦ šŸŸ¢ seems fine
ā€¢ "All exceptions raised by Flyte tasks or workflows will be caught and raised as an EagerException at runtime."
ā—¦ šŸŸ¢ weird, but fine
ā€¢ "All task/subworkflow outputs are materialized as Python values, which includes offloaded types like FlyteFile, FlyteDirectory, StructuredDataset, and pandas.DataFrame will be fully downloaded into the pod running the eager workflow. This prevents you from incrementally downloading or streaming very large datasets in eager workflows."
ā—¦ šŸŸ¢ totally fine, e.g. FDE passes all datasets around as s3 paths anyway instead of using flyte's df handling magic
ā€¢ "Flyte entities that are invoked inside of an eager workflow must be registered under the same project and domain as the eager workflow itself. The eager workflow will execute the latest version of these entities."
ā—¦ šŸŸ”/šŸ”“ iiuc what this means it seems whack to me (see msg above)
ā€¢ "Flyte console currently does not have a first-class way of viewing eager workflows, but it can be accessed via the task list view and the execution graph is viewable via Flyte Decks."
ā—¦ šŸŸ”/šŸ”“ also problematic, since clicking through tasks in the flyte UI is a very useful way to introspect what's happening, monitor task runtimes, monitor overall progress, grab intermediate inputs/outputs/paths, etc.
6 Views