Hello all I am having a issue when running the following: (I have simplified it)

from typing import List, Dict, Any
from flytekit import task, Resources, workflow

@task
def task2(b:List[Any]):
    return  b
@workflow
def wf():
    task2(b=["b","b"]) # trying to test translate_inputs_to_literals
if __name__ == "__main__":
    wf()
# running in sandbox, using default image
# pyflyte run  --remote ./test.py  wf

It seems to me that it can not get the pickle file from S3(file not exists) in the sandbox. So I am wondering if there is anything I didn't notice. Thank you in advance!

Folks if you would love to join the core contributor group for Flyte, please join the #contribute channel

Hi Flyte team, a general question, where do you guys keep the info of matching versions between propeller/admin and flytekit? say If I install an old version of flytekit may have issues registering workflows with new version of admin

No our goal is keep complete parity between flytekit and backend (all backend though is one unit) across a major version

Hey Flyte Fam! What's your definition of workflow orchestration? Share it as a reply to this tweet; would love to hear your thoughts! Thank you!

Hey People Following the private image guide: https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/private_images.html I am now attempting to specify the imagePullSecret in the custom pod template. However I am confused about step 1 and 2: 1. Add your

imagePullSecrets

configuration to this custom pod template. 2. Update FlytePropeller about the pod created in the previous step. Can anyone go into more details or translate what this means? - On a related note, is the expectation that one secret is to be created for each project namespace?

is it possible to run a workflow in a different namespace other than

project-domain

?

Hello, I’m inside a proxy and I can’t run any task in flyte because I receive the error: “E0331 224508.071917 73 pod_workers.go:951] “Error syncing pod, skipping” err=“failed to \“StartContainer\” for \“flytekit\” with ImagePullBackOff: \“Back-off pulling image \\\“ghcr.io/flyteorg/flytekit:py3.9-latest\\\“\”" pod=“default/py39-cacher” podUID=42b563ca-4747-4132-bcfc-2d576e4c4b2a” How can I run

flyte demo start

disabling SSL?

Hi, is there a way to add the project root to sys.path for

pyflyte package

? I tried adding

sys.path.append()

code in the workflow file, but it still says

No module named 'src'

during

pyflyte package

. Here’s the project dir structure where I run

pyflyte --pkgs <http://project.wf|project.wf> package ...

in the parent directory of the project (as previously instructed by the team):

/project
  /wf
    /src
      helpers.py
    main.py # where the workflow code is and some fns are imported via `src.helpers.some_fn`

This runs fine locally, but not via

pyflyte package

Thanks in advance for your help!

So looks like I can manually add the project root to PYTHONPATH, but when I do that,

pyflyte package

errors out with

Multiple definitions of the following tasks were found

for all tasks. I can confirm that without setting PYTHONPATH, the command runs successfully without issues.

This seems like a bug with

pyflyte package

-

pyflyte run

seems to work fine when I add the project root path to

sys.path

Hi all! I'm gearing up to move over to flyte and am curious about the best way to approach a rather niche requirement. Certain tasks of mine currently rely on a daemon running in my k8s cluster to execute. The tasks are k8s jobs that connect to this daemon via a url (exposed using k8s service) and a unix socket (exposed via a shared pvc). What would be the best way to make this daemon available to flyte tasks? My naive approach from reading through the docs is to manually deploy the daemon and then modify the tasks pod via flytekitplugins.pod; is there a better way? Thank you!

Hi Flyte family I am facing one issue. When I am adding any python print statement inside flyte task, UI is not showing those in log except error message I am adding print statements for debugging purpose. Any solution for this? Thank you in advance

I'm looking at specifying multiple containers for a task via this tutorial: https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/pod/pod.html#multiple-containers , however, I'd like to change the env variables to a pod based on inputs to the workflow. Is there a way to dynamically change the pod env variables?

Hi everyone! I’m working on a multi-step workflow and have noticed that my Flyte pods wait about 30-45s between the end of one and the beginning of another. This wouldn’t be an inconvenience when running a few steps but it adds up when trying to iterate on workflows of 20+ tasks. The time seems to be spent while the previous pod is still running, before the new one spins up. I’ve confirmed that my code is only taking a few seconds so AFAICT, the delay is due to something happening in Flyte iteself. Does anyone have insight into what’s going on here and if/how I can speed this up?

Hi all, so i have a task that returns a Dataclass and i want to refer to the field of that object in the workflow is it possible? for ex:

@dataclass
@dataclass_json
class MyDataClass:
   name: str

@task
def my_task() -> MyDataClass:
   return MyDataclass object

@task
def another_task(name: str) -> str:
   return f"Hello {name}"


@workflow
def my_wf() -> None:
   result = my_task()
   another_task(name=result.name)

when i try the above, it says

'Promise' object has no attribute 'name'

i just didnt wanna give the whole object to the task, i just want 1 field from it 😅

So folks I have a job thats been running for the last 16 minutes on the sandbox (typically takes around 2 min). I see that it says Parallelism

0

on the web ui. Am I missing something here ?

Also how do you people monitor the progress of a task ?

Hi folks I am attempting to set up azure blob through stow. My values for storage settings looks like so:

storage:
  type: custom
  enableMultiContainer: true
  limits:
    maxDownloadMBs: 500000
    type: custom
  bucketName: "{{ .Values.userSettings.azure.containerName }}"
  custom:
    container: "{{ .Values.userSettings.azure.containerName }}"
    enable-multicontainer: true
    connection: {}
    type: stow
    stow:
      kind: azure
      config:
        account: "{{ .Values.userSettings.azure.storageAccountName }}"
        key: "{{ .Values.userSettings.azure.storageAccountKey }}"

Trying to run the init workflow examples throws:

Traceback (most recent call last):                                                                                                                                                                                                                            │
│   File "/opt/venv/bin/pyflyte-execute", line 8, in <module>                                                                                                                                                                                                   │
│     sys.exit(execute_task_cmd())                                                                                                                                                                                                                              │
│   File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__                                                                                                                                                                          │
│     return self.main(*args, **kwargs)                                                                                                                                                                                                                         │
│   File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main                                                                                                                                                                              ││     rv = self.invoke(ctx)                                                                                                                                                                                                                                     ││   File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke                                                                                                                                                                            ││     return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                                            ││   File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke                                                                                                                                                                             ││     return __callback(*args, **kwargs)                                                                                                                                                                                                                        ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 470, in execute_task_cmd                                                                                                                                                      ││     _execute_task(                                                                                                                                                                                                                                            ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point                                                                                                                                                 ││     return wrapped(*args, **kwargs)                                                                                                                                                                                                                           ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 348, in _execute_task                                                                                                                                                         ││     _handle_annotated_task(ctx, _task_def, inputs, output_prefix)                                                                                                                                                                                             ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 291, in _handle_annotated_task                                                                                                                                                ││     _dispatch_execute(ctx, task_def, inputs, output_prefix)                                                                                                                                                                                                   ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 160, in _dispatch_execute                                                                                                                                                     ││     ctx.file_access.put_data(ctx.execution_state.engine_dir, output_prefix, is_multipart=True)                                                                                                                                                                ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 476, in put_data                                                                                                                                                       ││     raise FlyteAssertion(                                                                                                                                                                                                                                     ││ flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/flyte-z58pqpy5/sandbox/local_flytekit/engine_dir to <abfs://flyte-workflows/metadata/propeller/>...

Original exception: No plugin found for matching protocol of path <abfs://flyte-workflows/metadata/propeller/>...

Are there specific steps to enable/support the abfs protocol, beyond configuring stow as above?

Hi all, I am trying out the Databricks integration and was just want to verify that it requires Databricks Container Services enabled?

Hi team, if I recover an execution A in a new execution B, and then recover execution B in a new execution C, those tasks that are skipped in B (marked as status “Recovered”) will be executed in C. This doesn’t seems to be an expected behavior?

Hey! quick question, is it possible to use a subworkflow within a dynamic workflow?

Folks the Community sync is in progress - https://zoom.us/j/93875115830?pwd=YWZWOHl1ODRRVjhjVkxSV0pmZkJaZz09

Hello, Can map_task take pd.DataFrame as input? e.g.

@task 
def t1(data: pd.DataFrame):
    ...

@workflow
def wf():
   partitioned_data = some List of pd.DataFrame
   map_task(t1)(data=partitioned_data)

Note that the dataframes in the partitioned_data list is dynamically populated with data at runtime, although the size of the list is statically determined at compile time. I doubt map_task will work in this case.

Are office hours still happening on Wednesday mornings? I get a webpage with the content shown in the attached image when i click the join button from this page: https://docs.flyte.org/en/latest/community/index.html Super interested in learning from the experts!

Is there a recommended way to attach labels/annotations to pods/jobs in flyte such that they will show up in the prometheus metrics aggregations? Currently I am using the

labels

and

annotations

tags in a

LaunchPlan

however I am not seeing these show up in any of the metrics queries I am running against our prometheus server. Additionally any recommended ways for custom pod names would also be appreciated.

Hello, I have a workflow containing a Dynamic workflow:

@dynamic
def train_foreach_tenure(df, ...)
    for tenure in range(1, 25, 1):
        if tenure <= 23: 
            data = df[df['TENURE']==tenure]
        else:
            data = df[df['TENURE']>=tenure]

        training_task(df=data)

@workflow
def wf_train()
    ...
    train_foreach_tenure()
    ...