Hello all! Question around the right way of using dynamic workflows with mapped tasks inside them. I adapted the map_task_example to run within a dynamic workflow instead of the general workflow like so:

@dynamic
def wrapper_task(a: typing.List[int]) -> str:
    mapped_out = map_task(a_mappable_task)(a=a).with_overrides(
        retries=4,
    )
    coalesced = coalesce(b=mapped_out)
    return coalesced


@task
def a_mappable_task(a: int) -> str:
    inc = a + 2
    stringified = str(inc)
    return stringified


@task
def coalesce(b: typing.List[str]) -> str:
    coalesced = "".join(b)
    return coalesced


@workflow
def my_map_workflow(a: typing.List[int]) -> str:

    return wrapper_task(a=a)

    if __name__ == "__main__":
        result = my_map_workflow(a=[1, 2, 3, 4, 5])
        print(f"{result}")

I am seeing a type mismatch compilation error however at run-time:

User] malformed dynamic workflow, caused by: Collected Errors: 2
	Error 0: Code: MismatchingTypes, Node Id: dn1, Description: Variable [o0] (type [collection_type:<collection_type:<simple:STRING > > ]) doesn't match expected type [collection_type:<simple:STRING > ].

Any ideas on why the return is a list of lists from the mapped task? And how do I correctly pass these outputs onto other tasks?

Hi everyone, is torch a required dependency of flytekit? Or is it possible to install without the

extras

?

File "/home/ttheisen/repos/chariot/py/apps/v2-model-recommender/internal/workflow/secrets.py", line 1, in <module>
    from flytekit import Secret, current_context
  File "/home/ttheisen/.cache/pypoetry/virtualenvs/v2-model-recommender-KgGCWpXS-py3.8/lib/python3.8/site-packages/flytekit/__init__.py", line 187, in <module>
    from flytekit.extras import pytorch
  File "/home/ttheisen/.cache/pypoetry/virtualenvs/v2-model-recommender-KgGCWpXS-py3.8/lib/python3.8/site-packages/flytekit/extras/pytorch/__init__.py", line 18, in <module>
    import torch
  File "/home/ttheisen/.cache/pypoetry/virtualenvs/v2-model-recommender-KgGCWpXS-py3.8/lib/python3.8/site-packages/torch/__init__.py", line 228, in <module>
    _load_global_deps()
  File "/home/ttheisen/.cache/pypoetry/virtualenvs/v2-model-recommender-KgGCWpXS-py3.8/lib/python3.8/site-packages/torch/__init__.py", line 189, in _load_global_deps
    _preload_cuda_deps(lib_folder, lib_name)
  File "/home/ttheisen/.cache/pypoetry/virtualenvs/v2-model-recommender-KgGCWpXS-py3.8/lib/python3.8/site-packages/torch/__init__.py", line 154, in _preload_cuda_deps
    raise ValueError(f"{lib_name} not found in the system path {sys.path}")
ValueError: libcublas.so.*[0-9] not found in the system path ['/home/ttheisen/repos/chariot/py/apps/v2-model-recommender', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/ttheisen/.cache/pypoetry/virtualenvs/v2-model-recommender-KgGCWpXS-py3.8/lib/python3.8/site-packages']
make: *** [Makefile:34: run] Error 1

I have a question related to automatic downloads of FlyteFiles/FlyteFolders. I have a large amount of data (~1TB) spread across ~15,000 files stored in GCS. I have a Flyte workflow that, as a first step, downloads these files to be operated on locally. I noticed that the download speed is around 20 MB/s and uses about 1-2 CPU cores. I did some testing, and when I manually invoke

gcloud storage cp -r …

inside of a Flyte workflow, I am able to download the same exact files at approximately 1.7GB/s, and I see usage of ~20 CPU cores (the workflow requests 32 cpu cores). I observe the same behavior when uploading files to GCS as well. How can I still make use of FlyteFiles/FlyteFolders and the automatic download/upload functionality, but get faster download/upload speed?

Is there a way to specify the k8 service account to use when executing a task/workflow with FtelyRemote.execute?

i get

"failed to validate that project [studio] and domain [development] are registered, err: [project [studio] not found]\"}\"\n>>"}

[Resolved] Hello, I'm revising an earlier message with a more minimal example. I have run

pyflyte init

to initialize this directory structure below, which runs using

pyflyte run --remote ./workflows/example.py

just fine. However, when adding a custom image, I reach a

ModuleNotFoundError

, and flyte / the image cannot seem to access my code. Directory

.
├── docker_build.sh
├── Dockerfile
├── LICENSE
├── README.md
├── requirements.txt
└── workflows
    ├── example.py
    ├── __init__.py

example.py

Hello, I am trying to install flyte in a local kubernetes cluster. Is there a helm chart available for the same. The helm charts in the documentation is for GKE/EKS cluster only

Hi folks I am attempting to set up azure blob through stow. My values for storage settings looks like so:

storage:
  type: custom
  enableMultiContainer: true
  limits:
    maxDownloadMBs: 500000
    type: custom
  bucketName: "{{ .Values.userSettings.azure.containerName }}"
  custom:
    container: "{{ .Values.userSettings.azure.containerName }}"
    enable-multicontainer: true
    connection: {}
    type: stow
    stow:
      kind: azure
      config:
        account: "{{ .Values.userSettings.azure.storageAccountName }}"
        key: "{{ .Values.userSettings.azure.storageAccountKey }}"

Trying to run the init workflow examples throws:

Traceback (most recent call last):                                                                                                                                                                                                                            │
│   File "/opt/venv/bin/pyflyte-execute", line 8, in <module>                                                                                                                                                                                                   │
│     sys.exit(execute_task_cmd())                                                                                                                                                                                                                              │
│   File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__                                                                                                                                                                          │
│     return self.main(*args, **kwargs)                                                                                                                                                                                                                         │
│   File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main                                                                                                                                                                              ││     rv = self.invoke(ctx)                                                                                                                                                                                                                                     ││   File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke                                                                                                                                                                            ││     return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                                            ││   File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke                                                                                                                                                                             ││     return __callback(*args, **kwargs)                                                                                                                                                                                                                        ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 470, in execute_task_cmd                                                                                                                                                      ││     _execute_task(                                                                                                                                                                                                                                            ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point                                                                                                                                                 ││     return wrapped(*args, **kwargs)                                                                                                                                                                                                                           ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 348, in _execute_task                                                                                                                                                         ││     _handle_annotated_task(ctx, _task_def, inputs, output_prefix)                                                                                                                                                                                             ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 291, in _handle_annotated_task                                                                                                                                                ││     _dispatch_execute(ctx, task_def, inputs, output_prefix)                                                                                                                                                                                                   ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 160, in _dispatch_execute                                                                                                                                                     ││     ctx.file_access.put_data(ctx.execution_state.engine_dir, output_prefix, is_multipart=True)                                                                                                                                                                ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 476, in put_data                                                                                                                                                       ││     raise FlyteAssertion(                                                                                                                                                                                                                                     ││ flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/flyte-z58pqpy5/sandbox/local_flytekit/engine_dir to <abfs://flyte-workflows/metadata/propeller/>...

Original exception: No plugin found for matching protocol of path <abfs://flyte-workflows/metadata/propeller/>...

Are there specific steps to enable/support the abfs protocol, beyond configuring stow as above?

Hi, flyte team. I would like to ask a question about notification email. I have setup notification in values.yaml and build SQS & SNS in aws. I have checked logs that it is publishing message but it did not arriave in my mail box. would anyone can give some advice?

2022/08/18 08:08:18 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/execution_repo.go:61
[21.243ms] [rows:1] UPDATE "executions" SET "id"=358,"created_at"='2022-08-18 08:05:44.14',"updated_at"='2022-08-18 08:08:18.501',"execution_project"='flytesnacks',"execution_domain"='development',"execution_name"='f985b6eddf4c43d5f000',"launch_plan_id"=114,"workflow_id"=106,"phase"='SUCCEEDED',"closure"='<binary>',"spec"='<binary>',"started_at"='2022-08-18 08:05:49.251',"execution_created_at"='2022-08-18 08:05:44.139',"execution_updated_at"='2022-08-18 08:08:18.497',"duration"='2m29.245844975s',"mode"=1,"inputs_uri"='<s3://my-s3-bucket/metadata/flytesnacks/development/f985b6eddf4c43d5f000/inputs>',"user_inputs_uri"='<s3://my-s3-bucket/metadata/flytesnacks/development/f985b6eddf4c43d5f000/user_inputs>',"user"='a57ebf23-a6e8-45e7-bcb1-52b69f508f67',"state"=0 WHERE "execution_project" = 'flytesnacks' AND "execution_domain" = 'development' AND "execution_name" = 'f985b6eddf4c43d5f000'
{"json":{"exec_id":"f985b6eddf4c43d5f000","src":"execution_manager.go:1588"},"level":"debug","msg":"publishing notifications for execution [project:\"flytesnacks\" domain:\"development\" name:\"f985b6eddf4c43d5f000\" ] in state [SUCCEEDED] for notifications [[phases:SUCCEEDED email:\u003crecipients_email:\"<my email>\" \u003e ]]","ts":"2022-08-18T08:08:18Z"}
{"json":{"exec_id":"f985b6eddf4c43d5f000","src":"publisher.go:30"},"level":"debug","msg":"Publishing the following message [recipients_email:\"<my mail>\" sender_email:\"<sender mail setup in SES>\" subject_line:\"Notice: Execution \\\"flyte_fixedrate.positive_wf\\\" has succeeded in \\\"development\\\".\" body:\"Execution \\\\\\\"flyte_fixedrate.positive_wf [f985b6eddf4c43d5f000]\\\\\\\" has succeeded in \\\\\\\"development\\\\\\\". View details at \u003ca href=\\\\<http://flow.qraftpilot.com/console/projects/flytesnacks/domains/development/executions/f985b6eddf4c43d5f000>\u003e http://<flyte url>/console/projects/flytesnacks/domains/development/executions/f985b6eddf4c43d5f000\u003c/a\u003e. \\n\" ]","ts":"2022-08-18T08:08:18Z"}



{"json":{"exec_id":"faf3d92fb8453e24d000","node":"n0","src":"noop_notifications.go:32"},"level":"debug","msg":"call to noop publish with notification type [flyteidl.admin.NodeExecutionEventRequest] and proto message [event:\u003cid:\u003cnode_id:\"n0\" execution_id:\u003cproject:\"flytesnacks\" domain:\"development\" name:\"faf3d92fb8453e24d000\" \u003e \u003e producer_id:\"propeller\" phase:SUCCEEDED occurred_at:\u003cseconds:1660810338 nanos:222102672 \u003e input_uri:\"<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-faf3d92fb8453e24d000/n0/data/inputs.pb>\" output_uri:\"<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-faf3d92fb8453e24d000/n0/data/0/outputs.pb>\" task_node_metadata:\u003c\u003e spec_node_id:\"n0\" node_name:\"be_positive\" event_version:1 deck_uri:\"<s3://my-s3-bucket/metadata/propeller/flytesnacks-development-faf3d92fb8453e24d000/n0/data/0/deck.html>\" \u003e ]","ts":"2022-08-18T08:12:18Z"}

thank you!

hi team, happy Monday. We are getting

terminated in the background, manually

intermittently. No errors are reported in the POD log. Any suggestions on how to troubleshoot?