Hi. I'm trying to start a sandbox environment but it fails with following error messages:

Error: Get "<https://127.0.0.1:30086/api/v1/nodes>": dial tcp 127.0.0.1:30086: connect: connection refused

{"json":{},"level":"error","msg":"Get \"https://127.0.0.1:30086/api/v1/nodes\": dial tcp 127.0.0.130086 connect: connection refused","ts":"2023-01-24T122132-08:00"} Am I missing something? • Windows WSL2 with native filesystem. • flytekit 1.3.1 • flytectl 0.6.26

Is it possible to start a local execution from a launch plan (rather than directly invoking the workflow)? If so, is there a documentation link for this?

Hi, we are trying to install flyte kit 1.0 or higher, but facing compatibility issues with tensorflow which wants protobuf <3.2.0 - is there any way to mitigate this?

Hello! 🙂 Anyone know of a way to get around https://github.com/flyteorg/flyte/issues/3065 on a v1.3.0 demo cluster?

I’m curious if there is a way to configure StructuredDatasets to use an s3 bucket for the remote directories during local executions of tasks / workflows? Maybe there is a setting that I can override and pass in a base s3 path?

Hi. I'm still struggling to launch my workflow from a container inside of Sandbox environment. My workflow requires

verstack

so I included it to requirements.txt. Here is the outline what I did:

$ pyflyte init my_first_wf

$ ./bin/flytectl sandbox start --source .

Then I wrote some codes under workflows directory. I tested locally using python command and it worked without any problem, cuz verstack is inside venv library. Now I built an image using bundled docker_build.sh like this:

$ sh docker_build.sh -r localhost:30000

$ pyflyte --pkgs <http://workflows.my|workflows.my>_workflow package --image  localhost:30000/my_first_wf:fbc629e54b46c12d6396d39ea9d3f37307ba961d

At this moment, when I launch a container at localhost3000/my first wffbc629e54b46c12d6396d39ea9d3f37307ba961d, the python environment recognizes verstack.

$  docker run -it localhost:30000/my_first_wf:fbc629e54b46c12d6396d39ea9d3f37307ba961d sh

# python

>>> import verstack

(no problem)

However, when I try to run my workflow inside the demo cluster, it fails because python couldn't find verstack library.

pyflyte run --remote workflows/my_workflow.py  train --filename '$VVIX.csv' --test_size 0.25 --output_path model.model

The import statement fails because it couldn't find verstack, and attached is a part of the log.

Pod failed. No message received from kubernetes.
[fd4d75a833c134b5f9e1-n0-0] terminated with exit code (1). Reason [Error]. Message: 
l>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/root/workflows/my_workflow.py", line 11, in <module>
    from verstack import LGBMTuner 
ModuleNotFoundError: No module named 'verstack'

Am I successfully launching a pod that I made, or am I launching a different version of pod?

Hello Team! I have some problems, I made a workflow that has one shelltask. This shelltask have pod task config. But workflow gives errors that

failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute
│  handle for plugin [sidecar]: [BadTaskSpecification] invalid TaskSpecification [<nil>], Err: [nil Struct Object passed]

How can I handle it?

Hello everyone 🙂 I am quite new in flyte and to this channel. I am also a windows user. and I have couple of questions: • My first question is about flytectl. anyway to install flytectl on windows? • Second question is about pyflyte. I have a remote flyte server and I made a flyte_cluster_config.yaml file that I set the IP of the flyte server. Then I use the pyflyte to run my flyte wf on the cluster with following command:

pyflyte -c flyte_cluster_config.yml run --remote .\workflows\example.py wf --name minaexample

but I get the following error

status = StatusCode.UNIMPLEMENTED
        details = ""
        debug_error_string = "UNKNOWN:Error received from peer ipv4:10.32.16.101:80 {created_time:"2023-01-26T06:45:08.745870339+00:00", grpc_status:12, grpc_message:""}"

any ideas what could be a problem?

Has anyone done

FlyteDirectory

inputs to a

ContainerTask

? It seems like the download step doesn't correctly handle directories? with

inputs=dict(path=FlyteDirectory)

we're getting

Failed to download from ref [<gs://cradle-bio-pipelines/i9/f7aac4450113a425b905-n0-0/path>]

where that path does exist.

Hey everyone! I had a quick question around caching. I have a workflow in which one task takes over 13 hours, so leveraging caching would be a tremendous help. I understand that Flyte is caching based on the project, domain, cache version, and task signature -- all of these are constant between runs for me, so no issues there. Caching is also determined based on the input, and this is where I have a question. The input to the task is a pandas dataframe (with potentially many many rows), so I'm wondering how Flyte checks the input. For a dataframe input, does Flyte check every row and column and it's a cache miss if they don't exactly match the last input? In that case, would we expect for Flyte to take a significant amount of time to check for a cache hit/miss if the dataframe has many rows or columns?

Anyone able to set up debugger for flytekit in VScode? I kept seeing this:

Hey ya’ll! I’m currently looking into using Flyte as a solution for our DS and DE team- I’ve got a quick question about running the Getting Started Demo. For some reason the first task,

generate_normal_df

, is getting stuck on

running

and has been

queued

for a while now. This was done using the local demo Flyte Cluster. Running it locally (

pyflyte run example.py wf --n 500 --mean 42 --sigma 2

) the workflow executes fine. However, running it on the Flyte Cluster (

pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2

) doesn’t seem to work properly on my end. Any ideas?

Hi here, I am getting the following type error while trying to perform onehotencoding spark operation. The operation executes successfully in my local spark env but fails when I try to run with flyte:

TypeError: Failed to convert return value for var o0 for function ml_project_1.ohe.estimator with error <class 'pyarrow.lib.ArrowInvalid'>: ('Could not convert SparseVector(5, {2: 1.0}) with type SparseVector: did not recognize Python value type when inferring an Arrow data type', 'Conversion failed for column encoded_col_feat1 with type object')

I found a bug in docker_build.sh

while getopts a:r:v:h flag

should be

while getopts p:r:v:h flag

I made some modification so I will send PR later.

flytectl

question: Is it possible to use

flytectl create execution

to execute a workflow from an execution spec yaml (not relaunch or recover an existing run)? The docs only describe retrieving the execution spec of a task using

get tasks --execFile

, which isn't a supported option for

get workflow

. I tried using

get workflow-execution-config --attrFile

but this returns more of a workflow definition than execution spec.

Hi all, I'm trying to get my implementation of a Bazel/Flyte integration off the ground and I'm running into an issue on the last bit which is stumping me. I created a bazel macro called

flyte_library

(taking cues from @Rahul Mehta’s talk!). On run, that rule 1. Creates a py3_image with the workflow file, as well as pulling in the

aws

,

flytectl

, and

pyflyte-*

cli's. I wanted to keep things hermetic within the bazel env so I didn't create a base image with `awscli`/`pyflyte` pre-installed. 2. We add the

FLYTE_INTERNAL_IMAGE

tag, and then push this image to ECR. I'm still not 100% sure what

FLYTE_INTERNAL_IMAGE

does, but followed the examples I could find. 3. We then have a genrule which runs

docker run

using the image we just created, and calls a custom register script which wraps

pyflyte register

to register the workflow, and uses

flytectl

to enable/optionally execute any launchplans registered alongside the workflow. Registration works correctly as far as I can tell. The objects are created and viewable in the Console, but all tasks fail with:

[1/1] currentAttempt done. Last Error: UNKNOWN::Outputs not generated by task execution

I can see the pod starting, pulling the correct image, and the

pyflyte-fast-execute

command exiting successfully via

kubectl

. No logs are created before the script exits so I'm having a bit of trouble identifying the issue. Weirder still, the exact same

pyflyte-fast-execute

command runs fine if I run it in a docker container locally.

Any pointers for debug cache misses? I've started integrating caching into our core training tasks and am seeing that outputs are being correctly written to the cache, but it looks like only cache misses for downstream tasks. I also don't see anything super meaningful in the flytepropeller logs, only a few sparse log lines with:

{
  "json": {
    "exec_id": "run-name",
    "ns": "flytetester-development",
    "routine": "worker-11"
  },
  "level": "warning",
  "msg": "Workflow not found in cache.",
  "ts": "2023-01-26T17:37:00Z"
}

Are there any other locations to check/can I make the logging more verbose to see the cache keys that are being used for each execution?

@Pradithya Aria Pura / @Fredrik Lyford can you folks help me test this out https://github.com/flyteorg/flytekit/pull/1420

Hi everyone, 1. Is there a functional test that i can run to test a new flyte cluster? If so, i would like to extend to test my use cases too. 2. Is there a read-me that talks about how to make changes and test components? For example, flytectl, or flytekit. Thank you so much.

Hello! Could you say why Flytekit logging format is different from Flyte logging? https://github.com/flyteorg/flytestdlib/blob/master/logger/logger.go#L42-L50 https://github.com/flyteorg/flytekit/blob/master/flytekit/loggers.py#L66 This creates problem running the things on EKS for example, as you have to have multiple Fluent Bit parsers for each format to get correct logging level and timestamps later as you ingest the logs.

Quick question, does anyone know whether imperative workflows work with conditions? Thanks 🙂 And another question. Do you know whether I can make a dynamic workflow part of an imperatively defined workflow?

I'm seeing weird behavior where workflows I deploy with

-d staging

are still being registered to the

development

domain. I feel like I'm missing something basic. This workflow has been running in

development

. I'm thinking maybe that workflows/launchplans can't be re-registered in a new domain? CMD:

/usr/bin/pyflyte register -p flytetester -d staging -i <image> atomwise/examples/flyte/

Some output:

Running pyflyte register from /app/atomwise/examples/flyte/canonicalize_smiles_py_image.binary.runfiles/__main__ with images ImageConfig(default_image=Image(name='default', fqn='<account>/flyte_canonicalize_smiles', tag='<tag>'), images=[Image(name='default', fqn='<account>/flyte_canonicalize_smiles', tag='<tag>')]) and image destination folder staging on 1 package(s) ('/app/atomwise/examples/flyte/canonicalize_smiles_py_image.binary.runfiles/__main__/atomwise/exmples/flyte',)

Hi team, it appear that there a bug in flytekit on this line: https://github.com/flyteorg/flytekit/blob/master/flytekit/core/type_engine.py#L1141 When executing a task with Union-typed parameters using FlyteRemote, the above-mentioned line raises an Exception:

File "/home/dev/conda_dev/devenv/Linux/envs/devenv-3.8-c/lib/python3.8/site-packages/flytekit/remote/remote.py", line 847, in execute
    return self.execute_remote_task_lp(
  File "/home/dev/conda_dev/devenv/Linux/envs/devenv-3.8-c/lib/python3.8/site-packages/flytekit/remote/remote.py", line 924, in execute_remote_task_lp
    return self._execute(
  File "/home/dev/conda_dev/devenv/Linux/envs/devenv-3.8-c/lib/python3.8/site-packages/flytekit/remote/remote.py", line 715, in _execute
    type_hints[k] = TypeEngine.guess_python_type(input_flyte_type_map[k].type)
  File "/home/dev/conda_dev/devenv/Linux/envs/devenv-3.8-c/lib/python3.8/site-packages/flytekit/core/type_engine.py", line 856, in guess_python_type
    return transformer.guess_python_type(flyte_type)
  File "/home/dev/conda_dev/devenv/Linux/envs/devenv-3.8-c/lib/python3.8/site-packages/flytekit/core/type_engine.py", line 1125, in guess_python_type
    return typing.Union[tuple(TypeEngine.guess_python_type(v.type) for v in literal_type.union_type.variants)]
  File "/home/dev/conda_dev/devenv/Linux/envs/devenv-3.8-c/lib/python3.8/site-packages/flytekit/core/type_engine.py", line 1125, in <genexpr>
    return typing.Union[tuple(TypeEngine.guess_python_type(v.type) for v in literal_type.union_type.variants)]
AttributeError: 'LiteralType' object has no attribute 'type'

The fix seems to be changing

v.type

to just

v

Is there any way to define a

ContainerTask

with a toleration or affinity? It looks like I may have define a custom

Task

that overrides the pod spec construction method right now?

While deploying v1.2.1 of

flyte-core

through Helm on EKS when

workflow_notifications

are enabled it produces ill-formatted yaml. Steps to reproduce, `test-values.yaml`:

userSettings:
  accountRegion: us-east-1
  accountNumber: 123123123
  notifications:
    topicName: topic-name
    queueName: queue-name

workflow_notifications:
  enabled: true
  config:
    notifications:
      type: "aws"
      region: "{{ .Values.userSettings.accountRegion }}"
      publisher:
        topicName: "arn:aws:sns:{{ .Values.userSettings.accountRegion }}:{{ .Values.userSettings.accountNumber }}:{{ .Values.userSettings.notifications.topicName }}"
      processor:
        queueName: "{{ .Values.userSettings.notifications.queueName }}"
        accountId: "{{ .Values.userSettings.accountNumber }}"
      emailer:
        subject: "Flyte: {{ project }}/{{ domain }}/{{ launch_plan.name }} has '{{ phase }}'"
        sender: "{{ .Values.userSettings.notifications.sender }}"
        body: |
          "Execution {{ workflow.project }}/{{ workflow.domain }}/{{ workflow.name }}/{{ name }} has {{ phase }}.
           Details: <https://flyte.example.com/console/projects/{{> project }}/domains/{{ domain }}/executions/{{ name }}.
           {{ error }}"

Then run template:

helm template admin flyteorg/flyte-core --version v1.2.1 --values test-values.yaml | grep -A 4 "notifications.yaml"

Which outputs:

notifications.yaml: |
    notifications:
      type: aws
      region:us-east-1
      publisher:

By yaml spec the key must be separated with space, so

region:us-east-1

breaks the service deployment and should be

region: us-east-1

instead. Simple workaround is to put an extra space inside templated string

region: " {{ .Values.userSettings.accountRegion }}"

.

Hi! I have Flyte deployed on a K8s cluster and I also have Prometheus set to scrape FlyteAdmin and FlytePropeller metrics. Thing is, I currently have 3 replicas of the FlytePropeller pod and so I have 3 values for each metric. I'm worried this might impact negatively the utility of these metric. As an example, I get 3 different workflow average durations, one for each pod, but I would like to know the average duration of all executions. Is there a way to solve this? For some metrics I think I could just sum them by the instance label (which is the pod), but I don't know if this will work everytime and it will also make the queries more complex.

Hello! I want to know what is the best way to deal with image files in and between tasks, and if the Flytectl image files do this job in a optimized way. In addiction, when I create an task to download the images and do some operation, when the task is done, it will clean the local storage or do I need to do it manually?

Hi. In a code,

flytekit/clis/sdk_in_container/init.py

, there is a call to https://github.com/flyteorg/flytekit-python-template.git Does anybody know if this repository is public or not? I guess it's public but hidden?? I would like make a PR for this repo.

I'm trying to understand the difference between task and workflow at a fundamental level. My understanding was that tasks could not call other tasks. I wrote the following dummy example:

@task
def test(x: int) -> int:
    return x + 1

@task
def echo_integer(value: int) -> int:
    logging.error(f"I've got value {value}")
    return test(x=value + 5) + 1

@workflow
def wf(
    value: int,
):
    return echo_integer(value=value)

When run locally, we see type-errors because the

echo_integer

task now returns a promise instead of an int. Interestingly, it works fine when we run remotely, with the caveat that the called task is not run as a isolated container. I would have expected it to fail remotely as well. Any pointers about what I'm missing in my understanding?

Hello, Output data from a task can be consumed directly by a subsequent task in the same workflow as the input data. However, what about consuming data from a task by another task from a separate workflow? One way is to save the output data from the first task to S3 and the second task can read the data from S3. Is there a better way provided by Flyte?