Hi community, is there a way to use gpus in the sandbox cluster?

Hello! I am running a databricks task inside of a dynamic workflow. These have worked for about week, but for some reason I am experiencing this issue:

The Databricks Flyte tasks launch successfully, but hang indefinitely after the DB job finishes successfully.

So far I have tried looking at flytepropeller, but I haven't found any logs relating to the execution id in question. Any tips on ways to debug is much appreciated 🙏

Hi Flyte community, Is there a way to increase the verbosity level of pyflyte-execute? I'm currently debugging a task and the only output from running this command is

Killed

. New files aren't generated at the specified output prefix, raw output prefix, or checkpoint path locations.

Hi. Is it still true we have to use

<http://sigs.k8s.io/yaml|sigs.k8s.io/yaml>

in flytectl: https://github.com/flyteorg/flytectl/blob/cea39d9bdf2476f9a5313d1bf19bf08b3923237a/cmd/create/execution_util.go#L13. I'm asking because it depends on go-yaml v2, which does not support yaml 1.2. More specifically, it has weird handling of

n

,

y

,

no

,

yes

, etc. when coming to key name, see https://github.com/go-yaml/yaml/issues/214. If an input argument is named as

n

, when creating an execution via a execution spec file, that argument must be quoted as

"n"

, otherwise flytectl would fail because when unmarshalling

n

is parsed as

false

as the argument name.

Hi all I am investigating Flyte and are stuck on a problem. I uploaded a file to the minio s3 hosted by flytectl demo. I then supply the uri s3://my-s3-bucket/dataset/file.asf as Flytefile("s3://my-s3-bucket/dataset/file.asf") I get: Protocol not found Did you mean file:s3://my-s3-bucket/dataset/file.asf? I am new to S3 so maybe I am doing something wrong here Edit added task snippet:

@task(cache=True, cache_version="2.0")
def process_video() -> FlyteFile:
    """
    Run clip_extract on video
    """
    input_path = FlyteFile("<s3://my-s3-bucket/dataset/test.asf>")
    output_file = FlyteFile("output.mjpg")
    command = ['ffmpeg', '-i', str(input_path), '-c:v', 'mjpeg', '-q:v', '3', '-an', str(output_file)]
    subprocess.check_call(command)
    return output_file

so absolutely dying trying to get this deployed in EKS -- namely on minio charts are effectively the same so I am not sure why minio all the sudden just doesnt work

flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmp030l19_7/script_mode.tar.gz to <http://localhost:30084/my-s3-bucket/a/b/PJPRTCSJDCALQRYI44IFNWNQ5M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20230315%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230315T152958Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=9b449916e2b9ca077d45255d05f86fec463c3e999e14c2c38a0075928906afa0> (recursive=False).

Original exception: HTTPConnectionPool(host='localhost', port=30084): Max retries exceeded with url: /my-s3-bucket/a/b/PJPRTCSJDCALQRYI44IFNWNQ5M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20230315%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230315T152958Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=9b449916e2b9ca077d45255d05f86fec463c3e999e14c2c38a0075928906afa0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f205d87edc0>: Failed to establish a new connection: [Errno 111] Connection refused'))

noticed it was using a signed url override (not sure why it's doing that all of the sudden) and pretty much anything I change/update in the charts just doesnt apply, for example:

storage:
      # -- Sets the storage type. Supported values are sandbox, s3, gcs and custom.
      type: sandbox 
      # -- bucketName defines the storage bucket flyte will use. Required for all types except for sandbox.
      bucketName: my-s3-bucket 
      # -- settings for storage type s3
      signedUrl:
        stowConfigOverride:
          endpoint: <http://minio.flyte.svc.cluster.local:9000>

anyone have any thoughts?

Is there a glob like syntax for FlyteDirectory? If I have a directory that contains files with a prefix (along with other files), is it possible to only grab files matching the prefix?

Hello, I am following Using Secrets in a Task https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/use_secrets.html And I setup the secrets in K8S, and it works fine to retrieve the secrets.

FEATHR_SECRET_GROUP = 'aws'

@task(secret_requests=[Secret(group=FEATHR_SECRET_GROUP, key='s3_access_key'), Secret(group=FEATHR_SECRET_GROUP, key='s3_secret_key')])
def get_feathr_s3_secrets() -> Tuple[str, str]:
    context = current_context()
    s3_access_key = context.secrets.get(FEATHR_SECRET_GROUP, 's3_access_key')
    s3_secret_key = context.secrets.get(FEATHR_SECRET_GROUP, 's3_secret_key')

However, when I test with a key that doesn’t exist, the task is stuck in initializing state forever. Does anyone have an idea how to prevent this. I’d rather the task fails immediately. CC@Kamakshi Muthukrishnan

@task(secret_requests=[Secret(group=FEATHR_SECRET_GROUP, key='xxx'), Secret(group=FEATHR_SECRET_GROUP, key='yyy')])
def get_feathr_s3_secrets() -> Tuple[str, str]:
    context = current_context()
    s3_access_key = context.secrets.get(FEATHR_SECRET_GROUP, 'xxx')
    s3_secret_key = context.secrets.get(FEATHR_SECRET_GROUP, 'yyy')

Hi <!channel>, I hope you all are doing great. I am new to FLyte and I want to learn it as I really see a big potential in it. I just deployed Flyte in my AWS account using Opta. Now I am trying to create a project and push some workflows and run them. Any documentation link, articles or explanations will be very helpful to help me establish a connection with my deployed Flyte instance and be able to process tasks and workflows remotely. Many Tanks Ahmed

Hello everyone. I am getting this error during a workflow run, seem to be after completing a task and need to pass the result to the next task in the workflow. What could be the issue, please?

Message:

    Failed to convert return value for var o0 for function flyte.workflows.challenge_task_cpu with error <class 'AttributeError'>: __args__

SYSTEM ERROR! Contact platform administrators.

I'm also getting this error when trying to execute a workflow. I've checked the types, they seem to be consistent.

TypeTransformerFailedError: Type of Val 'typing.Dict' is not an instance of typing.Dict[str, str]

Hi team, I’m creating

n

tasks, each returning an `int`value, within a dynamic workflow, Is there an easy way for me to sum up / aggregate all the results, potentially in a single task? Currently, I have to do this in a cascade manner, and it’s not that efficient given a reasonably large

n

Hi team, I can’t for the life of me get the

pyflyte package

to work without causing module import issues - any help would be greatly appreciated. The project is in

/project

(root dir) and inside is

/wf

dir with

wf_10.py

where the workflow code lives, where I import tasks from

main.py

in the same dir. I’m running pyflyte from

/project

dir with this command

pyflyte --pkgs wf packages ...

and the packaging fails with:

Loading packages ['wf'] under source root /project
Failed with Unknown Exception <class 'ModuleNotFoundError'> Reason: No module named 'main'
No module named 'main'

If

--pkgs

flag is given, shouldn’t that put the path in PYTHON_PATH and be able to import python modules in that directory (in case you’re wondering, there is an empty

__init__.py

in

/wf

dir)?

I can’t seem to find a lot of docs regarding this and having trouble getting this to work - I tried multiple other combinations here but none of them would work all the way to execution without causing module import issues

Hi Team, What could possibly cause the workflows to go into an UNKNOWN state?

propeller logs:
---------------
{"json":{"src":"controller.go:157"},"level":"info","msg":"==\u003e Enqueueing workflow [examples-hbomax/f77eeb5cdbf1d432da25]","ts":"2023-03-16T18:00:23Z"}
{"json":{"src":"composite_workqueue.go:88"},"level":"debug","msg":"Subqueue handler batch round","ts":"2023-03-16T18:00:23Z"}
{"json":{"src":"composite_workqueue.go:98"},"level":"debug","msg":"Dynamically configured batch size [-1]","ts":"2023-03-16T18:00:23Z"}
{"json":{"src":"composite_workqueue.go:129"},"level":"debug","msg":"Exiting SubQueue handler batch round","ts":"2023-03-16T18:00:23Z"}
{"json":{"src":"composite_workqueue.go:88"},"level":"debug","msg":"Subqueue handler batch round","ts":"2023-03-16T18:00:25Z"}
{"json":{"src":"composite_workqueue.go:98"},"level":"debug","msg":"Dynamically configured batch size [-1]","ts":"2023-03-16T18:00:25Z"}
{"json":{"src":"composite_workqueue.go:129"},"level":"debug","msg":"Exiting SubQueue handler batch round","ts":"2023-03-16T18:00:25Z"}
{"json":{"src":"composite_workqueue.go:88"},"level":"debug","msg":"Subqueue handler batch round","ts":"2023-03-16T18:00:27Z"}

Hello all, I'm having a problem with Flyte on a Kubernetes cluster... Some workflows, when launched, keep stuck on first task "Initializing" status. Flytepropeller logs gives me this:

{
  "json": {
    "exec_id": "a67t5ktm7f6fhksmtcfr",
    "ns": "machine-learning",
    "routine": "worker-32"
  },
  "level": "warning",
  "msg": "Workflow namespace[machine-learning]/name[a67t5ktm7f6fhksmtcfr] has already been terminated.",
  "ts": "2023-03-16T18:10:39Z"
}

Can anyone help me with this? I'll be much appreciated

I was able to run

pyflyte register

and got the following output. But there is nothing registered from the console. Any idea?

Successfully serialized 12 flyte objects
[✔] Registration experiment.flyte.workflows.example.train_base_model type TASK successful with version 895hlD5tQnf8V_eJ_G4Dkg==
[✔] Registration experiment.flyte.workflows.example.train_cloak1 type TASK successful with version 895hlD5tQnf8V_eJ_G4Dkg==
[✔] Registration experiment.flyte.workflows.example.train_cloak2 type TASK successful with version 895hlD5tQnf8V_eJ_G4Dkg==
[✔] Registration experiment.flyte.workflows.example.score type TASK successful with version 895hlD5tQnf8V_eJ_G4Dkg==
[✔] Registration experiment.flyte.workflows.example.sgf_training_wf type WORKFLOW successful with version 895hlD5tQnf8V_eJ_G4Dkg==
[✔] Registration experiment.flyte.workflows.example.sgf_training_wf type LAUNCH_PLAN successful with version 895hlD5tQnf8V_eJ_G4Dkg==
[✔] Registration experiment.flyte.workflows.workflow.train_base_model type TASK successful with version 895hlD5tQnf8V_eJ_G4Dkg==
[✔] Registration experiment.flyte.workflows.workflow.train_cloak1 type TASK successful with version 895hlD5tQnf8V_eJ_G4Dkg==
[✔] Registration experiment.flyte.workflows.workflow.train_cloak2 type TASK successful with version 895hlD5tQnf8V_eJ_G4Dkg==
[✔] Registration experiment.flyte.workflows.workflow.test type TASK successful with version 895hlD5tQnf8V_eJ_G4Dkg==
[✔] Registration experiment.flyte.workflows.workflow.baby_training_wf type WORKFLOW successful with version 895hlD5tQnf8V_eJ_G4Dkg==
[✔] Registration experiment.flyte.workflows.workflow.baby_training_wf type LAUNCH_PLAN successful with version 895hlD5tQnf8V_eJ_G4Dkg==
Successfully registered 12 entities

Hello! Is it possible to run

flytectl demo start

on a VSI (virtual server instance) and be able to reach the UI from the public using the public ip of the server? For example:

<public_ip>:30080/console

. I’ve tried this and the demo is running from within the server but i cant seem to reach it from my own browser- do i need to setup a reverse proxy or do something else? I try to navigate to:

<public_ip>:30080/console

and nothing ends up loading despite the demo/sandbox running properly. I also curled the localhost:30080/console from inside the server and I get the html response back. Any sort of help or feedback would be greatly appreciated, thanks!! 🙏

question: what would be the potential cause for a sidecar_task not being retried even when retries are configured? has to do with UserError

Hello, there seems to be a bug with specifying a

map<string, struct>

input in the console. Specifically, when an entry is deleted from the map, the "Launch" button becomes grayed out as if the input is invalid, even though the other entries are formatted correctly. Re-adding the deleted entry doesn't fix the issue. Is there something I'm missing or should we file a bug?

Hi Flyte community, The ExecutionCreateRequest proto does not contain a field to pass the workflow id that we want to create an execution for. It only has a field for an ExecutionSpec that accepts a launch plan id. Does this mean that we need to create a launch plan for each workflow to use the Flyte Admin ExecutionCreate API operation? https://github.com/flyteorg/flytekit-java/blob/be2f4f664765c26b0b0eff94c81a2b9af32[…]e/flyteidl-protos/src/main/proto/flyteidl/admin/execution.proto

message ExecutionCreateRequest {
    // Name of the project the execution belongs to.
    // +required
    string project = 1;

    // Name of the domain the execution belongs to.
    // A domain can be considered as a subset within a specific project.
    // +required
    string domain = 2;

    // User provided value for the resource.
    // If none is provided the system will generate a unique string.
    // +optional
    string name = 3;

    // Additional fields necessary to launch the execution.
    // +optional
    ExecutionSpec spec = 4;

    // The inputs required to start the execution. All required inputs must be
    // included in this map. If not required and not provided, defaults apply.
    // +optional
    core.LiteralMap inputs = 5;
}

I started getting

Error: docker sandbox doesn't have sufficient memory available. Please run docker system prune -a --volumes

when starting the sandbox cluster. But there is enough memory according to docker info below. Any ideas?

...
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 62.5GiB
Name: Mercury
ID: fa8608e9-e110-482e-8c5f-908edce3debb
Docker Root Dir: /var/lib/docker
...

Hi, has default behavior of flytekit

1.4.1

changed for scikit-learn models? For return hint type, I used

RandomForestClassifier

in

1.2.7

and it was pickled, but now it seems to be

joblib

?

Hi Community, I am using bazel to build the flyte target but when I use bazel output binaries to serialize and register the workflows it lead to ModuleNotFoundError: No module named 'pyflyte' at the time of task execution. The error occurred because the value of

resolver

in task container argument changed from usual

flytekit.core.python_auto_container.default_task_resolver

to

pyflyte.pypi_flytekit.site-packages.flytekit.core.python_auto_container.default_task_resolver

. Can anyone help me in knowing where does flyte-cluster get this value of the resolver from ? is there a way to control this?

I’ve just realised all the environment variable defined within the inline config are becoming lowercase in the actual task pods. This snippet is from the

flyte-backend-flyte-binary-config

ConfigMap and everything looks correct

plugins:
  k8s:
    default-env-vars:
      AWS_METADATA_SERVICE_NUM_ATTEMPTS: 20
      AWS_METADATA_SERVICE_TIMEOUT: 5
      MLFLOW_TRACKING_UI: <http://mlflow.mlflow.svc.cluster.local>

However, looking at the manifest of a task using the Flyte MLFlow plugin, the env vars look like

- name: aws_metadata_service_timeout
  value: '5'
- name: mlflow_tracking_ui
  value: <http://mlflow.mlflow.svc.cluster.local>
- name: aws_metadata_service_num_attempts
  value: '20'

This is causing the MLFlow plugin to track locally as it can’t find MLFLOW_TRACKING_UI Is this a bug or am I doing something wrong?

Has anyone encountered the error when using Sub-Workflow that they might not

fail

even though the workflow failed? I think it’s only a UI error but we have subworflows that are still in

Queued

stage even though the main workflow has now failed. I don’t see any pods in the cluster either.

Hello, I have been applying .with_overrides() to workflows to increase the resource memory limit and it does seem to work. It doesn’t seem to work when I have a workflow has at least one task defined as subclass of PythonInstanceTask. Say I have a workflow with two tasks defined with @task decorator. The workflow failed the first task because of OOMKilled error. Then I apply .with_overrides(requests=Resources(cpu=“1”, mem=“25Gi”)) to the workflow, and both tasks will succeed. Then I change the second task to be a subclass of PythonInstanceTask and run it with the same .with_overrides(requests=Resources(cpu=“1”, mem=“25Gi”)) applied. Now the first task will fail with OOMKilled error again. Do you have any idea why? This lead me to think maybe .with_overrides() doesn’t work when there is a task defined as a subclass of PythonInstanceTask. Thanks

anyone knows how to change imagePullPolicy of a task or workflow?