flyte-org #ask-the-community

Abhinay Dronavally

06/13/2023, 6:29 AM

Hello, Flyte deployment is stuck even when postgres db is reachable -

Copy code

Normal  Pulled     61s   kubelet            Container image "postgres:15-alpine" already present on machine
  Normal  Created    61s   kubelet            Created container wait-for-db
  Normal  Started    61s   kubelet            Started container wait-for-db

Is there a way to bypass this check?

Nuno Martins

06/13/2023, 9:23 AM

Hello, I am trying to launch a Flyte workflow to run remotely on a specific image (

pyflyte run

using the flags

--remote

and

--image <IMAGE>

), but I always get an error saying that I am missing packages, for example scikit-learn, which I indeed am missing locally. How can I get around this error, since the workflows will not be running locally? Thanks

Robert Ambrus

06/13/2023, 9:42 AM

Hi all, I'm wondering how I can setup a Flyte plugin in

SANDBOX

environment. I've read the Plugin setup guide. As I understand, basically I need to add config changes to

values-override.yaml

and upgrade the flyte helm release with that overrides. Can you please tell me how to do this in the

SANDBOX

environment?

Albert Wibowo

06/13/2023, 9:52 AM

Hi all, I seem to have the following error when running flyte remotely. Anyone has encountered this before?

Copy code

Matplotlib created a temporary config/cache directory at /tmp/matplotlib-dqrfqnc3 because the default path (/home/flytekit/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.

My flyte workflow is quite simple and is as follow:

Copy code

@task()
def get_data() -> pd.DataFrame:
    data_dict = load_breast_cancer(as_frame=True)
    print(data_dict['data'].head())
    return data_dict['data']

@task()
def clean_data_pre_split(data:pd.DataFrame) -> pd.DataFrame:

    return data

@workflow()
def training_workflow():
    data  = get_data()
    data = clean_data_pre_split(data=data)
    return

user

06/13/2023, 4:00 PM

Reminder: 🔴 Community meeting is starting now - Info: https://www.addevent.com/event/EA7823958

Kishore Vikram

06/13/2023, 4:44 PM

[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last): File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point return wrapped(*args, **kwargs) File "/opt/venv/lib/python3.8/site-packages/flytekit/core/base_task.py", line 526, in dispatch_execute raise TypeError( Message: Failed to convert return value for var o0 for function FPD_Normal_DV.main_alternate.build_dataset_task with error <class 'flytekit.core.type_engine.TypeTransformerFailedError'>: Python value cannot be None, expected <class 'str'>/simple: STRING SYSTEM ERROR! Contact platform administrators. I am facing this issue

Ketan (kumare3)

06/13/2023, 9:28 PM

Interesting question, maybe it’s missing, as we assume it is known

Samhita Alla

06/14/2023, 4:40 AM

If you're importing the boto3 library in your flyte workflow, the default Docker image won't work. Have you tried using image spec? https://github.com/flyteorg/flytesnacks/pull/988

Kevin Su

06/14/2023, 7:10 AM

it complains boto3 not found

Does flyteConsole show this error? or this error happens locally

Robert Ambrus

06/14/2023, 9:42 AM

Hi @Evan Sadler @Kevin Su I'm trying to follow this guide and run a Flyte workflow on Databricks. Configured the dbx plugin based on this guide: https://docs.flyte.org/en/latest/deployment/plugins/webapi/databricks.html#deployment-plugin-setup-webapi-databricks I'm trying to take the easy way out first and run the workflow on an existing cluster. Installed

pyflyte

on the cluster and triggered the run like this:

pyflyte run -remote dbx_example_existing_cluster.py my_databricks_job

The Databricks job is started, but failing with this error:

TypeError: loader must define exec_module()

Robert Ambrus

06/14/2023, 9:42 AM

Did I miss some steps?

Victor Churikov

06/14/2023, 11:48 AM

Hey, I have a task I call four times with different inputs. Is it possible to set the display name for each task run individually, as shown in the graph in UI? I looked at some docs and didn’t find anything about it, maybe I missed it

Albert Wibowo

06/14/2023, 12:24 PM

Hello how do I import flytekitpluggins? Do I need to pip install it?

Ariel Kaspit

06/14/2023, 2:47 PM

Hey, How does your development process in Flyte look like? I mean, do you manage a separation between environments - dev and prod for example? If so, how? Thanks everyone!

Albert Wibowo

06/14/2023, 3:25 PM

Hello, I have multiple docker contexts running on my laptop, Is it possible to run

flytectl demo start

on specific docker context? Apologise for spamming questions here - still trying to learn the rope😄

Rezwan Abir

06/14/2023, 4:30 PM

hey folks, everytime i start flyte demo, all my workflows are still there, how do I start a clean flyte instance, where there are NO workflows

Rezwan Abir

06/14/2023, 4:51 PM

hey folks I have a requirements.txt file from which i build a docker image, and from that docker image i create a package

pyflyte --pkgs flyte.workflows package --image me/pdi:flyte-0.0.2.rez-dev

and from this package i do a register

flytectl register files --config /Users/rezwan.abir/.flyte/config.yaml --destinationDirectory . --project flytesnacks --domain development --archive flyte-package.tgz

but when I test workflow it says

ModuleNotFoundError: No module named 'boto3'

Rezwan Abir

06/14/2023, 4:51 PM

in my requrements.txt boto3 is def there. What am i missing?

Kevin Su

06/14/2023, 6:19 PM

@Byron Hsu qq: how do you install MPI operator? cc @yaroslav

Cody Scandore

06/14/2023, 6:23 PM

I was previously able to access the console and execute workflows on a AWS eks deployment. I had to redeploy yesterday, and I am able to access the console but

pyflyte

executions are failing.

Copy code

Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.INTERNAL
	details: failed to create a signed url. Error: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
	status code: 403, request id: 5efc9c88-fdcb-42ab-bea8-8de7a79101e9
	Debug string UNKNOWN:Error received from peer  {grpc_message:"failed to create a signed url. Error: WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403, request id: 5efc9c88-fdcb-42ab-bea8-8de7a79101e9", grpc_status:13, created_time:"2023-06-14T11:16:05.730571-07:00"}

Victor Churikov

06/14/2023, 6:35 PM

Using flyte with GCS. It stores a lot of my outputs between steps that I no longer need when the run is over, is there a way to configure retention for outputs?

Joseph Yi

06/14/2023, 6:38 PM

Hi I updated flytectl and am running in a sandbox environment with

flytectl sandbox start --source .

When I run:

flytectl sandbox exec -- docker build . --tag "flyte:image_tag"

it now gives me the error

Copy code

unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /Dockerfile: no such file or directory

Did anything change?

Mike Morgan

06/14/2023, 8:50 PM

Hello we are still working to get flyte running in our environment. Our IAM roles are pretty wide open at this point. Workflow get stuck in Running state but tasks do not get kicked off and no pods get started. Here is the errors we are getting when we kick of an individual task from the UI. Hope for any hints to help us move forward. Request failed with status code 500 failed to create workflow in propeller FlyteWorkflow.flyte.lyft.com "a79rf65mq9pmq4zhs67d" is invalid: [metadata.annotations: Invalid value: "": name part must be non-empty, metadata.annotations: Invalid value: "": name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName', or 'my.name', or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')] This workflow has no input parameters.

Rezwan Abir

06/14/2023, 9:34 PM

hey folks any idea why docker does not work

Copy code

flytectl demo exec -- docker build . --tag "flyte:v1"

Rezwan Abir

06/14/2023, 9:34 PM

Copy code

OCI runtime exec failed: exec failed: unable to start container process: exec: "docker": executable file not found in $PATH: unknown

Rezwan Abir

06/14/2023, 9:34 PM

but docker is def in my $PATH var, and i can run

docker

in the terminal without issues

Damon Snyder

06/14/2023, 9:46 PM

Hi Friends, I have another question. I'm trying to figure out a development process for iterating on the code for a workflow that depends on both k8s secrets and a custom docker image. What I've found is that if I run

pyflyte run --remote ... path/to/code.py workflow

then make changes to

path/to/code.py

including the

workflow

method or any of its dependencies those changes don't take effect if I

pyflyte run

the same workflow again. I think a) I don't fully understand the execution model and b) I need some pointers from the community 🙂 on how to iterate on developing workflows. Thanks!

Nuno Martins

06/15/2023, 8:00 AM

Hello, I am trying to create a custom image to use for flyte workflows through the

--image

parameter. I am interested in using whylogs, but if I include flyteplugins-whylogs (tried 1.4.2, 1.5.0, 1.6.1, 1.6.2 and 1.7.0), all my workflows fail without error message, even if I'm not importing whylogs in my code. On the other hand, if I remove this plugin, the workflows work normally. Is there something wrong with the plugin?

Albert Wibowo

06/15/2023, 9:48 AM

Hello, what's the difference between

flytectl demo start

and

flytectl sandbox start

? For context, I am NOT using Docker Desktop. I am using Docker Engine + Colima. As such, my docker host is in another path. To change it, I ran the following command as per @Marti Jorda Roca' suggestion:

Copy code

export DOCKER_HOST= .......
flytectl config init

When I ran

flytectl sandbox start

, it worked perfectly fine. But if I ran

flytectl demo start

instead, my workflow failed. Any idea what's happening under the hood? I also tried to use my own docker image when running

flytectl sandbox start

but turns out it did not spin up local registry. Any work around?

Raphael Guzman

06/15/2023, 12:44 PM

Working with the

@task

decorators and wondering if there is a way to combine it with another decorator I am already using. Here is a simple toy example. Does anyone have any ideas why this doesn't work when uncommenting the flyte decorator? I get an error that it doesn't recognize the new argument (see below). Any help is appreciated!

Copy code

╭──────────────────────────── Traceback (most recent call last) ─────────────────────────────╮
│ /workspaces/workflow-orchestration-poc/test2.py:54 in <module>                             │
│                                                                                            │
│ ❱  54 print(fun1(b=3))  # 3 - 1 + 2 + 7 = 11                                               │
│                                                                                            │
│ /usr/local/lib/python3.9/site-packages/flytekit/core/base_task.py:304 in __call__          │
│                                                                                            │
│ ❱ 304 │   │   return flyte_entity_call_handler(self, *args, **kwargs)  # type: ignore      │
│                                                                                            │
│ /usr/local/lib/python3.9/site-packages/flytekit/core/promise.py:1086 in                    │
│ flyte_entity_call_handler                                                                  │
│                                                                                            │
│ ❱ 1086 │   │   │   raise ValueError(                                                       │
╰────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Received unexpected keyword argument 'b' in function 'fun1'

example.py