flyte-org #ask-the-community

ask-the-community

Miha Garafolj

10/10/2022, 2:30 PM

Hello Flyte team! We are using Flyte to schedule spark driver and executor pods. The problem is that the tolerations we use somehow don't get applied to these pods. They however work fine for all the other pods. Example tolerations on a spark pod (driver and executor the same):

Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s

Example tolerations on another pod (also flyte created).

Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
                             <http://nvidia.com/gpu=present:NoSchedule|nvidia.com/gpu=present:NoSchedule>
                             <http://nvidia.com/gpu:NoSchedule|nvidia.com/gpu:NoSchedule> op=Exists
                             purpose=compute:NoSchedule

We already tried to restart the node pool. Is this somehow an expected behavior? Flytekit version:

1.1.0

Frank Shen

10/10/2022, 7:00 PM

Hello, I am new to Flyte. I’ve setup flyte locally and started the demo cluster. I created a launch plan below. How do I register it and activate it so that it will show up in the UI? The found the documentation not clear about LP registration. Thank yiou!

from flytekit import LaunchPlan, current_context, task, workflow

@task
def square(val: int) -> int:
    return val * val

@workflow
def my_wf(val: int) -> int:
    result = square(val=val)
    return result

default_lp = LaunchPlan.get_default_launch_plan(current_context(), my_wf)
square_3 = default_lp(val=3)

seunggs

10/11/2022, 3:23 AM

Hi, I’m just ran a workflow execution that outputs a numpy ndarray but I’m not sure how to retrieve the actual data? In the console, it just shows the s3 url with

.npy

extension:

seunggs

10/11/2022, 3:23 AM

{
  "o0": {
    "type": "single (NumpyArray) blob",
    "uri": "<s3://sidetrek-flyte-cluster-flyte-bucket/j7/s8df0ape621iv5k8jz0t-n3-0/5a5182507b2e1d3759ca877860fb780c/5d8c4cacddf4ae178bc3987478b0cb24.npy>"
  }
}

seunggs

10/11/2022, 3:25 AM

I suppose you can open it with

np.load

after downloading the file, but is there any easier way to retrieve the data?

Adwitiya

10/11/2022, 7:14 AM

thanks @Samhita Alla, will run using the docker image

varsha Parthasarathy

10/11/2022, 1:53 PM

HI team, QQ, for flyte map tasks, we have can use resources to specify the requests and limits, is it also possible to pass in the node pool to run this task on?

Krishna Agarwal

10/12/2022, 10:42 AM

Hello 👋 Sorry for being impatient but can someone review my PR, It's been more than 3 days. https://github.com/unionai-oss/unionml/pull/187 cc @Eduardo Apolinario (eapolinario) and @Niels Bantilan

Chris Hairfield

10/12/2022, 9:26 PM

Hello 👋, I recently joined a team using Flyte, and I’m tasked with introducing Flyte’s native scheduler. I’ve gotten my code running in the

sandbox

environment with

--fast

registration, which is pretty awesome. But now I’m stuck trying to set inputs specific to developing in

sandbox

(e.g., hostname of an external service running on my local machine). Question: How do I enable a schedule with specified inputs that override

default_inputs

whenever the scheduler invokes my workflow? What I think I want is a commandline like:

flytectl update launchplan -p {project} -d {domain} {lp_name} --version {version} --activate --execFile exec_spec.yaml

As a test, when I try to follow this line from the docs and omit a required input from the

LaunchPlan.get_or_create

call, then my schedule simply doesn’t run when actived:

Launch plans do not necessarily contain the entire set of required workflow inputs

I hope I’m just missing something! Thanks.

SeungTaeKim

10/13/2022, 1:37 AM

Hello, Flyte Team :flyte: I would like to ask a best usecase of making flyte project. Those who are currently processing ML work in my company are using flyte as a ml pipeline. You know, some research engineers who have not so much known software engineering have learning curve to learning flyte. So, my team builds CI process with github action, the example image is below. research engineers do not need to know how to write flytectl codes. However, there is a problem. This CI process basically made on flyte project per github repo, which make three namespaces in kubernetes (development/staging/production) (My team guess it will be hard to manage those namespaces all) It is too much building namespace per projct, so I would like get solutions or usecase of flyte users. Thank you so much

KS Tarun

10/13/2022, 5:24 AM

Hi, We've tried to setup flyte on an EKS cluster. On checking with

kubectl get pods -n flyte

, the output is as shown below:

NAME                                              READY   STATUS                   RESTARTS          AGE
flyte-deps-contour-contour-cd4d956d9-2xlwk        1/1     Running                  0                 19d
flyte-deps-contour-envoy-244xb                    2/2     Running                  0                 19d
flyte-deps-contour-envoy-rbwp7                    2/2     Running                  0                 19d
flyte-deps-kubernetes-dashboard-8b7d858b7-x9ncd   1/1     Running                  0                 19d
flyte-pod-webhook-85ff4b5f9d-t9l2f                1/1     Running                  0                 19d
flyteadmin-b46f578c7-qww88                        1/1     Running                  0                 17d
flyteconsole-77f665cc84-8gmv7                     1/1     Running                  0                 19d
flytepropeller-655664f9f9-6c69v                   1/1     Running                  0                 14d
flytepropeller-655664f9f9-l67tf                   0/1     Error                    0                 14d
flytepropeller-655664f9f9-zz2c2                   0/1     Error                    0                 19d
flytescheduler-6db897d545-mlntd                   0/1     ContainerStatusUnknown   1                 14d
flytescheduler-6db897d545-mmhtl                   1/1     Running                  563 (8m45s ago)   10d
minio-5cbbccf6cd-bm6ms                            1/1     Running                  0                 17d
postgres-565f59484-qcnjq                          1/1     Running                  0                 19d
syncresources-7b4b44f7cc-p4h9j                    1/1     Running                  0                 19d

But when we run any command to run/register a workflow

pyflyte run ........  etc.

, we're getting the below error:

Traceback (most recent call last):
  File "/home/tarun/.local/bin/pyflyte", line 5, in <module>
    from flytekit.clis.sdk_in_container.pyflyte import main
  File "/home/tarun/.local/lib/python3.8/site-packages/flytekit/__init__.py", line 259, in <module>
    load_implicit_plugins()
  File "/home/tarun/.local/lib/python3.8/site-packages/flytekit/__init__.py", line 253, in load_implicit_plugins
    discovered_plugins = entry_points(group="flytekit.plugins")
TypeError: entry_points() got an unexpected keyword argument 'group'

Can someone explain, what might be the cause of this error ?

Yash Panchwatkar

10/13/2022, 11:43 AM

hello guys i am trying to install flyte on aws eks i have created the initial cluster and nodes also i have created db as well but db connection is not getting establish some one please help

Yash Panchwatkar

10/13/2022, 12:21 PM

Thanks @Shivay Lamba issue is resolved.. will ping if there is any Thanks community

Yash Panchwatkar

10/13/2022, 1:30 PM

aws-load-balancer-controller has not scheduled or some one please verify the same weather I am on right path or not

Sanjay Chouhan

10/13/2022, 4:32 PM

We've tried to setup flyte on an EKS cluster. But getting some issue with Ingress as attached.

Yee

10/13/2022, 5:22 PM

do you have any error messages/stack traces you can paste?

Chandramoulee K V

10/14/2022, 5:39 AM

Hi, I have a private ECR image and i tried binding it to a task and register the workflow . It worked and now the task is binded with the private ECR image but when i run the workflow i get this following error:

containers with unready status: [a4tbv4qblclb2d8kdqg6-n1-0]|Back-off pulling image {ECR IMAGE}

What all accesses should be provided to the ECR image? Any idea on this

Jun Kwan

10/14/2022, 11:34 AM

Hi guys, I am currently following tutorial https://docs.flyte.org/projects/cookbook/en/stable/auto/integrations/flytekit_plugins/onnx_examples/pytorch_onnx.html However, when I want to save image to my current directory by changing to code to

final_img = Image.merge(
        "YCbCr",
        [
            img_out_y,
            img_cb.resize(img_out_y.size, Image.BICUBIC),
            img_cr.resize(img_out_y.size, Image.BICUBIC),
        ],
    ).convert("RGB")

    img_path = (
        Path("cat_superres_with_ort.jpg")
    )
    final_img.save(img_path)

Where can I find cat_superres_with_ort.jpg? Thanks!

Sanjay Chouhan

10/14/2022, 1:43 PM

Getting SSL error while running pyflyte register command for an AWS EKS setup. ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1131)

Sky Wolf

10/15/2022, 2:50 AM

Hello I was wondering if anyone could guide me through some basic open source contributions to flyte so I can get a handle on things.

Sky Wolf

10/15/2022, 2:51 AM

Please dm me if possible!

Yash Panchwatkar

10/15/2022, 6:47 AM

hello... i have a query ... is it okay to remove redishosturl from values-eks.yaml

Yash Panchwatkar

10/15/2022, 6:51 AM

btw all steps got well executed until now....

Yash Panchwatkar

10/15/2022, 6:53 AM

anyone....

Yash Panchwatkar

10/15/2022, 6:57 AM

Thanks @Samhita Alla for reply...🙏

Yash Panchwatkar

10/15/2022, 7:20 AM

hello guys I am trying to install Flyte with Flyte AWS Scheduler executed the command by replacing values in vales-eks.yaml here 2 pods were goes in the crashloopbackoff can any one help

Yash Panchwatkar

10/15/2022, 7:21 AM

Yash Panchwatkar

10/15/2022, 7:28 AM

here are the logs for same

Yash Panchwatkar

10/15/2022, 7:28 AM

Thusal Ranawaka

10/15/2022, 10:53 AM