Hi community I would like to contribute to the flytekit repo Flyte #ask-the-community

Hi community, I would like to contribute to the fl...

Hank Huang

07/13/2023, 9:37 AM

Hi community, I would like to contribute to the flytekit repo. When I following this doc, and try testing it with sandbox. But I’ve encounter the problem for the task would never stop, even though I tried to re-build the image / restart the cluster, and test on my master branch. Still the same. Any good suggestions? Here is my dockerFile

Copy code

FROM python:3.9-slim-buster
USER root
WORKDIR /root
ENV PYTHONPATH /root
RUN apt-get update && apt-get install build-essential -y
RUN apt-get install git -y
# The following line is an example of how to install your own plugins. In this case, it demonstrates how to install the 'deck' plugin.
# RUN pip install -U git+<https://github.com/Yicheng-Lu-llll/flytekit.git@>"demo#egg=flytekitplugins-deck-standard&subdirectory=plugins/flytekit-deck-standard" # replace with your own repo and branch
RUN pip install -U git+<https://github.com/hhcs9527/flytekit.git@master>
ENV FLYTE_INTERNAL_IMAGE "localhost:30000/flytekit:master"

Here is my workflow (relatively short, so I just paste it here)

Copy code

import typing
from typing import Tuple
from flytekit import task, workflow


@task(disable_deck=False)
def t1(a: int) -> typing.NamedTuple("OutputsBC", t1_int_output=int, c=str):
    return a + 2, "world"


@task(disable_deck=False)
def t2(a: str, b: str) -> str:
    return b + a


@workflow
def wf_deck_normal(a: int, b: str) -> Tuple[int, str]:
    x, y = t1(a=a)
    d = t2(a=y, b=b)
    return x, d

I think I get the problem, the running tasks shows the following, anyone know about it?

Copy code

7/13/2023 9:16:22 AM UTC task submitted to K8s

7/13/2023 9:16:22 AM UTC Unschedulable:0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

Yicheng Lu

07/13/2023, 5:26 PM

Seem like you do not have enough cpu resource for you task pod to run. the k3s will use whatever CPU resources are available on the host system.

Hank Huang

07/14/2023, 2:04 AM

Hi @Yicheng Lu , I think you are right.

Hank Huang

07/14/2023, 2:05 AM

But I am still figuring out how to fix it.

Yicheng Lu

07/14/2023, 2:13 AM

Could you increase the num of cpu for you vm, or eliminate other processes that may be consuming CPU resources. You can use

kubectl describe node

to see the actual cpu resources.

Hank Huang

07/14/2023, 2:16 AM

looks like we are able to allocate?

Hank Huang

07/14/2023, 2:18 AM

Hi @Yicheng Lu, just ask chatgpt, does the following mean my cpu resource is running out? • `ephemeral-storage: 252895796231`: The node has 252895796231 bytes (or about 252.9 GB) of ephemeral storage available. Ephemeral storage refers to the temporary storage available on the node. • `hugepages-1Gi: 0`: There are no 1GB huge pages available on this node. Huge pages are large memory pages that can be used to improve performance in specific workloads. • `hugepages-2Mi: 0`: There are no 2MB huge pages available on this node.

Hank Huang

07/14/2023, 2:20 AM

Here is the current cpu usage

Yicheng Lu

07/14/2023, 2:23 AM

Would you mind setting

disable_deck=True

, this may reduce the cpu usage

Hank Huang

07/14/2023, 2:25 AM

Sure! However, the issue I am addressing needs

disable_deck=True

to test the result.

Yicheng Lu

07/14/2023, 2:54 AM

Maybe try set:

Copy code

@task(
    requests=Resources(mem="1Gi",cpu="1"),
    limits=Resources(mem="1Gi",cpu="1"),
    disable_deck=False,
)

Hank Huang

07/14/2023, 2:55 AM

ok! thanks

Hank Huang

07/14/2023, 3:08 AM

it work! looks like the cpu is running out.

Hank Huang

07/14/2023, 3:08 AM

thanks for the help on the setting!

Hank Huang

07/14/2023, 3:12 AM

Hi @Yicheng Lu I turn off desk_disable for the performance measurement mentioned in this pr #1581 Measuring the time taken for each component when executing a task. but looks like the attachment not showing the result you did, do you know how to let it show?

Yicheng Lu

07/14/2023, 3:14 AM

image.png

Yicheng Lu

07/14/2023, 3:15 AM

The flyte deck button~

Hank Huang

07/14/2023, 3:16 AM

I see, so the result would upload it to s3? Looks like I am unable to test it on flyte-deployment, may be some s3 setting Here is what I have.

Copy code

NoSuchKeyThe specified key does not exist.metadata/propeller/flytesnacks-development-fbf6961c9f62546baa08/n1/data/0/deck.htmlmy-s3-bucket/my-s3-bucket/metadata/propeller/flytesnacks-development-fbf6961c9f62546baa08/n1/data/0/deck.html17719D7997601E5385f2f3ba-30a3-46d3-b10b-a0caf4074cef

Yicheng Lu

07/14/2023, 3:17 AM

Looks like deck.html is not in you S3

Yicheng Lu

07/14/2023, 3:18 AM

If you are using flytectl demo, You can access the mino: Access the minio console at: http://localhost:30080/minio/login. The default credentials are: - Username: ``minio`` - Password: ``miniostorage``

Hank Huang

07/14/2023, 3:21 AM

I am using that, should I do something to setup the minio with my env? I just login and re-run => but still no such key

Yicheng Lu

07/14/2023, 3:22 AM

Are you running the sample deck code from https://github.com/flyteorg/flytesnacks/blob/master/examples/basics/basics/deck.py

Hank Huang

07/14/2023, 3:22 AM

Yicheng Lu

07/14/2023, 3:22 AM

You do not need to

Hank Huang

07/14/2023, 3:22 AM

let me run the demo code, I think it would be better

Yicheng Lu

07/14/2023, 3:23 AM

Sure

Yicheng Lu

07/14/2023, 3:24 AM

In principle, as long as you set disable_deck=False, there should have a deck.html

Yicheng Lu

07/14/2023, 3:25 AM

should see deck.html in this path:

Hank Huang

07/14/2023, 3:26 AM

looks like we need flytekitplugins => do you know where to install it?

Yicheng Lu

07/14/2023, 3:28 AM

Copy code

# The following line is an example of how to install your own plugins. In this case, it demonstrates how to install the 'deck' plugin.
# RUN pip install -U git+<https://github.com/Yicheng-Lu-llll/flytekit.git@>"demo#egg=flytekitplugins-deck-standard&subdirectory=plugins/flytekit-deck-standard" # replace with your own repo and branch

Yicheng Lu

07/14/2023, 3:28 AM

https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-deck-standard

Hank Huang

07/14/2023, 3:54 AM

thanks! but I think we need to install plotly ModuleNotFoundError: No module named ‘plotly’ => not sure how you handle this in sand box? here is how I do.

Copy code

RUN pip install -U git+<https://github.com/hhcs9527/flytekit.git@>"master#egg=flytekitplugins-deck-standard&subdirectory=plugins/flytekit-deck-standard" plotly

Hank Huang

07/14/2023, 3:57 AM

still not fix the problem

Yicheng Lu

07/14/2023, 3:58 AM

Copy code

pip install plotly

Hank Huang

07/14/2023, 3:59 AM

you mean run in my running machine? (I’ve already installed)

Yicheng Lu

07/14/2023, 4:00 AM

in you dockerFile, You could: RUN pip install plotly

Hank Huang

07/14/2023, 4:01 AM

yeah… I think I should add a new line for it.

Hank Huang

07/14/2023, 4:01 AM

let me re-build it

Yicheng Lu

07/14/2023, 4:02 AM

sure

Hank Huang

07/14/2023, 4:20 AM

Thanks for the help! but I am still get (even I ran the sample code)

Copy code

NoSuchKeyThe specified key does not exist.metadata/propeller/flytesnacks-development-f41ce8dcaf22f41819b7/n0/data/0/deck.htmlmy-s3-bucket/my-s3-bucket/metadata/propeller/flytesnacks-development-f41ce8dcaf22f41819b7/n0/data/0/deck.html1771A1115FF4F6E685f2f3ba-30a3-46d3-b10b-a0caf4074cef

Hank Huang

07/14/2023, 4:23 AM

However my minio looks a bit different from yours

Hank Huang

07/14/2023, 4:23 AM

I have only deployment folder not metadata, Any good suggestion?

Yicheng Lu

07/15/2023, 4:07 AM

I am not sure what happened actually. Maybe try running the task with the default Flytekit image to see if you can see the deck.html.

Hank Huang

07/15/2023, 4:14 AM

I see, thanks!

Hank Huang

07/15/2023, 4:21 AM

By the way, @Yicheng Lu are you familiar with the dependency setting in flyte? I am import mashumaro to speed up the sterilize / de sterilize process. Here is the related PR(https://github.com/flyteorg/flytekit/pull/1735) The setup.py part is already set! But I found that there are many *requirement.txt need to add such information, that’s why I try to add it through the pip-compile command. Take lib marshmallow as example, not sure if I should do the same in my PR. Any good suggestion?

Yicheng Lu

07/15/2023, 5:31 AM

Sorry, I am not familiar with the dependency setting😞.

Hank Huang

07/15/2023, 5:46 AM

OK! thanks for the anwsering!

Open in Slack

Previous Next