https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
h

Hank Huang

07/13/2023, 9:37 AM
Hi community, I would like to contribute to the flytekit repo. When I following this doc, and try testing it with sandbox. But I’ve encounter the problem for the task would never stop, even though I tried to re-build the image / restart the cluster, and test on my master branch. Still the same. Any good suggestions? Here is my dockerFile
Copy code
FROM python:3.9-slim-buster
USER root
WORKDIR /root
ENV PYTHONPATH /root
RUN apt-get update && apt-get install build-essential -y
RUN apt-get install git -y
# The following line is an example of how to install your own plugins. In this case, it demonstrates how to install the 'deck' plugin.
# RUN pip install -U git+<https://github.com/Yicheng-Lu-llll/flytekit.git@>"demo#egg=flytekitplugins-deck-standard&subdirectory=plugins/flytekit-deck-standard" # replace with your own repo and branch
RUN pip install -U git+<https://github.com/hhcs9527/flytekit.git@master>
ENV FLYTE_INTERNAL_IMAGE "localhost:30000/flytekit:master"
Here is my workflow (relatively short, so I just paste it here)
Copy code
import typing
from typing import Tuple
from flytekit import task, workflow


@task(disable_deck=False)
def t1(a: int) -> typing.NamedTuple("OutputsBC", t1_int_output=int, c=str):
    return a + 2, "world"


@task(disable_deck=False)
def t2(a: str, b: str) -> str:
    return b + a


@workflow
def wf_deck_normal(a: int, b: str) -> Tuple[int, str]:
    x, y = t1(a=a)
    d = t2(a=y, b=b)
    return x, d
I think I get the problem, the running tasks shows the following, anyone know about it?
Copy code
7/13/2023 9:16:22 AM UTC task submitted to K8s

7/13/2023 9:16:22 AM UTC Unschedulable:0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
y

Yicheng Lu

07/13/2023, 5:26 PM
Seem like you do not have enough cpu resource for you task pod to run. the k3s will use whatever CPU resources are available on the host system.
h

Hank Huang

07/14/2023, 2:04 AM
Hi @Yicheng Lu , I think you are right.
But I am still figuring out how to fix it.
y

Yicheng Lu

07/14/2023, 2:13 AM
Could you increase the num of cpu for you vm, or eliminate other processes that may be consuming CPU resources. You can use 
kubectl describe node
to see the actual cpu resources.
h

Hank Huang

07/14/2023, 2:16 AM
looks like we are able to allocate?
Hi @Yicheng Lu, just ask chatgpt, does the following mean my cpu resource is running out? • `ephemeral-storage: 252895796231`: The node has 252895796231 bytes (or about 252.9 GB) of ephemeral storage available. Ephemeral storage refers to the temporary storage available on the node. • `hugepages-1Gi: 0`: There are no 1GB huge pages available on this node. Huge pages are large memory pages that can be used to improve performance in specific workloads. • `hugepages-2Mi: 0`: There are no 2MB huge pages available on this node.
Here is the current cpu usage
y

Yicheng Lu

07/14/2023, 2:23 AM
Would you mind setting 
disable_deck=True
, this may reduce the cpu usage
h

Hank Huang

07/14/2023, 2:25 AM
Sure! However, the issue I am addressing needs 
disable_deck=True
to test the result.
y

Yicheng Lu

07/14/2023, 2:54 AM
Maybe try set:
Copy code
@task(
    requests=Resources(mem="1Gi",cpu="1"),
    limits=Resources(mem="1Gi",cpu="1"),
    disable_deck=False,
)
h

Hank Huang

07/14/2023, 2:55 AM
ok! thanks
it work! looks like the cpu is running out.
thanks for the help on the setting!
Hi @Yicheng Lu I turn off desk_disable for the performance measurement mentioned in this pr #1581 Measuring the time taken for each component when executing a task. but looks like the attachment not showing the result you did, do you know how to let it show?
y

Yicheng Lu

07/14/2023, 3:14 AM
image.png
The flyte deck button~
h

Hank Huang

07/14/2023, 3:16 AM
I see, so the result would upload it to s3? Looks like I am unable to test it on flyte-deployment, may be some s3 setting Here is what I have.
Copy code
NoSuchKeyThe specified key does not exist.metadata/propeller/flytesnacks-development-fbf6961c9f62546baa08/n1/data/0/deck.htmlmy-s3-bucket/my-s3-bucket/metadata/propeller/flytesnacks-development-fbf6961c9f62546baa08/n1/data/0/deck.html17719D7997601E5385f2f3ba-30a3-46d3-b10b-a0caf4074cef
y

Yicheng Lu

07/14/2023, 3:17 AM
Looks like deck.html is not in you S3
If you are using flytectl demo, You can access the mino: Access the minio console at: http://localhost:30080/minio/login. The default credentials are: - Username: ``minio`` - Password: ``miniostorage``
h

Hank Huang

07/14/2023, 3:21 AM
I am using that, should I do something to setup the minio with my env? I just login and re-run => but still no such key
y

Yicheng Lu

07/14/2023, 3:22 AM
h

Hank Huang

07/14/2023, 3:22 AM
no
y

Yicheng Lu

07/14/2023, 3:22 AM
You do not need to
h

Hank Huang

07/14/2023, 3:22 AM
let me run the demo code, I think it would be better
y

Yicheng Lu

07/14/2023, 3:23 AM
Sure
In principle, as long as you set disable_deck=False, there should have a deck.html
should see deck.html in this path:
h

Hank Huang

07/14/2023, 3:26 AM
looks like we need flytekitplugins => do you know where to install it?
y

Yicheng Lu

07/14/2023, 3:28 AM
Copy code
# The following line is an example of how to install your own plugins. In this case, it demonstrates how to install the 'deck' plugin.
# RUN pip install -U git+<https://github.com/Yicheng-Lu-llll/flytekit.git@>"demo#egg=flytekitplugins-deck-standard&subdirectory=plugins/flytekit-deck-standard" # replace with your own repo and branch
h

Hank Huang

07/14/2023, 3:54 AM
thanks! but I think we need to install plotly ModuleNotFoundError: No module named ‘plotly’ => not sure how you handle this in sand box? here is how I do.
Copy code
RUN pip install -U git+<https://github.com/hhcs9527/flytekit.git@>"master#egg=flytekitplugins-deck-standard&subdirectory=plugins/flytekit-deck-standard" plotly
still not fix the problem
y

Yicheng Lu

07/14/2023, 3:58 AM
Copy code
pip install plotly
h

Hank Huang

07/14/2023, 3:59 AM
you mean run in my running machine? (I’ve already installed)
y

Yicheng Lu

07/14/2023, 4:00 AM
in you dockerFile, You could: RUN pip install plotly
h

Hank Huang

07/14/2023, 4:01 AM
yeah… I think I should add a new line for it.
let me re-build it
y

Yicheng Lu

07/14/2023, 4:02 AM
sure
h

Hank Huang

07/14/2023, 4:20 AM
Thanks for the help! but I am still get (even I ran the sample code)
Copy code
NoSuchKeyThe specified key does not exist.metadata/propeller/flytesnacks-development-f41ce8dcaf22f41819b7/n0/data/0/deck.htmlmy-s3-bucket/my-s3-bucket/metadata/propeller/flytesnacks-development-f41ce8dcaf22f41819b7/n0/data/0/deck.html1771A1115FF4F6E685f2f3ba-30a3-46d3-b10b-a0caf4074cef
However my minio looks a bit different from yours
I have only deployment folder not metadata, Any good suggestion?
y

Yicheng Lu

07/15/2023, 4:07 AM
I am not sure what happened actually. Maybe try running the task with the default Flytekit image to see if you can see the deck.html.
h

Hank Huang

07/15/2023, 4:14 AM
I see, thanks!
By the way, @Yicheng Lu are you familiar with the dependency setting in flyte? I am import mashumaro to speed up the sterilize / de sterilize process. Here is the related PR(https://github.com/flyteorg/flytekit/pull/1735) The setup.py part is already set! But I found that there are many *requirement.txt need to add such information, that’s why I try to add it through the pip-compile command. Take lib marshmallow as example, not sure if I should do the same in my PR. Any good suggestion?
y

Yicheng Lu

07/15/2023, 5:31 AM
Sorry, I am not familiar with the dependency setting😞.
h

Hank Huang

07/15/2023, 5:46 AM
OK! thanks for the anwsering!
Powered by