Hi. I've been reading the Flyte docs and have some questions around packaging and deployment. The documentation on this centers around how to package and register the worfklows and tasks. Assume my project includes a large number of python modules (say some kind of library, e.g. ml model code, data processing logic, etc) and most of the time I'm changing those and not the workflows or the tasks. Is there an assumption that I should be isolating my library as a python package and declare it as a dependency of the Flyte project? Wouldn't that make iteration very cumbersome? Note that most of the time I'm not iterating on the workflow or the tasks (which I expect will be quite stable), but instead iterating on the library code that those tasks depend on.

Why do you think it has to be isolated as a python package?

Let's say I want to have a code structure like this: repo \- requirements.in/.txt \- foo_lib [python modules] \- orchestration \- workflows [depends on foo_lib]

pyflyte run --remote orchestration/workflows/foo.py foo_workflow --name "Acme" --copy-all

let me try and will get back to you

do you have __`init__` file in the

orchestration

folder?

I’ll follow up here once I try the suggestions, if I still have an issue

are you using bazel?

@Kevin Su I am not using bazel. I made some progress. I am trying local demo cluster first. I started the demo cluster:

flytectl demo start

Everything is up and running. I did:

export FLYTECTL_CONFIG=~/.flyte/config-sandbox.yaml

My config contents:

admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: localhost:30080
  insecure: true
# This is not a needed configuration, only useful if you want to explore the data in sandbox. For non sandbox, please
# do not use this configuration, instead prefer to use aws, gcs, azure sessions. Flytekit, should use fsspec to
# auto select the right backend to pull data as long as the sessions are configured. For Sandbox, this is special, as
# minio is s3 compatible and we ship with minio in sandbox.
storage:
  connection:
    endpoint: <http://localhost:30002>
    access-key: minio
    secret-key: miniostorage

Netstat shows the port is active:

(ubuntu) ubuntu@ip-10-42-128-255:~/vfm$ netstat -an | grep 30080
tcp        0      0 0.0.0.0:30080           0.0.0.0:*               LISTEN

But when I create a project:

flytectl create project --id "my-hello-world-project" --labels "my-label=my-project" --description "My Flyte Hello World project" --name "My Hello World Project"

I get a connection error:

Error: Connection Info: [Endpoint: localhost:30080, InsecureConnection?: true, AuthMode: ClientSecret]: rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: connection failure
{"json":{},"level":"error","msg":"Connection Info: [Endpoint: localhost:30080, InsecureConnection?: true, AuthMode: ClientSecret]: rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: connection failure","ts":"2024-04-30T20:08:33Z"}

seems like a network issue, could you restart the demo cluster

flytectl demo teardown --volume
flytectl demo start

@Kevin Su Same issue.

@Buğra Gedik I can pull

Yes.

what!

The flyte agent is stuck and it is due to image pull failing

but all other images are downloaded fine?

Yes. Looks like this is the only one from gcr? I just spot checked a few others.

if you are not using agent, could you disable that first.

flytectl demo start --disable-agent

ya most likely you are not using agents @Buğra Gedik just disable it for now - also it is not gcr - it is also ghcr

Yeah, sorry for the typo

I think i know, it can be probably because you are running out of space for the docker daemon?

I'm sure it works for others. Something is off for me. I'll check the space a little later and report back.

but your error message is suspicious. But i think it may be red herring?

Running a hello world also gave a similar error within the task:

[1/1] currentAttempt done. Last Error: USER::Grace period [3m0s] exceeded|containers with unready status: [f092c13fb2ecf4c4fb31-n0-0]|Back-off pulling image "<http://cr.flyte.org/flyteorg/flytekit:py3.11-1.11.0|cr.flyte.org/flyteorg/flytekit:py3.11-1.11.0>"

S

+1

@Kevin Su, @Ketan (kumare3) Just to close the loop here, the problem with not being able to pull images had to do with DNS problems in the demo K8 cluster. It looks like K8 inherits DNS configs for kube-dns from /etc/resolv.conf on the host machine. However, on ubuntu (which I'm using), /etc/resolv.conf is an indirection to a local address. The problem is reported here: https://github.com/kubernetes/kubernetes/issues/23474 I had to follow this (https://askubuntu.com/questions/130452/how-do-i-add-a-dns-server-via-resolv-conf/51332#51332) to add a non-local DNS (like 8.8.8.8) to my /etc/resolv.conf. This way I was able to start the local Flyte demo cluster and also the test jobs passed.

thank you for the feedback. It might be good to capture this in a troubleshooting guide

@Kevin Su going back to my original question that started this thread:

if so, pyflyte run --copy-all will copy entire <repo> to s3, and you task will download it while running