flyte-org #ask-the-community

ask-the-community

seunggs

09/12/2022, 3:00 AM

I’m trying to execute a simple test workflow and getting this error:

Request failed with status code 500 failed to create workflow in propeller namespaces <project-name> not found

. The project already shows up in the flyte console and I was able to package/register the workflow, but I can’t seem to execute it?

seunggs

09/12/2022, 3:02 AM

I created the project via REST API (

POST /api/v1/projects

) with the following payload:

seunggs

09/12/2022, 3:03 AM

project: {
  id,
  name,
  description
}

Andrew Achkar

09/12/2022, 3:54 PM

Hi Flyte team, I noticed a strange error when attempting to run

pylint

on a python module contains

TypeVar('tar.gz')

which is used in specifying a

FlyteFile

. Details in 🧵

Ailin Yu

09/14/2022, 6:22 PM

Hello! I am interested in using the published Grafana dashboards to monitor Flyte. But even though my Flyte instance has these pod annotations:

<http://prometheus.io/path=/stats/prometheus|prometheus.io/path=/stats/prometheus>

<http://prometheus.io/port=15020|prometheus.io/port=15020>

<http://prometheus.io/scrape=true|prometheus.io/scrape=true>

And I’ve confirmed prometheus is scraping the pods with those annotations, I’m not seeing any Flyte provided metrics, so the dashboards are not populating. Is there some config I need to enable in flytepropeller in order for metrics to be emitted?

James Evers

09/14/2022, 6:38 PM

hey everyone, i'm trying to run ray on the flyte sandbox deployment, and am running into the following error. it looks like its basically an OOM error, but i'm not sure how to increase the default size of

/dev/shm

. what's the best approach to bumping this default in the sandbox deployment?

/usr/local/lib/python3.8/dist-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
2022-09-14 18:10:43,190	WARNING services.py:1882 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67108864 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=0.10gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.
2022-09-14 18:10:46,611	INFO worker.py:1509 -- Started a local Ray instance. View the dashboard at [1m[32m<http://127.0.0.1:8265> [39m[22m
[2022-09-14 18:10:57,864 E 1 1] <http://core_worker.cc:149|core_worker.cc:149>: Failed to register worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff to Raylet. IOError: [RayletClient] Unable to register worker with raylet. No such file or directory

Justin Tyberg

09/14/2022, 11:48 PM

I’m having trouble accessing flyteadmin from flytectl that is running in a pod on a GKE cluster. • flyteadmin running in

flyte

namespace • flytectl running in

foo

namespace If I run flytectl from within the pod in

foo

namespace, and hit the external endpoint (through the ingress), it works

bin/flytectl get projects \
  --admin.endpoint dns:///EXTERNAL_FQDN:443 \
  --admin.authType ClientSecret \
  --admin.clientId flytepropeller \
  --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
 ----- ------ ----------------- 
| ID  | NAME | DESCRIPTION     |
 ----- ------ ----------------- 
| dpp | dpp  | dpp description |
 ----- ------ ----------------- 
1 rows

However, if I try to hit the internal grpc endpoint, I get nada. No output.

bin/flytectl get projects \
>   --admin.endpoint dns:///flyteadmin.flyte.svc.cluster.local:81 \
>   --admin.authType ClientSecret \
>   --admin.clientId flytepropeller \
>   --admin.insecure true \
>   --admin.clientSecretLocation /etc/secrets/client_secret
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 

echo $?
0

🤔

KS Tarun

09/15/2022, 9:36 AM

Hi All, I have a script with a few tasks and the workflow defined. Certain information is stored in a config file which I am importing into this script. But when I run the workflow, it is giving a module not found error. What should I be doing to import other scripts ?

James Evers

09/15/2022, 5:53 PM

hi everybody, has anybody running on a mac m1 managed to run the tensorflow cookbook example?

seunggs

09/16/2022, 2:35 AM

Hi, I’m trying to execute a workflow in EKS and my simple add_one task is failing with this error but not sure how to debug it:

seunggs

09/16/2022, 2:35 AM

[4/4] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[a29fz6979nbn4dxsr7km-n0-3] terminated with exit code (1). Reason [Error]. Message: 
Traceback (most recent call last):
  File "/opt/venv/bin/pyflyte-execute", line 8, in <module>
    sys.exit(execute_task_cmd())
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 470, in execute_task_cmd
    _execute_task(
  File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
    return wrapped(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 348, in _execute_task
    _handle_annotated_task(ctx, _task_def, inputs, output_prefix)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 291, in _handle_annotated_task
    _dispatch_execute(ctx, task_def, inputs, output_prefix)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 80, in _dispatch_execute
    logger.debug(f"Starting _dispatch_execute for {task_def.name}")
AttributeError: 'function' object has no attribute 'name'

seunggs

09/16/2022, 4:18 AM

A quick question regarding flyte

/workflows

REST API endpoint - I see there’s an archive option in the dashboard but I don’t see a PUT endpoint for

/workflows

- is this expected or am I missing something?

Leong Shing Chew

09/16/2022, 2:16 PM

Hi, I have a question with regards to cluster resource templates, subsequent update to the templates(addition, update and removal) would be reflected in the k8s resources after the refresh interval right? Just want to confirm that deletion of the templates under "cluster resource manager" component would trigger the deletion of resources in the existing namespaces(project/domain).

Katrina P

09/16/2022, 4:32 PM

Hey everyone! I'm currently working on notifications and monitoring of Flyte. We have a little bit of a blocker on using SES right now in AWS, but I was wondering if anyone was working on either STMP or some direct Slack integration (would like to help out if so!). Additionally, is anyone else using datadog? I'm in the process of checking out getting the prometheus endpoint into the datadog configs and just curious if anyone else has successfully done this

George D. Torres

09/16/2022, 5:10 PM

Is there schema documentation for api requests that execute a launch plan? I found the protocol documentation , but I can't find any specific schema for what the inputs should look like when they are not primitives (e.g. a dataclass type that my workflow defines)

Zachary Kimble

09/16/2022, 5:22 PM

Hello, I'm trying to run a demo workflow with a custom image in local sandbox, but am having authentication issues puling the image from Azure container registry. I initialized the sandbox with

flytectl sandbox start

I created a secrets file

acr-secrets-flytesnacks-development.yaml

apiVersion: v1
data:
  .dockerconfigjson: ***Base64 encoded json***
kind: Secret
metadata:
  name: acr-pull-credentials
  namespace: flyte-development
type: <http://kubernetes.io/dockerconfigjson|kubernetes.io/dockerconfigjson>

I applied the secret in the project-domain namespace with

kubectl -n flytesnacks-development apply -f secrets/acr-secrets-flytesnacks-development.yaml

I patched the default service account within the project-domain namespace :

kubectl -n flytesnacks-development patch serviceaccount default -p '{"imagePullSecrets": [{"name": "acr-pull-credentials"}]}'

I run my workflow with

pyflyte run --remote --image ***.<http://azurecr.io/databricks_workflow:latest|azurecr.io/databricks_workflow:latest> databricks_wf.py databricks_workflow --sql 'select...'

The pods get stuck in pending and describe shows the following issue:

Failed to pull image "***.<http://azurecr.io/databricks_workflow:latest|azurecr.io/databricks_workflow:latest>": rpc error: code = Unknown desc = Error response from daemon: Head "https://*****.<http://azurecr.io/v2/databricks_workflow/manifests/latest|azurecr.io/v2/databricks_workflow/manifests/latest>": unauthorized: authentication required, visit <https://aka.ms/acr/authorization> for more information.

Placing the decoded secret in .docker/config.json allows

docker pull

to work locally. Thanks for a any suggestions!

seunggs

09/16/2022, 5:33 PM

So I’m stuck on S3 access - I created a service account called

flyte-executor

with the

flyte-user-role

(which has full s3 access) attached as an annotation and running Flyte executions with this service account, but it’s giving me PutObject access denied error. This service account is in the project+domain namespace. What am I doing wrong?

seunggs

09/16/2022, 5:37 PM

Name:                flyte-executor
Namespace:           myproject-development
Labels:              <http://app.kubernetes.io/managed-by=pulumi|app.kubernetes.io/managed-by=pulumi>
Annotations:         <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::xxx:role/flyte-user-role
Image pull secrets:  gcr-json-key
...

Alex Bain

09/16/2022, 10:00 PM

Flyte team, I would like to set a

priorityClassName

(and perhaps

preemptionPolicy

) on all the k8s pods for Flyte (and Spark executor) task pods that run in my EKS cluster. Could you give me a suggestion as to how to accomplish this?

Sebastian

09/19/2022, 7:10 AM

Hello, I am trying to schedule a workflow but the documentation feels quite spotty on this. I have a workflow

wf

and launch plan like

launch_plan.LaunchPlan.get_or_create(
    workflow=wf,
    name="my_lp",
    schedule=CronSchedule("0 0 ? * * *"),
    default_inputs={...},
    fixed_inputs={...},
)

how do I schedule this after registering it alongside my workflows with

pyflyte serialize

and

flytectl register

? I think the relevant docs are supposed to be https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_launchplan.html but running the specified commands fails with

launch plan prod failed to update due to rpc error: code = NotFound desc

. This command requires a version which I have none so it might be because of this. How should I proceed? Some side note feedback: It feels to me like activating launch plans is more naturally done in the web UI. However, trying to open my launch plan errors with

Error: Minified React error #31; visit <https://reactjs.org/docs/error-decoder.html?invariant=31&args[]=object%20with%20keys%20%7Brole_session_name%2C%20region_name%2C%20role_arn%7D&args[]=> for the full message or use the non-minified dev environment for full errors and additional helpful warnings.

Which probably comes from me using using struct arguments (role_session_name, region_name, role_arn are in a dataclass) but it nevertheless seems like there is a bug in the launch plan rendering.

KS Tarun

09/19/2022, 8:57 AM

Hi team, When running a working flow which requires the data to be pulled from multiple tables, I'm getting this OOM error. What can be done to avoid this ?

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[a5dqkgsjw2gf59jzhpgn-n1-0] terminated with exit code (1). Reason [OOMKilled]. Message: 
tar: Removing leading `/' from member names

Traceback (most recent call last):
  File "/opt/venv/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 507, in fast_execute_task_cmd
    subprocess.run(cmd, check=True)
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pyflyte-execute', '--inputs', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-staging-a5dqkgsjw2gf59jzhpgn/n1/data/inputs.pb>', '--output-prefix', '<s3://my-s3-bucket/metadata/propeller/flytesnacks-staging-a5dqkgsjw2gf59jzhpgn/n1/data/0>', '--raw-output-data-prefix', '<s3://my-s3-bucket/test/lh/a5dqkgsjw2gf59jzhpgn-n1-0>', '--checkpoint-path', '<s3://my-s3-bucket/test/lh/a5dqkgsjw2gf59jzhpgn-n1-0/_flytecheckpoints>', '--prev-checkpoint', '""', '--dynamic-addl-distro', '<s3://my-s3-bucket/gp/flytesnacks/staging/NTHIUJ632K3UNCS775EH722XVM======/fast31220425e74d24c7973f254bb8ecf02f.tar.gz>', '--dynamic-dest-dir', '/root', '--resolver', 'flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'get_io_f7', 'task-name', 'get_idv']' died with <Signals.SIGKILL: 9>.

Robin Eklund

09/19/2022, 11:30 AM

Hi! I am following this guide: https://docs.flyte.org/projects/cookbook/en/stable/auto/larger_apps/larger_apps_deploy.html I have built a Docker images, deployed it to AWS ECR, packaged and registered everything to flyte. When i execute the workflow i get

ModuleNotFoundError: No module named 'workflows'

. These are the commands i am running:

# build docker image
docker build -f path/to/Dockerfile -t ${TAG} . 

# deploy to ECR
docker push <docker image>

# package
pyflyte --pkgs workflows package -f --image <docker image>

# register 
flytectl register files --project <project> --domain <domain> --archive flyte-package.tgz --version <version>

# generate launchplan
flytectl get launchplan --project <project> --domain <domain> workflows.example.my_wf --latest --execFile execution_spec.yaml 

# execute workflow
flytectl create execution --project <project> --domain <domain> --execFile execution_spec.yaml

this is the

execution_spec.yaml

iamRoleARN: ""
inputs: {}
kubeServiceAcct: ""
targetDomain: ""
targetProject: ""
version: v6
workflow: workflows.example.my_wf

this are the files:

.
├── docker_build_and_tag.sh
├── Dockerfile
├── execution_spec.yaml
├── flyte.config
├── flyte-package.tgz
├── Makefile
├── README.md
├── requirements.test.txt
├── requirements.txt
└── workflows
    ├── example.py
    ├── __init__.py

All these files and folders are copied into /root in the docker image. Do anyone have some good way of debugging this? my initial thought was to start the docker container and then execute whatever Flyte is executing, but not sure what that is. Or if someone already now know what i am missing?

Sebastian

09/19/2022, 12:58 PM

Hello, does someone have an example of how to deploy workflows into prod/dev environments with CI/CD? I tried to split my resources into

flyte/workflows/{dev|prod}

but I can't find a way to tell flyte where to look for resources when serializing (!?) apart from the default

flyte/workflows

. What is the intended way to do this please?

Ketan (kumare3)

09/19/2022, 7:25 PM

@jeev infact pyflyte run uses Signed URLS to make it easy to run patches into your environment. Flyteadmin signs and returns the URL for a specific hash.

Matheus Moreno

09/19/2022, 8:33 PM

Hey, everyone. We're dealing with a weird bug here and we have no idea how to fix it. Basically, we have a task that's not finishing. We run a workflow with 8 tasks, and at the very last one it hangs. All of its code is being executed (a lot of `print`s from the very start all the way to the the

return

statement confirm that), but it hangs at the end and never actually finishes. We were able to reproduce it in our remote server and locally. On the remote server, none of the prints (or logs) are being shown on Stackdriver. What could be happening?

Robert Everson

09/19/2022, 9:03 PM

Hello, is there any method of Rate Limiting the number of resources that can get created on a k8s cluster (or just the number of tasks/nodes that start up in general)? We’re hitting an issue with crio on our clusters that is causing some of the nodes to fall over, but the nodes stay up in k8s so the resources don’t get returned to the cluster

Martin Hwasser

09/20/2022, 12:40 PM

I have a long running task that I want to cache. However, the underlying data may have changed in which case I don’t want to hit the cache. I can pass an md5 hash of the changed data to the task but that hash will only be used to change the task signature (input parameters) and will not actually be used by the task and this feels a bit ugly. Is there a cleaner way to achieve this?

Mike Carley

09/20/2022, 2:19 PM

How do you guys typically get your data sets? Is it mostly just queries from existing RDBMS?

Calvin Leather

09/20/2022, 4:53 PM

Hey @krishna Yerramsetty nice presentation... bioinformatics-oriented question... how have you been using FlyteFiles for indexed files? Do you have tabix indexes etc. on your bed of vcf files? This has been something my company has been thinking about/prototyping (using 2 flyte files is a bit annoying because the index doesn’t end up “next to” the file its indexing so in our shell tasks we have to cp the files + its index into a new temp folder, but what if we had something like FlyteDirectory that wrapped a file family with an expected number of files)

James Evers

09/21/2022, 4:54 PM

hi everyone, i'm having some trouble pulling images on a sandbox deployment. everything works fine with the base image (the one used when i started the deployment with

flytectl sandbox start --source .

), but when i try to build another image in the sandbox (

flytectl sandbox exec -- <docker build statement

), the pod fails to pull the image even though i can see that its been successfully build via

flytectl sandbox exec -- docker image ls

. anyone have any idea where i might be going wrong?

Title

James Evers

09/21/2022, 4:54 PM

hi everyone, i'm having some trouble pulling images on a sandbox deployment. everything works fine with the base image (the one used when i started the deployment with

flytectl sandbox start --source .

), but when i try to build another image in the sandbox (

flytectl sandbox exec -- <docker build statement

), the pod fails to pull the image even though i can see that its been successfully build via

flytectl sandbox exec -- docker image ls

. anyone have any idea where i might be going wrong?

Yee

09/21/2022, 5:51 PM

I feel like i’ve seen this recently - can you get the -o yaml dump for the pod

James Evers

09/21/2022, 5:51 PM

yep, just a sec

Yee

09/21/2022, 5:52 PM

and grep -i imagepullpolicy

that might not matter, just curious

James Evers

09/21/2022, 5:57 PM

imagePullPolicy: Always

Yee

09/21/2022, 5:58 PM

can you run

flytectl sandbox exec -- docker pull <image>

i’m hoping that fails.

James Evers

09/21/2022, 5:58 PM

ah it does fail!

Error response from daemon: repository <http://cr.flyte.org/flyteorg/tensorflow-kit|cr.flyte.org/flyteorg/tensorflow-kit> not found: name unknown: repository name not known to registry

Yee

09/21/2022, 5:59 PM

that’s the image you built?

James Evers

09/21/2022, 5:59 PM

yep, i also tried building it without the

<http://cr.flyte.org/flyteorg|cr.flyte.org/flyteorg>

prefix

Yee

09/21/2022, 6:00 PM

oh yeah you should probably just build locally without any registry

behind the scenes it assumed dockerhub, but yeah irrelevant

let me do some more digging around here.

James Evers

09/21/2022, 6:01 PM

just as a sanity check i'll try building it with the tag

tensorflow-kit:latest

actually, one thing that is popping up now that might be helpful is this error when i try to register the workflow:

executing:

pyflyte register --project flytesnacks --domain development --version v1.4 --image tensorflow-kit:latest src

throws:

AssertionError: Image Config with name tensorflow not found in the configuration

Yee

09/21/2022, 6:29 PM

did you start this with flytectl sandbox? or flytectl demo

James Evers

09/21/2022, 6:29 PM

flytectl sandbox start --source .

Yee

09/21/2022, 6:30 PM

oh it’s in the first message sorry

James Evers

09/21/2022, 6:30 PM

np!

i think i may not fully understand the relationship between some of the flyteadmin commands and the sandbox

Yee

09/21/2022, 6:31 PM

James Evers

09/21/2022, 6:31 PM

in this case, i have two images: 1. a base/default image 2. an image with some tensorflow stuff in it

Yee

09/21/2022, 6:31 PM

can you name your image something other than latest?

the version

:tst123

James Evers

09/21/2022, 6:32 PM

yep sure

ah interesting, let me try

while im waiting for the image to build -- how does

pyflyte register

use the image passed to that command?

for instance, if i have a deployment running on a default/base

image a

, but i want to register a workflow that will run on some specialized

image b

, i should register the workflow with

--image image-b

, right?

Yee

09/21/2022, 6:36 PM

currently it’s “fast register” only. so that means the image is set in the task’s image field. separately, your code is zipped up locally and uploaded to admin (which writes it to minio). at run time, your code is downloaded from minio unzipped and then task execution runs

James Evers

09/21/2022, 6:37 PM

oooo

Yee

09/21/2022, 6:38 PM

here

download this guy: https://blog.flyte.org/flyte-cheat-sheet the pdf

top right corner

multi-image workflows… that is how you can construct workflows with tasks that use two different images

though that doesn’t tell you the registration command

James Evers

09/21/2022, 6:39 PM

yeah i had been looking at that one

another thing im a little confused about here

Yee

09/21/2022, 6:40 PM

the registration command is

--image mydefaultimage:cafe123 --image other=nondefault:beef

James Evers

09/21/2022, 6:40 PM

ahhhhhh

okay that makes sense

so if my

flyte.config

file has

[images]
tensorflow = tensorflow-kit:v1

my register command should be:

pyflyte register --image default:v1 tensorflow=tensorflow-kit:tst123

Yee

09/21/2022, 6:42 PM

delete flyte.config

well you can delete it at least

if you don’t delete it then you need to specify it on the command line.

if you don’t specify it on the command line then

pyflyte --config xyz …

James Evers

09/21/2022, 6:44 PM

okay i just got the workflow to register with your suggestion

thanks for the help!!

View count: 2