flyte-org #ask-the-community

Mehtab Mehdi

10/30/2022, 9:22 PM

Hi Team, i am trying to integrate the databricks with flyte.. Data is coming and UI is coming when i am using task and plugin code in single file. while i am creating ECR image and trying to integrate than it is connecting on local system while i am running pyflyte --remote --image<ECR image name> myfly.py wf it is throwing the following Error

[1/1] currentAttempt done. Last Error: UNKNOWN::Outputs not generated by task execution

Can anyone help me to solve this issue.

SeungTaeKim

10/31/2022, 4:25 AM

Hi, flyte team! I would like to ask

Promise object

of task's return values.

Copy code

from flytekit import workflow, task, dynamic, conditional
from flytekit.core.promise import Promise

@task
def t1() -> bool:
    return True


@task
def t2() -> bool:
    return False


@workflow
def wf() -> bool:
    test = t1()
    print(test)
    return t2()


if __name__ == "__main__":
    print(wf())

when I run this sample codes through

pyflyte

, a task returns

Promise object

, and literal or objects I expect are wrapped in

Promise

. So, here is my question, how I unwrap this promise object? I would like to use the expected value from the function. Thank you!

Tarmily Wen

10/31/2022, 3:48 PM

Hi, I am struggling to inject secrets at execution time into tasks and workflows via the flytekit api. This is my task taken from the mnist training example

Copy code

@task(
    retries=2,
    cache=True,
    cache_version="1.0",
    requests=Resources(gpu=gpu, mem=mem, storage=storage),
    limits=Resources(gpu=gpu, mem=mem, storage=storage),
    secret_requests=[Secret(group="wandb-secrets", key="API_KEY")],
)
def pytorch_mnist_task(hp: Hyperparameters) -> TrainingOutputs:
    secrets = current_context().secrets
    wandb_api_key = secrets.get(group="wandb-secrets", key="API_KEY")

And I would like to call a remote execution and inject the secret at the same time like this:

Copy code

current_config = Config.auto()

remote = FlyteRemote(config=current_config)
flyte_workflow = remote.fetch_workflow(name=workflow_name, version="v1", project="flytesnacks", domain="development",)
workflow_execution = remote.execute(entity=flyte_workflow, inputs={"hp": Hyperparameters(epochs=2, batch_size=128)}, project="flytesnacks", domain="development",)

But after checking out the docs, I am confused about where to put the secret and the format it should be in. I noticed the SecretsConfig, but it isn't clear to me about how to utilize that here since it appears to require a file. Can I not specific an execution time environment variable?

Laura Lin

10/31/2022, 4:11 PM

Is there support for tasks with inputs that use Unions?

Union[str, flytekit.types.directory.FlyteDirectory]

. Trying to create a task that could take in a str or a flytedirectory but I get

Copy code

raise ValueError(f"Expected a directory. {source_path} is not a directory")
ValueError: Expected a directory. X is not a directory

AssertionError: Failed to Bind variable input for function

Hadi

11/01/2022, 10:31 AM

Is it possible to select the gpu or machine type when creating a task?

Yash Panchwatkar

11/01/2022, 11:38 AM

Hello community, how to use image from AWS ECR while running pyflyte register -i command .. by default it's pointing to the public docker hub .. is there any configuration given for the same.. i have tried imagePullSecret by attaching to the default service account still it's not working so...

Zhiyi Li

11/01/2022, 12:57 PM

Hey folks, I am trying to set up kafka to gather platform events for external consuming. I am wondering whether all the IDL messages would be sent out via the same topic as it writes https://github.com/flyteorg/flyteadmin/blob/46598814b2bcb277762f96a666b2c4649cbdbe8a/pkg/async/cloudevent/factory.go#L67 also I’m wondering what’s the purpose of this? (is this used for notification or external event?) https://github.com/flyteorg/flyteadmin/blob/bca622830ff563984bd3270708ec8b6a5ff83945/pkg/async/notifications/factory.go#L183

Laura Lin

11/01/2022, 6:51 PM

How do you change imagePullPolicy for existing deployments? By default, its ifnotpresent, but I would like to change it to always for all of my potential docker images/containers. Do I have to use https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#alwayspullimages?

Yash Panchwatkar

11/02/2022, 11:40 AM

Hi @Kevin Su when i try to run my workflow I get the following error. I am using private ecr image .

Copy code

[1/1] currentAttempt done. Last Error: USER::containers with unready status: [apv828cglh4qs76ftwgf-n0-0]|Back-off pulling image "<http://x.y.z.dkr.ecr.us-west-2.amazonaws.com/flyte_test:adiwala1|x.y.z.dkr.ecr.us-west-2.amazonaws.com/flyte_test:adiwala1>"

To make sure that imagePullSecret are present in service account I ran the following commands

kubectl get pod nginx -o=jsonpath='{.spec.imagePullSecrets[0].name}{"\n"}'

The output was

reg-ecr-cred

Can you please help me out here what am I missing?

Sampath Vaddadi

11/02/2022, 1:37 PM

Hi All, can anyone please tell me how can I control the type of instance that I can get as part of spark cluster when running a spark job through flyte? Is there such a provision? I see that we can specify spark_conf, does that take care of what type of instances are brought into kubernetes cluster for spark job? I have my kubernetes cluster on aws In databricks, I can control what type of ec2 instances by master and worker nodes are. I'm looking for similar feature in flyte

seunggs

11/02/2022, 6:00 PM

Hi, is it possible to send Platform Events to something like external Kafka cluster or is AWS SQS/SNS the only available option right now?

Laura Lin

11/02/2022, 9:36 PM

Are help strings for workflow inputs in the console UI still supported? I know it used to be possible (I think maybe through imperative workflows or something) but can't seem to find it in the docs.

Dennis O'Brien

11/02/2022, 10:09 PM

I'm just getting to adding a schedule to my first Flyte workflow and I have a few questions. My workflow takes two arguments, a datetime and an int.

Copy code

@workflow
def cohort_ltv_inference(inference_date: datetime, n_days_horizon: int) -> bool:
    ...

I had created a schedule in my launch plan but failed to provide

kickoff_time_input_arg

default_inputs

Copy code

launch_plan = LaunchPlan.get_or_create(
    name="pir_cohort_ltv_inference",
    workflow=cohort_ltv_inference,
    schedule=CronSchedule(
        schedule="45 17 * * *",  # At 17:45 daily.
    ),
)

In the workflow web ui, I see the schedule displayed:

Copy code

Schedules
At 05:45 PM

But when I noticed that no scheduled run was actually happening, I looked through the scheduler logs and saw messages like this:

Copy code

rpc error: code = InvalidArgument desc = expected_inputs inference_date missing

Ideally the end user would be alerted about this earlier. I saw a related issue on github here, and a PR here. I haven't worked in Go, but I think the function

validateLaunchSpec

doesn't handle the case of no

default_inputs

and no

fixed_inputs

passed to the launch plan

get_or_create

. In general I'm missing some of the familiar scheduler UI I am used to from my experience with Airflow. I know these are very different projects, but for almost all my use cases, I'll eventually be running the ML pipeline on a schedule.

Kamakshi Muthukrishnan

11/03/2022, 5:18 AM

Hi Team, I am getting module 'xgboost' has no attribute 'rabit' while using modin. Any specific version of xgboost/modin/ package will resolve this issues???

Kamakshi Muthukrishnan

11/03/2022, 5:18 AM

Copy code

ERROR worker.py:399 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::ModinXGBoostActor.train() (pid=29796, ip=172.16.54.211, repr=<modin.experimental.xgboost.xgboost_ray.ModinXGBoostActor object at 0x7fbef05837f0>)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.8/site-packages/modin/experimental/xgboost/xgboost_ray.py", line 166, in train
    with RabitContext(self._rank, rabit_args):
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.8/site-packages/modin/experimental/xgboost/utils.py", line 101, in __enter__
    xgb.rabit.init(self.args)
AttributeError: module 'xgboost' has no attribute 'rabit'

Kamakshi Muthukrishnan

11/03/2022, 6:15 AM

Yes.. I just added rabit as a seperate utils and fixed the above error

Willem Gillis

11/03/2022, 12:23 PM

Hi, our team is trying to use fast registration using

pyflyte register

but hitting a wall. The registration passes and it's worklows and launchplans even shows up in Flyte Console however when executing it we get error:

ModuleNotFoundError: No module named 'flyte'

When looking at the resulted generated file on the S3 minio we find that it is completely empty. Our full command is:

pyflyte --config ~/.flyte/cloud-config.yaml register flyte/workflows -p {{.PROJECT}} --domain {{.DOMAIN}} -v {{add .DOCKER_VERSION 1}} -i "{{.CONTAINER_REGISTRY}}/{{.PROJECT}}:{{.DOCKER_VERSION}}"

Folder structure see image Note that using the

pyflyte package

and

flytectl register files

in the same project does work as expected. Are we missing something?

Felix Ruess

11/03/2022, 2:37 PM

Is there a way to create ContainerTask with a string input and use that input in the command without writing cumbersome wrapper scripts which need to be part of the image? https://docs.flyte.org/projects/cookbook/en/stable/auto/core/containerization/raw_container.html#container-tasks

Edgar Trujillo

11/03/2022, 3:37 PM

Is anyone else seeing

FlytePropeller &  user defined pod executions

logs having for the most part static timestamps? We're running EKS with FluentBit deployed that pushes logs to Cloudwatch, but seeing the following

Felix Ruess

11/03/2022, 5:26 PM

Is there an alternative way to specify

FLYTE_AWS_SECRET_ACCESS_KEY

not as env var so that this secret is not visible to anyone who can see get the pod details?

varsha Parthasarathy

11/03/2022, 7:10 PM

Hi team, Is this expected? or are there workarounds?

Copy code

flytekit.core.type_engine.RestrictedTypeError: Transformer for type <class 'tuple'> is restricted currently

yujinlee

11/04/2022, 4:15 AM

Hello, can I get some advice regarding to contribution in flyte? I want to setup timezone as Asia/Seoul, so I'm considering to add feature to set timezone. I was thinking to set it up in

flyte_config.yaml

as follows:

Copy code

admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint:
  authType: Pkce
  insecure: false
timezone: ??

Any suggestion regarding with this feature? I also wonder which git repository to start in FlyteOrg. Thank You!

Sanjay Chouhan

11/04/2022, 7:22 AM

Can not run helm upgrade. Getting error 'Error: failed to download "flyteorg/flyte-core"'

Copy code

(base) sanjaychouhan@prod-ml-jenkins2:~$ helm upgrade -n flyte -f values-eks.yaml --create-namespace flyte flyteorg/flyte-core
Error: failed to download "flyteorg/flyte-core"
(base) sanjaychouhan@prod-ml-jenkins2:~$
(base) sanjaychouhan@prod-ml-jenkins2:~$ helm repo add flyteorg <https://flyteorg.github.io/flyte>
Error: repository name (flyteorg) already exists, please specify a different name

Felix Ruess

11/04/2022, 11:49 AM

I want to inject secrets as env vars into my raw container and thought I can do it via secret_requests: https://github.com/flyteorg/flytekit/pull/1307 But I was surprised that the injected env var is named prefix+group+key, the prefix I can set to empty, the group obviously not. This essentially forces me to create multiple k8s secrets (secret name becomes group) for each env var that I need injected that has a different prefix. So it seems it was not intended to be used this way (that the key becomes the env var name)? Point in case I need to inject

AWS_ACCESS_KEY_ID

and

AWS_SECRET_ACCESS_KEY

and thought I can get the correct ones by specifying the group (k8s secret name). But how it currently works I would need to use

AWS

as group name and hence can have only one k8s secret with that name....

Padma Priya M

11/04/2022, 1:10 PM

Hi, I am getting error while executing ray with optuna in flyte.

Felix Ruess

11/04/2022, 3:51 PM

Hi, I ran into the default memory limit:

Requested MEMORY default [2Gi] is greater than current limit set in the platform configuration [1Gi]. Please contact Flyte Admins to change these limits or consult the configuration

And I already increased the task resource limits but I still get the same error. Any pointers?

Louis DiNatale

11/04/2022, 4:20 PM

Hey Flyte team, happy Friday. When it comes to the maxParallelism config where should I be looking for errors as i start to increase this number?

Jay Ganbat

11/04/2022, 9:22 PM

Hi flyte folks, was there any significant performance change in Flyte console? we recently swtiched to flyte 1.2 milestone version and its correspodning flyte backend. Unfortunately now its taking a lot of time display the workflow tasks as opposed to what previously and from the browser log `

Copy code

vendor-a9fbc36b.js:2          GET https://.../fizlxcwy-n4-0-dn1-0/n1?limit=10000 net::ERR_INSUFFICIENT_RESOURCES

it might have been showing previously as well but never seem to be noticable. it takes solid min or 2 to even show what tasks were executed. previous version was flyteconsole

1.1.6

, propeller

1.1.15

and admin on

1.1.29

any idea on what have changed

Felix Ruess

11/05/2022, 5:19 PM

What endpoints do I need to scrape to get metrics? And what ingress rules might I need to set up for this? It doesn't say anything in the docs: https://docs.flyte.org/en/latest/deployment/cluster_config/monitoring.html

sapcode

11/06/2022, 7:50 AM

Dear flyte team, we are about to POC an on-premise full(no sandbox) installation in a local kubernetes cluster. However in the documentation we cant find the sequence of which helm values yamls are supposed to be used when the target is not AWS and not GCP. It would be very helpful to have a section for non cloud installation setup without sandbox. 1) Can someone post an easy installation script for minikube or bare metal kubernetes, without external iDP provider? 2) Is there a build in user management, where we can create users ? 3) is there a default user and password? Some blogs are mentioning foobar. 4) it would be great to have 1 installation yaml for keycloak and the setup values documented which needs to be changed to get it running. Thank you in advance!!!