flyte-org #ask-the-community

ask-the-community

Katrina P

04/21/2022, 4:33 AM

Hi! I tried joining the office hours tonight but had some connection issues unfortunately 😞. I had reached out previously here . I've tried running the sandbox with the k8s operator for spark instructions again with the most recent release, and I'm still running into the same issues. I've followed the instructions here and tried running the most recent release registered via:

flytectl register files --config ~/.flyte/config.yaml <https://github.com/flyteorg/flytesnacks/releases/download/v0.3.70/snacks-cookbook-integrations-kubernetes-k8s_spark.tar.gz> --archive -p flytesnacks -d development --version latest

(I can run the examples locally)

++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/bash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/bash ']'
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sed 's/[^=]*=\(.*\)/\1/g'
+ sort -t_ -k4 -n
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
Non-spark-on-k8s command provided, proceeding in pass-through mode...
+ '[' -n '' ']'
+ '[' '' == 2 ']'
+ '[' '' == 3 ']'
+ '[' -n '' ']'
+ '[' -z ']'
+ case "$1" in
+ echo 'Non-spark-on-k8s command provided, proceeding in pass-through mode...'
+ CMD=("$@")
+ exec /usr/bin/tini -s -- pyflyte-execute --inputs <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-kdu1tzd712/n0/data/inputs.pb> --output-prefix <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-kdu1tzd712/n0/data/0> --raw-output-data-prefix <s3://my-s3-bucket/72/kdu1tzd712-n0-0> --checkpoint-path <s3://my-s3-bucket/72/kdu1tzd712-n0-0/_flytecheckpoints> --prev-checkpoint '""' --resolver flytekit.core.python_auto_container.default_task_resolver -- task-module k8s_spark.dataframe_passing task-name create_spark_df
22/04/21 04:27:54 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
{"asctime": "2022-04-21 04:28:13,716", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! Begin System Error Captured by Flyte !!"}
{"asctime": "2022-04-21 04:28:13,717", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "Traceback (most recent call last):\n\n      File \"/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py\", line 165, in system_entry_point\n        return wrapped(*args, **kwargs)\n      File \"/opt/venv/lib/python3.8/site-packages/flytekit/core/base_task.py\", line 463, in dispatch_execute\n        new_user_params = self.pre_execute(ctx.user_space_params)\n      File \"/opt/venv/lib/python3.8/site-packages/flytekitplugins/spark/task.py\", line 124, in pre_execute\n        self.sess = sess_builder.getOrCreate()\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/sql/session.py\", line 228, in getOrCreate\n        sc = SparkContext.getOrCreate(sparkConf)\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 392, in getOrCreate\n        SparkContext(conf=conf or SparkConf())\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 144, in __init__\n        SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 339, in _ensure_initialized\n        SparkContext._gateway = gateway or launch_gateway(conf)\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/java_gateway.py\", line 108, in launch_gateway\n        raise RuntimeError(\"Java gateway process exited before sending its port number\")\n\nMessage:\n\n    Java gateway process exited before sending its port number\n\nSYSTEM ERROR! Contact platform administrators."}
{"asctime": "2022-04-21 04:28:13,717", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! End Error Captured by Flyte !!"}

Hampus Rosvall

04/21/2022, 11:33 AM

Hey, I am having troubles setting up Flyte as internal facing and use port-forwarding to register workflows to the backend? I am deploying the Ingress as internal facing. But I am getting an error when I hit

localhost:port

in the browser after port-forwarding to the console. However, I am able to query the host deployed by the Ingress inside the VPC and get information about e.g., projects.

Failed to fetch data
invalid wire type 4 at offset 1

Han

04/22/2022, 10:17 PM

Quick question. we have people working on a shared server. I have written a workflow that runs locally under my env. But when my colleague tries to run the same workflow on the server, he got the following error

$ python tasks/run_workflow.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/pathlib.py", line 1284, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/flyte/20220422_220444/sandbox/local_flytekit'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/pathlib.py", line 1284, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/flyte/20220422_220444/sandbox'

During handling of the above exception, another exception occurred:
...
...
PermissionError: [Errno 13] Permission denied: '/tmp/flyte/20220422_220444'

How do I ensure reproducibility for different local environments?

Erbene Castro

04/23/2022, 6:57 PM

Hi, how do I change the demo Flyte admin task resources limits? I used to change those in the kubernetes configmaps but this newer version doesn't have the same configmaps and I can't find the cluster and flyte admin configurations.

Rahul Mehta

04/24/2022, 5:16 PM

Hi, quick question -- where should the s3 credentials be set in values-eks.yaml? I see the references to `rdsHost`/`dbPassword` in the chart values (& will use an external secret instead of hard-coding them in there), but couldn't find the equivalent for S3. Also, we're looking to deploy flyte into a kops managed cluster (not quite on EKS yet). Would it be better to use values-eks.yaml or values.yaml as a starting point?

Rahul Mehta

04/24/2022, 11:00 PM

One more question that may be a bit naive: if there is a workflow in a project

test

, domain

development

& a version

v1

, how would I "promote" that workflow to

staging

in the same project w/ the same version? With what I've seen in the flytekit API, it only seems possible to register/create a workflow based on what's in the source code (ie. the decorated workflow function). Is it possible to "copy"/promote a workflow to a different domain only w/ the API? I also definitely could be misunderstanding how the domains abstraction is supposed to be applied to enable promotion/a proper CI/CD approach to pipelines

Hampus Rosvall

04/25/2022, 11:29 AM

Hey, what is the preferred way of providing project specific IAM Roles to a workflow? I am trying to run a workflow using an IAM role with permissions suitable for only that project. In my launch plan I enter some role e.g., as in the picture below but it doesn’t seem to be picked up when I describe the Pod. Does your default role e.g.,

eks-flyte-user-rule

assume the IAM Role you provide in the launch plan, or is it assumed by some service account that gets created?

Environment:
      ...
      AWS_ROLE_ARN:                       arn:aws:iam::<account_id>:role/eks-flyte-user-role

Chase Grisham

04/25/2022, 8:23 PM

Does anyone know if the

--pre

flag applies to the sandbox the way to does to demo? ie: does

flytectl sandbox start --source . --pre

download the same flyte version as

flytectl demo start --pre

Chase Grisham

04/26/2022, 3:53 PM

Having trouble running the k8s_spark.<http://dataframe_passing.my|dataframe_passing.my>_smart_structured_dataset
example. I've been able to run the other spark example

pyspark_pi

. I've set up the K8s Operator, built the Docker Image based on the Dockerfile in the

cookbook/integrations/kubernetes/k8s

folder. Getting this error about s3 :

[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):

      File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/opt/venv/lib/python3.8/site-packages/flytekit/core/base_task.py", line 527, in dispatch_execute
        raise TypeError(

Message:

    Failed to convert return value for var o0 for function k8s_spark.dataframe_passing.create_spark_df with error <class 'py4j.protocol.Py4JJavaError'>: An error occurred while calling o41.parquet.
: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3443)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
	at org.apache.hadoop.fs.FileSystem.get(FiInternal(DataFrameWriter.scala:355)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
	at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:781)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:748)


SYSTEM ERROR! Contact platform administrators.

Han

04/26/2022, 8:24 PM

Couple of problems I ran into while I try to go through this env setup tutorial. 1. I am seeing

<http://googletagmanager.com|googletagmanager.com>

and

<http://gstatic.com|gstatic.com>

for flytectl local sandbox dashboard links. Are they necessary? 2. What's the purpose of

pyflyte

? 3. cont'd

Hafsa Junaid

04/27/2022, 3:04 AM

Hey! No idea why pods are not successfully deployed, Following doc as it is https://docs.flyte.org/en/latest/deployment/gcp/manual.html#deployment-gcp-manual

Slackbot

04/29/2022, 2:24 AM

This message was deleted.

Hafsa Junaid

04/29/2022, 3:03 AM

Hey Team!!! Hello @Haytham Abuelfutuh As per the https://docs.flyte.org/en/latest/deployment/gcp/manual.html#deployment-gcp-manual docs command, the connection with psql is verified. But these logs kubectl logs -n flyte svc/flyteadmin -c run-migrations shows that connection ain't built for flyteadmin. The pods are failing and UI is not accessible. Any guide?

Stefan Avesand

05/04/2022, 12:32 PM

Good morning team! Trying to use the new StructuredDataset type to read data from BQ but can’t figure it out.

sd = StructuredDataset(uri='<bq://sp-one-model.quarterly>_forecast_2022F1.premium_revenue_tab_input_vat')
sd.open(pd.DataFrame).all()

AttributeError: 'NoneType' object has no attribute 'uri'

seunggs

05/05/2022, 7:31 AM

Hi! I have a quick question - I’m trying to create a flyte task from a web UI. Are there any tutorials or examples I can reference to do this? Any guidance on this would be appreciated!

Robin Kahlow

05/06/2022, 12:32 PM

Hey! I was following Flyte's auth setup with Google up to (not including) https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#oauth2-authorization-server. In Google Cloud Console's OAuth consent screen settings, the user type is set to external, the publishing status is testing and some users are added as test users. Am I correct in assuming only those test users will be able to log in on Flyte's console and not any other users? Thank you!

seunggs

05/06/2022, 9:13 PM

Hi, is there some documentation on running a part of the workflow? If there are 5 tasks in a workflow for example and I’d like to run the first 3 tasks to view the output of task #3, how could I do that? Also, where are the intermediate outputs for each task saved? I’m trying to visualize the intermediate task outputs and not sure if it’ll be performant enough to be near real-time for large datasets?

Michael Cheng Jan Kao

05/11/2022, 1:15 AM

hi, is there a recommended way of updating the resources after it has been set? I am currently doing this, wondering if this is acceptable and have any unintended side effects.

@task(requests=Resources(cpu="1", mem="4Gi"), limits=Resources(cpu="2", mem="32Gi"))
def hello_world():
    print('hello world')

hello_world._resources = ResourceSpec(requests=Resources(cpu='1', mem='16Gi', gpu=None, storage=None, ephemeral_storage=None), limits=Resources(cpu='2', mem='32Gi', gpu=None, storage=None, ephemeral_storage=None))

Robin Kahlow

05/11/2022, 12:31 PM

Hello again, is there a simple way to install specified Python packages before running a task without creating a new docker image?

seunggs

05/11/2022, 11:41 PM

Hi, is there a way to use a different ingress service? I’m using

emissary-ingress

and I was wondering if you had a reference implementation for that instead of using Amazon Loadbalancer Ingress Controller?

yujinlee

05/13/2022, 2:16 AM

Hi, may I ask a question here? Is there a way to get private docker image from Docker registry to register task? I wonder how to login docker registry in flyte to download private image.

Robin Kahlow

05/13/2022, 2:40 PM

I want to change the log url displayed in flyte console to where our logs are stored, so I added the entry below to the values. The url does change but it seems like podName is always empty so the url is https://my.papertrailapp.com/systems//events, is there something else I need to do to make the podName variable work? I just looked at the defaults which uses podName too https://docs.flyte.org/en/stable/deployment/cluster_config/flytepropeller_config.html#logs-logs-logconfig

task_logs:
    plugins:
      logs:
        kubernetes-enabled: true
        kubernetes-template-uri: <https://my.papertrailapp.com/systems/{{> .podName }}/events

Sebastia Agramunt

05/17/2022, 7:14 PM

Hi, newbie question… Do I really need kubernetes to run Flyte?. I need to run ML workflows on docker images for a small (but growing) project.

Attila Nagy

05/18/2022, 7:51 AM

Hi, Coming from https://docs.flyte.org/en/latest/deployment/aws/opta.html. I would like to customize the resources made by Opta (for example the parameters of AWS ASG worker nodes, like EC2 key pair name, auto public IP, or VPC ACLs), but can't see how to do that and where are they defined. Any ideas/pointers?

yujinlee

05/20/2022, 4:02 AM

Hi, How do I delete a registered task or workflow in a specific project-domain? First, I checked the CLI help. If you look at the

flytectl --help

, it says that you can delete tasks,worfklows.

delete      Terminates/deletes various Flyte resources such as tasks, workflows, launch plans, executions, and projects.

But in the available command of

delete

, I don't see an option to actually delete tasks or workflows.

Usage:
  flytectl delete [command]

Available Commands:
  cluster-resource-attribute Deletes matchable resources of cluster attributes.
  execution                  Terminates/deletes execution resources.
  execution-cluster-label    Deletes matchable resources of execution cluster label.
  execution-queue-attribute  Deletes matchable resources of execution queue attributes.
  plugin-override            Deletes matchable resources of plugin overrides.
  task-resource-attribute    Deletes matchable resources of task attributes.
  workflow-execution-config  Deletes matchable resources of workflow execution config.

HaoboGu

05/20/2022, 6:45 AM

Hi, I'm trying to test my workflow on local sandbox, but after I executed `flytectl sandbox start --source .``, I got the following message in container:

HaoboGu

05/20/2022, 6:45 AM

Release "flyte" does not exist. Installing it now.

W0520 06:19:59.185843     736 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob

W0520 06:21:41.614777     736 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob

NAME: flyte

LAST DEPLOYED: Fri May 20 06:19:57 2022

NAMESPACE: flyte

STATUS: deployed

REVISION: 1

TEST SUITE: None

Waiting for Flyte to become ready...

Terminated

Timed out while waiting for the flyteadmin rollout to be created

HaoboGu

05/20/2022, 6:46 AM

Then the container just quits.. does anybody know what's wrong with this sandbox? Thanks!

Katrina P

05/20/2022, 6:48 PM

I'm still working on figuring out how to deploy Flyte into our company's k8s infrastructure. I noticed this in the docs https://github.com/flyteorg/flyte/tree/v1.0.0/kustomize/overlays/eks#how-to-build-your-overlay -- anyone here have experience doing this? I'm having to install Flyte using kustomize into a shared services cluster and trying to figure out the smartest way to do this (pretty new to kustomize!)

Ruksana Kabealo

05/20/2022, 7:37 PM

Hi everyone! I'm trying to follow the "Deploying Workflows - Registration" doc here: https://docs.flyte.org/projects/cookbook/en/stable/auto/deployment/deploying_workflows.html I've built the docker image that contains my code. I'm trying to serialize it with pyflyte. I'm running the following command:

pyflyte -c /root/.flyte/config.yaml --pkgs delaine serialize --in-container-config-path /root/config.yaml --local-source-root ./ --image delaieine:2 workflows -f _pb_output/

And getting the following error traceback:

Traceback (most recent call last):
 File "/root/delaieine/bin/pyflyte", line 8, in <module>
  sys.exit(main())
 File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
  return self.main(*args, **kwargs)
 File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1055, in main
  rv = self.invoke(ctx)
 File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
 File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
 File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
  return ctx.invoke(self.callback, **ctx.params)
 File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 760, in invoke
  return __callback(*args, **kwargs)
 File "/root/delaieine/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
  return f(get_current_context(), *args, **kwargs)
 File "/root/delaieine/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/serialize.py", line 139, in workflows
  serialize_all(
 File "/root/delaieine/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
  return wrapped(*args, **kwargs)
 File "/root/delaieine/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/serialize.py", line 60, in serialize_all
  image_config=ImageConfig.auto(config_file, img_name=image),
 File "/root/delaieine/lib/python3.9/site-packages/flytekit/configuration/__init__.py", line 250, in auto
  for k, v in _internal.Images.get_specified_images(config_file).items()
 File "/root/delaieine/lib/python3.9/site-packages/flytekit/configuration/internal.py", line 25, in get_specified_images
  image_names = cfg.legacy_config.options("images")
AttributeError: 'NoneType' object has no attribute 'options'

NoneType leads me to believe it's not liking either the config or the docker image. I've checked the image/tag spelling. That's fine. The config exists, the path is right, and it looks like this:

admin:
 # For GRPC endpoints you might want to use dns:///flyte.myexample.com
 endpoint: localhost:30081
 authType: Pkce
 insecure: true

I'm using flytekit=1.0.2 and flytectl=0.5.25 I'm not sure how to troubleshoot this. Any help would be much appreciated!

Title

Ruksana Kabealo

05/20/2022, 7:37 PM

pyflyte -c /root/.flyte/config.yaml --pkgs delaine serialize --in-container-config-path /root/config.yaml --local-source-root ./ --image delaieine:2 workflows -f _pb_output/

And getting the following error traceback:

Traceback (most recent call last):
 File "/root/delaieine/bin/pyflyte", line 8, in <module>
  sys.exit(main())
 File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
  return self.main(*args, **kwargs)
 File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1055, in main
  rv = self.invoke(ctx)
 File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
 File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
 File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
  return ctx.invoke(self.callback, **ctx.params)
 File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 760, in invoke
  return __callback(*args, **kwargs)
 File "/root/delaieine/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
  return f(get_current_context(), *args, **kwargs)
 File "/root/delaieine/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/serialize.py", line 139, in workflows
  serialize_all(
 File "/root/delaieine/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
  return wrapped(*args, **kwargs)
 File "/root/delaieine/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/serialize.py", line 60, in serialize_all
  image_config=ImageConfig.auto(config_file, img_name=image),
 File "/root/delaieine/lib/python3.9/site-packages/flytekit/configuration/__init__.py", line 250, in auto
  for k, v in _internal.Images.get_specified_images(config_file).items()
 File "/root/delaieine/lib/python3.9/site-packages/flytekit/configuration/internal.py", line 25, in get_specified_images
  image_names = cfg.legacy_config.options("images")
AttributeError: 'NoneType' object has no attribute 'options'

NoneType leads me to believe it's not liking either the config or the docker image. I've checked the image/tag spelling. That's fine. The config exists, the path is right, and it looks like this:

admin:
 # For GRPC endpoints you might want to use dns:///flyte.myexample.com
 endpoint: localhost:30081
 authType: Pkce
 insecure: true

I'm using flytekit=1.0.2 and flytectl=0.5.25 I'm not sure how to troubleshoot this. Any help would be much appreciated!

Yee

05/20/2022, 8:14 PM

very sorry

this is a bug in flytekit’s attempt to handle both the .ini style of config file that is Python’s default, as well as the yaml style config that is shared with the

flytectl

tool.

some configuration entries though don’t make sense to have in the yaml file - image is one of them. the reason is that this is something that is set at compilation time (when your code is serialized into protobuf)

so we should just exclude that here.

https://github.com/flyteorg/flytekit/pull/1016

cc @Eduardo Apolinario (eapolinario)

Ruksana Kabealo

05/20/2022, 9:20 PM

Just to be clear, by "excluding that here" you mean not giving pyflyte a config at all correct? So no -c and no --in-container-config-path?

Yee

05/20/2022, 9:46 PM

oh no sorry

gimme a bit

https://pypi.org/project/flytekit/1.0.3/

okay - patched into this version. would you mind bumping the version and trying again please? sorry for the bug.

👍 1

Ruksana Kabealo

05/23/2022, 1:59 PM

@Yee Will do! Thank you!

@Yee This worked! Thanks so much!

❤️ 1

View count: 6