Katrina P
04/21/2022, 4:33 AMflytectl register files --config ~/.flyte/config.yaml <https://github.com/flyteorg/flytesnacks/releases/download/v0.3.70/snacks-cookbook-integrations-kubernetes-k8s_spark.tar.gz> --archive -p flytesnacks -d development --version latest
(I can run the examples locally)
++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/bash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/bash ']'
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sed 's/[^=]*=\(.*\)/\1/g'
+ sort -t_ -k4 -n
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
Non-spark-on-k8s command provided, proceeding in pass-through mode...
+ '[' -n '' ']'
+ '[' '' == 2 ']'
+ '[' '' == 3 ']'
+ '[' -n '' ']'
+ '[' -z ']'
+ case "$1" in
+ echo 'Non-spark-on-k8s command provided, proceeding in pass-through mode...'
+ CMD=("$@")
+ exec /usr/bin/tini -s -- pyflyte-execute --inputs <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-kdu1tzd712/n0/data/inputs.pb> --output-prefix <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-kdu1tzd712/n0/data/0> --raw-output-data-prefix <s3://my-s3-bucket/72/kdu1tzd712-n0-0> --checkpoint-path <s3://my-s3-bucket/72/kdu1tzd712-n0-0/_flytecheckpoints> --prev-checkpoint '""' --resolver flytekit.core.python_auto_container.default_task_resolver -- task-module k8s_spark.dataframe_passing task-name create_spark_df
22/04/21 04:27:54 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
{"asctime": "2022-04-21 04:28:13,716", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! Begin System Error Captured by Flyte !!"}
{"asctime": "2022-04-21 04:28:13,717", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "Traceback (most recent call last):\n\n File \"/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py\", line 165, in system_entry_point\n return wrapped(*args, **kwargs)\n File \"/opt/venv/lib/python3.8/site-packages/flytekit/core/base_task.py\", line 463, in dispatch_execute\n new_user_params = self.pre_execute(ctx.user_space_params)\n File \"/opt/venv/lib/python3.8/site-packages/flytekitplugins/spark/task.py\", line 124, in pre_execute\n self.sess = sess_builder.getOrCreate()\n File \"/opt/venv/lib/python3.8/site-packages/pyspark/sql/session.py\", line 228, in getOrCreate\n sc = SparkContext.getOrCreate(sparkConf)\n File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 392, in getOrCreate\n SparkContext(conf=conf or SparkConf())\n File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 144, in __init__\n SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)\n File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 339, in _ensure_initialized\n SparkContext._gateway = gateway or launch_gateway(conf)\n File \"/opt/venv/lib/python3.8/site-packages/pyspark/java_gateway.py\", line 108, in launch_gateway\n raise RuntimeError(\"Java gateway process exited before sending its port number\")\n\nMessage:\n\n Java gateway process exited before sending its port number\n\nSYSTEM ERROR! Contact platform administrators."}
{"asctime": "2022-04-21 04:28:13,717", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! End Error Captured by Flyte !!"}
Hampus Rosvall
04/21/2022, 11:33 AMlocalhost:port
in the browser after port-forwarding to the console. However, I am able to query the host deployed by the Ingress inside the VPC and get information about e.g., projects.
Failed to fetch data
invalid wire type 4 at offset 1
Han
04/22/2022, 10:17 PM$ python tasks/run_workflow.py
Traceback (most recent call last):
File "/usr/local/lib/python3.8/pathlib.py", line 1284, in mkdir
self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/flyte/20220422_220444/sandbox/local_flytekit'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/pathlib.py", line 1284, in mkdir
self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/flyte/20220422_220444/sandbox'
During handling of the above exception, another exception occurred:
...
...
PermissionError: [Errno 13] Permission denied: '/tmp/flyte/20220422_220444'
How do I ensure reproducibility for different local environments?Erbene Castro
04/23/2022, 6:57 PMRahul Mehta
04/24/2022, 5:16 PMRahul Mehta
04/24/2022, 11:00 PMtest
, domain development
& a version v1
, how would I "promote" that workflow to staging
in the same project w/ the same version? With what I've seen in the flytekit API, it only seems possible to register/create a workflow based on what's in the source code (ie. the decorated workflow function). Is it possible to "copy"/promote a workflow to a different domain only w/ the API? I also definitely could be misunderstanding how the domains abstraction is supposed to be applied to enable promotion/a proper CI/CD approach to pipelinesHampus Rosvall
04/25/2022, 11:29 AMeks-flyte-user-rule
assume the IAM Role you provide in the launch plan, or is it assumed by some service account that gets created?
Environment:
...
AWS_ROLE_ARN: arn:aws:iam::<account_id>:role/eks-flyte-user-role
Chase Grisham
04/25/2022, 8:23 PM--pre
flag applies to the sandbox the way to does to demo? ie: does flytectl sandbox start --source . --pre
download the same flyte version as flytectl demo start --pre
?Chase Grisham
04/26/2022, 3:53 PMk8s_spark.<http://dataframe_passing.my|dataframe_passing.my>_smart_structured_dataset
example. I've been able to run the other spark example pyspark_pi
. I've set up the K8s Operator, built the Docker Image based on the Dockerfile in the cookbook/integrations/kubernetes/k8s
folder. Getting this error about s3 :
[3/3] currentAttempt done. Last Error: SYSTEM::Traceback (most recent call last):
File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
return wrapped(*args, **kwargs)
File "/opt/venv/lib/python3.8/site-packages/flytekit/core/base_task.py", line 527, in dispatch_execute
raise TypeError(
Message:
Failed to convert return value for var o0 for function k8s_spark.dataframe_passing.create_spark_df with error <class 'py4j.protocol.Py4JJavaError'>: An error occurred while calling o41.parquet.
: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3443)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FiInternal(DataFrameWriter.scala:355)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:781)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:748)
SYSTEM ERROR! Contact platform administrators.
Han
04/26/2022, 8:24 PM<http://googletagmanager.com|googletagmanager.com>
and <http://gstatic.com|gstatic.com>
for flytectl local sandbox dashboard links. Are they necessary?
2. What's the purpose of pyflyte
?
3. cont'dHafsa Junaid
04/27/2022, 3:04 AMSlackbot
04/29/2022, 2:24 AMHafsa Junaid
04/29/2022, 3:03 AMStefan Avesand
05/04/2022, 12:32 PMsd = StructuredDataset(uri='<bq://sp-one-model.quarterly>_forecast_2022F1.premium_revenue_tab_input_vat')
sd.open(pd.DataFrame).all()
AttributeError: 'NoneType' object has no attribute 'uri'
seunggs
05/05/2022, 7:31 AMRobin Kahlow
05/06/2022, 12:32 PMseunggs
05/06/2022, 9:13 PMMichael Cheng Jan Kao
05/11/2022, 1:15 AM@task(requests=Resources(cpu="1", mem="4Gi"), limits=Resources(cpu="2", mem="32Gi"))
def hello_world():
print('hello world')
hello_world._resources = ResourceSpec(requests=Resources(cpu='1', mem='16Gi', gpu=None, storage=None, ephemeral_storage=None), limits=Resources(cpu='2', mem='32Gi', gpu=None, storage=None, ephemeral_storage=None))
Robin Kahlow
05/11/2022, 12:31 PMseunggs
05/11/2022, 11:41 PMemissary-ingress
and I was wondering if you had a reference implementation for that instead of using Amazon Loadbalancer Ingress Controller?yujinlee
05/13/2022, 2:16 AMRobin Kahlow
05/13/2022, 2:40 PMtask_logs:
plugins:
logs:
kubernetes-enabled: true
kubernetes-template-uri: <https://my.papertrailapp.com/systems/{{> .podName }}/events
Sebastia Agramunt
05/17/2022, 7:14 PMAttila Nagy
05/18/2022, 7:51 AMyujinlee
05/20/2022, 4:02 AMflytectl --help
, it says that you can delete tasks,worfklows.
delete Terminates/deletes various Flyte resources such as tasks, workflows, launch plans, executions, and projects.
But in the available command of delete
, I don't see an option to actually delete tasks or workflows.
Usage:
flytectl delete [command]
Available Commands:
cluster-resource-attribute Deletes matchable resources of cluster attributes.
execution Terminates/deletes execution resources.
execution-cluster-label Deletes matchable resources of execution cluster label.
execution-queue-attribute Deletes matchable resources of execution queue attributes.
plugin-override Deletes matchable resources of plugin overrides.
task-resource-attribute Deletes matchable resources of task attributes.
workflow-execution-config Deletes matchable resources of workflow execution config.
HaoboGu
05/20/2022, 6:45 AMHaoboGu
05/20/2022, 6:45 AMRelease "flyte" does not exist. Installing it now.
W0520 06:19:59.185843 736 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
W0520 06:21:41.614777 736 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
NAME: flyte
LAST DEPLOYED: Fri May 20 06:19:57 2022
NAMESPACE: flyte
STATUS: deployed
REVISION: 1
TEST SUITE: None
Waiting for Flyte to become ready...
Terminated
Timed out while waiting for the flyteadmin rollout to be created
HaoboGu
05/20/2022, 6:46 AMKatrina P
05/20/2022, 6:48 PMRuksana Kabealo
05/20/2022, 7:37 PMpyflyte -c /root/.flyte/config.yaml --pkgs delaine serialize --in-container-config-path /root/config.yaml --local-source-root ./ --image delaieine:2 workflows -f _pb_output/
And getting the following error traceback:
Traceback (most recent call last):
File "/root/delaieine/bin/pyflyte", line 8, in <module>
sys.exit(main())
File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/root/delaieine/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/root/delaieine/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/serialize.py", line 139, in workflows
serialize_all(
File "/root/delaieine/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
return wrapped(*args, **kwargs)
File "/root/delaieine/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/serialize.py", line 60, in serialize_all
image_config=ImageConfig.auto(config_file, img_name=image),
File "/root/delaieine/lib/python3.9/site-packages/flytekit/configuration/__init__.py", line 250, in auto
for k, v in _internal.Images.get_specified_images(config_file).items()
File "/root/delaieine/lib/python3.9/site-packages/flytekit/configuration/internal.py", line 25, in get_specified_images
image_names = cfg.legacy_config.options("images")
AttributeError: 'NoneType' object has no attribute 'options'
NoneType leads me to believe it's not liking either the config or the docker image. I've checked the image/tag spelling. That's fine. The config exists, the path is right, and it looks like this:
admin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: localhost:30081
authType: Pkce
insecure: true
I'm using flytekit=1.0.2 and flytectl=0.5.25
I'm not sure how to troubleshoot this. Any help would be much appreciated!Ruksana Kabealo
05/20/2022, 7:37 PMpyflyte -c /root/.flyte/config.yaml --pkgs delaine serialize --in-container-config-path /root/config.yaml --local-source-root ./ --image delaieine:2 workflows -f _pb_output/
And getting the following error traceback:
Traceback (most recent call last):
File "/root/delaieine/bin/pyflyte", line 8, in <module>
sys.exit(main())
File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/root/delaieine/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/root/delaieine/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/root/delaieine/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/serialize.py", line 139, in workflows
serialize_all(
File "/root/delaieine/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
return wrapped(*args, **kwargs)
File "/root/delaieine/lib/python3.9/site-packages/flytekit/clis/sdk_in_container/serialize.py", line 60, in serialize_all
image_config=ImageConfig.auto(config_file, img_name=image),
File "/root/delaieine/lib/python3.9/site-packages/flytekit/configuration/__init__.py", line 250, in auto
for k, v in _internal.Images.get_specified_images(config_file).items()
File "/root/delaieine/lib/python3.9/site-packages/flytekit/configuration/internal.py", line 25, in get_specified_images
image_names = cfg.legacy_config.options("images")
AttributeError: 'NoneType' object has no attribute 'options'
NoneType leads me to believe it's not liking either the config or the docker image. I've checked the image/tag spelling. That's fine. The config exists, the path is right, and it looks like this:
admin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: localhost:30081
authType: Pkce
insecure: true
I'm using flytekit=1.0.2 and flytectl=0.5.25
I'm not sure how to troubleshoot this. Any help would be much appreciated!Yee
05/20/2022, 8:14 PMflytectl
tool.Ruksana Kabealo
05/20/2022, 9:20 PMYee
05/20/2022, 9:46 PMRuksana Kabealo
05/23/2022, 1:59 PM