Quinn Romanek
12/10/2021, 4:03 PMQuinn Romanek
12/17/2021, 6:50 PMEndre Karlson
12/21/2021, 4:53 PMEugene Cha
12/27/2021, 2:43 AMKetan (kumare3)
Khuyen Tran
12/27/2021, 9:30 PMflytectl sandbox start --source .
. After doing that, I can access to Flyte UI. However, after stopping the container, I tried to run flytectl sandbox start --source .
to resume the existing sandbox cluster, but I can't access to Flyte UI. Here is the specific outputs I got:
➜ flytectl sandbox start
🧑🏭 Bootstrapping a brand new flyte cluster... 🔨 🔧
delete existing sandbox cluster [y/n]: n
Existing details of your sandbox👨💻 Flyte is ready! Flyte UI is available at <http://localhost:30081/console> 🚀 🚀 🎉
Add KUBECONFIG and FLYTECTL_CONFIG to your environment variable
export KUBECONFIG=$KUBECONFIG:/home/khuyen/.kube/config:/home/khuyen/.flyte/k3s/k3s.yaml
export FLYTECTL_CONFIG=/home/khuyen/.flyte/config-sandbox.yaml
I wonder what is the best way to resume the existing cluster?Brian Lorenz
02/04/2022, 3:51 PMRodrigo Baron
02/04/2022, 5:30 PMNon-spark-on-k8s command provided
?Jonas Tischer
02/22/2022, 6:08 PMpython deploy.py
I get the following error message:
Serializing Flyte workflows
Usage: pyflyte [OPTIONS] COMMAND [ARGS]...
Try 'pyflyte --help' for help.
Error: Invalid value for '-k' / '--pkgs': Illegal package value booster-team for parameter: <Option pkgs>. Expected for the form [a.b.c]
Registering Flyte workflows
Error: open flyte-package.tgz: no such file or directory
What is the best way to diagnose what’s going wrong?Joseph Curtin
02/23/2022, 3:34 PMHafsa Junaid
03/05/2022, 2:44 AMMatheus Moreno
03/11/2022, 6:43 PMtest_google_cloud
, but the system won't find it: if I try to run it, I'm met with the error AttributeError: module 'src.flyte.tasks' has no attribute 'test_google_cloud'
. I ran docker build
again, serialized the tasks again, registered them again, and still nothing. The task is registered, since I can see it on the UI, but the worker can't find it. What is happening?Erbene Castro
03/15/2022, 2:40 PMWeide Zhang
03/22/2022, 12:48 AMStephen McGroarty
03/22/2022, 4:11 AMhttps://${MY_DOMAIN}/console
however I can't seem to get it to work on the command line, when running a command I get:
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for ingress.local, not ${MY_DOMAIN}"
My config looks something like (i've tried a few variations):
admin:
endpoint: dns:///${MY_DOMAIN}
authType: Pkce
insecure: false
clientId: ${MY_ID}
logger:
show-source: true
level: 0
Any hints on how to get around this?Erbene Castro
03/25/2022, 2:46 PMErbene Castro
03/25/2022, 2:48 PMMatheus Moreno
03/25/2022, 3:56 PMdowstreet
03/25/2022, 6:32 PMMatheus Moreno
03/25/2022, 8:06 PMflytectl create
and flytectl get
from the container, but I can't build images, and I think the problem is on the sandbox side, since I can't do that even on my host machine. I get the error:
OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "docker": executable file not found in $PATH: unknown
This is the configuration for the sandbox. What am I missing?
services:
flyte-sandbox:
image: <http://cr.flyte.org/flyteorg/flyte-sandbox:latest|cr.flyte.org/flyteorg/flyte-sandbox:latest>
container_name: flyte-sandbox
privileged: true
volumes:
- ..:/usr/src # The docker-compose file is inside a sandbox/ dir
ports:
- "30081:30081" # Console at <http://localhost:30081/console>
- "30084:30084"
- "30088:30088"
Johnson Huynh
03/30/2022, 7:10 PMError: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}
This is the log from FlyteAdmin logs:
{
"json": {
"src": "token.go:37"
},
"level": "info",
"msg": "Error occurred in NewAccessRequest: invalid_client",
"ts": "2022-03-30T19:38:33Z"
}
My Flyte Scheduler configmap is listed below. Am I perhaps missing a step?
data:
admin.yaml: |
admin:
clientId: ExternalODICclientID
clientSecretLocation: /etc/secrets/flyte-secret-auth
endpoint: flyteadmin.flyte.svc:81
insecure: true
event:
capacity: 1000
rate: 500
type: admin
db.yaml: |
database:
dbname: postgres
host: 'postgres-postgresql.flyte.svc'
passwordPath: /etc/db/pass.txt
port: 5432
username: xxxx
logger.yaml: |
logger:
level: 4
show-source: true
server.yaml: |
scheduler:
metricsScope: 'flyte:'
profilerPort: 10254
This is our Flyte Admin configmap:
auth:
appAuth:
openId:
baseUrl: <https://ExternalODICbaseurl.com>
clientId: OurODICClientID
scopes:
- profile
- openid
- email
thirdPartyConfig:
flyteClient:
clientId: ExternalODICclientID
redirectUri: <http://localhost:53593/callback>
scopes:
- offline
- all
authorizedUris:
- <https://our.domain.com>
userAuth:
openId:
baseUrl: <https://ExternalODICbaseurl.com>
clientId: ExternalODICclientID
scopes:
- profile
- openid
- email
authorizedUris:
- <https://our.domain.com>
Matheus Moreno
03/31/2022, 2:32 PMTiansu Yu
04/07/2022, 9:39 AMHenri Palacci
04/10/2022, 1:48 AMflytectl
to work now - I get redirected to <http://localhost:53593/callback>
with the error message:
Flyte Authentication
Couldn't get access token due to error: Post "<https://flyte>.<mydomain.com>:443/oauth2/token": x509: "flyte.<mydomain.com>" certificate is not standards compliant
I also set insecureSkipVerify: true
in ~/.flyte/config.yaml
but to no avail.Katrina P
04/14/2022, 4:01 PMTraceback (most recent call last):
File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
return wrapped(*args, **kwargs)
File "/opt/venv/lib/python3.8/site-packages/flytekit/core/base_task.py", line 464, in dispatch_execute
new_user_params = self.pre_execute(ctx.user_space_params)
File "/opt/venv/lib/python3.8/site-packages/flytekitplugins/spark/task.py", line 123, in pre_execute
self.sess = sess_builder.getOrCreate()
File "/opt/venv/lib/python3.8/site-packages/pyspark/sql/session.py", line 228, in getOrCreate
sc = SparkContext.getOrCreate(sparkConf)
File "/opt/venv/lib/python3.8/site-packages/pyspark/context.py", line 392, in getOrCreate
SparkContext(conf=conf or SparkConf())
File "/opt/venv/lib/python3.8/site-packages/pyspark/context.py", line 146, in __init__
self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
File "/opt/venv/lib/python3.8/site-packages/pyspark/context.py", line 209, in _do_init
self._jsc = jsc or self._initialize_context(self._conf._jconf)
File "/opt/venv/lib/python3.8/site-packages/pyspark/context.py", line 329, in _initialize_context
return self._jvm.JavaSparkContext(jconf)
File "/opt/venv/lib/python3.8/site-packages/py4j/java_gateway.py", line 1585, in __call__
return_value = get_return_value(
File "/opt/venv/lib/python3.8/site-packages/py4j/protocol.py", line 334, in get_return_value
raise Py4JError(
Message:
An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext
SYSTEM ERROR! Contact platform administrators.
The full log from the pod:
++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/bash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/bash ']'
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sed 's/[^=]*=\(.*\)/\1/g'
+ sort -t_ -k4 -n
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' '' == 2 ']'
+ '[' '' == 3 ']'
+ '[' -n '' ']'
+ '[' -z ']'
+ case "$1" in
+ echo 'Non-spark-on-k8s command provided, proceeding in pass-through mode...'
+ CMD=("$@")
+ exec /usr/bin/tini -s -- pyflyte-execute --inputs <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-nw0xm6ntwz/n0/data/inputs.pb> --output-prefix <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-nw0xm6ntwz/n0/data/1> --raw-output-data-prefix <s3://my-s3-bucket/kb/nw0xm6ntwz-n0-1> --checkpoint-path <s3://my-s3-bucket/kb/nw0xm6ntwz-n0-1/_flytecheckpoints> --prev-checkpoint <s3://my-s3-bucket/vc/nw0xm6ntwz-n0-0/_flytecheckpoints> --resolver flytekit.core.python_auto_container.default_task_resolver -- task-module k8s_spark.pyspark_pi task-name hello_spark
Non-spark-on-k8s command provided, proceeding in pass-through mode...
22/04/14 15:19:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
ERROR:root:Exception while sending command.
Traceback (most recent call last):
File "/opt/venv/lib/python3.8/site-packages/py4j/clientserver.py", line 480, in send_command
raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/venv/lib/python3.8/site-packages/py4j/java_gateway.py", line 1038, in send_command
response = connection.send_command(command)
File "/opt/venv/lib/python3.8/site-packages/py4j/clientserver.py", line 503, in send_command
raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending or receiving
{"asctime": "2022-04-14 15:20:06,729", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! Begin System Error Captured by Flyte !!"}
{"asctime": "2022-04-14 15:20:06,729", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "Traceback (most recent call last):\n\n File \"/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py\", line 165, in system_entry_point\n return wrapped(*args, **kwargs)\n File \"/opt/venv/lib/python3.8/site-packages/flytekit/core/base_task.py\", line 464, in dispatch_execute\n new_user_params = self.pre_execute(ctx.user_space_params)\n File \"/opt/venv/lib/python3.8/site-packages/flytekitplugins/spark/task.py\", line 123, in pre_execute\n self.sess = sess_builder.getOrCreate()\n File \"/opt/venv/lib/python3.8/site-packages/pyspark/sql/session.py\", line 228, in getOrCreate\n sc = SparkContext.getOrCreate(sparkConf)\n File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 392, in getOrCreate\n SparkContext(conf=conf or SparkConf())\n File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 146, in __init__\n self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,\n File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 209, in _do_init\n self._jsc = jsc or self._initialize_context(self._conf._jconf)\n File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 329, in _initialize_context\n return self._jvm.JavaSparkContext(jconf)\n File \"/opt/venv/lib/python3.8/site-packages/py4j/java_gateway.py\", line 1585, in __call__\n return_value = get_return_value(\n File \"/opt/venv/lib/python3.8/site-packages/py4j/protocol.py\", line 334, in get_return_value\n raise Py4JError(\n\nMessage:\n\n An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext\n\nSYSTEM ERROR! Contact platform administrators."}
{"asctime": "2022-04-14 15:20:06,730", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! End Error Captured by Flyte !!"}
Yee
Sebastian
04/20/2022, 9:17 AMimport numpy as np
from flytekit import workflow
@workflow
def wf(date: str = "2022-01-01"):
date = np.datetime64(date) # this errors
Sören Brunk
04/20/2022, 12:25 PMfrom datetime import datetime
@task
def t(date: datetime = datetime(2022,1,1)):
date = np.datetime64(date)
...
@workflows
def wf(date: datetime):
t(date=date)
Han
04/20/2022, 5:21 PMflyte run workflow
on Google, the first result is a 404 page on the documentation site.caioau
04/20/2022, 6:18 PM