jeev
time="2022-02-28T20:30:40Z" level=info msg="Using config file: [/etc/flyte/config/allowed_origins.yaml /etc/flyte/config/cluster_resources.yaml /etc/flyte/config/core.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml /etc/flyte/config/logger.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage-limits.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resource_defaults.yaml]"
{"json":{"src":"migrate.go:107"},"level":"fatal","msg":"failed to connect to `host=cloud-sql-proxy user=flyte database=flyte`: server error (ERROR: SSL required (SQLSTATE 08P01))","ts":"2022-02-28T20:30:40Z"}
2022/02/28 20:30:40 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/database.go:102
[error] failed to initialize database, got error failed to connect to `host=cloud-sql-proxy user=flyte database=flyte`: server error (ERROR: SSL required (SQLSTATE 08P01))
jeev
Ketan (kumare3)
Sören Brunk
03/07/2022, 7:56 PM[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[vej152kn4b-n1-0] terminated with exit code (1). Reason [Error]. Message:
tar: /root/8143d4634b6d53c26072284ce429e69af1278102-fast1-fast32b90af686df921bd623ae6df68f5c48.tar.gz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
Traceback (most recent call last):
File "/opt/venv/bin/pyflyte-fast-execute", line 8, in <module>
sys.exit(fast_execute_task_cmd())
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/flytekit/bin/entrypoint.py", line 467, in fast_execute_task_cmd
_download_distribution(additional_distribution, dest_dir)
File "/opt/venv/lib/python3.9/site-packages/flytekit/tools/fast_registration.py", line 112, in download_distribution
result.check_returncode()
File "/usr/lib/python3.9/subprocess.py", line 460, in check_returncode
raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['tar', '-xvf', '/root/8143d4634b6d53c26072284ce429e69af1278102-fast1-fast32b90af686df921bd623ae6df68f5c48.tar.gz', '-C', '/root']' returned non-zero exit status 2.
.
Here's what's executing from the pod spec
spec:
containers:
- args:
- pyflyte-fast-execute
- --additional-distribution
- <gs://my-bucket/fast/8143d4634b6d53c26072284ce429e69af1278102-fast1-fast32b90af686df921bd623ae6df68f5c48.tar.gz>
...
I checked the bucket and the object exists so uploading during registration seems to be working correctly. Perhaps something wrong with the download or the path it is copied to in the container. Any ideas?Sören Brunk
03/09/2022, 10:15 PMSome node execution failed, auto-abort.
The task pods start, go into running state. I can see a few lines of task logs sometimes but then it looks like the pod is just deleted long before it's finished. It terminates and is cleaned up.
Log says:
Stopping container m4dydd0r8y-n0-0
So i checked flytepropeller logs and there's one error log a few seconds before the pod is deleted. Not sure if it's related:
Failed to update workflow. Error [Operation cannot be fulfilled on <http://flyteworkflows.flyte.lyft.com|flyteworkflows.flyte.lyft.com> "m4dydd0r8y": the object has been modified; please apply your changes to the latest version and try again]
I'm a bit at a loss here how to debug this further. Any hints much appreciated. I can also provide the full propeller log if that could help tracing this down.Sandra Youssef
Sandra Youssef
Joseph Wang
03/16/2022, 7:36 PMKetan (kumare3)
Ketan (kumare3)
komal azram
03/17/2022, 6:19 AMSören Brunk
03/17/2022, 10:10 AMFlyteDirectory
as final output. Via flyteconsole/API I get a reference to the object, like so:
o0:
type:
multi-part
uri:
<gs://flyte-bucket/data/q1/pjziwgu55i-n1-0/48f7d1af2f65a6ff4c03fac9191cbea9>
If I try to fetch that workflow via FlyteRemote
I get a local path as output. It's empty though.
remote = FlyteRemote.from_config()
exec = remote.fetch_workflow_execution("abc")
print(exec.outputs)
// /tmp/flyte3eg_mcpz/control_plane_metadata/local_flytekit/6f2223e1c864f6d1e88c8dbad090f349
What is the purpose of this local path and is there a way to get the original output location via FlyteRemote
?Hafsa Junaid
03/17/2022, 7:39 PMJoseph Wang
03/18/2022, 3:36 PMJoseph Wang
03/18/2022, 3:38 PMJoseph Wang
03/18/2022, 3:38 PMwhile continue(...):
next_candidate_sets = get_next_candidate_set(...)
results = run_jobs(next_candidate_sets)
store_results(results)
Joseph Wang
03/18/2022, 5:19 PMJoseph Wang
03/18/2022, 5:21 PMSandra Youssef
Mike Seid
03/18/2022, 6:23 PMKetan (kumare3)
Ketan (kumare3)
Hampus Rosvall
03/20/2022, 8:35 PMmodel_dir:
type:
multi-part
uri:
<s3://my-s3-bucket/ja/hfnqehz4ps-n0-0/9828036b2328e522488b3ff0311528e2>
Samhita Alla
Sandra Youssef
komal azram
03/22/2022, 3:47 PMKetan (kumare3)
Hafsa Junaid
03/22/2022, 5:00 PMFabian Baier
03/22/2022, 8:36 PMJing Li
03/23/2022, 7:38 AM