GitHub
11/02/2023, 9:26 PMremote.fetch_workflow_execution()
you get back a FlyteWorkflowExecution
instance. The data model backing this object mirrors the responses returned by Admin which was a good first start but we should improve the ux flow for humans.
Goal: What should the final outcome look like, ideally?
Rather than
we = remote.fetch_workflow_execution()
we.node_executions
the user should be able to do
we.nodes # to get a map of the nodes in the execution
we.nodes["n1"] # to get a node
we.nodes["n1"].execution.tasks
we.nodes["n1"].execution.subs??
For a more complicated example, let's say we have
@workflow
def my_wf(val: int) -> int:
result = square(val=val)
return result
@dynamic
def run_subwfs() -> typing.Tuple[int, int]:
x = my_wf(val=5)
y = my_wf(val=7)
return x, y
@workflow
def run_subwfs_wf() -> typing.Tuple[int, int]:
return run_subwfs()
If we ran run_subwfs_wf
and fetched the execution for it, we'd be able to navigate it like so:
we = r.fetch_workflow_execution()
we.nodes["n0"].kind <- pattern matches
# normal task
we.nodes["n0"].task.executions
# static subwf
we.nodes["n0"].workflow
we.nodes["n0"].workflow.nodes
# static lp
we.nodes["n0"].child_execution
# dynamic
we.nodes["n0"].dynamic.task.executions
we.nodes["n0"].dynamic.workflow.nodes
# branch
we.nodes["n0"].branch.conditions[0].node # really the node execution, .state
we.nodes["n0"].branch.conditions[0].expression # result or expression
consider FlyteNode -> FlyteNodeExecution
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
11/02/2023, 9:26 PMGitHub
11/02/2023, 9:26 PMflytectl get launchplan
command does not show the correct state of an activated launch plan.
Expected behavior
The activated launch plan state should be correctly returned by flytectl
Additional context to reproduce
flytectl version v0.3.20
Screenshots
Screen.Recording.2021-11-11.at.10.11.40.PM.mov
Screenshot 2021-11-11 at 10 09 39 PM▾
GitHub
11/02/2023, 9:26 PMGitHub
11/02/2023, 9:26 PMimage▾
GitHub
11/02/2023, 9:26 PMUNKNOWN
status. Ideal, would be if the workflow gets information about the system error being experienced.
{"json":{"exec_id":"uaben0erib","ns":"flytesnacks-development","res_ver":"52746460","routine":"worker-8","src":"executor.go:355","wf":"flytesnacks:development:core.control_flow.high_cpu_wf.single_integer_map_task"},"level":"info","msg":"Handling Workflow [uaben0erib], id: [project:\"flytesnacks\" domain:\"development\" name:\"uaben0erib\" ], p [Ready]","ts":"2021-10-29T04:17:44Z"}
{"json":{"exec_id":"uaben0erib","ns":"flytesnacks-development","res_ver":"52746460","routine":"worker-8","src":"executor.go:122","wf":"flytesnacks:development:core.control_flow.high_cpu_wf.single_integer_map_task"},"level":"info","msg":"Setting the MetadataDir for StartNode [s3://..../metadata/propeller/flytesnacks-development-uaben0erib/start-node/data]","ts":"2021-10-29T04:17:44Z"}
{"json":{"exec_id":"uaben0erib","node":"start-node","ns":"flytesnacks-development","res_ver":"52746460","routine":"worker-8","src":"protobuf_store.go:77","wf":"flytesnacks:development:core.control_flow.high_cpu_wf.single_integer_map_task"},"level":"error","msg":"Failed to write to the raw store [s3://.../metadata/propeller/flytesnacks-development-uaben0erib/start-node/data/0/outputs.pb] Error: Failed to write data [28b] to path [metadata/propeller/flytesnacks-development-uaben0erib/start-node/data/0/outputs.pb].: PutObject, putting object: WebIdentityErr: failed fetching WebIdentity token: \ncaused by: WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token\ncaused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied","ts":"2021-10-29T04:17:44Z"}
Expected behavior
The UI should show the error that is experienced by the system
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
11/02/2023, 9:26 PM2021-10-26 11_51_54-Metrics in prometheus_grafana▾
GitHub
11/02/2023, 9:26 PMGitHub
11/02/2023, 9:26 PMDELTA_CHECK=true make helm
to find any diff in the generated helm and doc files or any other uncommitted generated files.
When the same is done from local machine its possible the tools needed to run these are older and don't give the same results as in the CI.
While working on GCP deployment for flyte found the generated files had a diff in CI whereas locally it would work fine.
The discrepancy was due to older version of helm-docs which was used for generating docs.
make update_boilerplate was expected to solve this problem but it didn't
What if we do not do this?
Developer has to figure out the difference in failure in CI and local environments which can be cumbersome if the tool versions used in the two environments are different.
Related component(s)
flyte
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
11/02/2023, 9:26 PMopta/flyte.yaml
lead to nonexistent folder helm
flyte/opta/flyte.yaml
Line 46 in </flyteorg/flyte/commit/a8cb42015adda0e901d263a67a26def4d704cea2|a8cb420>
flyte/opta/flyte.yaml
Line 50 in </flyteorg/flyte/commit/a8cb42015adda0e901d263a67a26def4d704cea2|a8cb420>
Expected behavior
"../helm"
in both paths should be changed either to "../charts/flyte"
or "../charts/flyte-core"
(not sure which one)
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
11/02/2023, 9:26 PMGitHub
11/02/2023, 9:26 PMGitHub
11/02/2023, 9:26 PMdefault-annotations:
- annotationKey1: annotationValue1_{PROJECT}_{DOMAIN} # -> annotationValue1_flytesnacks_development
- key_{project}: True # -> key_flytesnacks: True
The idea would be to use predefined variable names (PROJECT
, DOMAIN
, `WORKFLOW`(?) ) that are anyway readily available when creating task pods.
Describe alternatives you've considered
The alternative would be to allow defining project/domain specific keys/values, which would allow more fine-grain control (maybe all projects don't need the key/value defined?) but at the same time would be less flexible with regards to including new projects. It basically serves a slightly different use case.
Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteGitHub
11/02/2023, 9:26 PMGitHub
11/02/2023, 9:27 PM@task(task_config=Pod(pod_spec=pod_spec, pimary_container_name="container_name"))
. This means that users need to know the configuration of the cluster ( like will any sidecars be added automatically) in order to successfully create/run a flyte workflow.
Goal: What should the final outcome look like, ideally?
Ideally a user of flyte wouldn't need to declare their task as a pod task in order to run it in a kubernetes cluster that uses mutating webhooks to add sidecars to pods. The user could declare their task like @task
and it would run the flyte task pod until the flyte task container finishes then the rest of the workflow would be run.
Describe alternatives you've considered
The only other alternative I've considered is continuing to use the workaround of declaring all workflows/tasks as pod tasks.
flyteorg/flyteGitHub
11/02/2023, 9:27 PMGitHub
11/02/2023, 9:27 PMtask = BlazingSQL(
input_schema=FlyteSchema,
outputs=...,
query=""" SELECT count(*) FROM taxi GROUP BY year(key)""",
use_gpus=False, # True by default
)
Type of Plugin
☑︎ Python/Java interface only plugin
☐ Web Service (e.g. AWS Sagemaker, GCP DataFlow, Qubole etc...)
☐ Kubernetes Operator (e.g. TfOperator, SparkOperator, FlinkK8sOperator, etc...)
☐ Customized Plugin using native kubernetes constructs
☐ Other
flyteorg/flyteGitHub
11/02/2023, 9:27 PMflytectl version
reports that a new release is available, even directly after flytectl upgrade
$ flytectl upgrade
You have already latest version of flytectl
$ flytectl version
A new release of flytectl is available: 0.3.4 → v0.3.4
{
"App": "flytectl",
"Build": "8ba75a6",
"Version": "0.3.4",
"BuildTime": "2021-10-01 16:08:44.983660372 +0200 CEST m=+0.019362263"
}
Minor nitpick: the flytectl upgrade
error message contains a typo (missing "the")
It seems that the version number specified in the build (used as stdlibversion.Version
in <cmd/version/version.go|`cmd/version/version.go`> incorrectly omits the v
of the Git tag for the release. This leads to IsVersionGreaterThan
in <pkg/util/util.go|`pkg/util/util.go`> incorrectly reporting that the latest Github version is greater than the local build version.
I haven't fully grasped the release process, but it seems that the Github release action sets the version number without the v
prefix in the Goreleaser action, which would explain the discrepancy.
Expected behavior
No newer release should be reported after upgrading to the latest version using flytectl upgrade
.
[Optional] Additional context
To Reproduce
1. Upgrade to latest release using flytectl upgrade
2. Perform update check through flytectl version
flyteorg/flyteGitHub
11/02/2023, 9:27 PMflytectl
inside a Gitlab CI pipeline. Gitlab supports supplying CI variables mounted as files, where the environment variable points to a file containing the value of the CI var - so naturally we tried setting FLYTECTL_CONFIG
through this mechanism.
However, this fails with the following error message when trying to execute any flytectl command:
Error:
Unsupported Config Type ""
ERRO[0000]
Unsupported Config Type ""
Apparently, this error message originates in the spf13/viper library used by flytectl through flytestdlib. The error is raised because viper tries to detect the type of the config file from its extension, which is unset in our CI environment (Gitlab names the file identical to the variable name, e.g. the FLYTECTL_CONFIG
variable will be stored under /builds/resources/repo_name.tmp/FLYTECTL_CONFIG
).
The error is not specific to Gitlab CI, but can be reproduced by pointing FLYTECTL_CONFIG
at any file without an extension, see below.
Steps to reproduce
# Note that ./config is a valid YAML flytectl config file
$ export FLYTECTL_CONFIG=$PWD/config
$ cat $FLYTYCTL_CONFIG
admin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: dns:///localhost:30081
insecure: true
logger:
show-source: true
level: 0
storage:
connection:
access-key: minio
auth-type: accesskey
disable-ssl: true
endpoint: <http://10.32.16.105:30084>
region: us-east-1
secret-key: miniostorage
type: minio
container: "my-s3-bucket"
enable-multicontainer: true
$ flytectl get projects
Error:
Unsupported Config Type ""
ERRO[0000]
Unsupported Config Type ""
$ flytectl version
A new release of flytectl is available: 0.3.4 → v0.3.4
{
"App": "flytectl",
"Build": "8ba75a6",
"Version": "0.3.4",
"BuildTime": "2021-10-01 15:37:24.613781861 +0200 CEST m=+0.029412676"
}
Proposed fix
Viper offers the SetConfigName
and SetConfigType
methods which could be used in flytestdlib instead of SetConfigFile
if the config file is always assumed to be a YAML file.
flyteorg/flyteGitHub
11/02/2023, 9:27 PMGitHub
11/02/2023, 9:27 PMflytectl sandbox status
Provide a possible output or ux example
$ flytectl sandbox status
...
To Connect to FlyteSandbox Kube cluster export:
...
To connect to FlyteSandbox using flytectl export:
...
flyteorg/flyteGitHub
11/02/2023, 9:27 PMGitHub
11/02/2023, 9:27 PMflytekit-resource-monitoring
that can load components that can output certain meta outputs.
For this, we will need support for meta-outputs.
flyteorg/flyteGitHub
11/02/2023, 9:27 PM@flytekit.task(
retries=1,
requests=flytekit.Resources(cpu="12", mem="32Gi"),
limits=flytekit.Resources(mem="64Gi", cpu="12"),
temp_disk_volumes=[flytekit.Volume(size="100G", mount_path="/tmpdisk_01")],
)
def foo():
download_large_file_to("/tmpdisk_01/abc")
Describe alternatives you've considered
Maybe flyte users can do it today with Sidecar task. But having a support of this parameter in every task would bring more convenience.
[Optional] Propose: Link/Inline OR Additional context
If you have ideas about the implementation please propose the change. If inline keep it short, if larger then you link to an external document.
flyteorg/flyteGitHub
11/02/2023, 9:27 PM{
"json": {
"exec_id": "grs2u5m2ns",
"node": "n0",
"ns": "flytesnacks-development",
"res_ver": "137261245",
"routine": "worker-2",
"src": "handler.go:313",
"tasktype": "python-task",
"wf": "flytesnacks:development:<http://core.control_flow.dynamics.wf|core.control_flow.dynamics.wf>"
},
"level": "warning",
"msg": "No plugin found for Handler-type [python-task], defaulting to [container]",
"ts": "2021-09-10T15:53:28Z"
}
There should be a document that explains how to filter logs to get only the logs for one specific workflow / execution id etc
flyteorg/flyteGitHub
11/02/2023, 9:27 PMGitHub
11/02/2023, 9:27 PMGitHub
11/02/2023, 9:27 PMGitHub
11/02/2023, 9:27 PMGitHub
11/02/2023, 9:27 PM